Roy's notes

Page move

After borrowing space on Malin's wiki for some years, further edits will be done here, so that next time the wiki goes down after something strange happened, she won't get a ton of mesages from me, but I'll have to fix it myself - oh well ;)

LVM, md and friends

mdadm

mdadm is Linux' software raid management tool, controlling this through the underlying kernel md driver. Through use of various RAID levels, you can achive a stable and fast system where you may be able to lose a drive or two without losing data.

Please keep in mind that RAID is not backup. If something goes wrong (if you do something stupid, if more than x drives die, if the house burns down etc), you'll still need an external backup to get your data back.

RAID levels

RAID is a way to combine several disks of the same size to a single blockdevice, that the OS understands as a disk. If you use disks of varying size, the array will only use the amount per drive equal to the smallest drive, per disk, in the RAIDset.

This is better explained on wikipedia, but I'll summerise as best I can below.

RAID-0

A RAID-0 or stripeset is a way to add two or more drives to a single md device without any redundancy. The pros are it is simple and fast and you get to use all the disk space available. The con is, if you lose a single drive, you lose the whole lot. Please only use this for testing, caching or other unimportant.

RAID-1

RAID-1 or a mirror is a way to combine two or more drives to a single md device with a total space as a single drive in the mirror. Typically, two drives are used in a mirror, but some use more. With RAID-1, you get the same write speed as a single drive and a bit more for reads, as Linux will attempt to read from both (or all) sides of the mirror in parallel. For real world data, this is not linear, though. You don't get double the read speed with two drives in a mirror and three with a three-way mirror, but still, it'll help, especially for random I/O.

RAID-2 and 3

Ancient stuff, not supported by md.

RAID-4

Same principle as RAID-0, only here we have an additional disk for storing parity data. With this setup, you need two or more drives and can lose any of them. If the parity data goes away, you still have the data and if one of the data drives dies, its data be reconstructed from parity.

RAID-5

RAID-5 is mostly the same as RIAD-4 except that here, we have rotating parity. Since RAID-4 uses a dedicated parity disk, this gets far more writes than the data disks and becomes a bottleneck. RAID-5 solves this.

RAID-6

In RAID-6, we take RAID-5 to a new dimension, literally, adding another set of parity.

RAID-10

RAID-10 is a non-standard RAID level some call RAID-1e or similar. It is like a mirror, but can grow over an odd number of drives and 'mirror' across them (see wikipedia link above for details).

Creating a mirror

To be continued

Linux' Logical Volume Manager (LVM)

LVM in general

LVM is designed to be an abstraction layer on top of physical drives or RAID, typically mdraid or fakeraid. Keep in mind that fakeraid should be avoided unless you really need it, like in conjunction with dual-booting linux and windows on fakeraid. LVM broadly consists of three elements, the "physical" devices (PV), the volume group (VG) and the logical volume (LV). There can be multiple PVs, VGs and LVs, depending on requirement. More about this below. All commands are given as examples, and all of them can be fine-tuned using extra flags in need of so. In my experience, the defaults work well for most usecases.

I'm mentioning filesystem below too. Where I write ext4, the samme applies to ext2 and ext3.

Create a PV

A PV is the "physical" parts. This does not need to be a physical disk, but can also be another RAID, be it an mdraid, fakeraid, hardware raid, a virtual disk on a SAN or otherwisee or a partition on one of those.

#  These add three PVs, one on a drive (or hardware raid) and another two on mdraids.
pvcreate /dev/sdb
pvcreate /dev/md1 /dev/md2

For more information about pvcreate, see the manual.

Create a VG

The volume group consists of one or more PVs grouped together on which LVs can be placed. If several PVs are grouped in a VG, it's generally a good idea to make sure these PVs have some sort of redundancy, as in mdraid/fakeraid or hwraid. Otherwise it will be like using a RAID-0 with a single point of failure on each of the independant drives. LVM has RAID code in it as well, so you can use that. I haven't done so myself, as I generally stick to mraid. The reason is mdraid is, in my opinion older and more stable and has more users (meaning bugs are reported and fixed faster whenever they are found). That said, I beleive the actual RAID code used in LVM RAID are the same function calls as for mdraid, so it may not be much of a difference. I still stick with mdraid. To create a VG, run

# Create volume group my_vg
vgcreate my_vg /dev/md1

Note that if vgcreate is run with a PV (as /dev/md1 above) that is not defined as a PV (like above), this is done implicitly, so if you don't need any special flags to pvcreate, you can simply skip it and let vgcreate do that for you.

Create an LV

LVs can be compared to partitions, somehow, since they are bounderies of a fraction or all of that of a VG. The difference between them and a partition, however, is that they can be grown or shrunk easily and also moved around between PVs without downtime. This flexibility makes them superiour to partitions as your system can be changed without users noticing it. By default, an LV is alloated "thickly", meaning all the data given to it, is allocated from the VG and thus the PV. The following makes a 100GB LV named "thicklv". When making an LV, I usually allocate what's needed plus some more, but not everything, just to make sure it's space available for growth on any of the LVs on the VG, or new LVs.

# Create a thick provisioned LV named thicklv on the VG my_vg
lvcreate -n thicklv -L 100G my_vg

After the LV is created, a filesystem can be placed on it unless it is meant to be used directly. The application for direct use include swap space, VM storage and certain database systems. Most of these will, however, work on filesystems too, although my testing has shown that on swap space, there is a significant performance gain for using dedicated storage without a filesystem. As for filesystems, most Linux users use either ext4 or xfs. Personally, I generally use XFS these days. See my notes below on filesystem choice.

# Create a filesystem on the LV - this could have been mkfs -t ext4 or whatever filesystem you choose
mkfs -t xfs /dev/my_vg/thicklv

Then just edit /etc/fstab with correct data and run mount -a, and you should be all set.

Create LVM cache

LVMcache is LVM's solution to caching a slowish filesystem on spinning rust with the help of much faster SSDs. For this to work, use a separate SSD or a mirror or two (just in case) an add the new disk/md dev to the same VG. In this case, we use a small mirror, md1. LVM is not able to cache to a disk or partition outside the volume group.

vgextend data /dev/md1
  Physical volume "/dev/md1" successfully created.
  Volume group "data" successfully extended

Then create two LVs, one for the data (contents of files) and one for the metadata (filenames, directories, attributes etc). Typically, the metadata part won't need to be very large. 100M should usually do unless you have ekstremely many small files. I guess there are ways to calculate it, but again, 1GB should do for everyone (or was that 640k?). As for the data cache, I chose to use 40G here for a 15TB RAID. It will only cache what actively use anyway, so no need for overkill.

lvcreate -L 1G -n _cache_meta data /dev/md1
lvcreate -L 100G -n _cache data /dev/md1

# If you want to use the rest of the cache device, here /dev/md1, replace -L 100G with -l +100%PVS, *not*, -l +100%FREE, as that will use all the available space in the VG regardless of there being a PV mentioned in the config.

# Some would want --cachemode writeback, but if paranoid, I wouldn't recommend it. Metadata or data can be easily
# corrupted in case of failure. On the other hand, most filesystems will recover from this these days since they use
# journaling and writeback caching ''really'' speeds up things.
lvconvert --type cache-pool --cachemode writethrough --poolmetadata data/_cache_meta data/_cache
lvconvert --type cache --cachepool data/_cache data/data

That should be it. If you benchmarked the disk before this, you should try again and you'll probably get a nice surprise. If not, well, see below how to turn this off again :)

Disabling LVM cache

If you want to change the cached LV, such as growing og shrinking or otherwise, you'll have to disable caching first and then re-enable it. There's no quick-and-easy fix, but you just do as you did when enabling in the first place.

Forst, disable caching for the LV. This will do it cleanly and remove the LVs earlier created for caching the main device.

lvconvert --uncache data/data

Uncaching an LV can sometimes lead to lvconvert hanging, printing "uncaching block xxx" repeatedly without the number increasing. If this happens, the solution I found was the "windows solution". Interrupt the uncaching with ctrl+c (it's made to do that nicely) and give the machine a reboot.

Now, since you now have a VG with an empty PV in it, earlier used for caching, I'd really recommend removing this for now to avoid expanding onto that as well. Since they're in the same VG, this might happen and if you get so far as to extend the lv and the filesystem on it, and this filesystem is XFS or similar filesystems without possibilities of reducing the size, you have a wee problem.

Remove that PV from the VG

vgreduce data /dev/md1

The PV is still there, but pvs should now show it not connected to a VG

PS: I once made this error after first extending a 2x16TB RAID10 to 3x16TB, removed the cache to allow lvextend to work, extended -l +100%FREE and xfs_growfs /dev/that/dev and then - oops - I had a filesystem on the raid *plus* a separate SSD. Since XFS can't be shrunk, and since there was a ton of data there and a full restore would take a week or three, I ended up getting yet another 16TB drive before doing it all the right way. So don't do this at home™ (or any other place).

Growing devices

LVM objects can be grown and shrunk. If a PV resides on a RAID where a new drive has been added or otherwise grown, or on a partition or virtual disk that has been extended, the PV must be updated to reflect these changes. The following command will grow the PV to the maximum available on the underlying storage.

# Resize the PV /dev/md (not the RAID, only the PV on the RAID) to its full size
pvresize /dev/md1

If a new PV is added, the VG can be grown to add the space on that in addition to what's there already.

# Extend my_vg to include md2 and its space
vgextend my_vg /dev/md2

With more space available in the VG, the LV can now be extended. Let's add another 50GB to it.

# Extend thicklv - add 50GB. If you know you won't need the space anywhere else, you may want to
# lvresize -l+100%FREE instead. Keep in mind the difference between -l (extents) and -L (bytes)
lvresize -L +50G my_vg/thicklv

PS: Cached LVs can't be shrunk directly, you'll have to uncache them first. Please see above

Growing filesystems

After the LV has grown, run xfs_growfs (xfs) or resize2fs (ext4) to make use of the new data. To grow the filesystem to all available space on ext4, run the below command. To partly grow the filesystem or to shrink it or otherwise, please see the manuals.

The filesystem can be mounted when this is run. If you for some reason get an access denied error when running this as root or with sudo, it's likely caused by a filesystem error. To fix this, unmount the filesystem first and run fsck -f /dev/my_vg/thicklv and remount it before retrying to extend it.

resize2fs /dev/my_vg/thicklv

Similarly, with XFS, but note that xfs_growfs doesn't take the device name, but the filesystem's mount point. In the case of XFS, also note that the filesystem must be mounted to be grown. This, in contrast to how it was in the old days when filesystems had to be unmounted to be grown.

xfs_growfs /my_mountpoint

Shrinking filesystems

Some filesystems, like ext4, support shrinking, meaning you can reduce the size of the filesystem and then shrink the size of the LV. XFS does not support this and neither do any other current Linux filesystems, so I'll stick to ext4 here.

There are probably a lot of ways to do this, but my advice is:

• Shrink filesystem to its minimum
• Shrink the lv to the wanted size
• Grow the filesystem to maximum available size (in the lv)

To do this, you'll need to unmount the filesystem first, since this operation can't be done while its mounted, unlike the other operations I have discussed above. If the filesystem is the root filesystem or you somehow can't unmount it because its got open files, you'll need to boot into single user mode (aka rescue mode). If it isn't the rootfs, check first with lsof to see if you can find open files on it and stop the processes holding those. Try to umount again and if that doesn't work, you'll need to boot into single. After that, it's quite straight forward. I'll use vg as the VG name here and lv as the LV name for simplicity.

umount /dev/vg/lv # if this succeeds, continue
fsck -f /dev/vg/lv # if this fails, try again. If things fail badly, you'll need to sort those out before going on
resize2fs -M /dev/vg/lv # resize to minimum.
lvresize -L <new size> /dev/vg/lv # keep in mind that if the new size is less than your current filesystem size, as reported by resize2fs, you *will* see data corruption, so make sure it's at least the current size or perhaps a few megs larger, just in case.
resize2fs /dev/vg/lv # grow the filesystem to fill the lv
fsck -f /dev/vg/lv # just doublecheck
mount /dev/vg/lv # mount it according to /etc/fstab and check if it looks right

Rename system VG

Some distros create VG names like those-from-a-sysadmin-with-a-well-developed-paranoia-not-to-hit-a-duplicate. Debian, for instance, uses names like "my-full-hostname-vg". Personally, I don't like these, since if I were to move a disk or vdisk around to somewhere else, I'd make sure not to add a disk named 'sys' to an existing system. If you do, most distros will refuse to use the VGs with duplicate names, resulting in the OS not booting until either one is specified by its UUID or just one removed. As I'm aware of this, I stick to the super-risky thing of all system VGs named 'sys' and so on.

If you do this, keep in mind that it may blow up your computer and take your girl- or boyfriend with it or even make either of them pregnant, allowing Trump to rule your life and generally make things suck. You're on your own!

# Rename the VG
vgrename my-full-hostname-vg sys

Now, in /boot/grub/grub.cfg, change the old lv path from something like '/dev/mapper/debian--stretch--machine--vg-root' to your new and fancy '/dev/sys/root. Likewise, in /etc/fstab, change occurences of '/dev/mapper/debian--stretch--machine--vg-' (or similar) with '/dev/sys/'. Here, this was like this - the old ones commented out.

#/dev/mapper/debian--stretch--box--vg-root      /               ext4            errors=remount-ro,discard       0       1
/dev/sys/root                                   /               ext4            errors=remount-ro,discard       0       1
# /boot was on /dev/vda1 during installation
UUID=myverylonguuidwithatonofgoodinfoinit       /boot           ext2            defaults                        0       2
#/dev/mapper/debian--stretch--box--vg-swap_1    none            swap            sw                              0       0
/dev/sys/swap_1                                 none            swap            sw                              0       0

Update /etc/initramfs-tools/conf.d/resume with similar data for swap.

You might want to run 'update-initramfs -u -k all' after this, but I'm not sure if it's needed.

Now, pray to the nearest god(s) and give it a reboot and it should (probably) work.

After reboot, run swapoff -a, then mkswap /dev/sys/swap_1 (or whatever it's called on your system) and swapon -a again. You may want to give it another reboot or two for kicks, but it should work well even without that.

Migration tools

At times, storage regimes change, new storage is added and sometimes it's not easy to migrate with the current hardware or its support systems. Once I had to migrate a 45TiB fileserver from one storage system to another, preferably without downtime. The server originally had three 15TiB PVs of which two were full and the third half full. I resorted to using pvmove to just move the data on the PVs in use to a new PV. We started out with creating a new 50TiB PV, sde, and then to pvmove.

# Attach /dev/sde to the VG my_vg
vgextend my_vg /dev/sde

# Move the contents from /dev/sdb over to /dev/sde block by block. If a target is not given in pvmove,
# the contents of the source (sdb here) will be put somewhere else in the pool.
pvmove /dev/sdb /dev/sde

This took a while (as in a week or so) - the pvmove command uses old code and logic (or at least did that when I migrated this in November 2016), but it affected performance on the server very little, so users didn't notice. After the first PV was migrated, I continued with the other two, one after the other, and after a month or so, it was migrated. We used this on a number of large fileservers, and it worked flawlessly. Before doing the production servers I also tried interrupting pvmove in various ways, including hard resets, and it just kept on after the reboot.

NOTE: Even though it worked well for us, always keep a good backup before doing this. Things may go wrong, and without a good backup, you may be looking for a hard-to-find new job the next morning.

Thin provisioned LVs

Thin provisioning on LVM is a method used for not allocating all the space given to an LV. For instance, if you have a 1TB VG and want to give an LV 500GB, although it currently only uses a fraction of that, a thin lv can be a good alternative. This will allow for adding a limit to how much it can use, but also to let it grow dynamically without manual work by the sysadmin. Thin provisioning adds another layer of abstaction by creating a special LV as a thin pool from which data is allocated to the thin volume(s)

Create a thin pool

Allocate 1TB to the thin pool to be used for thinly provisioned LVs. Keep in mind that the space allocated to the thin pool is in fact thick provisioned. Only the volumes put on thinpool are thin provisioned. This will create two hidden LVs, one for data and one for metadata. Normally the defaults will do, but check the manual if the data doesn't match the usual patterns (such as billions of files, resulting in huge amounts of metadata). I beleive the metadata part should be resizable at a later time if needed, but I have not tested it.

# Create a pool for thin LVs. Keep in mind that the pool itself is not thin provisioned, only the volumes residing on it
lvcreate --size 1T --type thin-pool --thinpool thinpool my_vg

Create a thin volume

You have the thinpool, now put a thin volume on it. It will allocate some space for metadata, but probably not much (a few megs, perhaps).

# Now, create a volume with a virtual (-V) size of half a terabyte named thin_pool, but using the thinpool (-T) named thin_pool for storage
lvcreate -V 500G -T my_vg/thin_pool --name thinvol

The new volume's device name will be /dev/thinvol. Now, create a filesystem on it, add to fstab and mount it. The df command will return available space according to the virtual size (-V), while lvs will show how much data is actually used on each of the thinly provisioned volumes.

Filesystem dilemmas

XFS or ext4 or something else

The choice of filesystem varies for your use. Distros such as RHEL/CentOS has moved to XFS by default from v7. Debian/Ubuntu still sticks to ext4, like most other. I'll discuss my thoughts below on ext4 vs XFS and leave the other filesystems for later.

bcachefs

bcachefs is the latest newcomer, which started out as bcache, a caching system for linux, using SSDs or NVMEs or similar to cache slower HDDs. It has later evolved to a full filesystem, supporting similar to mirroring, but across multiple drives of different sizes, as some may know the commercial unraid does. It supports full checksumming and healing of data when errors occur. It stripes data across available media, making sure there's at least two (or more, depending on configuration) drives holding the data to allow for failures. Bcachefs also supports similar to RAID levels similar to RAID5 or RAID6 with erasure coding. The latter is not quite stable at the time of writing (2022-09-25), but should work well soon (fingers crossed).

btrfs

btrfs, pronounced "butterfs" or "betterfs" or just spelled out, is a filesystem that mimics that of ZFS. It has been around for 10 years or so, and has proven stable as a dropin replacement for xfs and ext4 and so on, but its real magic in building a full stack with a full RAID subsystem, hasn't yet proven to be stable. In fact, after 15 years of development, it's hardly usable for that. So you might want to use it for its snapshots and compression, btrfs send/receive etc, but preferably with something below, like mdraid and lvm to help out with redundancy and perhaps SSD caching.

ext4

ext4 is probably the most used filesystem on linux as of writing. It's rock stable and has been extended by a lot of extensions since the original ext2. It still retains backward compatibility, being able to mount and fsck ext2 and ext3 with the ext4 driver. However, it doesn't handle large filesystems very well. If a filesystem is created for <16TiB, it cannot be grown to anything larger than 16TiB. This has changed around 2020, though, with mot filesystems created for 64bit use. You still can't change a 32bit filesystem to 64bit. One other issue is handling errors. If a large filesystem (say 10TiB) is damaged, an fsck may take several hours, leading to long downtime. When I refer to ext4 further in this text, the same applies to ext2 and ext3 unless otherwise specified. As mentioned below, ext4 easily outperforms xfs on metadataoperations, so if you have a filesystem with millions of files, ext4 is your friend. However, ext4 still uses static inode allocation and if a filesystem's inode tables are full (one inode per file/directory/symlink etc), so is the filesystem. Read the manual and lookup the -N flag in mkfs.ext4.

XFS

XFS, originally developed by SGI some time in the bronze age, but has been worked on heavily after that. RHEL and Centos version 7 and forward, uses XFS as the default filesystem. SuSE does the same. It works well, but for metadata operations, it's significantly slower than that of ext4, so if you need fast file access for millions of files on a filesystem, ext4 is still your friend. Just read the notes in the above paragraphs. XFS still scales well, but it lacks at least two things ext4 has. It cannot be shrunk (not that you normally need that, but nice if you have a typo or you need to change things). Also, it doesn't allow for automatic fsck on bootup. If something is messed up on the filesystem, you'll have to login as root on the console (not ssh), which might be an issue on some systems. The fsck equivalent, xfs_repair, is a *lot* faster than ext4's fsck, though.

ZFS

ZFS is like a combination of RAID, LVM and a filesystem, supporting full checksumming, auto-healing (given sufficient redundancy), compression, encryption, replication, deduplication (if you're brave and have a ton of memory and can afford some risks) and a lot more. It's a separate chapter and needs a lot more talk than paragraph here.

Low level disk handling

This section is for low-level handling of disks, regardless of storage system elsewhere. These apply to mdraid, lvmraid and zfs and possibly other solutions, with the exception of individual drives on hwraid (and maybe fakeraid), since there, the drives are hidden from Linux (or whatever OS).

S.M.A.R.T. and slow drives

Modern (that is, from about 1990 or later) have S.M.A.R.T. (or just smart, since it's easier to type) monitoring built in, meaning the disk's controller is monitoring itself. This is handy and will ideally tell you in advance that a drive is flakey or dying. Modern (2010+) smart monitoring is more or less the same. It can be trusted about 50% of the time. Usually, if smart detects an error, it's a real error. I don't think I've seen it reporting a false positive, but others may know better.

smartmontools

First, install smartmontool and run smartclt -H /dev/yourdisk (/dev/sda or something) to get a health report. Then perhaps run smartctl -a /dev/yourdisk and look for 'current pending sectors' This should be zero. Also check the temperature, which normally should be much above 50.

single, slow drive

What may happen, is smart not seeing anything and life goes on like before, only slower and less prouductive, without telling anyone. If you stubler over this issue, you'll probably see it during a large rsync/zfs send/receive or a resync/rebuild/resilver of the raid/zpool. During this, it feels like everything is slow and your RAID has turned into a RAIF (redundant array of inexpensive floppies). To check if you have a single drive messing up it all, check with iostat -x 2, having 2 being the delay between each measurement. Interrupt it with ctrl+x (if you're like Neo) or ctrl+c if you're someone else. If there's a rougue drive there, it should stand out like sdg below

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.38    0.00    1.75    0.00    0.00   97.87

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sda              0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00
sdb              0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00
sdc              0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00
sdd              0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00
sde              0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00
sdf              0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00
sdg              3.50    0.00   2352.00      0.00     0.00     0.00   0.00   0.00 14306.29    0.00  13.85   672.00     0.00 285.71 100.00
sdh              0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00

In ZFS land, you should be able to get similar data with zpool iostat -v <pool>

Now you know the bad drive (sdg), pull it from the raid with mdadm --fail /dev/mdX /dev/sdX. Now test the drive's performance manually, just simply:

# hdparm -t /dev/sdg

/dev/sdg:
 Timing buffered disk reads:   2 MB in  5.37 seconds = 381.46 kB/sec

Bingo - nothing you'd want to keep. For a good drive, it should be something like 100-200MB/s

PS: Keep in mind that you have a degraded RAID by now in case you had a spare waiting for things to happen.

Try to replace the cable and/or try the disk on another port/controller. If the result from hdparm persists, it's probably a bad cable

So, then, just psycially locate the drive, pull it out, take out the discs and magnets and recycle the rest.

"Unplug" drive from system

# Find the device unit's number with something like
find /sys/bus/scsi/devices/*/block/ -name sdd
# returning /sys/bus/scsi/devices/3:0:0:0/block/sdd here, 
# meaning 3:0:0:0 is the SCSI device we're looking for.

# Given this is the correct device, and you want to 'unplug' it, do so by
echo 1 > /sys/bus/scsi/devices/3:0:0:0/delete

Rescan disk controllers

If a drive is added and for some reason doesn't get detected, or is removed by the command above, it can be rediscovered with a sysfs command to the scsi host to which it is connected. Since there may be quite a few of these, an easy way is to just scan them all and check the output of dmesg -T after running it.

# Make a list of controllers and send a rescan message to each of them. It won't do anything 
# for those where nothing has changed, but it will show new drives where they have been added.

for host_scan in /sys/class/scsi_host/host*/scan
do
    echo '- - -' > $host_scan
done

Rescan disks

Just like disk controllers, if something changes, it isn't always visible without a reboot. This may be something like extending a vmware drive. If you add one, the linux guest will recognize it immediately, but if you resize it, this may not be the case. To force a rescan, try this.

for disk_rescan in /sys/class/block/sd*/device/rescan
do 
    echo 1 > $disk_rescan
done

For both this and the controller rescans, just run lsblk to check for updates. You should also find the changes in dmesg -T

Fail injection with debugfs

https://lxadm.com/Using_fault_injection

mdadm stuff

Resync

To resync a raid, that is, check, run

echo check > /sys/block/$dev/md/sync_action

where $dev is something like md0. If you have several raids, something like this should work

for dev in md{0,1}
do
  echo repair > /sys/block/$dev/md/sync_action
done

Also useful in a one-liner like

for dev in md{0,1} ; do echo repair > /sys/block/$dev/md/sync_action ; done

Different raids on different drives will do this in parallel. If the drives are shared somehow with partitions or similar, the check or repair will wait for the other to finish before going on.

As for repair vs check, this article describes it well. I stick to using repair unless it's something rare where I don't want to touch the data.

If you need to stop the check or repair, just

echo idle > /sys/block/$dev/md/sync_action

Spare pools

In the old itmes, mdadm supported only a dedicated spare per md device, so if working with a large set of drives (20? 40?), where you'd want to setup more raid sets to increase redundancy, you'd dedicate a spare to each of the raid sets. This has changed (some time ago?), so in these modern, heathen days, you can change mdadm.conf, usually placed under /etc/mdadm, and add 'spare-group=somegroup' where 'somegroup' is a string identifying the spare group. After doing this, run update-initramfs -u and reboot and add a spare to one of the raid sets in the spare group, and md will use that or those spares for all the raidsets in that group.

As some pointed out on #linux-raid @ irc.freenode.net, this feature is very badly documented, but as far as I can see, it works well.

Example config

# Create two raidsets

mdadm --create /dev/md1 --level=6 --raid-devices=6 /dev/vd[efghij]
mdadm --create /dev/md2 --level=6 --raid-devices=6 /dev/vd[klmnop]

# get their UUID etc

mdadm --detail --scan
ARRAY /dev/md/1 metadata=1.2 name=raidtest:1 UUID=1a8cbdcb:f4092350:348b6b80:c054d74c
ARRAY /dev/md/2 metadata=1.2 name=raidtest:2 UUID=894b1b7c:cb7eba70:917d6033:ea5afd2b

# Put those lines into /etc/mdadm/mdadm.conf and and add the spare-group

ARRAY /dev/md/1 metadata=1.2 name=raidtest:1 spare-group=raidtest UUID=1a8cbdcb:f4092350:348b6b80:c054d74c
ARRAY /dev/md/2 metadata=1.2 name=raidtest:2 spare-group=raidtest UUID=894b1b7c:cb7eba70:917d6033:ea5afd2b

# update the initramfs and reboot

update-initramfs -u
reboot

# add a spare drive to one of the raids

mdadm --add /dev/md1 /dev/vdq

# fail a drive on the other raid

mdadm --fail /dev/md2 /dev/vdn

# check /proc/mdstat to see md2 rebuilding with the spare from md1

Recovery

The other day, I came across a problem a user had on #linux-raid@irc.freenode.net. He had an old Qnap NAS that had died and tried plugging the drives in to his Linux machine and got various errors. The two-drive RAID-1 did not come up as it should, dmesg was full of errors from one of the drives (both connected on USB) and so forth.

We went on and started by taking down the machine and just connecting one of the drives. I had him check what was assembled with

cat /proc/mdstat

That returned /proc/md127 assembled, but LVM didn't find anything. So running a manual scan

pvscan --cache /dev/md127

This made linux find the PV, VG and a couple of LVs, but probably because the mdraid was degraded, the LVs were deactivated, according to lvscan. Activating the one with a filesystem manually

lvchange -a y /dev/<vg>/<lv>

Substitute <vg> and <lv> with the names listed by vgs and lvs.

This worked, and the filesystem on /dev/<vg>/<lv> could be mounted correctly.

Renaming a RAID

Not finished

The badblock list

md has an internal badblock list (BBL), added around 2010, to list the bad blocks on the disk to avoid using those. While this might sound like a good idea, the badblock list is updated in case of a read error, which can be caused by several things. To make things worse, the BBL is replicated to other drives when a drive fails or the array is grown. So if you start out with sda and sdb and sdb suffers some errors, nothing wrong, md will read from sda, but flag those sectors on sdb as bad. So after a while, perhaps sda dies and gets replaced, with the same content, including the BBL. Then, at some point you might to replace sdb and it, too gets the BBL. So it sticks - forever. Also, there's no real point of keeping a BBL list, since the drive has its own and even if a sector error occurs, the array will have sufficient redundancy to recover from that (unless you use RAID-0 which you shouldn't). If a drive finds a bad sector, it'll be reallocated by the drive itself without the user's/admin's/system's knowledge. If a sector can't be reallocated, it means the drive is bad and should be replaced.

To check if you have a badblock list on a drive, run

mdadm --examine-badblocks /dev/sdX # replace X with something reasonable

There's no official way to remove it and although you can assemble it with the no-bbl option, this will only work if there's an empty BBL. Also, this will require downtime. As far as I can see, the only way to do this currently, is with a wee hack that should work. Just replace the names of mddev/diskdev with your own.

#!/bin/bash

mddev="/dev/md0"
diskdev="/dev/sda"
mdadm $mddev --fail $diskdev --remove $diskdev --re-add $diskdev --update=no-bbl

If there's a BBL on the drive already, replace the --update part to 'force-no-bbl'. This is not documented anywhere as per se, but it works.

I haven't found a way to turn this off permanently, but there may be something coming around when the debate on the linux-raid mailing list closes up on this. There's a bbl=no in mdadm.conf, but that's only for creation. I will continue extensive testing for this.

For more info about this topic, take a look at this article

mdadm tweaks

Migrate from a mirror (raid-1) to raid-10

Create a mirror

mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb /dev/sdc

LVM (pv/vg/lv) on md0 (see above), put a filesystem on the lv and fill it with some data.

Now, some months later, you want to change this to raid-10 to allow for the same amount of redundancy, but to allow more disks into the system.

mdadm --grow /dev/md0 --level=10
mdadm: Impossibly level change request for RAID1

So - no - doesn't work. But - we're on

Since we're on lvm already, this'll be easy. If you're using filesystems directly on partitions, you can do this the same way, but without the pvmove part, using rsync or whateever you like instead. I'd recommend using lvm for for the new raid, which should be rather obvious from this article. Now plug in a new drive and create a new raid10 on that one. If you two new drives, install both.

# change "missing" to the other device name if you installed two new drives
mdadm --create /dev/md1 --level=10 --raid-devices=2 /dev/vdd missing

Now, as described above, just vgextend the vg, adding the new raid and run pvmove to move the data from the old pv (residing on the old raid1) to the new pv (on raid10). Afte rpvmove is finished (which may take awile, see above), just

vgreduce raidtest /dev/md0
pvremove /dev/md0
mdadm --stop /dev/md0

…and your disk is free to be added to the new array. If the new raid is running in degraded mode (if you created it with just one drive), better don't wait too long, since you don't have redundancy. Just mdadm --add the devs.

If you now have /dev/mdstat telling you your raid10 is active and have three drives, of which one is a spare, it should look something like this

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md1 : active raid10 vdc[3](S) vdb[2] vdd[0]
      8380416 blocks super 1.2 2 near-copies [2/2] [UU]

Just mdadm --grow --raid-devices=3 /dev/md1

RAID section is not complete. I'll write more on migraing from raid10 to raid6 later. It's not straight forward, but easier than this one :)

Interrupting an mdadm --grow

While working on a friend's machine, we started growing a raid-10 with two 18TB drives, adding a third. While this worked fine-ish some time, it was slow, possibly beause of a slow USB dock (although everything was USB3 according to lsusb etc). At the worst, we saw speeds of 200kB/s for resync, meaning it'd take 18 months or so to finish, which isn't very enjoying So, we'll need to see how md handles interruption of this resync.

On a similar setup of hers on a test vm, I added a disk to a 5-disk raid-10 and issued mdadm --grow --raid-devices 6 and let it run for some seconds. I then rebooted the machine, reboot, and waited for it to come up. It did, but the filesystem on the raid-10 wasn't mounted. Since it was flagged as nofail in fstab, it didn't bother the boot process, but it was nonetheless unavailable. The status in /proc/mdstat showed auto-read-only and pvs just showed is as (unknown). I tried to echo check > /sys/block/md1/md/sync_action, and this activated the RAID and it continued from where it was interrupted. Apparently, even sending it an idle command is sufficient to wake it up. After this, I could mount the filesystems s usual. I also tried reboot -f and echo b > /proc/sysrq-trigger, the latter being the best analogue to using a hard reset switch. This wikipedia article has more on sysrq. It all worked the same way and after checking sha512sum, the data stored on the original filesystem, was untouched.

Tuning

While trying to compare a 12-disk RAID (8TB disks) to a hwraid, mdraid was slower, so we did a few things to speed things up:

While testing with fio, monitor /sys/block/md0/md/stripe_cache_active, and see if it's close to /sys/block/md0/md/stripe_cache_size. If it is, double the size of the latter

echo 4096 > /sys/block/md0/md/stripe_cache_size

Continue until stripe_cache_active stabilises, and then you've found the limit you'll need. You may want to double that for good measure.

(to be continued)

Copying partition layout

While it's strictly not related to md, it's mostly used there. If you have a, say, set of two 500GB drive in a mirror and one of them dies, you plugin a new 1TB drive and want to use partitions (typically for the boot raid), you may want to restrict the space used on the disk so that the leftovers can be used for other things. The best is then to just use sfdisk to copy the old drive's partition table to the new one. Later, you can just add more partitions on the free space on the larger one. We'll call the 500GB disk /dev/sdg and the 1TB disk /dev/sdh here, for reference.

# sfdisk -d /dev/sdg | sfdisk /dev/sdh

That's it - just add /dev/sdh1 to the md raid.

Links

Below are a few links with useful info

• Irreversible mdadm failure recovery

ZFS

ZFS can do a lot of things most filesystems or volume managers can't. It checksums all data and metadata and so long that the system uses ECC enabled memory (RAM), it will have complete control over the whole data set, and when (not if) an error occurs, it'll fix the issue without even throwing a warning. To fix the issue, you'll obviously need sufficient redundancy (mirrors or RAIDzN).

ZFS send/receive between machines

ZFS has a send/receive mechanism that allows for snapshots to be sent over the wire to a receiving end. This can be a full filesystem, or an incremental change since last send/reveive. In a WAN scenario, you'll probably want to use VPN for this. On a controlled network, sending in cleartext is also possible. You may as well use ssh, but keep in mind that ssh was never designed to be used as a fullblown vpn solution and is just too slow or the task with large amounts of data. You may, though, use mbuffer, if on a loal LAN.

# Allow traffic from the sending IP in whatever firewall interface you're using (if you're using a firewall at all, that is)
ufw allow from x.x.x.x
# 
# Start the receiver first. This listens on port 9090, has a 1GB buffer,
    and uses 128kb chunks (same as zfs):

mbuffer -s 128k -m 1G -I 9090 | zfs receive data/filesystem

To be continued one day… See this for some more. I'll get back with details on it.

Ceph

manuell variant her: https://docs.ceph.com/en/latest/install/manual-deployment/

Gjøre ceph-deploy mulig:

• lage brukere for ceph
• gi nevnte brukere sudo
• lage et keypair på noden som skal kjøre ceph-deploy
• Legge inn dette i authorized_keys for brukeren på samtlige noder

Workarounds så langt:

• Unsupported: Endre /etc/debian_version til 10 midlertidig
• Resolve error: Kommenter vekk 127.0.1.1 i /etc/hosts

ceph-deploy install --no-adjust-repos raidtest.karlsbakk.net
# eventuelt flere mon-instanser (quorum, dvs oddetall er best. 3 er et godt tall)
# i så fall, endre ceph.conf i hjemmemappen til ceph-deploy sin bruker og legg til i initial_nodes
ceph-deploy mon create-initial

# noder som skal kunne kjøre ceph-kommandoen (cluster admins)
ceph-deploy admin raidtest.karlsbakk.net

# noder som skal kjøre ceph mds (metadata)
ceph-deploy mds raidtest.karlsbakk.net

# noder som skal kjøre ceph mgr (mer metadata)
ceph-deploy mgr raidtest.karlsbakk.net

# noder som skal tilby http-gateway inn mot objektlagringen (s3, swift)
ceph-deploy rgw raidtest.karlsbakk.net

Vise tilgjengelige disker

eph-deploy disk list raidtest.karlsbakk.net # list opp disker via ceph-deploy

nuke partisjonstabell/lvm/osv på en eksisterende disk

ceph-deploy disk zap HOST DISK

ceph-deploy osd create --data /dev/sdh raidtest.karlsbakk.net
ceph-deploy osd create --data /dev/sdi raidtest.karlsbakk.net

ceph config set global osd_pool_default_size 2

Konvertere crushmap til enkeltnodebruk: https://docs.ceph.com/en/latest/rados/operations/crush-map-edits/ ceph osd getcrushmap -o crushmap.bin crushtool -d crushmap.bin -o crushmap.txt

General Linux system control

Add more CPU cores (hotplug)

If working with virtual machines, adding a new core can be useful if the VM is slow and the load is multithreaded or multiprocess. To do this, add a new CPU in the hypervisor (be it KVM or VMware or Hyper-V or whatever). Some hypervisors, like VMware, will activate it automatically if open-vm-tools is installed. Please note that open-vm-tools is for VMware only. I don't know of any such thing for KVM or other hypervisors (although I beleive VirtualBox has its own set of tools, but not as a package). If this doesn't work, run this one to add all available cores.

for cpu in /sys/devices/system/cpu/cpu[0-9]*/online
do
    state=$( cat $cpu )
    if [ "$state" -eq 0 ]
    then
        echo 1 > $cpu
    fi
done

Where X is the CPU number you want to add. You can 'cat /sys/devices/system/cpu/cpuX/online' to check if it's online.

Add more memory (hotplug)

As with cores above, the same applies to memory. If the guest can't be informed, you can add all available memory diretly with this

for f in /sys/devices/system/memory/memory[0-9]*/state
do
    state=$( cat $f )
    if [ "$state" = "offline" ]
    then
        echo online > $f
    fi
done

Mount partitions on a disk image

Sometimes you'll have a disk image, either from recovering a bad drive after hours (or days or weeks) of ddrescue, and sometimes you just have a disk image from a virtual machine where you want to mount the partitions inside the image. There are three main methods of doing this, which are more or less synonmous, but some easier than the other.

Basics

A disk image is usually like a disk, consisting of either a large filesystem, a PV or a partition table with one or partitions onto which resides a filesystem, a pv, swap or something else. If it's partitioned, and you want your operating system to mount a filesystem or allow it to see whatever LVM stuff lies on it, you need to isolate the partitions.

Manual mapping

In the oldern days, this was done manually, something like this

root@mymachine:~# fdisk -l /dev/vda
Disk /dev/vda: 25 GiB, 26843545600 bytes, 52428800 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xc37b7ea5

Device     Boot    Start      End  Sectors  Size Id Type
/dev/vda1  *        2048 50333311 50331264   24G 83 Linux
/dev/vda2       50333312 52426369  2093058 1022M  5 Extended
/dev/vda5       50333314 52426369  2093056 1022M 82 Linux swap / Solaris
root@mymachine:~#

To calculate the start and end of each partition, you just take the start sector and multiply it by the sector size (512 bytes here) and you have the offset in bytes. After this, it's merely an losetup -o <offset> /dev/loopX <nameofimagefile> and you have the partition mapped to your /dev/loopX (having X being something typically 0-255. After this, just mount it or run pvscan/vgscan/lvscan to probe whatever lvm config is there, and you should be able to mount it like any other filesystem.

kpartx

kpartx does the same as above, more or less, just without so much hassle. Most of it should be automatic and easy to deal with, except it may fail to disconnect the loopback devices sometimes, so you may have to losetup -d them manually. See the manual, kpartx (8). Please note that partx works similarly and I'd guess the authors of the two tools are arguing a lot of which one's the best.

guestmount

guestmount is something similar again, but also supports file formats like qcow2 and possibly other, proprietary filesystems like vmdk and vdi, but don't take my word for it - I haven't tested. Again, see the manual or just read up on the description here. It seems rather trivial.

Linux on laptops

Wifi with HP EliteBook 725/745 on Ubuntu

The HP EliteBook 725/745 and similar are equipped with a BCM4352 wifi chip. As with a lot of other stuff from Broadcom, this lacks an open hardware description, so thus no open driver exist. The proper fix, is to replace the NIC with something with an open driver, but again, this isn't always possible, so a binary driver exists. In ubuntu, run, somehow connect the machine to the internet and run

sudo apt-get update
sudo apt-get install bcmwl-kernel-source
sudo modprobe wl

If you can't connect the machine to the internet, download the needed packages on another machine and put it on some usb storage. These packages should suffice:

# apt-cache depends bcmwl-kernel-source
bcmwl-kernel-source
  Depends: dkms
  Depends: linux-libc-dev
  Depends: libc6-dev
  Conflicts: <bcmwl-modaliases>
  Replaces: <bcmwl-modaliases>

Then install manually with dpkg -i file1.deb file2.deb etc

After this, ip/ifconfig/iwconfig shouldd see the new wifi nic, probably wlan0, but due to a old, ignored bug, you won't find any wifi networks. This is because the driver from Broadcom apparently does not support interrupt remapping, commonly used on x64 machines. To turn this off, change /etc/default/grub and change the line GRUB_CMDLINE_LINUX_DEFAULT="quiet splash", adding intremap=off to the end before the quote: GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intremap=off". Save the file and run sudo update-grub and reboot. After this, wifi should work well.

Wifi with HP Compaq Presario CQ57 on Ubuntu

The HP Compaq Presario CQ57 uses the RT5390 wifi chipset. This has been supported for quite some time now, and I was quite surprised to find it not working on a laptop a friend was having. The driver loaded correctly, but Ubuntu insisted of the laptop being in flight mode. Checking rfkill, it showed me two NICs.

# rfkill list all
0: phy0: Wireless LAN
	Soft blocked: no
	Hard blocked: yes
1: hp-wifi: Wireless LAN
	Soft blocked: yes
	Hard blocked: no

Turning rfkill on and off resulted in nothing much, except some error messages in the kernel log (dmesg)

[fr. jan. 24 15:10:20 2020] ACPI Error: Field [B128] at bit offset/length 128/1024 exceeds size of target Buffer (160 bits) (20170831/dsopcode-235)
[fr. jan. 24 15:10:20 2020]
                            Initialized Local Variables for Method [HWCD]:
[fr. jan. 24 15:10:20 2020]   Local0: 00000000a34b7928 <Obj>           Integer 0000000000000000
[fr. jan. 24 15:10:20 2020]   Local1: 00000000b0aa6865 <Obj>           Buffer(8) 46 41 49 4C 04 00 00 00
[fr. jan. 24 15:10:20 2020]   Local5: 00000000a9811efd <Obj>           Integer 0000000000000004
[fr. jan. 24 15:10:20 2020] Initialized Arguments for Method [HWCD]:  (2 arguments defined for method invocation)
[fr. jan. 24 15:10:20 2020]   Arg0:   0000000078957d1b <Obj>           Integer 0000000000000001
[fr. jan. 24 15:10:20 2020]   Arg1:   00000000725c9c6e <Obj>           Buffer(20) 53 45 43 55 02 00 00 00
[fr. jan. 24 15:10:20 2020] ACPI Error: Method parse/execution failed \_SB.WMID.HWCD, AE_AML_BUFFER_LIMIT (20170831/psparse-550)
[fr. jan. 24 15:10:20 2020] ACPI Error: Method parse/execution failed \_SB.WMID.WMAD, AE_AML_BUFFER_LIMIT (20170831/psparse-550)

After upgrading BIOS and testing different kernels, I was asked to try to unload the hp_wmi driver. I did, and things changed a bit, so I tried blacklisting it, creating the file /etc/modprobe.d/blacklist-hp_wmi.conf with the following content

# Blacklist this - it doesn't work!
blacklist hp_wmi

I gave the machine a reboot and afte that, the hp-wifi entry was gone in the rfkill output, and things works.

Raspberry pi

I couldn't quite decide if the raspberry pi should be a separate section or part of Linux, but hell, it can run more than Linux, although 99,lots% of people just uses Linux on them. This is a small list of things to know about them; things not on the front page of the manual or perhaps hidden even better.

Which pi?

I just setup three raspberry pi machines and although some are quite different, like v1 to vEverythingelse or the Pi zero versions to the normal ones, not all of us remember how to distinguish between them, and sometimes they're in a nice 3d printed (or otherwise) chassis or otherwise hidden. So, how to check three different ones:

root@home-assistant:~ $ cat /proc/device-tree/model ; echo
Raspberry Pi 2 Model B Rev 1.1

root@green-pi:~ $ cat /proc/device-tree/model ; echo
Raspberry Pi 3 Model B Rev 1.2

roy@yellow-pi:~ $ cat /proc/device-tree/model ; echo
Raspberry Pi 4 Model B Rev 1.1

While this works well, for the latter, the pi4, it doesn't tell how much memory I have. It comes in variants of 1, 2 and 4GB, and this is the latter.

For a more detailed list, the command awk '/^Revision/ {sub("^1000", "", $3); print $3}' /proc/cpuinfo will give you a revision number, detailing the type of pi, who produced it etc. Please see https://elinux.org/RPi_HardwareHistory for a full table.

Growing or shrinking the rootfs

Inhibit growing the rootfs

Raspberry Pi OS (former Raspbian) grows its partition table and rootfs to the media size (usually an SD card) at first boot. While this can be a good idea for most work, sometimes it's not. If you want to change this behaviour, mount the SD card's boot partition, typically a small partition at the start of the medium and change the file cmdline.txt. It should look something like this

console=serial0,115200 console=tty1 root=PARTUUID=f4481065-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait quiet init=/usr/lib/raspi-config/init_resize.sh splash plymouth.ignore-serial-consoles

To

console=serial0,115200 console=tty1 root=PARTUUID=f4481065-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait

Post-install growing of the fs

If you have installed the system and not yet grewn the fs, you'll need to grow the partition table first. The most basic way to do this is to just remove the large partition holding the current OS and creating a new, larger one, starting at the exact same sector and then growing the filesystem in the partition.

root@raspberrypi:~# fdisk /dev/mmcblk0

Welcome to fdisk (util-linux 2.33.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): p
Disk /dev/mmcblk0: 14.9 GiB, 15931539456 bytes, 31116288 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xf4481065

Device         Boot  Start     End Sectors  Size Id Type
/dev/mmcblk0p1        8192  532479  524288  256M  c W95 FAT32 (LBA)
/dev/mmcblk0p2      532480 7774207 7241728  3.5G 83 Linux

Command (m for help): d
Partition number (1,2, default 2): 2

Partition 2 has been deleted.

Command (m for help): n
Partition type
   p   primary (1 primary, 0 extended, 3 free)
   e   extended (container for logical partitions)
Select (default p):

Using default response p.
Partition number (2-4, default 2):
First sector (2048-31116287, default 2048): 532480
Last sector, +/-sectors or +/-size{K,M,G,T,P} (532480-31116287, default 31116287): +4G

Created a new partition 2 of type 'Linux' and of size 4 GiB.
Partition #2 contains a ext4 signature.

Do you want to remove the signature? [Y]es/[N]o: n

Command (m for help): w

The partition table has been altered.
Syncing disks.

root@raspberrypi:~# reboot

After the machine is back up, open a terminal and you should see everything as before, except lsblk should show a larger mmcblk0p2 partition. To resize the filesystem, just run this as root

resize2fs /dev/mmcblk0p2
resize2fs 1.44.5 (15-Dec-2018)
Filesystem at /dev/mmcblk0p2 is mounted on /; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
The filesystem on /dev/mmcblk0p2 is now 1048576 (4k) blocks long.

root@infoskjerm:~# df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       3.9G  3.0G  739M  81% /

Monitoring

vcgencmd from /opt/vc/bin, but symlinked to /usr/bin so it's available in path, is useful for fetching hardware info about the pi. For example, measuring the temperature

root@yellow-pi:~# vcgencmd measure_temp
temp=63.0'C

Or read directly from the sensor - result is given in millidegrees (Celsius)

root@lagus:~# cat /sys/class/thermal/thermal_zone0/temp
50464

The command is generally badly documented and parts of it seems outdated for newer hardware. Here's the output from a Raspberry pi 4 with 4GB RAM

roy@yellow-pi:~ $ vcgencmd get_mem arm
arm=948M
roy@yellow-pi:~ $ vcgencmd get_mem gpu
gpu=76M

Netbooting a pi

On Raspberry pi 2 (a somewhat late revision), 3 (preferably 3+) and 4, you can boot directly off the net. To do this, you need to prepare the pi for netboot, to setup a boot server in the dhcp scope on the dhcp server and setup an NFS server from which to boot. I have a router running OpenWRT, which uses dnsmasq to serve DHCP and I've setup a separate linux vm to serve the boot files over NFS, using LVM with read-write snapshots for each clients, as this serves the role of thin provisioning the system. Details follow below.

Configuring the pi for netboot

The command rpi-eeprom-config is used to setup the pi's eeprom (somewhat similar to that of the BIOS in a PC). We're mostly interested in the boot order here, and several modes may (and should) be given, telling the pi to start trying the first one and then go on further. The default value on a pi4 is 0xf41, so decoded from right, it says "Try SD card, then USB mass storage before falling back to a reboot, restarting the cycle. I chose 0xf241 to allow sd card booting if present, but then falling back to networking. To change this, run

$ sudo rpi-eeprom-config -e

and edit the fields you want to change.

NOTE: I have not tried setting the boot order to 0xf or something like that and I don't know if this will render the pi unsable or not. Better not try it yourself unless you're knee deep in old pi's.

And then, give the pi a reboot.

I'll start by going through pi4, since that's what I have right now. This comes with booting supported out of the box, but not enabled.

Raspberry pi 4 boot modes

Value	Mode	Description
0x1	SD CARD	SD card (or eMMC on Compute Module 4)
0x2	NETWORK	Network boot
0x3	RPIBOOT	RPIBOOT - See usbboot (since 2020-09-03)
0x4	USB-MSD	USB mass storage boot (since 2020-09-03)
0x5	BCM-USB-MSD	USB 2.0 boot from USB Type-C socket or USB Type-A socket on CM4 IO board. (since 2020-12-14)
0xe	STOP	Stop and display error pattern (since 2020-09-03). A power cycle is required to exit this state.
0xf	RESTART	Start again with the first boot order field. (since 2020-09-03)

DHCP

dnsmasqd

To configure the dhcp server, login to the router using ssh, since these can't be set in Lucy, the webui, you just edit the dhcp config in /etc/config/dhcp to something like this

config dhcp 'lan'
; blablabla
        list dhcp_option '66,192.168.10.56'
        option filename 'bootcode.bin'

Note dhcp option 66 - that's 'server name', meaning the server address to the dhcp server.

Now just restart dnsmasq

# /etc/init.d/dnsmasq restart

The pi will contact the tftp server given and request the files there with a prefix of its serialnumber followed by slash. You can find the serialnumber in /proc/cpuinfo thus

cat /proc/cpuinfo |awk '/^Serial/ { print substr($3,9) }'

ISC DHCP Server

If using the isc-dhcpd, the following cofig should work well

# dhcp-options
subnet 10.x.x.0 netmask 255.255.255.0 {
    range 10.x.x.10 10.x.x.199;
    option routers 10.x.x.1;
    default-lease-time 3600;
    max-lease-time 7200;
    filename "bootcode.bin";
    server-name "x.y.z.199";
    next-server x.y.z.199;
    option vendor-class-identifier "PXEClient";
# The one below is "Raspberry Pi Boot" hex (I think).
#   option vendor-encapsulated-options 6:1:3:a:4:0:50:58:45:9:14:0:0:11:52:61:73:70:62:65:72:72:79:20:50:69:20:42:6f:6f:74:ff;
# Apparently this should work like the above, only better, since the three spaces are necessary for older pi revisions.
    option vendor-encapsulated-options "Raspberry Pi Boot   "
}

Thin-provisioning clients

In one setup, I installed a new version of Raspbian on a pi2 with a 64GB SD card. The installer makes sure the partitioning and fs size is changed after the initial bootup, so we'll need to fix that. This can be disabled somewhere (google it), but I forgot that, so I installed the SD card in another machine and checked with lsblk where it was located (sdb, with root on sdb2), so I ran a mandatory fsck -f first, then resized the filesystem to its minimum, removed the partition on which it was located, create a new one with the same starting point, but 10GB in size, added another to fill the gap, grew the current rootfs to the partition's full size and setup LVM.

Below from another card for documentation purposes, this time on sda:

# fsck -f /dev/sda2
fsck from util-linux 2.33.1
...
# resize2fs -M /dev/sda2
Resizing the filesystem on /dev/sda2 to 1239086 (4k) blocks.
The filesystem on /dev/sda2 is now 1239086 (4k) blocks long.
# printf "Filesystem size is %d MB\n" $(( 1239086 / 256 ))
Filesystem size is 4840 MB
# fdisk /dev/sda

Welcome to fdisk (util-linux 2.33.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): p
Disk /dev/sda: 14.9 GiB, 15931539456 bytes, 31116288 sectors
Disk model: STORAGE DEVICE
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xea7d04d6

Device     Boot  Start      End  Sectors  Size Id Type
/dev/sda1         8192   532479   524288  256M  c W95 FAT32 (LBA)
/dev/sda2       532480 31116287 30583808 14.6G 83 Linux

Command (m for help): d
Partition number (1,2, default 2): 2

Partition 2 has been deleted.

Command (m for help): n
Partition type
   p   primary (1 primary, 0 extended, 3 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (2-4, default 2): 2
First sector (2048-31116287, default 2048): 532480
Last sector, +/-sectors or +/-size{K,M,G,T,P} (532480-31116287, default 31116287): +8g

Created a new partition 2 of type 'Linux' and of size 8 GiB.
Partition #2 contains a ext4 signature.

Do you want to remove the signature? [Y]es/[N]o: n

Command (m for help): p
Disk /dev/sda: 14.9 GiB, 15931539456 bytes, 31116288 sectors
Disk model: STORAGE DEVICE
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xea7d04d6

Device     Boot  Start      End  Sectors  Size Id Type
/dev/sda1         8192   532479   524288  256M  c W95 FAT32 (LBA)
/dev/sda2       532480 17309695 16777216    8G 83 Linux

Command (m for help): n
Partition type
   p   primary (2 primary, 0 extended, 2 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (3,4, default 3):
First sector (2048-31116287, default 2048): 17309696
Last sector, +/-sectors or +/-size{K,M,G,T,P} (17309696-31116287, default 31116287):

Created a new partition 3 of type 'Linux' and of size 6.6 GiB.

Command (m for help): t
Partition number (1-3, default 3):
Hex code (type L to list all codes): 8e

Changed type of partition 'Linux' to 'Linux LVM'.

Command (m for help): p
Disk /dev/sda: 14.9 GiB, 15931539456 bytes, 31116288 sectors
Disk model: STORAGE DEVICE
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xea7d04d6

Device     Boot    Start      End  Sectors  Size Id Type
/dev/sda1           8192   532479   524288  256M  c W95 FAT32 (LBA)
/dev/sda2         532480 17309695 16777216    8G 83 Linux
/dev/sda3       17309696 31116287 13806592  6.6G 8e Linux LVM

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

Repartitioning done, now grow the shrunken filesystem to fill up the new, smaller partition

# resize2fs /dev/sda2
resize2fs 1.44.5 (15-Dec-2018)
Resizing the filesystem on /dev/sda2 to 2097152 (4k) blocks.
The filesystem on /dev/sda2 is now 2097152 (4k) blocks long.

This leaves us with sda3, the new one, so we'll create a new volume group on it along with a logical volume to be used as the base later

# vgcreate data /dev/sda3
# lvcreate -n _client_ -L 10g data
# mkfs -t ext4 /dev/data/_client_

BIOS upgrade from Linux

Generally, BIOS upgrades have been moved to the BIOS itself these days. This saves a lot of work and quite possibly lives after those who earier rammed their heads through walls, now can upgrade directly from the internet instead of using obscure operating systems best avoided. Sometimes, however, one must use these nevertheless. This is a quick run-through on your options.

DOS

Most non-automatic BIOS upgrades are installed from DOS. Doing a BIOS upgrade this way, is generally non-problematic. Just install a USB thingie with FreeDOS, copy your file(s) onto the thing and boot on it. The commandline is the same as old MS/DOS, except a wee bit more userfriendly, but not a lot.

Windows

Some macahines, like laptops from HP, may have BIOS upgrades that are made to be installed from Windows and won't run in FreeDOS or MS/DOS, instead throwing an error, saying This program cannot be run in DOS mode. This means you'll have to boot on a Windows rescue disc and do the job from there. Googling this, tells you it's quite easy, you just choose 'make rescue disc' from Windows, and it's all good, except if you don't run Windows, that is. To remedy this problem, do as follows:

Download Windows

Since you don't run Windows and possibly don't have a spouse or friend around with such unhealthy habits either, you need to get it. It's quite easy - just google 'download windows' and it'll send you to somewhere like this.

Burn it!

Now it's time to burn the installer on a DVD ROM. You'll need a double-layered one, since the OS is > 4.7GiB, but I'm sure you have a ton of those lying around. Now, just place your optical medium in your optical drive and burn, baby, burn!

Or make a USB thing?

Not a lot of machines have optical drives these days, so you may want to use an USB pen drive or something instead. This is easy, just make the USB bootable with the Windows application Microsoft has made for this. Unfortunately, this only runs on Windows, and unlike stuff like Debian, where the iso file can be dd'ed directly onto a USB drive to make it bootable, the Windows isos don't come with this luxury. There are lots of methods to make bootable USB things from iso files out there, but the only thing most of them have in common, is that they generally don't work.

WoeUSB

After hours of swearing, it's nice to come across software like WoeUSB. It may not be perfect, it relies on a bunch of GUI stuff you'll never need and so on, but it works, and that's the important part! Just clone the repo and RTFM and it's all nice. It's slow, but hell, getting a windows machine and/or an optical drive might be slower.

WoeUSB does nothing fancy, but it does work. It will create a filesystem, FAT32 or NTFS (better use NTFS, FAT32 has its limits). It creates normal filesystems and so on. Just make sure to copy in the BIOS update file, usually an exe file, to run when Windows is up and running.

Install BIOS

Boot on the Windows USB thing and choose 'repair computer' and then 'command prompt'. Find the install file, and if you're lucky, you'll find it needs a 32bit OS, just like I did. Since the 64bit version doesn't include 32bit compatibility in the installer, revert to downloading the 32bit version of windows and start over. Pray to your favourite god(s) not to get across such machines again - it might help.

3D printer stuff

Below are a number of not very ordered paragraphs about 3D printer related stuff, mostly based on my experience. Their content may be interesting or perhaps practical knowledge, but then, that depends on the reader.

3D printer related introductions on youtube or elsewhere

First off, I'd like to mention some youtube videos that are all (or mostly) a nice start if you're new to 3D printers.

Thomas Salanderer's 10-video series on 3D printing basics

Thomas Salanderer is a an experienced 3D printer user/admin/fixer/etc and he has a lot of interesting videos. Some accuse him for being a [Prusa https://www.prusa3d.com/] fanboy, which he quite obviously is, but that doesn't stop him from making interesting and somewhat neutral videos. The series walks you thought the first steps of how things work and so on and hopefully opens more doors than anything else I've seen.

Teaching Tech's "3D printer tuning reborn"

Teaching Tech is, like Salanderer or even more, good pedagogically and explains a lot. They (or he) do a bunch of testing and describing pros and cons with different things, just like other youtubers in the sme business. The mentioned video shows how to tune a 3D printer after you have first learned the basics, and leans on a webpage made to help you out without too much manual work.

This was a short list, but I guess it'll grow over time.

Hydrophilic filament

Most popular filament types, including PLA, PETG, PP or PA (Polyamide, normally known as Nylon) and virtualy any filament type named poly-something (and thus with an acronym of P-something) are hydrophilic (also (somewhat incorrectly) called hydroscopic), meaning so long they're dryer than the atmosphere around them, they'll do their best to suck out the atmosphere's humidity. This is why most filament are shrink-wrapped with a small bag of silica gel in it. If such filament is exposed for normal humid air for some time, it'll become very hard to use. Most of my experience is with PLA, which can handle a bit (depending on make), but still not a lot. With PLA, if you end up with prints where the filament seems not to stick, it may help with a higher temperature. Otherwise, the filament must be baked. If you don't want to use your oven, try something like a fruit and mushroom dryer. Then get a good, sealble plastic box and add something like half a kilogram of silica gel to an old sock, tie it and put it in the your new dry box.

Please note that ABS is hydrophobic, so you won't have to worry too much there. Also, remember that PA/Nylon is extremely hydrophilic. Even a couple of days in normal air humidity can ruin it completely and will require baking.

Upgrading the firmware on an Ender 3

This is about upgrading the firmware, and thus also flashing a bootloader to the Creality Ender 3 3D printer. Most of this is well documented on several place, like o the video from Teaching tech and elsewhere, but I'll just summarise some issues I had.

Using an arduino Nano as a programmer

Quick note before you read on. If you have an Ender 3 controller version 1.1.5 or newer (or perhaps even a bit older), a CR-10S or some other more or less modern printers or controllers, they come with a bootloader installed already, so don't bother flashing one. The one that's in there already, should work well. So if you have one of these, just skip this and jump to the next section.

However, the normal CR-10 (without the S), Ender 3 and a few other printers from Creality come without a bootloader, so you'll need to flash one before upgrading the firmware. Well, strictly speakning, you don't need one, you can save those 8kB or so for other stuff and use a programmer to write the whole firmware, but then you'll need to wire up everything every time you want a new firmware, so in my world, you do need the bootloader, since it's easier.

To install a bootloader, you'll need a programmer, and I didn't have one available. Having some el-cheapo china-copies of Arduinos around, I read I could use one and tried so. Although the docs mostly mention the Uno etc, it's just as simple with the Nano copy.

So, grab an arduino, plug it to your machine with a USB cable, and, using the Arduino IDE, use the ArduinoISP sketch there (found under Examples) to burn the software needed for the arduino to work as a programmer. When this is done, connect a 10µF capacitor between RST and GND to stop it from resetting when used as a programmer. Connect the MOSI, MISO and SCK pins from the ICSP block (that little isolated 6-pin block on top of the ardu). See here for a pinout. Then find Vcc and GND either on the ICSP block or elsewhere. Some sites say that on some boards you shouldn't use the pins on the ICSP block for this. I doubt it matters, though. Then connect another jumper wire from pin 10 on the ardu to work as the reset pin outwards to the chip being programmed (that is, on the printer). The ICSP jumper block pin layout is the same on the printer and on the ardu, and probably elsewhere. Sometimes standardisation actually works…

When it's all wired up, change the setup in the Arduino IDE to use the Sanguino board (which may need to be installed there), the Atmega 1284 16MHz board with the programmer set to "Arduino as ISP". Also make sure to choose the right serial port. When all is done, choose Tools | Burn programmer and it should take a few seconds to finish.

Flashing the printer

When the boot loader is in place, possibly after some wierd errors from during the process, the screen on the 3d printer will go blank and it'll behave like being bricked. This is ok. Now disconnect the wires and connect the USB cable between computer and printer. If you haven't installed support for the Sanguino board, that is, the variant of the 1284-chip and surrounding electronics that is the core of the, you need to install it first. In the Arduino IDE, choose the Tools / Boards / Boards manager boot option and search for Sanguino. After this, you should find the Sanguino or Sanguino 1284p in the boards menu. After this, you should be able to communicate with the printer's controller. Download the TH3D unified firmware, making sure it's the last version. Uncompress the zip and with Finder/Explorer/whateversomething'scalledonlinux, browse to TH3DUF_R2/Firmware/TH3DUF_R2.ino and open it. It should open directly in the Android IDE. Keep in mind that the names TH3DUF_R2 and TH3DUF_R2.ino will vary between versions, but you get the point. Now configure it for your particular needs. You'll find most of this in the Configuration.h tab (that's really a file). Most of the other files won't be necessary unless you're doing some more advanced stuff, like replacing th controller with something non-standard or similar, but then again, that's another chapter (or book, even). The video mentioned above describes this well. After flashing the new firmware, you'll probably get some timeouts and errors, since apparently it resets the printer after giving it the new firmware, but without notifying the flashing software of it doing so. If this happens, calm down and reboot the printer and check its tiny monitor - it should work. If it doesn't work, and if you flashed the bootload yourself, try to flash it again, and if that doesn't work, try flashing the programmer again yet another time, just for kicks. Again, if you have a newer controller like the v1.1.5, don't touch the bootloader, as the one already installed, should work well as it is.

PS: I tried enabling MANUAL_MESH_LEVELING, which in the code says that If used with a 1284P board the bootscreen will be disabled to save space. This effectively disabled the whole printer, and apparently may be making the firmware image a wee bit too large for it to fit the 1284P on the ender. It works well without it, though, and it may be fixed by now, since I tested this back in early 2019.

And then…

With the new firmware in place, the printer should work as before, although with thermal runaway protection to better avoid fires in case the shit hits the fan, and to allow for things like autolevelling. With any luck, this thing may be ready for use by the time you read this - it surely looks nice!

Octoprint/Octopi

This could have been under some other section, but again, I just branch out even though most of it is about sections mentioned elsewhere.

When setting up a 3d printer for the first time, you usually get an SD card for storage and transfer data to the printer using Sneakernet. Some printers use micro SD cards, while other fullsize SD cards, which imho is better, since they don't get lost that easily, but otherwise do the same thing. This works, but is somewhat tedious. If you want to control the printer from a PC or phone, there's a simple trick for that as well - octoprint.

Octoprint, written by Gina Häußge, runs on most platforms including linux, windows, BSD*, macos etc. It supports controlling a camera to some extent out of the box and there's a large number of plugins availablee to do a lot of things you possibly haven't thought of (and quite often, you'd ever need). The easiest way to install it, is to just grab a Raspberry pi - preferably some of the newer models (will get to that later) and download octopi and follow the instructions on that page.

Performance issues

Some report (and I have seen it myself) issues with the pi's performance with regards to both handling the octoprint processes (which should be as close to realtime as possible) and handling other stuff like I/O, networking etc. The point is that at pi3 or older, has all peripherals connected to an internal USB hub. This might be practical, but performance-wise it's not. If you in addition connect a webcam to USB, you're asking for trouble, since USB will be your bottleneck. With a raspberry pi camera, the ones connected to the pi directly with a ribbon cable, this should not use USB, so it's better. Still, again, for older pi's, there might be some shortage in therms of cpu or i/o. The octopi runs something like raspbian which is a fork of debian which is a linux distro, so it all boils down to knowing linux.

Your typical octopi will run octoprint, a streamer, usually mjpg-streamer, a simple, lightweight videostreamer that outputs a stream of jpeg-images (about the same format as cinemas use - that is - not MPEG). This may be a bit tough on the cpu and potentially i/o. Add inn a dash of USB overload and you're may see trouble.

Still - a pi3 should be able to handle the load, but it might help to prioritise corretly. The octoprint process is the important part here, so we could give that full priority both in CPU and I/O and. Unfortunately, it doesn't seem like the current octopi has been migrated to using systemd for the startup. We'll deal with this later. To fix this in the old SysV init style file present there, the following patch should work against the file /etc/init.d/octoprint

--- octoprint.old	2020-07-20 18:46:10.166651965 +0200
+++ octoprint	2020-07-20 18:53:21.724803211 +0200
@@ -22,6 +22,9 @@
 PIDFILE=/var/run/$NAME.pid
 PKGNAME=octoprint
 SCRIPTNAME=/etc/init.d/$PKGNAME
+NICELEVEL=-19                        # Top CPU priority
+IONICE_CLASS=1                       # Realtime I/O class
+IONICE_PRIORITY=0                    # Realtime I/O priority (probably not needed with class=realtime)

 # Read configuration variable file if it is present
 [ -r /etc/default/$PKGNAME ] && . /etc/default/$PKGNAME
@@ -70,10 +73,11 @@

    is_alive $PIDFILE
    RETVAL="$?"
-
    if [ $RETVAL != 0 ]; then
        start-stop-daemon --start --background --quiet --pidfile $PIDFILE --make-pidfile \
-       --exec $DAEMON --chuid $OCTOPRINT_USER --user $OCTOPRINT_USER --umask $UMASK -- serve $DAEMON_ARGS
+       --exec $DAEMON --chuid $OCTOPRINT_USER --user $OCTOPRINT_USER --umask $UMASK \
+       --nicelevel $NICELEVEL --ioched $IONICE_CLASS:$IONICE_PRIORITY \
+       --serve $DAEMON_ARGS
        RETVAL="$?"
    fi
 }

This ismply upgrades the process to be allowed to use most of the resources on the pi, regardless of what other processes are whining about.

PS: Code not tested - I don't have a pi right here. Will test later, but I'm pretty sure it'll work. After all, it's just a few new arguments to start-stop-daemon

Samba

Apple Time Machine

Time machine is a nice backup tool from Apple, running on their macOS operating system. It was introduced with Mac OS X 10.5 Leopard in October 2007. Apple's own documentation tells us to get an Apple Time Capsule (or Airport Time Capsule), that is, a small, network-attached external drive. Either that, or a normal USB drive of sorts. Some of us have home servers, so apparently, it's not hard to setup that with Samba (Windows file server emulator - ish).

For each mac computer, create a Samba account and for each account, create a share. This will be setup with a few extra flags, as described below. Note that it must be a separate share per backed up computer - a subfolder per user will not do. Apple Time Machine must be able to beleive it ows that 'disk'.

Add this to your /etc/samba/smb.conf:

[time_machine_someuser]
    comment = Time Machine backup for Mac
    path = /data/time_machine/someuser
    browseable = yes
    read only = no
    valid users = someuser
    vfs objects = catia fruit streams_xattr
    fruit:time machine = yes

Reload samba with systemctl reload smb and connect to it from your mac (CMD+k in Finder, write smb://ip.addr.of.server/, press enter, browse to time_machine_someuser and connect) and open your Time Machine settings under System Settings and click Add Drive

roy