From the Trenches, Tips & Tricks Edition: Hacking "/ on ZFS" and GELI Encrypted Drives, the Old-School Way

Glen Barber is back to kick off our latest From The Trenches series: The Tips and Tricks Edition. 

All my personal machines run FreeBSD.

In fact, all my personal machines run FreeBSD-CURRENT. I do this primarily to keep track of changes that get committed to the head branch, so I can personally test changes (for the things I use, at least) before they get merged to the stable branches.

As one of the Release Engineers, I find it essential that, whenever possible, I find issues so they can be corrected before they are part of a release.

My primary work machine is a laptop, currently a Lenovo Thinkpad T540p. I picked this laptop, and all the other laptops before it, because it met my minimum requirements for a primary workstation: it is capable of supporting a large amount of RAM (16GB for my Thinkpad, 8GB for all previous laptops), an Intel Core i7 CPU, and I could replace the DVD drive with a second hard drive.

In addition to these hardware requirements, I also have a few personal requirements of any workstation - the drives must be encrypted, and the underlying filesystem must be ZFS.

For me, it is not so much about the data I have *on* the laptop that I need to protect, but the kinds of things within the FreeBSD Project I am permitted access. Without encrypted drives, a lost or stolen laptop would absolutely be my worst possible nightmare, because I only have my login passphrase protecting my data (GPG key, SSH keys, and so on).

Recent FreeBSD releases allow "/ on ZFS" installation with the option to enable GELI-based encryption. This predates my original installation, however, since each laptop I have purchased for the past several years used the hard drives from the previous laptop. According to zpool history, the installation was at least two and a half years ago, but I know it is much longer than that, because of zfs recv being one of the first things zpool history reports.

So, I needed to do things the old-fashioned way, and manually create the GELI-backed providers and perform the "/ on ZFS" installation myself.

While bsdinstall(8) may now cover the majority of use cases for such installations, there may be cases where someone specifically needs to do something a certain way that the installer does not provide.

Because I only had one hard drive in the system when the system was initially installed (a long time ago), I will only refer to one hard drive when describing the steps I used to perform the installation, for now.

I installed the system using the 9.0-RELEASE or 9.1-RELEASE memory stick installer (memstick.img), I cannot remember which, but that detail is not as important, since I did not use the installer anyway.

When I booted from the memory stick, the two drives recognized on the system were the internal hard drive, /dev/ada0, and the external USB flash drive for the installation, /dev/da0. The first menu screen has three options available: "Install", "Shell", "Live CD".

I selected "Live CD", and logged in as root (no password is necessary for the "Live CD" functionality). The hard drive did not have an operating system. Because I purchased the hard drive, in addition to the laptop, with the intention of replacing the laptop's drive, I did not need to remove any partitions from an existing installation. If I did need to remove partitions, I would have done so with:

# gpart destroy -F ada0

Here is where some technical details become important:

• While you can install "/ on ZFS" on a drive partitioned with MBR (Master Boot Record), using GPT is far easier. In fact, I have forgotten much about how MBR partitioning is actually done.
• When doing full disk encryption, you must keep /boot contents separate, otherwise loader(8) and the kernel will not be available when the BIOS hands over control to the operating system. As such, /boot should be given its own partition on the disk left unencrypted, and the rest of the system on its own encrypted partition.

I created four partitions on the drive. The first partition is for the boot blocks (not to be confused with the /boot contents), the second partition is for /boot, the third is for the encrypted system, and the fourth is for swap.

# gpart create -s gpt ada0
# gpart add -t freebsd-boot -s 512k -i 1 -l gptboot ada0
# gpart add -t freebsd-zfs -s 10G -i 2 -l bootfs ada0
# gpart add -t freebsd-swap -s 10G -i 3 -l swapfs ada0
# gpart add -t freebsd-zfs -s 180G -i 4 -l rootfs ada0

I decided to put the swap partition between the /boot partition and the rest of the system, in case I needed to increase or decrease the size of the /boot partition, it would be far easier (and safer) to do.

Then, I loaded the necessary kernel modules for ZFS and GELI:

# kldload /boot/kernel/opensolaris.ko
# kldload /boot/kernel/zfs.ko
# kldload /boot/kernel/geom_eli.ko

Now that GELI functionality is available, I created the backend provider for the ZFS dataset:

# geli init -b -a HMAC/SHA256 -e AES-CBC -l 256 \
    -s 4096 /dev/ada0p4

Then I attached the GELI provider, and wrote data from /dev/random to the new device /dev/ada0p4.eli:

# geli attach ada0p4
# dd if=/dev/random of=/dev/ada0p4.eli bs=4096

This took a while on the system this hard drive was originally installed, so I probably got coffee at this point. :-)

When the dd(1) command finished, I continued the installation.

I created temporary directories to use to import the pools after they were created:

# mkdir /tmp/zroot
# mkdir /tmp/zboot

Keep in mind, I am installing from a memory stick image, which by default, is read-only. The /tmp directory is writable, however, because it is a md(4)-backed memory disk filesystem.

# zpool create -O checksum=fletcher4 -O atime=off \
    -m /tmp/zboot zboot /dev/ada0p2
# zpool create -O checksum=fletcher4 -O atime=off \
    -m /tmp/zroot zroot /dev/ada0p4.eli

Then I made a few ZFS datasets for various paths:

# for i in var var/log var/tmp var/db usr usr/home \
    usr/compat usr/ports \
    usr/local tmp; do \
    zfs create zroot/${i} \
    done

I also made a separate ZFS dataset for the "bootfs" contents, and set the mountpoint to the /boot directory in the temporary working directory:

# zfs create zboot/boot
# zfs set mountpoint=/tmp/zroot/boot zboot/boot

On the memory stick installation media, the distribution sets are located in /usr/freebsd-dist. I extracted their contents into the newly-created filesystem:

# cd /tmp/zroot
# for i in base kernel lib32; do \
    tar -xf /usr/freebsd-dist/${i}.txz -C . \
    done

Then I wrote the bootcode to the first partition of the drive:

# gpart bootcode -b /tmp/zroot/boot/pmbr \
    -p /tmp/zroot/boot/gptzfsboot -i 1 ada0

Because the "bootfs" (/boot) and "rootfs" (everything else) are both ZFS, I needed to use the gptzfsboot bootcode for the "freebsd-boot" partition.

Now the system is installed, but I needed to make a few modifications before I was ready to reboot. In particular, set a root password, edit /etc/fstab to enable swap, edit /etc/rc.conf to enable the zfs rc(8) startup script, and edit /boot/loader.conf to load the geom_eli.ko, opensolaris.ko, and zfs.ko kernel modules at boot.

# chroot /tmp/zroot
# passwd root
[enter password]
# echo '/dev/gpt/swapfs none swap sw 0 0' \
    >> /etc/fstab
# echo 'zfs_enable="YES"' >> /etc/rc.conf
# echo 'geom_eli_load="YES"' >> /boot/loader.conf
# echo 'zfs_load="YES"' >> /boot/loader.conf
# exit

Before rebooting, I needed to make a few adjustments to where /boot from the zboot/boot dataset would be mounted at boot.

# zfs umount zboot/boot
# zfs set mountpoint=/realboot zboot/boot

This now makes the /boot directory mount as /realboot, so I then needed to point /boot in the zroot dataset to the correct place. This was easily solved with a symbolic link:

# cd /tmp/zroot
# ln -s boot /realboot

Now when the system boots, the filesystem will look something like this:

/bin
/sbin
/boot -> /realboot
/realboot
[...]

Finally, I needed to unmount the zroot dataset, and fix its mountpoints. I only needed to change the zroot mountpoint itself, since all children datasets adjusted their paths automatically.

# zfs umount -a
# zfs set mountpoint=/ zroot

At this point, the installation was complete. I rebooted the laptop, entered the GELI passphrase for /dev/ada0p4.eli when prompted, and was greeted by the "login: " prompt we have all grown to love.