Part 1

Before continuing, be sure to read part 1 of this blogpost.

What you will need

1 at least 1GB USB stick for the FreeBSD installer image
1 or more xGB USB sticks for the boot files and encryption keys

You should create multiple copies of the USB stick that holds the boot files and encryption keys. If you lose the stick or the data gets corrupted and you don’t have another copy, all your data stored on the encrypted disks is lost.

Booting the FreeBSD Installer

I’m using a USB stick with the FreeBSD 10 memstick image to boot into the FreeBSD installer. See here for a Mac OS X guide on how to get the memstick image onto a USB stick.

Now after your system finishes booting from the USB stick, it should present you with a blue, text-based installer giving you three options:

Install
Shell
Live CD

We will start by dropping into the shell and run su - to get a root shell.

SSHd

I will assume that your server is connected to your LAN during the installation. That way we can start an SSH daemon from the installer image and use our Mac or PC to enter the setup commands or copy files to the server.

So, on the shell on your server, run

ifconfig
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
	ether XX:XX:XX:XX:XX:XX
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 
	inet 127.0.0.1 netmask 0xff000000 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

to identify your network interface name. In my case it’s bge0.

And then:

dhclient <your-network-interface-name>
DHCPDISCOVER on bge0 to 255.255.255.255 port 67 interval 8
DHCPOFFER from 192.168.1.1
DHCPREQUEST on bge0 to 255.255.255.255 port 67
DHCPACK from 192.168.1.1
bound to 192.168.1.45 -- renewal in 21600 seconds.

to get an IPv4 address, in my case 192.168.1.45.

If your LAN does not offer you an IP address via DHCP, run man ifconfig and read up on how to configure a network card manually.

Let’s say your server is now connected to your LAN and has an IPv4 address. We can now start an SSH daemon by running:

mkdir /tmp/etc
mount_unionfs /tmp/etc /etc
echo "PermitRootLogin yes" >> /etc/ssh/sshd_config
passwd root
service sshd onestart

The root password you are asked to enter is just for the installer; it’s not the root password you will use later for your installation.

Now to login to the installer image by running ssh root@<ip-address-of-your-server> on your Mac or PC.

This guide should also work if you enter all the commands on the command line yourself, but doing it over SSH is more convenient.

Identifying your disks

Now let’s see which storage devices are connected to your server:

camcontrol devlist
<VB0250EAVER HPG9>                 at scbus0 target 0 lun 0 (pass0,ada0)
<VB0250EAVER HPG9>                 at scbus1 target 0 lun 0 (pass1,ada1)
<ST4000VN000-1H4168 SC43>          at scbus2 target 0 lun 0 (pass2,ada2)
<ST4000VN000-1H4168 SC43>          at scbus3 target 0 lun 0 (pass3,ada3)
<Sony USB Stick>                   at scbus6 target 0 lun 0 (pass4,da0)

In my case I have four hard disks and one USB stick (da0) We’ll create two zpools, one for the OS installation and for data. In my case I’ll use the disks ada0 and ada1 for the OS and ada2 and ada3 for my data.

The device names are probably different on your system. Please consult FreeBSD Disk device names to find out how FreeBSD names attached storage devices.

Randomizing

We will start by writing random data to the two operating system disks.

dd if=/dev/random of=/dev/ada0 bs=1m &
dd if=/dev/random of=/dev/ada1 bs=1m

This will take a very long time, depending on how big your disks are.

Partitioning

Now let’s start partitioning the disks. This is what the layout will look like in the end:

| Hard Disk Device | Partition 1               | Partition 2                        |
-------------------------------------------------------------------------------------
| ada0             | ada0p1 freebsd-swap       | ada0p2 freebsd-zfs OS installation |
| ada1             | ada1p1 freebsd-swap       | ada1p2 freebsd-zfs OS installation |
-------------------------------------------------------------------------------------

As I said, we are going to store the bootcode, kernel and keyfiles on a USB stick, so there is no need for a boot partition.

You might also notice that we’ll create separate swap partitions and won’t use a ZVOL for swap. Here is why.

The next steps will destroy any data on your drives:

To better understand the following commands it would be a good idea to read the manpage of gpart: man gpart.

Clean the drives of existing partition tables:

gpart destroy -F ada0
gpart destroy -F ada1

If you get a message like gpart: arg0 'ada2': Invalid argument, that’s fine and you can ignore it. It just means that there was no partition table on the disk anyway.

Create a GPT partition table on each disk:

gpart create -s gpt ada0
gpart create -s gpt ada1

Nowadays, disks (especially very large ones) use a sector format called “Advanced Format”. Long story short, even if you don’t have Advanced Format disks, we are going to align the partitions with 4K sectors. This blog post explains it quite well in a ZFS/FreeBSD context.

Next we are going to create the swap partition. You will have to choose how big your swap partition is going to be. I’ll create a swap partition of the same size as the memory I’m planning on having in the server, so I’ll choose 16GB. You might have different needs. https://wiki.freebsd.org/SystemTuning#Swap suggests the following:

Size swap space to approximately twice the size of main memory on systems with less than 4GB RAM and the size of main memory for systems with more than 4GB. If in doubt, allocate more swap; allocating insufficient swap is far worse than allocating too much. If the system has multiple disks, reduce swap I/O contention by spreading swap across the disks, ideally in equally sized partitions.

We are using labels so we can more easily replace hardware later.

gpart add -l swap0 -t freebsd-swap -a 1m -s 16G ada0 # start at 4096 * 256 byte
gpart add -l swap1 -t freebsd-swap -a 1m -s 16G ada1 # start at 4096 * 256 byte

Now for the OS partition. Because I still have a few spare 160GB drives, I’m only going to use 140GB for the OS partition, so that if one of the current disks fails, I can easily replace it with one of my 160GB spares.

gpart add -l zroot0 -s 140G -t freebsd-zfs ada0 # only use 140GB
gpart add -l zroot1 -s 140G -t freebsd-zfs ada1 # only use 140GB

If you want to use all the remaining space on your disks (which is what I would normally do), run this instead:

gpart add -l zroot0 -t freebsd-zfs ada0 # use the whole remaining space
gpart add -l zroot1 -t freebsd-zfs ada1 # use the whole remaining space

What have we done?

root@:~ # gpart show ada0
=>       34  390721901  ada0  GPT  (186G)
         34       2014        - free -  (1.0M)
       2048   33554432     1  freebsd-swap  (16G)
   33556480  293601280     2  freebsd-zfs  (140G)
  327157760   63564175        - free -  (30G)

root@:~ #  gpart show ada1
=>       34  488397101  ada1  GPT  (233G)
         34       2014        - free -  (1.0M)
       2048   33554432     1  freebsd-swap  (16G)
   33556480  454840655     2  freebsd-zfs  (217G)

We will come back to the two empty data disks later.

SWAP Raid 1

Since we have two OS disks that…

kldload geom_mirror
gmirror label -b load -F swap /dev/gpt/swap0 /dev/gpt/swap1

Encryption

We are going to use GELI for the encryption. Basically it will encrypt each sector transparently. ZFS itself doesn’t know it is being encrypted:

| ZFS                |
| GELI Encryption    | 
| Physical Hard Disk |

Preparation

Load the kernel modules that are needed for GEOM and ZFS:

kldload opensolaris
kldload zfs
kldload geom_eli

Swap

geli onetime -d -e AES-XTS -l 256 -s 4096 /dev/mirror/swap

Operating System Partitions

Insert the USB stick that you plan to use as a boot device. Mine is da1. You will usually see some debug output about the just-connected USB stick on the server shell (not via SSH). It should also show the device name.

But you can always check your devices again with:

camcontrol devlist
<VB0250EAVER HPG9>                 at scbus0 target 0 lun 0 (pass0,ada0)
<VB0250EAVER HPG9>                 at scbus1 target 0 lun 0 (pass1,ada1)
<ST4000VN000-1H4168 SC43>          at scbus2 target 0 lun 0 (pass2,ada2)
<ST4000VN000-1H4168 SC43>          at scbus3 target 0 lun 0 (pass3,ada3)
<Sony USB Stick>                   at scbus6 target 0 lun 0 (pass4,da0)
<Sony USB Stick>                   at scbus7 target 0 lun 0 (da1,pass5)

Create the boot partition and install bootcode:

gpart destroy -F da1
gpart create -s gpt da1

gpart add -l gptboot0 -s 512k -t freebsd-boot da1

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1

Let’s create the ZFS partition and boot pool:

gpart add -l boot0 -t freebsd-zfs da1

mkdir -p /tmp/mnt/bootpool

zpool create -m /tmp/mnt/bootpool bootpool /dev/gpt/boot0

mkdir -p /tmp/mnt/bootpool/boot/zfs

mount_nullfs /tmp/mnt/bootpool/boot/zfs /boot/zfs

Create OS encryption key:

mkdir /tmp/mnt/bootpool/boot/keys
dd if=/dev/random of=/tmp/mnt/bootpool/boot/keys/zroot_encryption.key bs=64 count=1

Encrypt OS disks:

Choosing your password how long

mkdir /tmp/mnt/bootpool/boot/metadata_backup

geli init -b -s 4096 -e AES-XTS -l 256 -K /tmp/mnt/bootpool/boot/keys/zroot_encryption.key -B /tmp/mnt/bootpool/boot/metadata_backup/ada0p2.eli /dev/ada0p2
geli init -b -s 4096 -e AES-XTS -l 256 -K /tmp/mnt/bootpool/boot/keys/zroot_encryption.key -B /tmp/mnt/bootpool/boot/metadata_backup/ada1p2.eli /dev/ada1p2

geli attach -k /tmp/mnt/bootpool/boot/keys/zroot_encryption.key /dev/ada0p2
geli attach -k /tmp/mnt/bootpool/boot/keys/zroot_encryption.key /dev/ada1p2

ZFS zroot pool

mkdir -p /tmp/mnt/zroot

zpool create -m none zroot mirror /dev/ada0p2.eli /dev/ada1p2.eli

zfs set checksum=on zroot
zfs set atime=off zroot
zfs create zroot/ROOT
zfs create -o mountpoint=/tmp/mnt/zroot zroot/ROOT/default

because the eli devices have a sector size of 4096, zpool create automatically uses ashift=12

check it with zdb

Remount boot pool:

umount /boot/zfs
umount /tmp/mnt/bootpool

mkdir /tmp/mnt/zroot/bootpool
zfs set mountpoint=/tmp/mnt/zroot/bootpool bootpool
zfs mount bootpool

mount_nullfs /tmp/mnt/zroot/bootpool/boot/zfs /boot/zfs

Mounts should look like:

root@:~ # mount
/dev/iso9660/FREEBSD_INSTALL on / (cd9660, local, read-only)
devfs on /dev (devfs, local, multilabel)
/dev/md0 on /var (ufs, local)
/dev/md1 on /tmp (ufs, local)
<above>:/tmp/etc on /etc (unionfs, local)
zroot/ROOT/default on /tmp/mnt/zroot (zfs, local, nfsv4acls)
bootpool on /tmp/mnt/zroot/bootpool (zfs, local, nfsv4acls)
/tmp/mnt/zroot/bootpool/boot/zfs on /boot/zfs (nullfs, local)

ZFS zroot filesystems

zfs create -o mountpoint=/tmp/mnt/zroot/tmp -o compression=lz4 -o exec=on -o setuid=off zroot/tmp
chmod 1777 /tmp/mnt/zroot/tmp

zfs create -o mountpoint=/tmp/mnt/zroot/usr -o canmount=off zroot/usr

zfs create zroot/usr/home
cd /tmp/mnt/zroot; ln -s /usr/home home

zfs create -o compression=lz4 -o setuid=off zroot/usr/ports
zfs create -o compression=lz4 -o exec=off -o setuid=off zroot/usr/src

zfs create -o mountpoint=/tmp/mnt/zroot/var zroot/var
zfs create -o compression=lz4 -o exec=off -o setuid=off zroot/var/crash
zfs create -o compression=lz4 -o exec=off -o setuid=off zroot/var/log

zfs create -o compression=lz4 -o atime=on zroot/var/mail

zfs create -o compression=lz4 -o exec=on -o setuid=off zroot/var/tmp
chmod 1777 /tmp/mnt/zroot/var/tmp

Installing the base system

Available packages:

ls /usr/freebsd-dist/
MANIFEST    base.txz    doc.txz     games.txz   kernel.txz  lib32.txz   ports.txz   src.txz

cd /tmp/mnt/zroot

unxz -c /usr/freebsd-dist/base.txz | tar xpf -
unxz -c /usr/freebsd-dist/kernel.txz | tar xpf -
unxz -c /usr/freebsd-dist/src.txz | tar xpf -
unxz -c /usr/freebsd-dist/ports.txz | tar xpf -

Now let’s chroot into the new system:

chroot /tmp/mnt/zroot

Setup /boot:

cd /
rm -r boot/zfs
mv boot/* bootpool/boot/
rm -r boot
ln -sf bootpool/boot

And an initial /boot/loader.conf that will load ZFS, encryption and settings for encrypted disks on boot:

echo 'zfs_load="YES"' > /boot/loader.conf
echo 'aesni_load="YES"' >> /boot/loader.conf
echo 'geom_mirror_load="YES"' >> /boot/loader.conf
echo 'geom_eli_load="YES"' >> /boot/loader.conf
echo 'geli_ada0p2_keyfile0_load="YES"' >> /boot/loader.conf
echo 'geli_ada0p2_keyfile0_type="ada0p2:geli_keyfile0"' >> /boot/loader.conf
echo 'geli_ada0p2_keyfile0_name="/boot/keys/zroot_encryption.key"' >> /boot/loader.conf
echo 'geli_ada1p2_keyfile0_load="YES"' >> /boot/loader.conf
echo 'geli_ada1p2_keyfile0_type="ada1p2:geli_keyfile0"' >> /boot/loader.conf
echo 'geli_ada1p2_keyfile0_name="/boot/keys/zroot_encryption.key"' >> /boot/loader.conf
echo 'vfs.root.mountfrom="zfs:zroot/ROOT/default"' >> /boot/loader.conf
echo 'zpool_cache_load="YES"' >> /boot/loader.conf
echo 'zpool_cache_type="/boot/zfs/zpool.cache"' >> /boot/loader.conf
echo 'zpool_cache_name="/boot/zfs/zpool.cache"' >> /boot/loader.conf

Set root password:

passwd root

Set timezone:

tzsetup

Setup /etc/rc.conf

Create file and enable ZFS:

echo 'zfs_enable="YES"' > /etc/rc.conf

Set keymap:

kbdmap

Select your keymap and then write the output to /etc/rc.conf

echo 'keymap="german.iso.kbd"' >> /etc/rc.conf

Set hostname:

set HOSTNAME=<name-of-your-host>
echo hostname="$HOSTNAME" >> /etc/rc.conf
hostname -s "$HOSTNAME"

Setup services. This is how I did it; you can change the ‘YES’ to ‘NO’ if you don’t want a service to be running on boot:

echo 'sshd_enable="YES"' >> /etc/rc.conf
echo 'moused_enable="NO"' >> /etc/rc.conf
echo 'ntpd_enable="YES"' >> /etc/rc.conf
echo 'powerd_enable="YES"' >> /etc/rc.conf
echo 'dumpdev="NO"' >> /etc/rc.conf

Setup network:

Again I assume that you have DHCP for IPv4 and router advertisements for IPv6. Don’t forget to use the correct device name, in my case bge0:

echo 'ifconfig_bge0="DHCP"' >> /etc/rc.conf
echo 'ifconfig_bge0_ipv6="inet6 accept_rtadv"' >> /etc/rc.conf

Setup mail:

cd /etc/mail
make aliases

Setup /etc/fstab

printf "# Device\t\tMountpoint\tFStype\tOptions\t\tDump\tPass#\n" > /etc/fstab
printf "/dev/mirror/swap.eli\t\tnone\tswap\tsw\t\t0\t0\n" >> /etc/fstab

Exit chroot:

exit

Unmount filesystems:

cd /
umount /boot/zfs
zfs unmount -a

Setup ZFS mountpoints

zfs set mountpoint=legacy zroot/ROOT/default
zfs set mountpoint=/tmp zroot/tmp
zfs set mountpoint=/usr zroot/usr
zfs set mountpoint=/var zroot/var
zfs set mountpoint=/bootpool bootpool

Reboot into the new system

reboot

Creating a backup boot USB stick

gpart destroy -F da1
gpart create -s gpt da1

gpart add -l gptboot0 -s 512k -t freebsd-boot da1
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1

gpart add -l boot0 -t freebsd-zfs da1

mkdir /mnt/boot
cp -r /bootpool/boot/* /mnt/boot/

zpool export bootpool

mkdir /mnt/newbootpool
zpool create -m /mnt/newbootpool bootpool /dev/da1p2

cd /mnt/newbootpool
mv /mnt/boot .

cd
zfs unmount bootpool
zfs set mountpoint=/bootpool bootpool

ZFS Full Disk Encryption with FreeBSD 10 - Part 2