TL;DR: No, keep them on separate swap partitions.

–

I’m working on a secure ZFS installation for my home NAS and plan to use ZFS as the filesystem. One goal is to have everything on disk encrypted.

While the new FreeBSD 10 installer can automatically partition your hard disk with an encrypted ZFS setup, the swap partition is not encrypted.

I also plan to mirror the swap space, so that if one disk goes down in my RAID 1 (or zmirror) setup, the system will keep running. If you placed a swap partition on each disk in a two-disk setup and they were not mirrored, the system would crash if one disk became unresponsive.

So my first idea was just to store the swap space in the ZFS pool instead of creating a separate swap partition that would need to be mirrored and encrypted.

Now, there are two options to store swap space on a ZFS pool:

1. Use a file as swap space that is stored on a ZFS filesystem
2. Store the swap space in a ZVOL

The first option is a no-go, because in case of low memory, ZFS needs memory to manage the disk writes to the swap file, but as there is no memory available it needs to write to the swap file, but ZFS needs memory to manage the disk writes to the swap file . . .

The second option the internetz suggests is to create a ZVOL and use that as swap space. That seems to work on first sight, and even Sun/Oracle suggested it in a blog post.

Also, the FreeBSD Wiki has something to say on how to create a ZVOL usable for swap space:

zfs create -V 2G -o org.freebsd:swap=on -o checksum=off -o compression=off -o dedup=off -o sync=disabled -o primarycache=none zroot/swap

But they added a note:

If there is no real memory available, the system might become unresponsive.

Because storing the swap space in a ZVOL would be much more convenient (at least if you don’t need kernel crash dumps, which wouldn’t work this way), I wanted to try for myself.

I wrote a little C program that just allocates memory and then waits for 10 minutes so that the memory is not immediately released:

#include <stdio.h>
#include <stdlib.h>
#include <time.h>

void alloc_one_meg() {
    int number_of_bytes_of_int = (int)sizeof(int);
    int number_of_ints_for_one_megabyte = 1024*1024/number_of_bytes_of_int;
    int *megabyte = malloc(sizeof(int) * number_of_ints_for_one_megabyte);

    for (int i=0; i < number_of_ints_for_one_megabyte; i++) {
        megabyte[i] = rand();
    }
}

void waitFor (unsigned int secs) {
    printf("waiting for %d seconds\n", secs);

    int retTime = time(0) + secs;
    while (time(0) < retTime);
}

int main ( int argc, char *argv[] )
{
    if ( argc != 2 ) {
        printf( "usage: %s <megabytes>\n", argv[0] );

    } else  {

        int number_of_megabytes;
        sscanf (argv[1],"%d",&number_of_megabytes);

        for (int i=0; i < number_of_megabytes; i++) {
            alloc_one_meg();
        }

        printf("allocated %d MB\n", number_of_megabytes);

        waitFor(600);
    }
}

Download link: memory.c

You can compile and run the program on a vanilla FreeBSD installation by running:

# clang -o memory memory.c
# ./memory <number of megabytes>

My test system is a VMware virtual machine with 256MB RAM.

SWAP partition

Now let’s see how the system behaves if we run memory.c on an installation with a normal 2GB swap partition:

The system stays responsive and kills memory.c as soon as it runs out of swap space.

ZVOL

Now let’s try the same thing with the swap space on a ZVOL:

It still replies to ping requests, but it has stopped being responsive. You cannot kill memory.c or do anything else useful.

As you can see in the top output, which I had running on the second console, the system has 256MB RAM and 512MB swap space. So the 300MB shouldn’t even have used up the whole swap space.

And if you can trust the top output, it crashed with only about 60MB of swap space used.

Conclusion

Some people have suggested reserving memory via kernel parameters (in that case ZFSOnLinux), but I’m aiming for a stable, maintainable system and reserving memory might just delay the problem, if it helped at all.

So the way to go is to keep swap space off ZFS and use separate partitions - like the FreeBSD 10 ZFS installer already does.