Using virtiofsd to Setup the rootfs of an ArchLinux Virtual Machine for Linux Kernel Testing

Tutorial on how to use virtiofsd to setup an ArchLinux system without having a separate disk for easier testing of the Linux Kernel. This way you can easily share files between host and guest and do some easier management tricks.

Sometimes you want to have a comfortable testing environment for your own home-based personal contributions to the Linux Kernel. I haven't done any of those in quite some time, for lack of time, and my laptop's disk is kinda full, my SSD hardware failed a while ago, so I decided to use my "development server" (an old machine I bought for using in embedded development, where I would connect electronic devices projected by myself therefore I needed it to avoid ruining my USB ports - and yeah, compiling a kernel there is painful).

But why using virtiofsd? Well, there are several advantages in my opinion, but the main one is that it's quite comfortable since you have the guest filesystem accessible within the host. You can easily install, for instance, kernel modules from within the host while running the VM, for instance, among several other use cases. I used to have a shell script for that in the past, but I managed to lose it, probably from the old SSD that failed.

There are some disadvantages as well, like running a complete standard system like this for kernel testing is way too disk consuming compared to using those small images generated with tools like buildroot using i.e. `busybox` to have a system. But you gain all the flexibility of using a regular system that you are used to, so the convenience pays off IMHO.

So, how to set it up? This tutorial will cover the steps to do so from an ArchLinux based host to install an ArchLinux based guest. There are plenty of tutorials on how to do it using other distros, like Fedora but nothing for ArchLinux. On my steps I will use `doas` to get root privileges, but you can use `sudo` or whatever you're used to instead.

Installing Necessary Packages on the Host

First, let's start with the packages we're going to need:

  • virtiofsd: well, the main thing we are going to use
  • qemu-system-x86: assuming you want to run a x86 based system (replace it for other architectures, and beware you'll need to get some arch-based stuff for that architecture)
  • arch-install-scripts: for guest setup

If you run ArchLinux I assume you know how to install packages, but in case you're lazy:

$ doas pacman -Sy virtiofsd qemu-system-x86 arch-install-scripts

Setting Up a Directory for your Guest System

In my example we will have a single filesystem for everything. One of the main advantages of using this kind of setup is that you can have stuff shared across multiple VMs, so for kernel testing it could be useful to have a `/boot`, for instance, shared across multiple VMs. Or you could use a shared `/home` so you can create your user login for something you want. I don't know if there are side effects of using a entire system like this on multiple concomitant VMs, I imaging you might face trouble with stuff like database files, unix domain sockets, etc, you'll have to handle that carefully, but I think with proper handling you can have it. I won't dedicate attention to this, however, and will leave that to you.

So, the first thing we're going to need is a directory in a place where we have enough disk space. In the case of my "development server", I have a disk partition at `/srv` where I created a `vms` directory to have them. On my old laptop I had a similar dis structure with subdirectories for different filesystems (i.e. `boot`, `rootfs`, etc). Here I adopted a simpler setup, however. If you wanna do something more complex I will leave that to you as well.

Let's start by creating the directories (as root):

$ doas mkdir -p /srv/vms/vm-tutorial

If you have a partition dedicated to this system, or a btrfs subvolume, I strongly encourage you to use it and do a separate mount. I don't have it, I use ext4 on that machine so no btrfs subvolume as well, making it hard for me to show you how to do that (`btrfs subvolume` and `mount` commands are your friends, though).

Yes, I decided to call it `vm-tutorial`, sue me. Next we need to setup the `pacman` for the VMs using `pacstrap`:

$ doas pacstrap -K /srv/vms/vm-tutorial/ base vim tmux binutils openssh less

You can change your choice of packages, I just installed a couple of tools I kinda use a lot right in the pacstrap command. You can install, uninstall or update anything else you want later from the host with the command:

$ doas pacman -r /srv/vms/vm-tutorial/ [your-pacman-arguments]

Chrooting into the Guest System

It's not a mandatory step to chroot, you could do all the operations directly, but doing so make your life easier. Now you are basically doing some of the steps you would when setting up any ArchLinux system, like pointed out by the Installation Guide. You can skip a lot of things for this setup, so I opted to write the steps I did.

You can always chroot to make operations as well, and if you have a dedicated partition you can manage packages through `chroot` too. In the past I used to make that without a dedicated partition, but apparently `pacman` these days will fail if it doesn't detect enough disk space, and it won't detect disk space properly without a partition. 

To chroot into the VM:

$ doas arch-chroot /srv/vms/vm-tutorial/

The reason why using `arch-chroot` is better than just regular `chroot` is that it mounts for you stuff like `/proc`, `/sys`, `/run`, `/dev/pts`, etc, among several other things. If you used just `chroot` you would have to mount all of them and do all the labor manually and it's annoying.

Once you are there you can do all sorts of things, but here we'll do the ArchLinux setup stuff only. Let's start with setting the default timezone.

# ln -sf /usr/share/zoneinfo/Your_Continent/Your_Timezone_Town /etc/localtime

While this command is just something you won't need `chroot` at all to run, some others you might need, like:

# locale-gen

And then:

[root@kurupi-arch /]# cat > /etc/locale.conf
LANG=en_US.UTF-8
[root@kurupi-arch /]#

I have shown the complete shell output so you can see the contents I `cat`ed into `/etc/locale.conf`.

For me those are the only necessary steps, since I will boot with `/bin/bash` as init and won't use networking at all, although I could. You might also have noticed I didn't installed a kernel, well, since it's for kernel testing purposes there is no point in doing so anyway. You could install it every time from your kernel tree, and I do it for modules, but it's just an unnecessary extra step. Booting from the kernel tree directory itself is way easier, and I will show you how to do that in the Booting the Guest section of this tutorial.

Running the virtiofsd Daemon

To run the `virtiofsd` daemon we need to pass some parameters. First one, we need a tag for the filesystem, that is just the name you're going to use to identify it when mounting it in the guest. Moreover, you need to define the path for a socket file, so `qemu` can connect to it. If you would like some security like me you could opt for sandboxing it, and you would want to start `qemu` as a regular user rather than root, so the socket file must be readable by your user. I achieved that with the following command:

$ doas /usr/lib/virtiofsd --socket-path=/tmp/vm-tutorial-viofsd.sock --shared-dir /srv/vms/vm-tutorial/ --tag rootfs --sandbox chroot --socket-group agatha

You might wanna replace `agatha` by your local user's group.

Booting the Guest

To boot your guest, for my usage case scenario where I don't need anything but the shell and some basic tools, all you need is your recently built kernel. You will need a init ramdrive generated by `mkinitcpio` if you don't wanna enable fuse and virtiofs as built-in in your kernel, but that will add some seconds of delay to your boot and I personally don't wanna that. So I built my kernel with some config options:

CONFIG_VIRTIO=y
CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_BLK=y
CONFIG_VIRTIO_NET=y
CONFIG_FUSE_FS=y
CONFIG_VIRTIO_FS=y

Moreover, there are some options to `qemu` that you MUST have:

  • Shared Memory: You need to setup shared memory between the host and the guest, otherwise `virtiofsd` won't work. It also requires numa. All that will become several memory backend parameters that we're going to see.
  • Standard stuff: a cpu and kvm for a good performance.
  • virtiofsd mapping: you need to create the virtiofsd device into the machine.
  • A serial console: Unless you wanna something more sophisticated, but for most people doing this sort of thing this is more than enough.
  • Specify the kernel: Because that's what we are doing all this for, to test a specific kernel, so we don't wanna go through installing a bootloader and all the stuff that will just waste time.
  • Specify kernel parameters: Indicate how to mount the root filesystem, the console, etc.

So, the `qemu` command-line in my case looks like this:

$ qemu-system-x86_64 \
-m 2G -object memory-backend-file,id=mem,size=2G,mem-path=/dev/shm,share=on -numa node,memdev=mem \
-cpu host -enable-kvm \
-chardev socket,id=char0,path=/tmp/vm-tutorial-viofsd.sock -device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=rootfs \
-nographic -serial mon:stdio \
-kernel arch/x86/boot/bzImage \
-append "rootfstype=virtiofs root=rootfs rw console=ttyS0 init=/bin/bash"

Yeah, it's a long line, I used some parameters per line in the same sequence they appeared on my listed parameters so you can understand what is each of them. The kernel parameter assumes you're running from the top of your kernel build tree. For those not doing kernel development, it could be adapted to the path of a kernel in the virtual machine's `/boot`, but chances are you would also need to setup an initrd with `mkinitcpio` or something alike to load the required modules. I won't cover it here, but it's quite easy, you just need the fuse and the virtiofs kernel modules and some adjusts to the kernel command-line.

As for the kernel command-line (the one after the `-append` option), you need to specify the root filesystem's type (virtiofs), mounting tag (rootfs), mode (rw). Also, to use a serial console you need to specify it. And too boot directly into `/bin/bash` you can use the `init=` argument. You'll be limited to single-user mode, but it will save you precious initialization time and won't need to log in.

You can run regular init by omitting this option if you want, you just need to set a root password with `passwd` when you `chroot` into it so you can log in. You'll need some other configs that I can't recall as well, but you might easily figure them out, I think.

Here's a gif of mine booting up.

Installing Modules into the Guest

The "extra" is installing modules into the guest. I mean, if you have kernel modules you'll need that. But it's pretty easy, from the kernel tree run:

$ INSTALL_MOD_PATH=/srv/vms/vm-tutorial/ doas make -j3 modules_install

Replace `-j3` according to your number of cpus and modules installed. You can do it while the VM runs if you'd like (as long as the modules are compatible).