scratch directories with overlayfs

October 15, 2017

One of the nice things about Concourse is that everything gets a normal, read-write directory tree to work in, but changes made aren’t persisted, so you don’t have to worry about temporary files, scratch work, mistakes, etc., interfering with other jobs down the line. It turns out you can do this yourself, and it’s not super hard.

overlayfs is a newer Linux filesystem, the new default Concourse filesystem driver, and pretty cool. To use it, you don’t need anything fancier than good old mount and mkdir. There is, however, some setup, so let’s walk through the steps.

First, let’s get a git repo-

$ git clone https://github.com/hfinucane/jsonproc.git

Then, we’ll make some directories for overlayfs to do its work in-

$ mkdir Lower Upper Work

and finally, lets make the overlayed directory we’re going to be using-

$ mkdir ScratchBuildDir

And we’ll put it together-

$ sudo mount -t overlay overlay -o lowerdir=jsonproc:Lower -o upperdir=Upper -o workdir=Work ScratchBuildDir

The error message situation isn’t great, make sure to run dmesg | tail if something goes wrong. That said, lets look at what we can do now-

$ cd ScratchBuildDir
$ go build -o jproc
$ ls
jproc  LICENSE  main.go  main_test.go  README.md

Now lets go look at our original directory-

$ cd ..
$ ls jsonproc
LICENSE  main.go  main_test.go  README.md

It’s not there, all our work is isolated in the scratch build directory. Other directories, however, have been affected-

$ ls Upper
jproc

When you tear down the overlay, they will remain-

$ sudo umount ScratchBuildDir
$ ls Upper
jproc

so if you’re building your own system with safe, ephemeral working directories, you’ll need unique Upper directories, or you’ll need to clear them out between uses.

I haven’t really touched on the Lower directory. overlayfs doesn’t want to let you run without doing any overlaying, so you have to overlay something on something, even if it’s just an empty directory for this example. If you were writing a container-based build system, Lower might be the OS tree, and you’d want the git checkout to be nested inside of var/tmp/build or something. You’re not limited to two directories either- jsonproc:Lower:Lowest will stack jsonproc on top of Lower on top of Lowest.