One of the nice things about Concourse is that everything gets a normal, read-write directory tree to work in, but changes made aren’t persisted, so you don’t have to worry about temporary files, scratch work, mistakes, etc., interfering with other jobs down the line. It turns out you can do this yourself, and it’s not super hard.
overlayfs is a newer Linux filesystem, the new default Concourse filesystem driver, and pretty cool.
To use it, you don’t need anything fancier than good old
mkdir. There is, however, some setup, so let’s walk through the steps.
First, let’s get a git repo-
$ git clone https://github.com/hfinucane/jsonproc.git
Then, we’ll make some directories for
overlayfs to do its work in-
$ mkdir Lower Upper Work
and finally, lets make the overlayed directory we’re going to be using-
$ mkdir ScratchBuildDir
And we’ll put it together-
$ sudo mount -t overlay overlay -o lowerdir=jsonproc:Lower -o upperdir=Upper -o workdir=Work ScratchBuildDir
The error message situation isn’t great, make sure to run
dmesg | tail if
something goes wrong. That said, lets look at what we can do now-
$ cd ScratchBuildDir $ go build -o jproc $ ls jproc LICENSE main.go main_test.go README.md
Now lets go look at our original directory-
$ cd .. $ ls jsonproc LICENSE main.go main_test.go README.md
It’s not there, all our work is isolated in the scratch build directory. Other directories, however, have been affected-
$ ls Upper jproc
When you tear down the overlay, they will remain-
$ sudo umount ScratchBuildDir $ ls Upper jproc
so if you’re building your own system with safe, ephemeral working directories, you’ll need unique Upper directories, or you’ll need to clear them out between uses.
I haven’t really touched on the
overlayfs doesn’t want to
let you run without doing any overlaying, so you have to overlay something on
something, even if it’s just an empty directory for this example. If you were
writing a container-based build system,
Lower might be the OS tree, and you’d
want the git checkout to be nested inside of
var/tmp/build or something.
You’re not limited to two directories either-
jsonproc:Lower:Lowest will stack
jsonproc on top of
Lower on top of