12 Sep 2020, 15:35

which where what?

Finding a binary in your $PATH can sometimes be confusing. Especially when which mybin and whereis mybin don’t find it, but command -v mybin does, and worse, your shell finds it- so what is wrong with which?

It has to do with how you define your path.

export PATH=~/bin/:$PATH

will work with bash and command -v, but which and whereis aren’t hip to shell metacharacters, and won’t pick up anything in ~/bin/. Solving this is pretty easy, too-

export PATH=$HOME/bin:$PATH

19 Apr 2020, 09:53

Caddy and Cloudflare

self signed certs for running Caddy behind Cloudflare

I saw some goofy logs this morning-

acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: Cannot negotiate ALPN protocol "acme-tls/1" for tls-alpn-01 challenge, url:
[ERROR] Renewing: acme: Error -> One or more domains had a problem:
[INFO] Unable to deactivated authorizations: https://acme-v02.api.letsencrypt.org/acme/authz-v3/4017030008
[INFO] acme: Trying to solve TLS-ALPN-01
[INFO] acme: use tls-alpn-01 solver
[INFO] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/4017030008
[INFO] acme: Obtaining bundled SAN certificate
[INFO] acme: Trying renewal with -3768 hours remaining

I had a Caddy server doing ACME challenges behind Cloudflare, it turned out. That wasn’t really working. I restarted Caddy, and then it just sat there trying to do its ACME challenge and not serving any pages.

Anyway, in case it ever helps anyone else, the magic Caddyfile incantation is

mydomain.com:443 {
    proxy  / localhost:1234 {
    tls self_signed

That :443 is the real trick, because without it, Caddy wants to run self-signed domains on port 2015, which doesn’t do anyone any good.

Systems administration. The gift that keeps on giving.

10 Dec 2019, 19:10

Really Canonical
 * Overheard at KubeCon: "microk8s.status just blew my mind".

Last login: Thu Nov 21 08:36:04 2019 from
hank@tinyserver:~$ microk8s.status
microk8s is running
cilium: disabled
dashboard: enabled
dns: enabled
fluentd: disabled
gpu: disabled
helm: disabled
ingress: enabled
istio: disabled
jaeger: disabled
juju: disabled
knative: disabled
kubeflow: disabled
linkerd: disabled
metallb: disabled
metrics-server: disabled
prometheus: disabled
rbac: disabled
registry: enabled
storage: enabled

I mean, really? That blew your mind?

I resent folksy advertising. Just plop a static ‘upgrade here’ link in the MOTD and leave me alone.

05 Oct 2019, 09:53

Weekend Update

I don’t know what I’m doing with my life but this doesn’t really seem like it should be it. Crouched with my head aching over a podcast and a mug of cold coffee staring at the sun outside dreading the resumption of duties and obligations. Happy Saturday everybody.

28 Sep 2019, 19:44

Self Care Dont Care

The thing I really need to do this week is take care of myself. Watch my sleep schedule, get my exercise, conduct my business in a fulfilling and sustainable way. This is something I have told myself a lot. It’s my own little Mount Everest. Lets give it another shot.

02 Jun 2019, 11:17

Code Reviews

I’m a big believer in code reviews, possibly more than is warranted. It’s also true that it is hard work, and it is often hard to get started. So here are a list of the things I try to do, maybe they will help someone else.

Say nice things about nice code. We all have our ups and downs. Code reviews are often about preventing things from going wrong in the future- and it can be hard on the people in the present. I value people doing the work, so I want to communicate that. If a function looks clean, say so. If there is a workaround for an ugly wart in the language or framework, commiserate. Think of it like NBA players high-fiving their teammates after free throw attempts- everyone is a professional, no one technically needs it, it’s a little rote- but even perfunctory social gestures help. Even if everyone knows they’re a little forced.

File tickets for tech debt. As a reviewer, it’s more polite to create real tickets for focused followup work than to unload a dump truck of scope creep. I’m sort of in “treat the new hires well” mode in my personal life, but the same applies to long-time developers. The exception is when someone is doing a ton of work in the same section of code and just kind of creating a mess. If your org is too dysfunctional to allow you to work on tech debt tickets occasionally… I don’t know how to help, to be honest. That sounds like its own problem.

Double check that people aren’t re-implementing existing code. New developers on a project are particularly susceptible to this- they don’t know the codebase yet. It’s fine, it’s just about communicating.

Double check the names for naming conventions. This is the right time to make sure that everything isn’t called FooBar except for the one new feature where everything is BarFoo.

Run through the security checklist. Don’t build strings and send them to the shell, especially if they have user input in them. Use your language’s execve(2). Don’t build strings and send them to a SQL database, use parameterized queries. User supplied data has got to get escaped in the template. Etc.

Run through the test checklist. Is it tested? Where is it tested? Do the tests exercise the usual boring edge cases- too much, too little, garbage input.

Run through the integration input. Does this require new monitoring? Is there a companion change to the monitoring configuration somewhere?

Anyway, once you’ve run through this list, you’ve read the code a couple times, you’ve thought about it a little, and you’re probably in a better place to think about it as a holistic thing. And all of that needs to get done anyway.

16 Dec 2018, 17:13

CSS Grid

I don’t really do frontend work any more, so I never got around to actually using it until now. It’s amazing though-

<label name="title">Title:</label> <input name="title"></input>
<label name="question">Question:</label> <input name="question"></input>
<label name="answer">Answer:</label> <input name="answer"></input>
form {
    display: grid;
    grid-template-columns: 100px 1fr;
    grid-gap: 10px;
    padding: 10px;
    background-color: #eee

form label {
    grid-column: 1; /* put the labels on the left */
    text-align: right;

form input {
    grid-column: 2; /* put the inputs on the right */

And bam, a totally passable form:

That would have been… either a ton of extra markup to put it inside a table, or a fiddly float hellscape ten years ago.

The web is nice, I’ve missed it in my long night of operations plumbing.

03 Sep 2018, 09:33

a quick Hugo plugin for gnome builder

I whipped this together to try and work on my blog less in vim. It wasn’t that bad, I should write more of these.

19 Dec 2017, 18:00

rkt beginner notes

Since I collect abandonware container systems:

getting started with rkt

from quay, the coreos dockerhub competitor-

# sudo rkt fetch quay.io/coreos/alpine-sh
# sudo rkt run --interactive quay.io/coreos/alpine-sh --exec=/bin/sh

from dockerhub-

# sudo rkt --insecure-options=image fetch docker://alpine
# sudo rkt run --interactive docker://alpine --exec=/bin/sh

the dockerhub stuff also creates a fake rkt registry for docker-

# sudo rkt run --interactive registry-1.docker.io/library/alpine --exec=/bin/sh

will also work.

Finally, Quay mirrors the default Docker library under quay.io/dockerlibrary, so

# sudo rkt fetch quay.io/dockerlibrary/debian:9

Gets you debian.

mounted volumes

syntax got me for a while, key is to realize the inside/outside distinction-

# rkt run --volume logs,kind=host,source=/srv/logs \
    example.com/app1 --mount volume=logs,target=/var/log \
    example.com/app2 --mount volume=logs,target=/opt/log

“volume” is outside, “mount” is inside, easy-peasy.

quality of life things

Clean everything up real quick-

rkt gc --grace-period=0s

15 Oct 2017, 10:24

scratch directories with overlayfs

One of the nice things about Concourse is that everything gets a normal, read-write directory tree to work in, but changes made aren’t persisted, so you don’t have to worry about temporary files, scratch work, mistakes, etc., interfering with other jobs down the line. It turns out you can do this yourself, and it’s not super hard.

overlayfs is a newer Linux filesystem, the new default Concourse filesystem driver, and pretty cool. To use it, you don’t need anything fancier than good old mount and mkdir. There is, however, some setup, so let’s walk through the steps.

First, let’s get a git repo-

$ git clone https://github.com/hfinucane/jsonproc.git

Then, we’ll make some directories for overlayfs to do its work in-

$ mkdir Lower Upper Work

and finally, lets make the overlayed directory we’re going to be using-

$ mkdir ScratchBuildDir

And we’ll put it together-

$ sudo mount -t overlay overlay -o lowerdir=jsonproc:Lower -o upperdir=Upper -o workdir=Work ScratchBuildDir

The error message situation isn’t great, make sure to run dmesg | tail if something goes wrong. That said, lets look at what we can do now-

$ cd ScratchBuildDir
$ go build -o jproc
$ ls
jproc  LICENSE  main.go  main_test.go  README.md

Now lets go look at our original directory-

$ cd ..
$ ls jsonproc
LICENSE  main.go  main_test.go  README.md

It’s not there, all our work is isolated in the scratch build directory. Other directories, however, have been affected-

$ ls Upper

When you tear down the overlay, they will remain-

$ sudo umount ScratchBuildDir
$ ls Upper

so if you’re building your own system with safe, ephemeral working directories, you’ll need unique Upper directories, or you’ll need to clear them out between uses.

I haven’t really touched on the Lower directory. overlayfs doesn’t want to let you run without doing any overlaying, so you have to overlay something on something, even if it’s just an empty directory for this example. If you were writing a container-based build system, Lower might be the OS tree, and you’d want the git checkout to be nested inside of var/tmp/build or something. You’re not limited to two directories either- jsonproc:Lower:Lowest will stack jsonproc on top of Lower on top of Lowest.