How to reclaim disk space used by docker (volumes), but keep a few important ones

Docker is a nice tool, but it's quite the disk hog. On my laptop, it takes up 50G. It would take more, but my /var partition is limited to 60G (and the rest is already used by various postgres DBs...).

So I learned quite early to run docker system prune from time to time (or every time docker complains that there is no space left, or a project crashes because postgres cannot write to disk (Note to future self: Put docker and postgres on different partitions!)). docker system prune "removes unused data" (according to the docs). It will remove all stopped containers and build caches (which take up place on disk), networks (which don't take up disk space) and all "dangling images" (i.e. images that are not used).

But it worked less and less for me. I had to run docker system prune more and more often, and the reclaimed space got smaller and smaller.

So I had to investigate and do something one (well, I) rarely does: Read the docs properly!

Remove all the volumes!

There I found the --volumes flag "to prune volumes as well". And a note that "this flag was added in Docker 17.06.1. Older versions of Docker prune volumes by default". (which is why things worked before I updated docker some time ago).

docker system df (another useful command I learned in my doc-reading-spree) verified that most of my disk space was used up by volumes. So I ran docker system prune --volumes, confirmed that I want to remove all that stuff, and, well, docker removed all that stuff.

Including volumes containing databases of apps I wasn't currently working on. Which is all fair, because that's what I ordered it to do (delete all volumes unless the volume is used by a running container). I now had a lot of free space, but starting one of those projects now took long, because docker first had to recreate the database volume (and while I try to keep my dev DBs small (~1G), this still takes a while). (I don't have any real / important data in docker volumes on my dev laptop, thanks for asking...).

So I was now looking for a better solution.

prune && labels

As with most things nowadays, you can tag volumes using labels. And I also discovered (after reading even more docs...) that there is another command to remove volumes: docker volume prune. And this command can filter the volumes to prune based on labels (or the absence of a label). So my plan was now to label my database volumes, and (from time to time) prune all volumes missing that label.

I use docker-compose for most of my projects, and adding a label to a (database) volume managed by docker-compose is quite simple:

docker-compose.yaml
services:
  db:
    image: postgres:11.5
    volumes:
      - winxle_db:/var/lib/postgresql/data
volumes:
  winxle_db:
    labels:
      dont_auto_prune: ""

I define a service db using a postgres image and a named volume winxle_db for the postgres data dir. And in the volumes section I add a label named dont_auto_prune without a value (thus it behaves boolean-ish).

After rebuilding everything, I can now filter the volumes:

~$ docker volume ls --filter label=dont_auto_prune
DRIVER              VOLUME NAME
local               winxle-mono_winxle_db

If you actually want to see the labels you have to use this really ugly way to pass a template to docker volume ls:

docker volume ls --filter label=dont_auto_prune \
                 --format '{{.Name}}: {{.Labels}}"'

This looks a lot like Helm charts to me (probably because it's also using golang templating, urks...), so I'm very near to getting sick. Let's move on quickly...

docker volume prune also takes a --filter option, so I can prune all volumes that don't have my label set via:

docker volume prune --filter 'label!=dont_auto_prune'"

Because I cannot be bother to remember this, and don't want to google how it's done only to end up at my own blog, I did a quick realias to add a new bash alias:

alias docker-prune="docker system prune && \
                    docker volume prune --filter 'label!=dont_auto_prune'"

And now all I need to do is run docker-prune from time to time (or when my system complains), and I get my free space back without having to wait a few minutes to rebuild my costly DB volumes!

PS: Perl Conference in the Cloud

Next Thursday (25th June, 18:00 CEST UTC+2) I will give a talk at the Perl Conference in the Cloud on Writing a good Dockerfile for Perl apps. You can attend the talk via zoom (if you pay the very low conference attendance fee) or watch the live stream on youtube/yapcna.