Make and Go for Fun and Profit
I’ve been somewhat interested in Go for quite a while now. It’s gotten to the point where it has replaced Ruby for me in those places where I write command line utilities which are too involved for them to make sense to be a shell script. I don’t have too many opinions about the language itself, but I like the static type system and that it’s a compiled language. And to be honest, the build system and how to utilize it have been the most interesting bits for me so far. One thing I especially like is the fact that go provides a bunch of tooling to do different things but how you tie them together is up to you. So this gives rise to some fun use cases for a nice Makefile.
The Basics
Every project I start gets this Makefile with some basic setups and variable definitions that I always want.
export GO15VENDOREXPERIMENT = 1
# variable definitions
NAME := coolthings
DESC := a nice toolkit of helpful things
PREFIX ?= usr/local
VERSION := $(shell git describe --tags --always --dirty)
GOVERSION := $(shell go version)
BUILDTIME := $(shell date -u +"%Y-%m-%dT%H:%M:%SZ")
BUILDDATE := $(shell date -u +"%B %d, %Y")
BUILDER := $(shell echo "`git config user.name` <`git config user.email`>")
PKG_RELEASE ?= 1
PROJECT_URL := "https://github.com/mrtazz/$(NAME)"
LDFLAGS := -X 'main.version=$(VERSION)' \
-X 'main.buildTime=$(BUILDTIME)' \
-X 'main.builder=$(BUILDER)' \
-X 'main.goversion=$(GOVERSION)'
For the most part this just defines a whole bunch of meta data that gets compiled into the binaries via linker flags. This is a pattern I have seen in a lot of Go projects and I really like that this is somewhat of a standard thing to do. Especially with the static nature of Go binaries, the more helpful information you can compile into the binary the better it is when you have to figure out where a binary comes from.
I also always have a handful of tasks defined that are helpful for running tests and such, especially to have a uniform and documented way how they are run locally and on CI.
# development tasks
test:
go test $$(go list ./... | grep -v /vendor/ | grep -v /cmd/)
PACKAGES := $(shell find ./* -type d | grep -v vendor)
coverage:
@echo "mode: set" > cover.out
@for package in $(PACKAGES); do \
if ls $${package}/*.go &> /dev/null; then \
go test -coverprofile=$${package}/profile.out $${package}; fi; \
if test -f $${package}/profile.out; then \
cat $${package}/profile.out | grep -v "mode: set" >> cover.out; fi; \
done
@-go tool cover -html=cover.out -o cover.html
benchmark:
@echo "Running tests..."
@go test -bench=. $$(go list ./... | grep -v /vendor/ | grep -v /cmd/)
These make heavy use of go list to determine existing packages to run tests
for. The rules also exclude the vendor folder as I don’t want to run those
tests and the cmd folder which I will describe more in the next section.
Structure for multiple binaries
Go has this defacto standard of how to structure code if your build produces
multiple executables. Since your main entry point in the app is always the
main package and there can only be one per directory (which is also true for
any other package btw) you need to separate different executables by
directory. The pattern here is basically to have a cmd folder that contains
subfolders for each executable which in turn just contain a main.go file.
This is a pretty nice pattern, once you get used to it and is a convention
that lets you easily create make rules for building those executables via the
make wildcarding support.
CMD_SOURCES := $(shell find cmd -name main.go)
TARGETS := $(patsubst cmd/%/main.go,%,$(CMD_SOURCES))
%: cmd/%/main.go
go build -ldflags "$(LDFLAGS)" -o $@ $<
This piece just finds all main.go files under the cmd folder and creates
targets from them located at the top level of the repo. Then there is a rule
to build those targets via a rule that ties them back to the source file via
wildcarding again and runs go build with the linker flags from before.
Of course it’s good habit to provide man pages for your tools. So we can rig up a similar set of rules for building man pages for each executable:
MAN_SOURCES := $(shell find man -name "*.md")
MAN_TARGETS := $(patsubst man/man1/%.md,%,$(MAN_SOURCES))
%.1: man/man1/%.1.md
sed "s/REPLACE_DATE/$(BUILDDATE)/" $< | pandoc -s -t man -o $@
all: $(TARGETS) $(MAN_TARGETS)
.DEFAULT_GOAL:=all
This lets us write man pages in markdown under the man/man1/folder named as
${cmd}.1.md and again uses wildcards in make to generate them top level via
an implicit rule. I also added an all target there which is the default and
builds all binaries and man pages.
Over time I’ve come to the conclusion that it’s really a good practice to have
your main.go files be as slim as possible. Ideally all they should be
concerned with is flag parsing, calling a method from your library packages,
and formatting and printing the output to the terminal. Any actual logic
should live in library modules somewhere else in your repo. This maintains a
good code layout to extend, makes sure code is reusable, and provides good
conventions for testing.
Installation
So now that we have rules to build the binaries, we also want to be able to
install them to the PREFIX we have defined at the top. Go comes with an
install command already (go install) which will put binaries in your
$GOPATH/bin but there is no need to have to rely on that. Plus on a multi
user system you want to provide tools for everyone anyways. Also let’s be
real, go install is not a replacement for a real package manager. Just
because go builds are fast and produce a static binary doesn’t mean it’s not a
good idea to be able to build packages. Plus you want your man pages to be
installed with your software as well of course. So let’s write some generic
install commands:
INSTALLED_TARGETS = $(addprefix $(PREFIX)/bin/, $(TARGETS))
INSTALLED_MAN_TARGETS = $(addprefix $(PREFIX)/share/man/man1/, $(MAN_TARGETS))
# install tasks
$(PREFIX)/bin/%: %
install -d $$(dirname $@)
install -m 755 $< $@
$(PREFIX)/share/man/man1/%: %
install -d $$(dirname $@)
install -m 644 $< $@
install: $(INSTALLED_TARGETS) $(INSTALLED_MAN_TARGETS)
local-install:
$(MAKE) install PREFIX=usr/local
We’re adding the PREFIX to all targets and man targets here to generate the
paths to install. Then we write another implicit wildcarding rule that has the
original targets as dependencies and performs install commands to put them
into the prefix. This is a quick and easy way to have a generic make install
target and also lets us easily add a local install target that we can use as a
dependency for building packages later on.
Dependencies, Oh My!
If you’ve spent time with Go and make before, you will maybe have noticed a flaw in the building step of the Makefile so far. To revisit, we are building binaries from the source in the cmd folder with this implicit rule.
%: cmd/%/main.go
go build -ldflags "$(LDFLAGS)" -o $@ $<
However this only tells make about the first level of direct dependencies for
the binary to the cmd source. Chances are you are using library and vendored
code in those. This means while go build technically knows about all
dependencies, make doesn’t. And it will refuse to rebuild the binaries if
something other than the cmd source changes. This is annoying but fortunately
also fixable. A simple fix would be to just not have dependencies in make for
the executables and mark them as .PHONY so that they are always regarded out
of date. This pushes all dependency resolution back to the go tool chain which
is nice, but kinda defeats half of the purpose of a Makefile as it will just
run all the commands all the time. To be clear, in practice this is a fine
solution and the downsides are mostly academic with the speed of a usual go
build.
However it’s fun to figure out how to make things work and while we’re here
already, lets utilize make to its full extent and make it aware of all
dependencies. The details for the make side of things I got from this awesome
blogpost which gives a great overview over automatic dependency
management in makefiles. So now all we need is a way to get a list of all
dependencies for a go source file. And of course, go files to the rescue
again! As it not only lets us print packages for passing to the test runner,
but also can print out all dependencies of a file. And with its -f parameter
it also supports basic templating for printing out the results. Utilizing that
we only need to do a small amount of post processing to print it in make
dependency format and we are good to go.
# source, dependency and build definitions
DEPDIR = .d
$(shell install -d $(DEPDIR))
MAKEDEPEND = echo "$@: $$(go list -f '{{ join .Deps "\n" }}' $< | awk '/github/ { gsub(/^github.com\/[a-z]*\/[a-z]*\//, ""); printf $$0"/*.go " }')" > $(DEPDIR)/$@.d
$(DEPDIR)/%.d: ;
.PRECIOUS: $(DEPDIR)/%.d
-include $(patsubst %,$(DEPDIR)/%.d,$(TARGETS))
%: cmd/%/main.go $(DEPDIR)/%.d
$(MAKEDEPEND)
go build -ldflags "$(LDFLAGS)" -o $@ $<
The makedepend command here grabs all dependencies that come from github
(which was a good enough approximation for me to filter out the std lib), cuts
off the project prefix and appends /*.go to each dependency. With the go
rules of having a package per folder, this also is pretty accurate most of the
time and only occasionally serves false positives to result in a rebuild. We
then adapt the implicit build rule to require the dependency file as well but
also rebuild it on each build. And BOOM our Makefile knows almost perfectly a
out all source dependencies.
Packaging and Documentation
I always aim for providing packages and good documentation for my Go projects. But I’ve already written about those things more generally here, so if you’re interested in the details of it, give that blog post a read. The important part is that the Makefile also holds the logic for building docs and packages, so they can be easily triggered from CI.
Cleanup
Since it’s also always good to make it easy to clean up artifacts and generated intermediate and output files, all makefiles also get some clean up tasks.
# clean up tasks
clean-docs:
rm -rf ./docs
clean-deps:
rm -rf $(DEPDIR)
clean: clean-docs clean-deps
rm -rf ./usr
rm $(TARGETS)
rm $(MAN_TARGETS)
.PHONY: all test rpm deb install local-install packages govendor coverage docs jekyll deploy-docs clean-docs clean-deps clean
Equipped with those Make tricks I’ve been having tons of fun building Go code. Some of that is surely more involved than it has to be and especially the dependency resolution stuff is very bonus round. But it’s been super interesting to rig it up and I learned a lot of things about Make. And in the end that’s what it’s all about for me. (Besides having projects with a super nice to use structure :)