Ljubljana, Slovenia

Embedding data in Go executables

If you’ve been following along, you know that I’ve been developing Pendulum editor as part of the #100DaysOfCode challenge, where one commits to writing code at least one hour per day. Pendulum is a web based editor which is great for editing simple text and Markdown files.

I’m writing this article with it, in fact. It’s made with a Go back-end and a VueJS front-end. I want to make it easy for users and provide a single executable which will contain everything, so they don’t need to download an installer or extract zip files. I needed a way how to pack up everything together. I decided to use go-bindata and resort to code generation to add all the data into the executable with go build.

Code generation?

Well, sure. It’s simple enough. The go-bindata tool already enables us to create a .go file from a public_html folder, for example. This is perfect for my use case. But why resort to a bash script or Makefile to produce it? We can use Go’s code generation tool and just call go generate before we call go build.

If you want to get started with code generation, all you need to do is include a single comment somewhere in your source code, main.go for example:

package main

//go:generate echo "Hello world"

func main() {
}

When you run go generate, you will find out that it prints “Hello world”. It’s not really a requirement that you would generate any code with go generate. Whatever you put after the //go:generate text will be executed. You can even run go build if you wanted.

package main

//go:generate echo "Hello world"
//go:generate go run main.go

func main() {
    println("Hello world from Go")
}

Running it will produce the expected result:

# go generate
Hello world
Hello world from Go

Inception! Well, go generate is… interesting. Node programs are using babel to provide ES6/ES7 syntax capabilities to Node ES5 runtime, and people are attempting to use the same approach to provide Go with functionality beyond the current scope of the language.

For example: The genny project is an example more directed at generating typed-code so you don’t really have to copy paste aggressively, but projects like Have went closer to what Babel is doing with Node - providing a language which transpiles to Go. I don’t know of any other attempts that got more traction, yet. The discussions about Go2 and generics seem to suggest however, that there’s some interest for this.

Well, for our case, we’re slightly more boring, we’re just trying to package some data in our application, so let’s interrupt this social commentary and continue:

//go:generate go-bindata -prefix front/src -o assets/bindata.go -pkg assets -nomemcopy front/src/dist/...

That’s a bit of a long line, let’s break it down just for visual inspection:

  • //go:generate - hint for go generate,
  • go-bindata - main command that executes,
  • -prefix front/src - exclude “front/src” from package,
  • -o assets/bindata.go - generated output file location,
  • -pkg assets - name of the package we’re generating,
  • -nomemcopy - an optimization for a lower memory footprint,
  • front/src/dist/... - the location we’re packing.

This creates an assets package in your application folder, which you can import with a short import, ie app/assets, where app matches your application folder.

Serving embedded files via HTTP

This is where things get a little bit complicated. Or simple, after you read a bit of documentation. If you wanted to serve local files, you would use something like the following line of code:

folder := http.Dir("/")
server := http.FileServer(folder)
http.Handle("/", server)

In fact, the package go-bindata-assetfs provides an implementation for http.FileServer. Using it is simple enough:

import "github.com/elazarl/go-bindata-assetfs"
import "app/assets"
// ...
func main() {
    // ...
    files := assetfs.AssetFS{
        Asset:     assets.Asset,
        AssetDir:  assets.AssetDir,
        AssetInfo: assets.AssetInfo,
        Prefix:    "dist",
    }
    server := http.FileServer(&files)
    // ...
}

There is only a slight hiccup. I am using a VueJS app with pushHistory enabled. This means, that when the user navigates the app, they will see links without a shebang (hash, #), but plain ordinary absolute links like /blog/about.md. Which don’t exist in this asset filesystem, but are handled with the application.

Well, turns out the solution is simple enough. The assetfs.AssetFS structure has functions AssetInfo (which is the equivalent of os.Stat), and the function Asset (sort of like ioutil.ReadFile). With this it’s possible to check if a file exists in the asset filesystem, and to output a different file if it doesn’t:

// Serves index.html in case the requested file isn't found
// (or some other os.Stat error)
func serveIndex(serve http.Handler, fs assetfs.AssetFS) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        _, err := fs.AssetInfo(path.Join(fs.Prefix, r.URL.Path))
        if err != nil {
            contents, err := fs.Asset(path.Join(fs.Prefix, "index.html"))
            if err != nil {
                http.Error(w, err.Error(), http.StatusNotFound)
                return
            }
            w.Header().Set("Content-Type", "text/html")
            w.Write(contents)
            return
        }
        serve.ServeHTTP(w, r)
    }
}

In case the file is found, I use the provided ServeHTTP method, instead of providing my own implementation. All it takes to use this is a bit of a change in the handler which we defined before:

http.HandleFunc("/", serveIndex(server, assets))

The function serveIndex returns a http.HandlerFunc, and this line was changed accordingly. This provides a full implementation of how to serve your data which you add to your application with go generate and go-bindata. And if you want to skip the //go:generate part and just put it in your CI scripts, that’s fine too!

And with this I implemented a single-executable release for Pendulum. Grab it from the GitHub releases page to check it out.

Edit: Improved serveIndex example() thanks to @hipone on Reddit.

While I have you here...

It would be great if you buy one of my books:

I promise you'll learn a lot more if you buy one. Buying a copy supports me writing more about similar topics. Say thank you and buy my books.

Feel free to send me an email if you want to book my time for consultancy/freelance services. I'm great at APIs, Go, Docker, VueJS and scaling services, among many other things.