Embedding Assets in Go

In this article, I'm going to show you how to embed assets in Go. In particular, I'll walk you through the process I used to embed a shell script into chef-runner, one of my open-source Go projects. My goal is to enable you to apply this useful technique to your own projects -- by using the right tools for the job and combining them in the best possible way. Let's get started!

The goal of chef-runner is to speed up development and testing of Chef cookbooks. I originally developed the tool as a fast alternative to the painfully slow vagrant provision command. chef-runner has since evolved and can now be used to rapidly provision not only local Vagrant machines but also remote hosts like EC2 instances.

One of the things chef-runner can do is to install Chef on a machine before provisioning it. This makes it possible to set up bare servers that have nothing installed but the base operating system. For this feature, chef-runner used to download the Omnibus installer -- better known as install.sh script -- to a local folder before copying it to the target machine, where it will be executed to install Chef.

Later, I decided to change the mechanism a bit. Instead of downloading the installer script from the internet, I thought it would be better to embed the script and make it part of chef-runner's source code. This would yield the following benefits:

• Simplicity. The code ends up being simpler. No download logic. No clever caching.

• Transparency. Always use the same script that is checked into version control.

• Speed. No need to download the script again for each project.

At the same time, I accepted the fact that users might not benefit from improvements to the installer immediately, simply because it gets updated so rarely.

chef-runner is written in Go. I knew that it's possible to embed assets in Go, and this seemed like the perfect opportunity to get familiar with the tooling. Here's how I ended up doing it using a combination of go-bindata and go generate.

go-bindata converts any text or binary file into Go source code, making it the perfect tool for embedding data into Go programs. You can use it, for example, to embed assets such as CSS, JavaScript, and image files into your web application. The result will be a fully stand-alone binary that can be deployed just as easily as any other Go program.

I used go-bindata to add the Omnibus installer as an asset to chef-runner's omnibus package. For this, all I had to do was download the installer script once and then invoke the go-bindata command-line tool for generating Go code from it. The exact steps are as follows:

# inside $GOPATH/src/github.com/mlafeldt/chef-runner
$ cd chef/omnibus/
$ mkdir assets
$ curl https://www.chef.io/chef/install.sh > assets/install.sh
$ go-bindata -pkg omnibus -o assets.go assets/

Afterwards I had to adapt chef/omnibus/omnibus.go to use the asset directly instead of downloading it. Asset data can be accessed via the Asset function, which is included in the generated assets.go source file. The resulting code ended up looking like this:

// chef/omnibus/omnibus.go
package omnibus
type Installer struct {
    ChefVersion string
    SandboxPath string
    RootPath string
    Sudo bool
}
func (i Installer) writeOmnibusScript() error {
    script := path.Join(i.SandboxPath, "install.sh")
    log.Debugf("Writing Omnibus script to %s\n", script)
    data, err := Asset("assets/install.sh")
    if err != nil {
        return err
    }
    return ioutil.WriteFile(script, []byte(data), 0644)
}

That's really all I had to do as far as embedding is concerned; I was surprised how little effort it took.

However, there's one thing I don't like about the Asset function generated by go-bindata: It will show up in your package's Godoc, which might be a little confusing, to say the least. One possible workaround is to store assets in a separate package.

If you'd like to learn more about go-bindata and its features, check out the project's README. Among other things, there's a special debug mode in which asset data will be loaded from the original file on disk -- via the same asset API. This is useful during development when you don't want to redeploy your code every time you touch your assets.

Another related project, go-bindata-assetfs, can be used to serve files that have been embedded with go-bindata via the net/http package (by implementing the http.FileSystem interface). Once again, Go's focus on composability makes for good complementary solutions.

While tools like go-bindata are valuable on their own, you still need to integrate them into your build process. For this, you might consider using a build tool like Make to glue all the pieces together. But wouldn't it be great if Go itself would provide the ability to automatically generate source code? Well, it does!

Go 1.4 introduced a new command, go generate, to automate the generation of source code before compilation. The tool works by scanning Go source code for special comments that define commands to run. This way you can declare build instructions in your code, keeping everything together in a nice way.

This is the comment I added to chef-runner's omnibus package:

// chef/omnibus/omnibus.go
package omnibus
//go:generate go-bindata -pkg $GOPACKAGE -o assets.go assets/

Now, when running go generate, it will pick up the command and execute go-bindata with the specified parameters ($GOPACKAGE will be replaced with the actual package name, i.e., "omnibus"). Let's generate some source code:

$ go generate -x ./chef/omnibus
go-bindata -pkg omnibus -o assets.go assets/

The -x flag causes go generate to print commands as they are executed. The one command shown above should look familiar.

As with most Go tools, you can run go generate ./... to process all packages of your project at once. If your Go project happens to contain a Makefile -- as most do these days -- it's a good idea to provide make generate, both as a shorthand and for use by other Make targets.

There is one thing you need to be aware of when using go generate, though. The tool isn't integrated with go get, as one might expect. Because of that, your project will only be "go gettable" if you check in all sources created by go generate. That's why I put the mentioned assets.go file under version control.

For more detailed information on go generate, consult go help generate or, better yet, read this article on the Go blog.

All in all, embedding assets in Go is straight forward thanks to existing tooling. There's go-bindata, which can convert any files into embeddable assets, and there's go generate, which automates the generation of Go code in a nicely integrated way. I'm sure we'll see more of these convenient tools in the not-so-distant future.

Last but not least, in one of my other open-source projects, I used a Python script to generate a Go map of distributions supported by Packagecloud. By doing so before compilation, I managed to save a rather expensive API call that would otherwise be made at runtime. I recommend checking out the full source code if you're interested in seeing another practical example of embedding assets in Go.