Since 2018, we've been using Dep to manage all the dependencies (473 at the time of counting) in our microservices monorepo. In 2020, Dep was deprecated in favour of the officially-supported Go modules system, but we weren’t able to migrate because of the new way it calculates the dependency tree; migrating to Go modules would have changed over 1.5 million lines of third-party code. This is extremely risky, difficult to review and hard to roll back, but continuing to use Dep meant we were relying on tooling that was no longer maintained and not sticking with the Go ecosystem which could cause problems down the line.

We wanted to find a way to switch to Go modules safely so we turned the operation into a ‘no-op’ by freezing dependency versions. We prepared our monorepo by iteratively updating our dependencies using the existing Dep tooling, which in turn reduced the risk of dependency changes during and after the migration to Go modules.

Managing dependencies with Go modules

Dep was intended as an experiment by the Go community to learn about dependency management and has since been replaced by the officially supported Go modules system (Go 1.11). There is some migration automation built in via go mod init which will detect existing dep-managed dependencies (in Gopkg.toml) but because the tools resolve dependencies differently, it’s expected that the versions will be different. To understand why, we need to understand some core Go modules concepts.

Go understands three distinct source units which you might refer to as a ‘dependency’:

files, which can be toggled using build constraints (or tags)
packages, which are binaries compiled from several files
modules, which are versioned groups of packages

The go.mod file defines a name for this module that other modules might reference to add a dependency. Since major versions are API incompatible by definition, Go requires major version suffixes to be defined in the module names, as if different major versions were completely different modules. This is necessary to solve the diamond dependency problem, which Dep cannot do.

The go.mod file is used to specify all the dependencies of your project, even if the project itself isn't used as a dependency. In our case, we manage our monorepo as a single module.

Diagram showing that a Go module contains packages and files. The module name specifies the major version in its path and the module contains two packages and two files: go.mod and go.sum. The first package, named foo, contains two files: foo.go and foo_test.go. foo_test.go is only built during tests. The second package, named bar, contains three files: bar.go, bar_test.go and bar_windows.go. Bar_windows.go is only built when the windows build flag is set (for example).

Minimal Version Selection

Go dynamically calculates the dependency graph, which it calls the build list, using an algorithm called Minimal Version Selection (MVS). This takes into account the semantic version requirements of this module’s main packages (our services) and their dependencies. The important thing to remember is that a module's requirements specify the minimum version required, but Go might select a higher minor or patch version depending on indirect dependency requirements. The Go modules reference has a great explanation of this.

Go modules uses the require directive to select module versions in go.mod, but it also provides the replace directive to override versions selected by minimal version selection with local or specific versions. The replace directive is the sledgehammer of the Go modules world and can be (ab)used to restrict module versions where go mod tidy would otherwise upgrade them. In general, it’s best to let Go handle dependency management with require, but we found it useful to fix older versions of libraries and applied it liberally for our migration. More on that later.

The go.sum file is new too. It contains hashes of the module’s direct and indirect dependencies to ensure packages can be built deterministically. In order to perform minimal version selection, Go must sometimes download and check several versions of a module, which means the go.sum file might contain several hashes for each version of a module. This means it cannot be used as a source of truth for which dependencies are used. It’s not comparable to Gopkg.lock.

Vendoring

The final piece of the puzzle is vendoring. But first, we need to know about the lifetime of a module. Modules are tagged and released on various git forges (GitHub, GitLab, Bitbucket), and then proxied and cached through proxy.golang.org, which is the main module mirror. When you run go get, it checks the GOPROXY environment variable, and downloads the module to the local Go module cache (go env GOMODCACHE) and when you run the Go build commands, it reads modules from this cache.

If you run go mod vendor, it copies all of the packages required by this module into an in-tree directory called vendor. It doesn’t copy packages that aren’t used by this build or tests of dependencies. When the vendor directory is present, the Go build commands operate in “vendor mode” which means that the build commands read from the vendor dir instead of the local Go module cache or the internet.

We find there are a few benefits to vendoring, even if it is a practice left over from previous dependency management systems. After all, go.sum now ensures module authenticity. Vendoring indirectly provides a package cache and without it we would need to mirror proxy.golang.org in order to ensure the reproducibility of our builds. When a version of a module changes, we can track and can audit the source changes too. This was particularly useful when migrating from Dep, as we were able to precisely see what dependencies had changed.

Finally, when creating the vendor directory, Go creates a listing called vendor/modules.txt. This isn’t supposed to be read by machines or humans and there is no public spec for the file format, but if we look at the source we can see that for each module it prints the name, version (and replacement) and the specific packages used in this project. This means that the vendor/modules.txt file is the static source of truth for our monorepo dependencies.

Diagram showing the lifetime of a Go module. It shows that Go modules start in a git forge such as GitHub and are proxied through proxy.golang.org. When you run `go get` Go downloads the module from proxy.golang.org to the Go module cache on your laptop. When you run `go mod vendor` the module is copied to the vendor directory in your module (in our case the monorepo). When you run `go build` or `go list` the module is referenced from the vendor directory if it is present, otherwise it is referenced from the Go module cache

Dependency changes

For us, the most important difference between Go modules and Dep is the way they choose which version to use. In particular, the cascading effect Minimal Version Selection has when changing direct dependencies, which in turn cause changes to their dependencies.

To make things even more confusing, some Go projects don’t support Go modules yet and some of our dependencies are old enough that they don’t expect Go modules semantics. For example, we use a forked version of etcd which predates Go modules. Dep has calculated a dependency on go-systemd v22 but the old etcd source doesn’t import go-systemd with the major version suffix. This causes Go modules to downgrade go-systemd to a pseudo-version that is one commit before the go.mod file is introduced. This is actually a sensible decision because the old version of etcd isn’t likely to know about future versions of go-systemd anyway. We can consider Minimal Version Selection to be the source of truth but it’s important that we don’t introduce many changes like this at once. Deploying these kinds of changes is straightforward, but debugging and rolling back thousands of them is not.

Another problem we had was with the gopls language server, which affected engineers that use LSP-based editors like VS Code or Neovim. When the LSP server detects a project is using Go modules it searches for the go.mod file to find the root of the project. Then it parses the entire monorepo into memory using memory-heavy Go libraries (intended for code compilation). This grinds an engineer’s laptop to a halt. The workaround is to change the root_dir pattern in the gopls client settings, so it stops searching when it finds a service’s main.go. gopls still parses package imports so features like jump-to-definition still work.

Planning the migration

This isn’t the first time Monzo engineers have tried to move to Go modules. There are several historic branches performing the change since Go modules were introduced. The change itself is straightforward, but the difficult part is the migration.

At Monzo, we use proposals to coordinate complex and high risk changes like this. Any engineer can write a proposal suggesting a change to the platform. The proposal is peer reviewed and modified, just like a pull request, until we’re aligned and happy to execute.

Initially our proposal involved maintaining two dependency trees which different sets of services could depend on while we made a slow migration from one to another. Another approach was to review the mountain of changes and merge them to master, then block all feature deployments while we carefully monitored deploying the dependency changes. The problem with both of these is they involve complicated big-bang style changes and a drawn out deployment process. This blocks product engineers and creates a headache for platform engineers. Not only does this process scale poorly (we have lots of services), but if we detect a problem, it’s difficult to rollback half way through.

These ideas were scrapped during the proposal review. We agreed the safest way to make platform-wide changes is to reduce them so that they barely change anything at all and we changed our approach before writing any code. In order to safely migrate to Go modules we decided to bring the dependency changes close to zero, by changing them before and after the tooling change. Without the proposal process, we might have wasted weeks working on a different solution.

In addition, our proposal covered the social impact of the change. We wanted to make sure engineers knew about Go modules before they had to use them. We would deliver a Go modules introduction at our regular engineering knowledge-sharing sessions and write new quickstart docs. We would also update engineers on Slack, which would close the feedback loop and allow squads to raise any concerns about the services they own. These steps are vital if you’re doing a large migration like this.

Migrating to Go modules

There are three ways we prevented dependencies changing when we switched from Dep to Go modules:

upgrade some before the switch as we already know what version Go modules will select
use the replace directive to force versions that we intend to keep, for example private forks and old indirect dependencies
use the replace directive to force a version we intend to change later, as we already know that version works and it’s quicker to change these using Go modules rather than Dep

The last point is an anti-pattern as we shouldn’t force minimal version selection into an unnatural dependency graph, but this is the tradeoff we made so that we can move incrementally after the switch. Initially our go.mod file looked like the block on the left and we are working towards the block on the right. Gopkg.toml represents intentionally-selected versions such as private forks and old versions, and Gopkg.lock represents modules we will want to unrestrict later.

Diagram showing the layout of our go.mod file. The dependencies from Gopkg.toml have been included in one replace-directive block and the dependencies from Gopkg.lock have been included in another replace-directive block. The vendor directory remains the same. Later, we intend to trim the first replace-directive block and remove the second block.

Ultimately we’re interested in the changes to the vendor directory (there shouldn’t be any) and we can use git diff to check it. We converted Gopkg.toml and Gopkg.lock into a list of replace directives but there were still an enormous number of changes to do with incompatible dependencies. The problem is that git diff doesn’t clearly summarise the semantic versions differences between the modules before (Dep) and after (Go modules).

A screenshot showing git diff with 9501 files changed, 1581996 insertions and 917451 deletions.

We created a tool called depdiff that compares versions between the Gopkg.lock (converted to JSON) and the vendor/modules.txt file, which is the static representation of the build list. The tool outputs semantic version comparisons in a human-readable and line processor-friendly format. This allowed us to make changes to go.mod and quickly check that the vendored packages were equivalent. Once we achieved a reduced set of modules that had to change during the switch (major version changes to import paths) we could use git diff to prove to our security team that the switch was effectively a no-op.

To make sure go.mod is reproducible we run go mod tidy and go mod vendor in CI and block the pull request if changes are detected. This is important because it’s possible to select module versions that wouldn’t be selected with Minimal Version Selection. Running go mod tidy makes sure versions are explicitly set with replace or can be reliably generated.

One tricky problem with go mod tidy is that it also tidies away packages that aren’t dependencies of services, for example commands to generate source such as github.com/vektra/mockery which we run in CI. To solve this, we used the tools.go pattern, which imports the extra packages in a file toggled with a build flag. Since this file might be compiled, go mod tidy can’t remove it.

Screenshot of the tools.go pattern. This pattern prevents `go mod tidy` from removing modules that we need but aren’t referenced in the build.

We also have a check to prevent multiple major versions in go.mod with an exceptions list. Although this feature is supported by Go modules, and in some cases necessary (think about the diamond dependency problem), it’s simpler to maintain one major version per third-party dependency across the entire monorepo.

By separating dependency updates from the tooling change we were able to safely switch to Go modules without affecting deployments. This allowed us to independently move and test the dependencies before and after the switch, which is safer and more scalable than deploying a mass change. Now our dependency management process is simpler, idiomatic and well documented.

If you find yourself wanting to switch from Dep to Go modules, here are some more useful resources:

We found the Cockroach Labs post to be an invaluable resource
Go blog post about migrating to Go modules
Go modules reference page

Have questions about this project, Go Modules, or Monzo engineering in general? The author Tom Preston is taking questions over on the Monzo Community.