Stream Is Deprecating Virtual Go – It’s Time to Move to Go Modules

Engineering
Go
3 min read
Tommaso B.
Tommaso B.
Published January 16, 2020 Updated August 31, 2020

TL;DR

Stream released vg almost three years ago, and since its inception, Go has improved quite a bit in regards to its dependency management. Since Go has come so far, we’re now advocating that the community use native Go modules rather than vg.

The Gist

Stream provides an API as a service to build highly scalable feeds and white-label chat for any application or messaging service. In the beginning days of Stream, the entire infrastructure utilized Python on the backend; however, the amount of throughput demanded by Stream's customers, in addition to the growing size of the development team, made Python inadequate for continued use. With that, the team decided to switch to a more performant programming language – Go.

We have already written about our decision-making process a couple of years on our blog, there were many things we liked about Go at that time. Unfortunately, dependency management was not one of them – there were no lockfiles, no clear way to version dependencies, etc.

For this reason, we wrote a small library called "virtualgo" that wrapped dep and made it dead simple for developers to work on several Go projects at the same time without dependency clashing.

Note: Everything that motivated us 3 years ago does not apply anymore to dependency management in Go. Modules have solved all these problems and added additional functionality as well such as enforcing semver, immutability and checksum database, etc.

We were very excited about Go modules and tried them as soon as possible (Go 1.11). Sadly we had to postpone using them as many libraries did not support them at all; in July of 2019, all libraries we needed had support for modules, and we started migrating projects over.

Migrating to Go modules is incredibly easy; in our case, because we used vg and dep, we could use go mod itself to migrate the dependency manifests. Surprisingly that was pretty much all we had to do.

A few items to expand:

  • Cgo isn’t Go and very problematic. It’s almost always a good idea to stay away from it because most of the guarantees of Go, for example, in terms of portability, don’t exist.
  • We have used tools package pattern to track the exact version of our necessary tools such as stringer, protocol buffers, etc. By this, there is no divergence between developer environments. This package is also guarded by tools build flag, which doesn’t add any overhead into the build speed or output.
  • Thanks to modules we were able to drop some build optimizations that made our build pretty complex.
  • We’re using sentry for error reporting, which can generate context snippets whenever an error occurs, we have used vendor mod to generate these snippets for 3rd party dependencies, too.

Summary

  1. Two years ago we switched from Python to Go
  2. We did not find a solution to dependency management that we liked (we tried Glide, dep, vendor, etc.)
  3. We built virtualgo and open-sourced it ❤️
  4. Virtualgo solved the necessary need for Go dependency management and was/is used by many Go developers; (1.1k stars as of writing this on GitHub)
    1. Virtualgo solved these problems
      1. GOPATH based solution
      2. No vendoring
      3. No shared dependencies between projects
      4. Simplify integration with Dep
  5. Go added support for modules in 1.11
  6. Initial support for modules was experimental; however, it, later on, became stable, but many libraries broke in the process
    1. Go generate did not work; no stringer
  7. Go modules are now very well supported; at this point any standard and well-maintained library supports it
  8. Why modules are excellent and better than vg
    1. Blessed by the community (one of dep’s author was upset about this 🔥)
    2. Lockfile (go.sum)
    3. Fast
    4. Supports proxies out of the box
    5. Private Github repo; no problem
    6. Put any Go project anywhere you want
    7. No more “github.com” in your import path
  9. What’s next
    1. vg will stay around as long as GitHub will allow it
    2. We are not using it anymore; other devs might so if needed we are happy to add a new maintainer and review / merge PRs
    3. we are going to add a disclaimer + link to this article on the Github readme
    4. how to migrate from vg to modules
      1. First, have a look at Go official docs to get an idea about modules
      2. vg wraps dep; migration is super simple (we did that, and it was painless)
Recommended posts
Stream Is Deprecating Virtual Go – It’s Time to Move to Go ModulesHow a Go Program Compiles down to Machine CodeGo 1.11 Rocket TutorialFixing the billion dollar mistake in Go by borrowing from RustGo Client Now AvailableWhy we switched from Python to GoTesting Go at StreamBuilding a Performant API using Go and Cassandra