Go's early growth 2012-2016

In this article, I’ll talk about how I got involved with the Go programming language, my first several years working on the Go team, and the factors that contributed to Go's early growth from 2012-2016.

I first encountered Go in 2010, when Rob Pike came to the Google New York office to give a tutorial on the language. At the time I was working on globally distributed storage and indexing systems. In that work I had created a pair of C++ classes to read and write replicated bigtables in a way that ensured read-after-write consistency, even when some replicas were unavailable. Each of my C++ classes was 700 lines of tricky multithreaded code. The code worked, but it was very hard to understand and maintain.

Rob Pike's Go tutorial introduced me to a new way of thinking about concurrency, communicating sequential processes (CSP), and Go provided an elegant syntax and implementation of CSP. I immediately tried porting my C++ classes to Go and was amazed to find that I could reduce each down to 100 lines, a 7x reduction in code size! Not only that, but because the code was expressed using higher-level concurrency primitives and no longer had to do explicit memory management, it was much easier to consolidate common code and extend the routines with new logic.

I was smitten. I immediately started trying to write more of my Google code in Go, but I quickly discovered that Go's support for Google's internal libraries and protocols was nascent. Thankfully, the culture inside Google supported my spending some time contributing to improving these libraries, and in doing so I got to know several members of the Go team. Near the end of 2011, Russ Cox invited me to join the Go team to work full time on making Go great for Googlers.

I joined the Go team in January 2012. At the time, the team was focused on shipping Go 1.0, a milestone release that would guarantee backwards compatibility and encourage many more companies (including Google) to adopt Go for real production workloads. I personally had very little to do with Go 1.0—my focus was on Go inside Google.

Unlike most of the Go team, I had spent several years developing, operating, and maintaining production systems inside Google, mainly written in C++. I was very familiar with the “production contact surface” that jobs needed to expose to be “operator friendly”: communication protocols, command line flags, logs, diagnostics, status pages, exported metrics, crash reporting, traces, and so on. I was also familiar with the tools Google developers used to get their jobs done. This knowledge gave me confidence in identifying the requirements that Go would need to satisfy to work well inside Google, but I also wanted to consult the experts: Google's Site Reliability Engineers (SREs).

I called up some of the SRE friends I'd made during my pager-carrying years and asked them to help us identify what Go would need to do to satisfy a “Production Readiness Review”, which was the process at the time for determining whether a system was mature enough for SRE support. The SREs were delighted with the request: this was the first time they had been invited to review a new language well before it would be deployed in production. Google's SREs helped us get the requirements right, and in particular they identified the need to reduce Go's garbage collection pause latency dramatically from where it was in Go 1.0. These requirements greatly influenced the evolution of Go’s low-latency concurrent GC.

Many people across Google and the Go team contributed to making Go a great language for Google's internal use, and Google's use of Go grew rapidly from 2012 to 2016. However, we noticed that adoption was uneven: organizations that were well served by C++ or Java tended to stick with those languages, while organizations that were dissatisfied with their current language or were writing mostly greenfield code were eager to adopt Go. In 2013, Google's SRE organization decided to adopt Go for all their own new programs, replacing Python as their language of choice.

Another group that eagerly adopted Go was Cloud. Go was booming outside Google as the language of open source cloud infrastructure, starting with Docker, Vitess, and Kubernetes and expanding to the majority of Cloud Native Computing Foundation (CNCF) projects. Google created Kubernetes, and Google Cloud wrote many of their internal systems in Go. Soon, Go began growing as a language for cloud applications, first in startups, then later in mainstream enterprises.

Reflecting on this period, there were several factors that contributed to Go's early growth:

1. Greenfield code. Most professional programming is maintenance of existing code. Paradigm shifts, like cloud development, create the opportunity to write new code in new languages, largely independent of existing programs and libraries.

2. Suitability to task. Go was designed for building production systems at Google. This includes not just the language and runtime but also the libraries: the Go standard library includes everything you need to build production-grade HTTP and REST servers, and Go gRPC enables building high-performance RPC services. Go provides simple cross compilation and easy deployment with static binaries. Go arrived just in time to meet widespread demand to build new cloud infrastructure and applications, and it provided a simpler programming platform than the incumbents.

3. Distributed systems. The separation of applications into processes connected over a network created the opportunity to write different parts of the system in different languages, which made it easier to adopt Go incrementally. Go became popular not only for services and microservices, but also for command line tools and web backends.

4. Open source. Go was open sourced in 2009, and several elements of its design fostered open source collaboration: a standard code format, a standard tool for builds and testing, and a decentralized package system. Go was well timed to grow with GitHub and serve as the language for open source cloud infrastructure and DevOps.

5. The Go team. Go has always had an amazing team at Google, and Google has provided steadfast support for Go since 2007. Many of the early Go team members were well known and widely respected by the developer community, which certainly helped get people interested in Go.

6. The Go community. Go's community has been incredible in advocating for the language and building Go's rich ecosystem. Contributors have extended Go to many new platforms, operating systems, and application domains.

Many of these factors may apply to the introduction of any new programming language or to the expansion of an existing language into a new application domain. An additional factor is interoperability, which is not a strength for Go but has been critical to the adoption of Kotlin (for Java), Swift (for Objective C), and C++ (for C).

My first four years on the Go team is when I made most of my coding contributions to Go and wrote many of the articles listed on ajmani.net/go. These cover topics like concurrency, package structure, gRPC, and context, a data type I designed to support cancellation and request-scoped data in distributed systems.

In a future article, I'll talk about Go from 2016 till now. These are the years that I've been managing the Go team, during which I've learned a lot about running an open source team within a large company.