Introducing Gizmo

At The New York Times, our development teams have been adopting the Go programming language over the last three years to build better back-end services. In the past I’ve written about using Go for Elastic MapReduce streaming. I’ve also talked about using Go at GothamGo for news analysis and to improve our email and alert systems at the Golang NYC Meetup. We use Go for a wide variety of tasks, but the most common use throughout the company is for building JSON APIs.

When we first began building APIs with Go, we didn’t use any frameworks or shared interfaces. This meant that they varied from team to team and project to project with regard to structure, naming conventions and third-party tools. As we started building more and more APIs, the pains of microservices started to become apparent.

Around the time we reached this point, I came across Peter Bourgon’s talk from FOSDEM 2015, “Go and the Modern Enterprise,” and its accompanying blog post. A lot of what Peter said hit close to home for me and seemed very relevant to our situation at The Times. His description of the “Modern Enterprise” fit our technology teams quite well. We’re a consumer-focused company whose engineering group has more than doubled in size to a few hundred heads in recent years, and we have had a service-oriented architecture for a long time. We have also run into the same problems he brought up. As the need for more and more microservices arose, we needed a common set of tools to orchestrate and monitor them, as well as a common set of patterns and interfaces that enable developers to concentrate on business problems. An RFC for a new toolkit named “Go Kit” came out of the talk, and eventually open source development of it was under way.

Peter’s talk and the concept of Go Kit made me very excited but also somewhat dismayed. I knew a lot of the RPC and tracing technology involved would likely take a long time to be adopted throughout the company without some stepping stones to get us there. We also didn’t have a whole lot of time to wait around for the toolkit to be completed, so we decided to build our own set of tools that could bridge the gap and hopefully complement Go Kit by implementing some of its “non-goals.”

It’s my pleasure to announce that as of today our toolkit, Gizmo, is open source. Gizmo offers four packages to help developers quickly configure and build microservice APIs and pubsub daemons.



The Gizmo logo was created by Jean Kim, based on the Go mascot designed by Renée French and copyrighted under the Creative Commons Attribution 3.0 license.

package config

The config package provides a set of common, composable structs for working with tools common to technology currently at The New York Times:

  • MySQL
  • MongoDB
  • Oracle
  • AWS (SNS, SQS, S3)
  • Kafka
  • Gorilla’s securecookie
  • Gizmo Servers

The package also has a generic Config type that contains all of the above types. It’s meant to be a catchall struct that most applications should be able to use.

config also contains a set of functions to help populate data into config types from JSON files, JSON blobs in Consul’s key/value store or, with the help of Kelsey Hightower’s envconfig, environment variables. The helper functions work with any structs, so you aren’t required to use a gizmo/config type to use them.

package server

The server package contains the bulk of the toolkit. It provides a set of interfaces that define how a server and service should look and interact with one another starting with the Server interface:

type Server interface {
    Register(Service) error
    Start() error
    Stop() error
}

The Server implementation is in charge of providing health checks, monitoring, logging, graceful shutdowns and hooks for adding in middleware handlers for any registered services. There are currently two implementations of the Server interface in Gizmo. The first and most used implementation is called SimpleServer. The SimpleServer is composed of the mux and context packages from the Gorilla web toolkit, go-metrics from Richard Crowley and logrus from Simon Eskildsen. The second implementation is the RPCServer, an experimental server that exposes endpoints over gRPC on one port and JSON over another. This server is what we hope to be our next stepping stone on our path toward RPC and Go Kit.

Servers are meant to host services that (at a very minimum) must implement the Service interface:

type Service interface {
    // Prefix's result will be prefixed to all endpoint routes within the Service
    Prefix() string
    // Middleware provides a hook for servicewide middleware.
    Middleware(http.Handler) http.Handler
}

The SimpleServer can accept three different flavors of Service interfaces. The first is SimpleService, which is exposed via http.HandlerFuncs:

type SimpleService interface {
    Service
 
    // route - method - func
    Endpoints() map[string]map[string]http.HandlerFunc
}

The next service type is JSONService. It is meant to cut out a lot of the boilerplate code behind a pure JSON response. It also provides an additional middleware function to have more control over how a service responds at a global level.

type JSONService interface {
    Service
 
    // route - method - func
    JSONEndpoints() map[string]map[string]JSONEndpoint
    // JSONMiddleware provides a hook for servicewide middleware around JSONEndpoints.
    JSONMiddleware(JSONEndpoint) JSONEndpoint
}

A JSONEndpoint is a function that accepts an http.Request and responds with the HTTP status code, an object to marshal as JSON and an optional error. It is converted to an http.Handler via Gizmo’s helper function, JSONToHTTP, which sets the appropriate Content-Type header and status code, then encodes the response as JSON.

type JSONEndpoint func(*http.Request) (int, interface{}, error)

The last service that the SimpleServer can register is a mixture of the simple and JSON services. It’s called MixedService and can be useful in cases where a service may need some pure JSON endpoints and some endpoints that do more generic web tasks like dropping cookies.

type MixedService interface {
    Service
 
    // route - method - func
    Endpoints() map[string]map[string]http.HandlerFunc
 
    // route - method - func
    JSONEndpoints() map[string]map[string]JSONEndpoint
    JSONMiddleware(JSONEndpoint) JSONEndpoint
}

When it comes to the RPCServer, only the RPCService can be accepted. It offers service creators the option to expose their gRPC service over JSON:

 
type RPCService interface {
    Service
    // Service will return a gRPC service's description and an implementation.
    Service() (grpc.ServiceDesc, interface{})
 
    // route - method - func
    JSONEndpoints() map[string]map[string]JSONEndpoint
    JSONMiddleware(JSONEndpoint) JSONEndpoint
}

For some basic examples of how to implement, configure and structure Gizmo servers, take a look at the examples/servers directory in the repository.

package web

This is the smallest package and consists only of handy functions for parsing types from request queries and payloads. Most of the functions involve extracting integer and date types from input but there is also a function to parse ‘truthy/falsy’ to help with transitions away from legacy services.

package pubsub

The pubsub package provides a high-level abstraction for building publisher and subscriber services and implementations for the two messaging systems in use at The New York Times: Amazon SNS/ SQS and Apache Kafka. The Amazon implementations are widely used in production here, but the Kafka implementation is still somewhat experimental. To pair well with gRPC, the pubsub interfaces default to accepting protobuf messages but also allow users to work with byte slices. The Publisher interface is pretty small as publishing is a fairly basic concept:

type Publisher interface {
    Publish(string, proto.Message) error
    PublishRaw(string, []byte) error
}

The Subscriber interface is somewhat more involved as it needs to be able to offer the ability to acknowledge processed messages and also shut down gracefully.

type Subscriber interface {
    // Start will return a channel of messages
    Start() <-chan SubscriberMessage
    // Err will contain any errors returned from the consumer connection.
    Err() error
    // Stop will initiate a graceful shutdown of the subscriber connection
    Stop() error
}

A SubscriberMessage interface provides users with access to the message payload and a hook to mark a message as done. The underlying SQS implementation will send an SQS delete message request on Done(); Kafka will emit the message’s offset.

type SubscriberMessage interface {
    Message() []byte
    Done() error
}

To help Gizmo users write tests for pubsub services, the toolkit also provides a set of test implementations of the interfaces that can be found in pubsub/pubsubtest. For some examples of how to use the pubsub package, take a look at the examples/pubsub directory in the repository.

Next Steps

We currently have five teams using Gizmo to get stuff done at The New York Times, but we know the toolkit is far from done and far from perfect. While we’re open sourcing it to help others adopt Go, we’d also like the awesome Go community to take the toolkit to the next level. See something you’re unsure of or any missing configurations or tools? Please create an issue or a pull request and feel free to contribute to our new public github repository:

//github.com/nytimes/gizmo

We’d love to get some help making other Server and Service implementations. Over the next year, I’d like to work together on the following additions to the toolkit:

Keep an eye for Open blog posts about Gizmo’s server and pubsub packages. Also, if you have any questions you can reach out to us on the Gopher Slack community under the #gizmo channel.