Simplified JSON Handling in Go

One of my first Go projects involved taking a bunch of JSON test fixtures and feeding them to the APIs we’d built. Another team created these fixtures in order to provide expected inputs and outputs to an API with implementations across a number of different languages.

JSON is always tricky in a typed language — it’s a format with strings, numbers, dictionaries, and arrays. If you’re coming from a language like javascript, python, ruby, or PHP a big selling point of JSON is the ability to parse and encode data without needing to think about types.

// in PHP
$object = json_decode('{"foo":"bar"}');

// in javascript
const object = JSON.parse('{"foo":"bar"}')

In a typed language, someone needs to decide how a JSON object’s strings, numbers, dictionaries, and arrays are handled. In Go, the built-in APIs are designed such that you, the end user programmer, need to decide how a JSON file is best represented as a Go data structure. Dealing with JSON in Go is a deep topic that I won’t get into fully, but here’s two code samples that demonstrate the challenges. These borrow heavily from the Go by Example examples.

Parsing/Unmarshalling into a map[string]interface

First, consider this program

package main

import (
    "encoding/json"
    "fmt"
)


func main() {

    byt := []byte(`{
        "num":6.13,
        "strs":["a","b"],
        "obj":{"foo":{"bar":"zip","zap":6}}
    }`)
    var dat map[string]interface{}
    if err := json.Unmarshal(byt, &dat); err != nil {
        panic(err)
    }
    fmt.Println(dat)

    num := dat["num"].(float64)
    fmt.Println(num)

    strs := dat["strs"].([]interface{})
    str1 := strs[0].(string)
    fmt.Println(str1)

    obj := dat["obj"].(map[string]interface{})
    obj2 := obj["foo"].(map[string]interface{})
    fmt.Println(obj2)

}

Here we’re Unmarhsaling (i.e. parsing, decoding, etc) the JSON from the byt variable into the map/dictionary object named dat. This is similar to what we do in other languages, with the exception that our input needs to be an array of bytes (vs. a string) and each value of the dictionary needs to have a type assertion applied in order to use/access that value. These type assertions can start to get tedious and verbose when we’re dealing with a deeply nested JSON object.

Parsing/Unmarshalling into a map[string]interface

Your second option looks like this

package main

import (
    "encoding/json"
    "fmt"
)

type ourData struct {
    Num   float64 `json:"num"`
    Strs []string `json:"strs"`
    Obj map[string]map[string]string `json:"obj"`
}

func main() {
    byt := []byte(`{
        "num":6.13,
        "strs":["a","b"],
        "obj":{"foo":{"bar":"zip","zap":6}}
    }`)

    res := ourData{}
    json.Unmarshal(byt, &res)
    fmt.Println(res.Num)
    fmt.Println(res.Strs)
    fmt.Println(res.Obj)
}

Here we’re unmarshalling the bytes in byt into the instantiated struct of type ourData. This uses the tags feature of Go structs.

Tags are strings that following the definition of a struct field. Consider our definition

type ourData struct {
    Num   float64 `json:"num"`
    Strs []string `json:"strs"`
    Obj map[string]map[string]string `json:"obj"`
}

Here you can see a tag of json:"num" for the Num field, a tag of json:"strs" for the Str field, and a tag of json:"obj" for the Obj field. The strings above use back ticks (sometimes called back quotes) to define the tags as string literals. You don’t need to use back ticks, but using double quoted strings leads to some messy escaping

type ourData struct {
    Num   float64 "json:\"num\""
    Strs []string "json:\"strs\""
    Obj map[string]map[string]string "json:\"obj\""
}

Tags are not required in a struct definition. If your struct does include tags, it means the Go reflection APIs can access the value of that tag. Packages can then use these tags to do things.

The Go encoding/json package uses these tags to determine where each top-level JSON field goes when it unmarshals those fields into an instantiated struct. In other words, when you define a struct like this

type ourData struct {
    Num   float64   `json:"num"`
}

you’re telling Go

Hey, if someone wants to json.Unmarshal some JSON into this struct, take the top level JSON num field and drop it into the struct’s Num field.

This can make your Unmarshal code a bit easier in that the client programmer doesn’t need to explicitly type assert every field. However, it still has limitations/drawbacks.

First — you can only use top level fields in tags — nested JSON requires nested types (ex. Obj map[string]map[string]string ...), so you’re really just shifting the verboseness around.

Second — it presumes your JSON is consistently structured. If you run the above program you’ll notice the "zap":6 doesn’t make it into the Obj field. You could handle this by making the type map[string]map[string]interface{}, but then you’re back to needing type assertions to get at the values.

This was the state of things when I took on my first Go project, and it was a bit of a slog. Fortunately, things seem a bit better today.

SJSON and GJSON

Go’s built-in JSON handling hasn’t changed, but there are a number of mature packages for handling JSON in a way that’s more in spirit with the format’s easy-to-use intentions.

SJSON (for setting values) and GJSON (for getting values) are two packages developed by Josh Baker that allow you to directly extract or set a value in a JSON string. To borrow the code samples from the project’s READMEs — here’s all you need to do to fetch a nested value out of a JSON string using GJSON

package main

import "github.com/tidwall/gjson"

const json = `{"name":{"first":"Janet","last":"Prichard"},"age":47}`

func main() {
    value := gjson.Get(json, "name.last")
    println(value.String())
}

Similarly, here’s the sample program for using SJSON to “set” a value in a JSON string by returning a new string with the set value.

package main

import "github.com/tidwall/sjson"

const json = `{"name":{"first":"Janet","last":"Prichard"},"age":47}`

func main() {
    value, _ := sjson.Set(json, "name.last", "Anderson")
    println(value)
}

If SJSON and GJSON aren’t your speed, there’s a number of other third party libraries for making real-world JSON a little less unpleasant to deal with in Go.