Skip to content

ymz-ncnk/musgo

Repository files navigation

[Deprecated] MusGo

MusGo is no longer supported, please consider using mus-go instead. mus-go benefits:

  1. It is very simple and can be easily implemented for other programming languages.
  2. More flexible.
  3. The test coverage is 100%.
  4. It is faster.
  5. Does not use code generation.

Please note that the serialization format of the mus-go is not compatible with the serialization format of the MusGo, due to the float data type.

MusGo is an extremely fast serializer based on code generation. It supports validation, different encodings, aliases, pointers, and private fields.

Fail Fast Serializer

When unmarshalling, MusGo fails fast with validation error as soon as it realizes that the data is invalid. This leaves most of the data untouched and saves system resources. More info you can find in Validation section.

Binary serialization format

Go to the MusGen documentation.

Architecture

MusGen performs the main task of the code generation. MusGo, on the other hand, is tasked with creating the type description needed for MusGen + persists the generated code.

Tests

It is also MusGo's responsibility to test the code generated by MusGen for Golang. You can find these tests in the musgen_..._test.go files and corresponding test data in testdata/musgen. The test coverage of the generated code is about 80%.

MusGo itself is pretty well tested, test coverage is about 90%.

Benchmarks

github.com/alecthomas/go_serialization_benchmarks

Backward compatibility

Go to the MusGen documentation.

Versioning

Go to the MusGen documentation.

How to use

First, you should download and install Go, version 1.18 or later.

Create in your home directory a foo folder with the following structure:

foo/
 |‒‒‒gen/
 |    |‒‒‒mus.go
 |‒‒‒validator/
 |    |‒‒‒validator.go
 |‒‒‒foo.go

foo.go

//go:generate go run gen/mus.go
package foo

type Foo struct {
  num int `mus:"validator.Positive"` // Private fields are supported
  // too. It will be checked with Positive validator while unmarshalling.
  arr []int `mus:",,validator.Positive"` // Every slice element will be checked
  // with Positive validator.
  Alias StringAlias // Alias types are supported too.
  Bool bool     `mus:"-"` // This field will be skiped.
}

type StringAlias string

validator/validator.go

package validator

import "errors"

var ErrNegative error = errors.New("negative")

func Positive(n int) error {
  if n < 0 {
    return ErrNegative
  }
  return nil
}

gen/mus.go

//go:build ignore

package main

import (
  "foo"
  "reflect"

  "github.com/ymz-ncnk/musgo/v2"
)

func main() {
  // MusGo can generate code for struct or alias types.
  musGo, err := musgo.New()
  if err != nil {
    panic(err)
  }
  // You should "Generate" for all involved custom types.
  unsafe := false // To generate safe code.
  var alias foo.StringAlias
  // Alias types don't support tags, so to set up validator we use
  // GenerateAliasAs() method.
  conf := musgo.DefAliasConf
  conf.MaxLength = 5 // Restricts length of StringAlias values to 5 characters.
  err = musGo.GenerateAliasAs(reflect.TypeOf(alias), conf)
  if err != nil {
    panic(err)
  }
  // reflect.Type could be created without the explicit variable.
  err = musGo.Generate(reflect.TypeOf((*foo.Foo)(nil)).Elem(), unsafe)
  if err != nil {
    panic(err)
  }
}

Run from the command line:

$ cd ~/foo
$ go mod init foo
$ go get github.com/ymz-ncnk/musgo/v2
$ go get github.com/ymz-ncnk/muserrs
$ go generate

Now you can see Foo.mus.go and StringAlias.mus.go files in the foo folder. Pay attention to the location of the generated files. The data type and the code generated for it must be in the same package. Let's write some tests. Create a foo_test.go file:

foo/
 |‒‒‒...
 |‒‒‒foo_test.go

foo_test.go

package foo

import (
  "foo/validator"
  "reflect"
  "testing"

  "github.com/ymz-ncnk/muserrs"
)

func TestFooSerialization(t *testing.T) {
  foo := Foo{
    num:   5,
    arr:   []int{4, 2},
    Alias: StringAlias("hello"),
    Bool:  true,
  }
  buf := make([]byte, foo.SizeMUS())
  foo.MarshalMUS(buf)

  afoo := Foo{}
  _, err := afoo.UnmarshalMUS(buf)
  if err != nil {
    t.Error(err)
  }
  foo.Bool = false
  if !reflect.DeepEqual(foo, afoo) {
    t.Error("something went wrong")
  }
}

func TestFooValidation(t *testing.T) {
  t.Run("Validator", func(t *testing.T) {
    var (
      foo = Foo{
        num:   -11,
        arr:   []int{1, 2},
        Alias: "hello",
      }
      want = muserrs.NewFieldError("num", validator.ErrNegative)
    )
    buf := make([]byte, foo.SizeMUS())
    foo.MarshalMUS(buf)

    afoo := Foo{}
    _, err := afoo.UnmarshalMUS(buf)
    if err == nil {
      t.Error("validation error expected")
    }
    if err.Error() != want.Error() {
      t.Fatalf("unexpected error, want '%v' actual '%v'", want, err)
    }
  })

  t.Run("Element validator", func(t *testing.T) {
    var (
      foo = Foo{
        num:   3,
        arr:   []int{1, -12, 2},
        Alias: "hello",
      }
      want = muserrs.NewFieldError("arr", muserrs.NewSliceError(1,
        validator.ErrNegative))
    )
    buf := make([]byte, foo.SizeMUS())
    foo.MarshalMUS(buf)

    afoo := Foo{}
    _, err := afoo.UnmarshalMUS(buf)
    if err == nil {
      t.Error("validation error expected")
    }
    if err.Error() != want.Error() {
      t.Fatalf("unexpected error, want '%v' actual '%v'", want, err)
    }
  })

  t.Run("Max length", func(t *testing.T) {
    var (
      foo = Foo{
        num:   8,
        arr:   []int{1, 2},
        Alias: "hello world",
      }
      want = muserrs.NewFieldError("Alias", muserrs.ErrMaxLengthExceeded)
    )
    buf := make([]byte, foo.SizeMUS())
    foo.MarshalMUS(buf)

    afoo := Foo{}
    _, err := afoo.UnmarshalMUS(buf)
    if err == nil {
      t.Error("validation error expected")
    }
    if err.Error() != want.Error() {
      t.Fatalf("unexpected error, want '%v' actual '%v'", want, err)
    }
  })
}

More advanced usage you can find at https://github.com/ymz-ncnk/musgotry.

When encoding multiple values, it is impractical to create a new buffer each time, it takes too long. Instead, you can use the same buffer for each Marshal:

...
buf := make([]byte, FixedLength)
for foo := range foos {
  if foo.Size() > len(buf) {
    return errors.New("buf is too small")
  }
  i = foo.MarshalMUS(buf)
  err = handle(buf[:i])
  ...
}

To gain more performance, the recover() function can be used:

...
defer func() {
  if r := recover(); r != nil {
    return errors.New("buf is too small")
  }
}()
buf := make([]byte, FixedLength)
for _, foo := range foos {
  i = foo.MarshalMUS(buf)
  err = handle(buf[:i])
  ...
}

It will intercept every panic, so use it with careful.

Supported Types

Supports following types:

  • bool
  • byte
  • int
  • int8
  • int16
  • int32
  • int64
  • uint
  • uint8
  • uint16
  • uint32
  • uint64
  • float32
  • float64
  • string
  • array
  • slice
  • map
  • struct
  • alias

Pointers are supported as well. But aliases to pointer types are not, Go doesn't allow methods for such types.

Private fields

You could encode and decode private fields too.

Unsafe code

You could generate fast unsafe code. Read more about it in the MusGen documentation.

Nil pointers support

Take a note, that nil pointers are encoded with zeros:

type PtrInt struct {
  Value *int
}

var ptr PtrInt
// Buf will not equal to an empty slice here.
buf := make([]byte, ptr.SizeMUS())
ptr.MarshalMUS(buf) // buf == []{0}

There is no any reason, for example, to persist a buf of zeros, instead you can persist an empty slice:

...
var buf []byte
if ptr.Value == nil {
  buf = []byte{}
} else {
  buf = make([]byte, ptr.SizeMUS())
  ptr.MarshalMUS(buf)
}
persist(buf)

And on Unmarhsal:

...
var ptr PtrInt
if len(buf) != 0 {
  _, err = ptr.UnamrshalMUS(buf)
  ...
}

Validation

For every structure field you can set up validators using the mus:"Validator,MaxLength,ElemValidator,KeyValidator" tag , where:

  • Validator - it's a name of the function that will validate the current field.
  • MaxLength - if the field is a string, array, slice or map, MaxLength will restrict its length. Must be a positive number.
  • ElemValidator - it's a name of the function that will validate field elements, if the field is an array, slice or map.
  • KeyValidator - it's a name of the function that will validate field keys, if the field is a map.

All tag items, except MaxLength, must have the "package.FunctionName" or "FunctionName" format.

Decoding(and encoding) is performed in order, from the first field to the last one. That's why, it will stop with a validation error on the first not valid field. There is no practical reason for decoding the rest of the structure when we already know that it is not valid.

For an alias type, you can set up validators with help of the MusGo.GenerateAliasAs() method.

Validators

Validator is a function with the following signature func (value Type) error, where Type is a type of the value to which the validator is applied.

A few examples:

// Validator for the field.
type Foo struct {
  Field string `mus:"StrValidator"`
}

func StrValidator(str string) error {...}

// ElemValidator for the slice field.
type Bar struct {
  Field []string `mus:",,StrValidator"`
}

// KeyValidator for the map field.
type Zoo struct {
  Field map[string]int `mus:",,,StrValidator"`
}

// Validator for the field of a custom pointer type.
type Far struct {
  Field *Foo `mus:FooValidator`
}

func FooValidator(foo *Foo) error {...}

// Validator for the alias field.
type Ror []string

type Pac struct {
  Field Ror `mus:RorValidator`  // you can't set MaxLength or 
  // ElemValidator here, they should be applied for the Ror type.
}

func RorValidator(ror Ror) error {...}

Errors

Often validation errors are wrapped by one of the predefined error (from the MusErrs):

  • FieldError - happens when the field validation failed. Contains the field name and cause.
  • SliceError - happens when the validation of the slice element failed. Contains the element index and cause.
  • ArrayError - happens when the validation of the array element failed. Contains the element index and cause.
  • MapKeyError - happens when the validation of the map key failed. Contains the key and cause.
  • MapValueError - happens when the validation of the map value failed. Contains the key, value and cause.

Encodings

All uint, int and float types support Varint and Raw encodings. By default Varint is used. You can choose Raw encoding using the #raw in mus:"Validator#raw,MaxLength,ElemValidator#raw,KeyValidator#raw" tag.

For example:

// Set up the Raw encoding without a validator for the field.
type Foo struct {
  Field uint64 `mus:"#raw"`
}

// Set up the validator and Raw encoding for the field.
type Foo struct {
  Field uint64 `mus:"Positive#raw"`
}

Raw encoding has better speed and worse size. Only on large numbers (> 2^48 in uint representation) it has same or lesser size as Varint.

For an alias type, you can set up encoding with help of the MusGo.GenerateAliasAs() method.

Single number serialization

If all you want is to serialize a single number you can use:

About

[Deprecated] Provides serialization with validation support for Golang

Topics

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages