This ended up being a release to cleanup old sins in the decoder internals and change some defaults how binary values work with JSON and string functions.

It also adds a new Time Zone Information Format decoder tzif (Thanks Takashi Oguma @bitbears-dev) and a new Apple BookmarkData decoder apple_bookmark decoder (Thanks David McDonald @dgmcdona). Also a new function from_ns_keyed_archiver was added to convert NSKeyedArchiver encoded objects into JSON.

A possible breaking change is that now all from/to prefix functions now has a from_/to_ prefix, ex: from_mp3 instead of frommp3. There are some few exceptions to this. Note that the functions named just be the format name, ex mp3 are still around.

In other fq related news jq-lsp got some fixed and additions and seems to work fine with neovim. It's also possible to use jq-lsp with vscode using vscode-jq.

Changes

• All functions that had a from/to prefix now has the prefix from_/to_. This is to be easier to read and more consistent, there are still some exceptions like tovalue, torepr, tobytes etc but in general anything that does not deal with primitive types is now snake_case. #535
• Change default bit_formats option value (how raw bits values are represented in JSON) from snippet to string. snippet meant truncated bits as base64. Now all bits are included as a UTF-8 string. The string will be binary safe (not lose any data) when used internally in fq but will lose data when represented in JSON as some bytes can't be encoded as UTF-8. #499
• Don't auto convert to binary for string/regexp functions, turned out this is very confusing. Now you have to manually use tobytes etc to convert to binary value. #540

  # This used to not work as test/1 would convert decode values to the source bytes
  # (0x00 0x00 0x00 0x01) in this case. Now the jq value (symbolic in this case) will
  # be used instead. You can do ".test | tobytes" to get old behavior.
  #
  # find all types with a "mdta." prefix
  $ fq -o line_bytes=10 'grep_by(.type | test(`^mdta\.`))' file.mp4
       │00 01 02 03 04 05 06 07 08 09│0123456789│.boxes[3].boxes[2].boxes[0].boxes[2].boxes[0]{}: box
  0x528│      00 00 00 1c            │  ....    │  size: 28
  0x528│                  00 00 00 01│      ....│  type: "mdta.title" ("\x00\x00\x00\x01")
  0x532│00 00 00 14 64 61 74 61 00 00│....data..│  boxes[0:1]:
  0x53c│00 01 00 00 00 00 74 65 73 74│......test│
       │00 01 02 03 04 05 06 07 08 09│0123456789│.boxes[3].boxes[2].boxes[0].boxes[2].boxes[1]{}: box
  0x546│00 00 00 25                  │...%      │  size: 37
  0x546│            00 00 00 02      │    ....  │  type: "mdta.encoder" ("\x00\x00\x00\x02")
  0x546│                        00 00│        ..│  boxes[0:1]:
  0x550│00 1d 64 61 74 61 00 00 00 01│..data....│
  0x55a│00 00 00 00 4c 61 76 66 35 39│....Lavf59│
  0x564│2e 32 37 2e 31 30 30│        │.27.100│  │

• Fix panic when cancel (ctrl-c etc) before interpreter is executing. Thanks @pldin601 for reporting. #495
• Fix error using JQValue:s in assign/update paths, ex .[<JQValue here>] = 123 #509
• Rename fields added for bit-ranges not used by a decoder from unknown# to gap#. "unknown" is probably a useful field name in some formats and "gap" describe better what it is. #500
• Big decode API internals refactor to split scalars types into their own go types so they can store per type specific values. This also opens up for more ways to make fq both faster and more memory efficient. It also makes the decode API more type safe and makes it possible to experiment with decode DLS that uses chained methods etc. #523

Decoder changes

• apple_bookmark New Apple BookmarkData decoder. Thanks David McDonald @dgmcdona. #493
• bplist
  • Fix decoding of UID types
  • Adds a lost_and_found array with unused values
  • Fix an endian issue for unicode strings
  • Add NSKeyedArchiver to JSON helper function from_ns_keyed_archiver, see bplist docs for details on how to use it. Thanks David McDonald @dgmcdona. #493

  # decode bplist, from_ns_keyed_archiver converts to JSON plist and then into object data as JSON, find app bookmarks keys and expand them as bookmark data and convert to represented JSON, and finally build path to applications
  $ fq -r 'from_ns_keyed_archiver | (.. | .Bookmark? // empty) |= (apple_bookmark | torepr) | .. | .target_path? // empty | join("/")' recentapps.sfl2
  System/Applications/Utilities/Terminal.app
  Applications/Spotify.app
  System/Applications/Calculator.app
  System/Applications/Preview.app
  Applications/Alacritty.app
  Applications/DB Browser for SQLite.app
  System/Applications/System Preferences.app
  System/Library/CoreServices/Applications/Directory Utility.app
  System/Applications/Utilities/Activity Monitor.app
  Applications/Safari.app
  

• tzif new Time Zone Information Format decoder. Thanks Takashi Oguma @bitbears-dev. #498
• mp4
  • Map mdta metadata namespace and key names for ilst child boxes. #521

  # create object with all ilst key/value pairs
  $ fq 'grep_by(.type=="ilst").boxes | map({key: .type, value: .boxes[0].data}) | from_entries' file.mp4
  {
    "mdta.encoder": "Lavf59.27.100",
    "mdta.title": "test"
  }
  # query specific value
  $ fq -r 'grep_by(.type=="mdta.encoder").boxes[0].data | tovalue' file.mp4
  Lavf59.27.100

  • Support sidx version 1. #506
  • Add description and symbolic values for traf sample flags, makes it easier to see and query for I-frames etc. #514

  # which boxes has depends_on flags
  $ fq 'grep_by(.sample_depends_on) | parent' fragmented.mp4
  

  • Support PNG codec mapping. #492
  • Decode pdin boxes. #524
  • Decode hnti boxes. #513
• mp3_tags Add VBRI support and split into into mp3_frame_xing and mp3_frame_vbri decoders. #525

Changelog

• 7fa8b63 Add related file format projects to README
• 4fdb736 Update docker-golang to 1.19.4 from 1.19.3
• 519eff6 Update github-go-version to 1.19.4 from 1.19.3, 1.19.3, 1.19.3
• 2a91d29 Update gomod-golang/text to 0.5.0 from 0.4.0
• cb15b37 added checks to prevent infinite looping and recursion
• c244533 added some sfl2 test files to bplist package
• 7d13cf7 adds flag parsing to applebookmark
• 71b17d0 apple bookmarkdata decoder initial commit
• 8f39ef6 bplist: Harmonize ns_keyed_archive jq style a bit
• cba72db bplist: added overload for from_ns_keyed_archiver jq func
• 129b4b7 bplist: doc: update docs to reflect changes to ns_keyed_archiver
• 9dab3c6 bplist: minor fix to from_ns_keyed_archiver
• 448c3ef bplist: update docs with from_ns_keyed_archiver reference, add error case to function
• a9047c0 bplist: updates from_ns_keyed_archiver to do automatic torepr based on format detection
• 4a28e44 changes decoder package name from bookmark to apple_bookmark
• d0b044c converts to snake_case and refactors decode helper
• d199793 created stack type
• e77f776 decode,interp: Rename unknown gap fields from "unknown#" to "gap#"
• a85da29 decode: Make FieldFormat usage more consistent
• 9b81d4d decode: More type safe API and split scalar into multiple types
• 3ec0ba3 decode: add ns_keyed_archiver, restructure apple decoder into apple package
• 330d5f7 decode: apple_bookmark: simplifies flag decoding
• 93f2aa5 decode: change PosLoopDetector to use generics
• 7e98b53 decode: fix type on defer function call, test: add loop.fqtest
• a873819 decode: fixes endian of unicode strings
• f747873 decode: implements lost and found for unreferenced objects
• b45f9fa decode: improve stack push/pop
• a162e07 decode: minor change to method receiver name
• 3232f9c decode: moves PosLoopDetector into its own package
• 7c9504c decode: moves macho decoder to apple package
• 7083467 decode: remove dead code from ns_keyed_archiver
• 7ab4466 decode: remove unused field from decoder, unused parens from torepr
• bdb8166 decode: removed unnecessary struct
• 98eab8c decode: rename parameter for consistency
• 04379df decode: revert decode.D back, place posLoopDetector in apple_bookmark
• 7fb674b decode: unexport methods
• fa368bb decode: updates all.go with correct macho path
• 0287ffa decoding well but torepr needs work
• 42debe5 dev,doc,make: Cleanup makefile and have proper targets for *.md and *.svg
• 423bab9 dev,test: Use jqtest code from jqjq for jq tests
• 6fc84a8 doc,dev: Add more usage and dev tips
• 2fc16ae doc: Add some padding margin to formats table to make it less likely to cause git conflicts
• 62f377c doc: fixes snippet for recursive bookmark searching
• 22064f5 doc: remake
• 4aad2fd doc: remake
• b872b1a doc: remake
• 1e1fc55 fixed one more snake_case letter
• d0b76ca fixes broken test and removes long link from markdown body
• 5146f28 fixes broken test for all.fqtest
• 253033c fixes broken uid parsing in plist decoder
• f535ad3 fixes spacing in jq files
• 64351e8 fixes tests and adds torepr test
• c7d00b8 fixes unknown bit ranges
• 8f930aa forgot to add bookmark.jq in last commit
• 164e527 gojq: Update rebased fq fork
• 6c86945 gojq: Update rebased fq fork
• 578b84d interp,display: Add workaround for go 1.18 when escaping 0x7f
• 42d9f2c interp,help: Properly count line length when breaking on whole words
• 8d69f1f interp: Change default bits_format=string
• 6c229d7 interp: Don't auto convert to binary for string functions, is just confusing
• 568afff interp: Fix panic when trigger before any context has been pushed
• e3ae144 interp: Rename to/from functions to to_/from_
• ba88a68 interp: mimic jq: if expr arg is given read stdin even if tty
• 9bd65f9 migrates tests to per-sample files
• f7d7a49 missed a letter on last commit - converting to snake_case
• 2f37cb5 mod: Update modules not tracked with bump
• 55f4f1a moved a flag bit fields into correct positions
• 9e5a072 mp3_frame_tags: Covert to decode group and split to mp3_frame_{xing,vbri} decoders
• 48522e3 mp3_tags,mp3: Add VBRI header support and rename tags to tag as there is only one
• 83ccedc mp4,decode: Properly decode ilst items (both mdta and mdir)
• 1dea40e mp4,doc: Add JSON box tree example and reorder a bit
• b1b3b63 mp4: Add namespace to mdta ilst boxes
• 7b60b24 mp4: Add pdin box support
• ef2d523 mp4: Add png mapping
• 5fb81a1 mp4: Add sym and description for traf sample flags
• 1d6ce2c mp4: Decode hint and hnti child boxes
• 9ac453a mp4: Fix typo in sample flags sample_is_depended_on description
• a23fe61 mp4: sidx version 1 segment_duration is s64
• 3942db7 pkg/decode/D: Adds PushAndPop, Push, Pop methods. doc: adds help_applebookmark.fqtestdecode: converts applebookmark to use new d.PushAndPop method
• 0c216df refactors some decoder logic in apple_bookmark for better querying
• 34db9d7 regenerated docs, added tests, fixed torepr
• 0a72635 remade documentation
• 1352598 removed commented out line
• 8126943 removed unnecessary conversions
• 5b1455e removed unused function
• 63a3ca2 removes underscore from apple_bookmark package name
• a351c34 removes unused function
• 2ee6360 support tzif (time zone information format)
• 8d5dcff test: applebookmark: adds problematic test case
• 63a4e80 test: fixed doc test
• 47a568e text,test: Unbeak base64 tests
• 44c91d8 tweaks apple_bookmark markdown documentation
• fd22426 tzif: add help_tzif.fqtest
• c4e7fc7 tzif: moved document to tzif.md
• abde823 tzif: use PeekFindByte() to find end of the string
• 4481a77 tzif: use scalar.Fn() to define a mapper ad hoc
• dbc6fcc updated doc with apple reference
• f5e25fc updated docs
• 6f4d1cb updated documentation
• b2aeac6 updates bplist fq tests
• a23ac8f updates fqtest for torepr in apple_bookmarkdata