Immudb 1.2.1 highlights

This release of immudb comes with many new features and improvements. The most important ones are highlighted below.

Improved transaction support

Starting with this version of immudb, SQL clients can now use true SQL transactions. It is now possible to start a transaction, perform multiple read and write operations and then commit or rollback changes. Before that each transaction had to be executed within one command sent to the server.

Logical deletion and expiration

KV and SQL records can now be deleted. This was enabled by introduction of metadata attached to entries that can store information about deletion or expiration. Such deleted entries are no longer available when trying to read KV or SQL entries but one can still lookup the history of a key or perform an SQL statement using time travel to fetch historical values before deletion.

In adddition to deletion, KV entries also support setting data expiration. Once expired, values of KV entries can no longer be retrieved. Expiration is a much stronger concept than deletion and once expired, immudb will protect against reading of the value for given key even if the history of an entry is retrieved.

Much improved SQL support

Due to introduction of metadata, immudb now supports the DELETE statement. Deleted rows do not appear in standard SQL queries anymore, however one can still query them using time travel feature.

This release also add support for a standard UPDATE statement that enables modifying only some columns of a row, before that the only way to update an existing entry was with the UPSERT statement that required values for all columns and could only be used with a primary key.

An improvement to the UPSERT allows updating rows for tables that have secondary indexes.

There are many more smaller improvements in the SQL engine for this release such as: indexes for columns with NULL values, native TIMESTAMP type, SELECT DISTINCT, IS [NOT] NULL, IN, basic CAST, LIKE NOT. We also improved SQL compatibility for table aliases and COUNT function.

Important notes

Breaking changes on the SQL layer

In order to support nullable columns, a breaking change for the on-disk format was introduced. After upgrading to immudb 1.2.1, the old data will not be available through the SQL interface even though immudb still contains the low-level KV representation of those entries. If you need to migrate the old data to the new format, please contact us on our discord channel.

Breaking changes in the SDK

One of the strengths of immudb is that immudb can prove to the client that the record was immutably stored in the database and that it was not tampered with. Verification of such proof is done on the client side.

Introduction of the entry metadata requires a slightly modified version of the proof that also needs to consider such metadata. An updated version of the SDK will be needed to correctly verify proofs for new entries.

There are also many changes in the API related to new transaction support. Because transaction requires server-side state, the client can now operate in a context of a server-side session. There are also many new commands related to transaction support.

Avoiding downtime during the upgrade

To avoid downtime during the upgrade, components should be upgraded in the following order:

• immudb clients
• immudb follower replica instances
• immudb main database

Changelog

[v1.2.1] - 2021-12-13

Bug Fixes

• protobuf: Fix compatibility with 1.1 version
• database: Internal consistency check on data reads
• database/meta: Do not crash on history with deleted items
• pkg/database: history skipping not found entries

Changes

• cmd/immuadmin/command: add super user login hint
• embedded/sql: use sql standard escaping with single quotes
• embedded/sql: support for escaped strings
• embedded/store: prevent value reading of expired entries
• embedded/store: fix expiration error declaration
• embedded/store: dedicated expiration error
• embedded/store: improve metadata serialization/deserialization methods
• embedded/store: validations during metadata deserialization
• embedded/store: return data corrupted error when deserialization cannot proceed
• embedded/store: use fixed time during the lifespan of a tx
• embedded/store: easily extendable meta attributes
• embedded/store: mandatory expiration filter
• embedded/store: reduce attribute code size
• pkg/auth: require admin permission to export and replicate txs
• pkg/integration: remove useless compilation tag on tests
• pkg/server: deprecate GetAuth and WithAuth
• pkg/server/sessions: session max inactivity time set to 3m and minor stat collecting fix
• pkg/server/sessions: tuning sessions params
• pkg/server/sessions: session timeout set to 2 min

Features

• embedded/store: logical entries expiration
• pkg/api: logical entries expiration
• pkg/client: expirable set

[v1.2.0-RC1] - 2021-12-07

Bug Fixes

• Update jaswdr/faker to v1.4.3 to fix build on 32-bit systems
• Makefile: Fix building immudb for specific os/arch
• Makefile: Use correct version of the grpc-gateway package
• embedded/sql: ensure determinism and no value overlaps distinct rows
• embedded/sql: fix inserting calculated null values
• embedded/sql: correct max key length validation based on specified col max length
• embedded/sql: fix rollback stmt
• embedded/sql: normalize parameters with lower case identifiers
• embedded/sql: param substitution in LIKE expression
• embedded/sql: Use correct statement for subquery
• embedded/sql: Do not modify value returned by colsBySelector
• embedded/sql: Fix SELECT * when joining with subquery
• embedded/sql: distinct row reader with limit argument
• embedded/store: release lock when tx has a conflict
• embedded/store: read conflict validation
• embedded/store: typo in error message
• pkg/auth: Fix password tests
• pkg/client: fix database name saving on token service
• pkg/database: sql exec on provided tx
• pkg/server: fix keep alive session interceptor
• testing: using pointers for job channels
• webconsole: Fix html of the default missing page.

Changes

• refining sdk client constructor and add readOnly tx guard
• Update build/RELEASING.md documentation.
• fix more tests
• decoupled token service
• token is handled internally by sdk. Remove useless code
• remove token service from client options and fix tests
• cmd/immuadmin/command: fix immuadmin token name on client creation
• cmd/immuclient: deleteKeys functioin and updates after metadata-related changes
• cmd/immuclient: temporary disable displaying hash in non-verified methods
• embeddded/tbtree: leverage snapshot id to identify it's the current unflushed one
• embedded/multierr: minor code simplification
• embedded/sql: set INNER as default join type
• embedded/sql: minor update after rebasing
• embedded/sql: Alter index key prefixes
• embedded/sql: wip rw transactions
• embedded/sql: remove opt_unique rule to ensure proper error message
• embedded/sql: minor code simplification
• embedded/sql: kept last snapshot open
• embedded/sql: Simplify row_reader key selection
• embedded/sql: Better error messages when (up|in)serting data
• embedded/sql: standardized datasource aliasing
• embedded/sql: wip sqlTx
• embedded/sql: de-duplicate tx attributes using tx header struct
• embedded/sql: fix nullable values handling
• embedded/sql: rollback token
• embedded/sql: set parsing verbose mode when instantiating sql engine
• embedded/sql: unsafe snapshot without flushing
• embedded/sql: reusable index entries and ignore deleted index entries
• embedded/sql: bound stmt execution to a single sqltx
• embedded/sql: delay index sync until fetching row by its pk
• embedded/sql: changes on tx closing
• embedded/sql: leverage metadata for logical deletion
• embedded/sql: use order type in scanSpecs
• embedded/sql: cancel non-closed tx
• embedded/sql: expose Cancel method
• embedded/sql: wip interactive sqltx
• embedded/sql: limit row reader
• embedded/sql: return map with last inserted pks
• embedded/sql: use int type for limit arg
• embedded/sql: defer execution of onClose callback
• embedded/sql: sql engine options and validations
• embedded/sql: standard count(*)
• embedded/sql: ddl stmts not counted in updatedRows
• embedded/sql: method to return sql catalog
• embedded/sql: non-thread safe tx
• embedded/sql: use current db from ongoing tx
• embedded/sql: postponing short-circuit evaluation for safetiness
• embedded/sql: wip sql tx preparation
• embedded/store: entryDigest calculation including key len
• embedded/store: conservative read conflict validation
• embedded/store: non-thread safe ongoing tx
• embedded/store: wip tx header versioning
• embedded/store: expose ExistKeyWithPrefix in OngoingTx
• embedded/store: set header version at commit time
• embedded/store: set tx as closed even on failed attempts
• embedded/store: strengthen tx validations
• embedded/store: GetWith method accepting filters
• embedded/store: handle watchersHub closing error
• embedded/store: remove currentShapshot method
• embedded/store: tx header version validations and increased max number of entries
• embedded/store: threadsafe tx
• embedded/store: filter out entries when filter evals to true
• embedded/store: ongoing tx api
• embedded/store: early tx conflict checking
• embedded/store: simplified ExistKeyWithPrefix in current snapshot
• embedded/store: reorder tx validations
• embedded/tbtree: remove ts from snapshot struct
• embedded/tools: upgrade sql stress tool
• embedded/tools: update stress_tool after metadata-related changes
• embedded/tools: upgrade stress tool using write-only txs
• pkg/api: consider nil case during tx header serialization
• pkg/api: changes in specs to include new metadata records
• pkg/api/schema: increase supported types when converting to sql values
• pkg/client: updates after metadata-related changes
• pkg/client: avoid useless tokenservice call and add tests
• pkg/client: check if token is present before injecting it
• pkg/client: omit deleted flag during value decoding
• pkg/client/clienttest: fix immuclient mock
• pkg/client/tokenservice: handlig error properly on token interceptor and fix leftovers
• pkg/database: updates after metadata-related changes
• pkg/database: improve readability of Database interface
• pkg/database: snapshots should be up to current committed tx
• pkg/database: revised locking so to ensure gracefully closing
• pkg/database: implement current functionality with new tx supportt
• pkg/database: return a specific error in querying
• pkg/database: enforce verifiableSQLGet param validation
• pkg/database: use new transaction support
• pkg/database: limit query len result
• pkg/errors: invalid database name error converted to immuerror
• pkg/errors: useDatabase returns notFound code when error
• pkg/integration: updates after metadata-related changes
• pkg/server: use upgraded database apis
• pkg/server: updates after metadata-related changes
• pkg/server: error when tx are not closed
• pkg/server/sessions: polish logger call
• pkg/server/sessions: add sessions counter debug messages
• pkg/stdlib: general improvements and polishments
• pkg/stdlib: remove context injection when query or exec
• pkg/stdlib: improve connection handling and allow ssl mode in connection string
• pkg/stdlib: fix unit testing
• pkg/stdlib: handling nil pointers when converting to immudb named params
• pkg/stdlib: increase pointer values handling and testing
• stress_tool_sql: add sessions and transaction mode
• stress_tool_worker_pool: add long running stress tool
• test: test backward compatibility with previous release (v1.1.0)
• tsting: add index compactor in long running stress tool

Features

• helm chart for deploying immudb on kubernetes (#997)
• embedded/appendable: method for reading short unsigned integer
• embedded/sql: wip sqltx at engine with autocommit
• embedded/sql: support for not like
• embedded/sql: increased expression power in LIKE and IN clauses
• embedded/sql: null values for secondary indexes
• embedded/sql: engine as tx executor
• embedded/sql: support value expression in like pattern
• embedded/sql: create index if not exists
• embedded/sql: Detect ambigous selectons on joins
• embedded/sql: WIP un-restricted upsert
• embedded/sql: support for IN clause
• embedded/sql: delete from statement
• embedded/sql: sql update statement
• embedded/sql: distinct row reader
• embedded/store: initial commit towards full tx support
• embedded/store: wip enhanced tx support
• embedded/store: included filters in key readers
• embedded/store: including metadata records
• embedded/store: conservative tx invalidation
• embedded/store: functional constraints
• embedded/store: logical key deletion api
• embedded/store: keyReader in tx scope
• embedded/tbtree: read as before returns history count
• embedded/tbtree: implements ExistKeyWithPrefix in snapshots
• sql: Add support for IS NULL / IS NOT NULL expressions
• sql/index-on-nulls: Update on-disk format to support nullable values
• sql/timestamp: Add timestamp support to embedded/sql
• sql/timestamp: Add timestamp to protobuf definition
• sql/timestamp: Add timestamp to stdlib
• sql/timestamp: Add CAST from varchar and integer to timestamp

Downloads

Docker image
https://hub.docker.com/r/codenotary/immudb

Immudb Binaries

Immuclient Binaries

Immuadmin Binaries