Storage Adapters¶

Every adapter implements the same storage.StorageAdapter interface, so swapping one for another is a one-line change to the factory call. You get CRUD, pagination, Lucene-backed search, count, and raw query escape hatches — plus schema and migration helpers.

Picking an adapter¶

Adapter type	Constant	Backing store	Use it when
In-memory	`storage.MEMORY`	An in-process SQLite database (no file on disk)	Tests, demos, prototypes. All data is lost on process exit.
SQL	`storage.SQL`	Postgres / MySQL / SQLite (via GORM)	Anything production-ish that wants relational queries and JSON columns.
DynamoDB	`storage.DYNAMODB`	Amazon DynamoDB	AWS-native services that prefer single-table design.
CosmosDB	`storage.COSMOSDB`	Azure CosmosDB	Azure-native services.

Cassandra was supported historically but is currently disabled in the factory (see storage/storage.go); the source file is preserved as storage/cassandra.go.backup.

Building an adapter¶

main.go

import "github.com/tink3rlabs/magic/storage"

adapter, err := storage.StorageAdapterFactory{}.GetInstance(
    storage.SQL,
    map[string]string{
        "provider": "postgresql",
        "host":     "localhost",
        "port":     "5432",
        "user":     "blox",
        "password": "secret",
        "dbname":   "blox",
        "schema":   "public",
    },
)
if err != nil {
    log.Fatal(err)
}

The returned adapter is always wrapped with a telemetry-instrumented adapter. If you need the concrete *SQLAdapter — for example, to register a custom GORM plugin — call storage.UnwrapAdapter(adapter). See UnwrapAdapter on pkg.go.dev.

Same code, different backend

The whole point: swap storage.MEMORY for storage.SQL (and pass a config map) and your handler code keeps working. Lucene filters, cursor pagination, typed errors — all unchanged.

Configuration by adapter¶

In-memory¶

adapter, _ := storage.StorageAdapterFactory{}.GetInstance(storage.MEMORY, nil)

No config keys. Use only for tests and prototypes. Internally it's an in-memory SQLite database — so SQLite quirks apply, e.g. boolean Lucene filters must be written done:1, not done:true (see Search).

SQL (Postgres / MySQL / SQLite)¶

// Postgres
config := map[string]string{
    "provider": "postgresql",
    "host":     "localhost",
    "port":     "5432",
    "user":     "blox",
    "password": "secret",
    "dbname":   "blox",
    "schema":   "public",
}

// MySQL
config := map[string]string{
    "provider": "mysql",
    "host":     "localhost",
    "port":     "3306",
    "user":     "blox",
    "password": "secret",
    "dbname":   "blox",
}

// SQLite
config := map[string]string{
    "provider": "sqlite",
    "path":     "/path/to/database.db",
}

adapter, _ := storage.StorageAdapterFactory{}.GetInstance(storage.SQL, config)

The SQL adapter uses GORM internally. Connection pooling, migrations, and schema creation are handled for you via CreateSchema(), CreateMigrationTable(), etc. The schema key is Postgres-specific and becomes a TablePrefix on the GORM config.

Values are escaped when the connection string is built, so passwords may contain spaces, quotes, @, / or ?. For MySQL, driver options can follow the database name: "dbname": "app?parseTime=true".

Quote secrets in YAML

If the config map comes from a YAML file, quote the values. Unquoted, 0123 is read as the number 83, ~ as null, and p@ss #x as p@ss. Single quotes keep everything literal: password: 'p@ss #x'.

DynamoDB¶

config := map[string]string{
    "provider":   "dynamodb",
    "region":     "eu-west-1",
    "endpoint":   "http://localhost:8000",   // optional, for DynamoDB Local or LocalStack
    "access_key": "...",                     // optional — falls back to the default AWS credential chain
    "secret_key": "...",                     // optional — same as above
}
adapter, _ := storage.StorageAdapterFactory{}.GetInstance(storage.DYNAMODB, config)

When access_key / secret_key are empty, the adapter uses the standard AWS credential provider chain (env vars, IRSA, instance role).

DynamoDB takes these per-call params in the variadic params ...map[string]any argument:

Param key	Constant	Applies to	Meaning
`sort_direction`	`storage.SortDirectionKey`	`List`, `Search`	`"ASC"` (default) or `"DESC"`. Same key as the other adapters. PartiQL only honours `ORDER BY` on a Query (the `WHERE` equals the partition key) against the table or index sort key.
`index`	`storage.IndexKey`	`List`, `Search`	Read from this global or local secondary index (`FROM "table"."index"`) instead of the base table. Must match `[A-Za-z0-9_.-]{3,255}`. Other adapters ignore it.
`parameters`	`storage.QueryParametersKey`	`Query`	Values for the statement's `?` placeholders, as a slice in placeholder order.

index.go

var rules []Rule
next, err := adapter.List(&rules, "id",
    map[string]any{"principal": principal},
    50, cursor,
    map[string]any{storage.IndexKey: "principal_index"},
)

List filter keys are written into the statement, so each dot-separated part must match [A-Za-z0-9_-]+; anything else returns an error before a request is sent. Each part is quoted, so keys that are reserved words (status, name) or contain hyphens work, and address.city is a nested path ("address"."city"). The sortKey argument is quoted the same way (ORDER BY "name" ASC). A nil value renders IS NULL, a slice renders IN (...), and an empty slice returns an empty page without querying.

Invalid filter keys, IndexKey, and SortDirectionKey return a plain error, not *errors.BadRequest: those values come from application code. Search query text is user input, so every parse error from DynamoDB Search is a BadRequest (HTTP 400).

CosmosDB¶

// Endpoint + key
config := map[string]string{
    "provider": "cosmosdb",
    "endpoint": "https://your-account.documents.azure.com:443/",
    "key":      "...",
    "database": "blox",
}

// Or use a single connection string
config := map[string]string{
    "provider":          "cosmosdb",
    "connection_string": "AccountEndpoint=https://...;AccountKey=...;",
    "database":          "blox",
}

// Or Microsoft Entra ID (managed identity, workload identity, Azure CLI, ...):
// an endpoint with no key uses azidentity.NewDefaultAzureCredential
config := map[string]string{
    "provider": "cosmosdb",
    "endpoint": "https://your-account.documents.azure.com:443/",
    "database": "blox",
}

// Local emulator — disable TLS verification
config := map[string]string{
    "provider":        "cosmosdb",
    "endpoint":        "https://localhost:8081/",
    "key":             "...",
    "database":        "blox",
    "skip_tls_verify": "true",   // local testing only
}

// Regional private endpoint: route requests to the regions you can reach
config := map[string]string{
    "provider":          "cosmosdb",
    "endpoint":          "https://your-account-westus.documents.azure.com:443/",
    "key":               "...",
    "database":          "blox",
    "preferred_regions": "West US",   // ordered, comma-separated
}

adapter, _ := storage.StorageAdapterFactory{}.GetInstance(storage.COSMOSDB, config)

Preferred regions¶

The Cosmos DB SDK reads the account's regions from the configured endpoint and, unless told otherwise, sends requests to the account's primary region, not the region of the endpoint you configured. A workload that can only reach one region (for example through a regional private endpoint) then fails, including the readiness Ping.

Set preferred_regions to an ordered, comma-separated list of the regions the workload can reach, such as "West US" or "West US, Central US". The adapter doesn't infer a region from the endpoint hostname.

Region names are matched ignoring case and spaces, so West US and westus are the same. A name that isn't one of the account's regions is silently ignored, and if none match, requests go to the primary region as if the key weren't set. Check the names against the account's regions in the Azure portal; the adapter logs the configured list at startup.
It applies whichever way the adapter authenticates: endpoint and key, connection string, or Microsoft Entra ID.
On an account with a single write region, writes always go to that region; preferred regions only route reads. Workloads that must write through a regional endpoint need an account that writes in that region.

CosmosDB partition key is per-call, not global

Azure CosmosDB requires you to pass the partition key on every operation, not just at adapter construction. The adapter exposes this via the variadic params ...map[string]any argument. If you forget, queries either fail or cross-partition-fan-out (slow and expensive). See the table and example below.

CosmosDB takes per-call params for the partition key. Pass them in the variadic params ...map[string]any argument on Create / Get / Update / Delete / List / Search:

Param key	Meaning
`pk_field`	Field name to use as the partition key in your document (default `"pk"`).
`pk_value`	Value for that partition key.
`sort_direction`	`"ASC"` (default) or `"DESC"` for `List` / `Search`. Same key as `storage.SortDirectionKey`.

storage.go

params := map[string]any{
    "pk_field": "tenant",
    "pk_value": "acme-corp",
}
err := adapter.Create(user, params)
err  = adapter.Get(&user, map[string]any{"id": "user-123"}, params)

Read-your-writes¶

The adapter keeps the session token Cosmos DB returns from each Create / Update / Delete, per container and partition key, and sends it on later reads, so on an account with session consistency those reads see that write. A token is also taken from a failed write's response, so after a 409 Conflict a Get sees the item that already exists.

Reads with pk_field / pk_value send the token for that partition only.
Reads without a partition key, and every Query, run across partitions and send one token for each partition range of the container.
Tokens for the same partition range are merged per region, as the Cosmos DB SDKs do, so on an account that writes in more than one region a read still sees writes accepted in any of them.
Tokens are held in memory and are not shared between processes.

Config key	Default	Meaning
`session_tokens`	`true`	`"false"` turns read-your-writes tracking off.
`session_token_ttl`	`5m`	How long a token is kept after the last write to its partition (a Go duration).
`session_token_max`	`10000`	Most container/partition pairs held; the least recently written is dropped first.

A value that can't be parsed or isn't positive logs a warning and keeps the default.

Session not available

A read that carries a session token can reach a region that hasn't received the write yet. Cosmos DB then answers 404 with substatus 1002 ("read session not available") instead of returning older data, and the SDK retries: on a multi-write account in your preferred regions, on a single-write account once against the write region. If every attempt fails, the read returns an error (not ErrNotFound). Reads and writes that go to the same region, as they do on a multi-write account with preferred_regions set, rarely hit this. To go back to reads without session tokens, set "session_tokens": "false".

Known limitation: partition splits

After Cosmos DB splits a partition range, a response for a write sent to the old range that arrives after one from the new range replaces the new range's token. The next read may then not see the newer write. Fixing this needs the partition-range layout, which the adapter doesn't track.

Common patterns¶

Cursor pagination¶

List and Search return a cursor string. Pass "" on the first call; pass whatever the previous call returned for each subsequent page. An empty cursor on the response means there are no more pages.

paginate.go

var page []Task
cursor := ""
for {
    var err error
    cursor, err = adapter.List(&page, "created_at", nil, 100, cursor)
    if err != nil {
        return err
    }
    process(page)
    if cursor == "" {
        break
    }
}

Sort direction¶

sort.go

import "github.com/tink3rlabs/magic/storage"

_, err = adapter.List(
    &page,
    "created_at",
    nil,
    100,
    "",
    map[string]any{storage.SortDirectionKey: "DESC"},
)

The value is case-insensitive ("asc" and "ASC" are equivalent). Anything other than asc/desc returns an error.

Filter vs search¶

List(dest, sortKey, filter, limit, cursor, params...) — filter is a map[string]any of exact equalities. Multiple keys are ANDed. A slice value matches any of its elements on DynamoDB and CosmosDB.
Search(dest, sortKey, query, limit, cursor, params...) — query is a Lucene query string. See Search (Lucene).

For an HTTP GET /tasks?filter=... endpoint, pass the raw filter query string straight to Search. magic handles validation, error reporting, and parameterization. On DynamoDB, Search reports those errors as *errors.BadRequest; invalid List filter keys do not, because the filter map is built by the service, not by the user.

Count¶

n, err := adapter.Count(&Task{}, map[string]any{"status": "in_progress"})

An empty filter counts every item. On CosmosDB, Count runs a server-side SELECT VALUE COUNT(1) and requires pk_field and pk_value in params: the Go SDK cannot run cross-partition aggregates, so a count is always scoped to one partition and returns an error without them. On CosmosDB, Get, List and Count check filter keys before querying: each must be an identifier ([a-zA-Z_][a-zA-Z0-9_]*) or a dotted path of them (address.city) for a nested property. Keys are written as quoted accessors (c["order"]), so property names that are Cosmos keywords, such as value, order or top, work. pk_field must be a plain identifier, because Create stores the partition value under that top-level key. A nil filter value matches documents where the property is null or missing, like SQL's IS NULL. An empty slice matches nothing, so the count is 0.

Update¶

On the SQL and Memory adapters, Update writes only the row identified by the item's primary key, and only if that row also matches filter:

err := adapter.Update(&Task{Id: id, TenantID: tenant, Title: "new"}, map[string]any{"tenant_id": tenant})
if errors.Is(err, storage.ErrNotFound) {
    // no task with that id in this tenant; nothing was written
}

Update never creates a row (use Create) and returns an error when the item has no primary key value.

Not-found¶

err := adapter.Get(&task, map[string]any{"id": "missing"})
if errors.Is(err, storage.ErrNotFound) {
    // 404
}

Migrations¶

Each adapter implements CreateMigrationTable, GetLatestMigration, and UpdateMigrationTable. magic ships an embed.FS (storage.ConfigFs) that adapters use to load SQL migration files; populate it from your own embed'd migrations directory at startup.

A typical bootstrap:

if err := adapter.CreateSchema(); err != nil {
    log.Fatal(err)
}
if err := adapter.CreateMigrationTable(); err != nil {
    log.Fatal(err)
}
latest, err := adapter.GetLatestMigration()
// run any newer migrations in order...

The SQL adapter wraps each migration in a transaction. DynamoDB and CosmosDB do not have schema migrations in the relational sense; the helpers are no-ops on those adapters.

Escape hatches¶

When you need a raw query that doesn't fit the interface, use:

adapter.Execute(statement) — fire-and-forget DDL/DML.
adapter.Query(dest, statement, limit, cursor, params...) — not implemented on the SQL adapter (returns a "not implemented yet" error today).

These bypass the Lucene layer entirely. You are responsible for parameter binding. Prefer List / Search whenever possible: Query statements are written in the adapter's own query language, so code that uses them doesn't port to other adapters.

On DynamoDB, pass values for the statement's ? placeholders under storage.QueryParametersKey, in order. Never write values into statement:

query.go

next, err := adapter.Query(&rules,
    `SELECT * FROM "rules" WHERE "tenant" = ? AND ("principal" = ? OR "status" = ?)`,
    50, cursor,
    map[string]any{storage.QueryParametersKey: []any{tenant, principal, "escalated"}},
)

Adapter-specific limitations¶

Memory — an in-memory SQLite database: data lost on restart, single process only, and SQLite's limitations (below) apply.
SQLite — no pg_trgm, so fuzzy Lucene search is unsupported (returns an explanatory error). Use wildcards instead.
MySQL — fuzzy search uses SOUNDEX, which ignores the distance hint (~2) and works only on ASCII pronunciations.
DynamoDB — the Lucene compiler targets PartiQL; fuzzy search is not supported. Equality, comparisons, ranges, value* and *value* wildcards (rendered as begins_with/contains), and boolean composition work. Ends-with (*value) and ? wildcards return an error. See Search (Lucene).
CosmosDB — see storage/cosmosdb.go for the supported subset and partition-key handling.