The Data Substrate

The previous page followed a single call from generate() to a recorded row. This page is about where that row lands, and why the shape of the storage — not a policy document — is what makes somm's privacy and cost promises hold.

One project-local database: .somm/calls.sqlite

Constructing a Repository creates the database with owner-only file permissions, applies migrations automatically, and configures WAL, foreign keys, and per-thread, fork-aware connections (packages/somm-core/src/somm_core/repository.py:177). The write path never requires a running service: a per-process WriterQueue batches short writes into the database, and repeated lock failures spill to permission-restricted JSONL files that are later replayed atomically (notes/PLAN.md:1179, packages/somm/src/somm/telemetry.py:270). Even provider cooldown state lives in SQLite, with separate model-level and provider-wide entries, so it survives restarts (docs/BLUEPRINT.md:187).

flowchart LR
  subgraph immutable["Append-only records"]
    calls[("calls")]
    updates[("call_updates")]
    pv[("prompt_versions")]
    rev[("workload_revisions")]
    receipts[("eval receipts")]
  end
  subgraph pointers["Mutable pointers"]
    wl[("workloads — live row")]
    labels[("prompt labels")]
  end
  subgraph optin["Opt-in bodies"]
    samples[("samples")]
    ds[("datasets")]
  end
  calls -- "late metadata" --> updates
  wl -- "every mutation appends a snapshot" --> rev
  labels -- "point at" --> pv
  calls -- "deterministic sampling" --> samples
  samples -- "promoted idempotently" --> ds
  ds -- "graded" --> receipts
  decisions[("decisions")]
  decisions -. "best-effort mirror" .-> global[("~/.somm/global.sqlite")]
    

Append-only telemetry: immutable calls, mutable pointers via call_updates

Telemetry is append-only: late metadata goes into call_updates, preserving deterministic audit history.— docs/BLUEPRINT.md:56

Everything downstream respects this contract. Evaluation and recommendation services consume calls to produce their own records without rewriting the original telemetry (docs/BLUEPRINT.md:171). Failures are recorded the same way — deliberately visible and bounded in telemetry rather than silently swallowed (README.md:162) — which is why somm calls --status error is the diagnostic route bug reports ask for (.github/ISSUE_TEMPLATE/bug_report.md:28).

Prompt versions, workload revisions, and audit history

The schema is at version 19, which evolved the original workload/prompt/call ledger into decisions, workload revisions, datasets, evaluation receipts, campaigns, and canonical model aliases (packages/somm-core/src/somm_core/version.py:7). ensure_schema() applies the packaged SQL migrations automatically (packages/somm-core/src/somm_core/schema.py:80), and each migration commits its DDL and version stamp in the same transaction, so a database can never end up partially upgraded (packages/somm-core/src/somm_core/schema.py:91).

The append-only discipline extends beyond calls, using a consistent pattern: immutable snapshots underneath, mutable pointers on top (notes/GAMEPLAN-2026-07.md:95).

• Workloads. Routing constraints live on the workload's live row for fast reads, but every mutation appends a revision snapshot. Rollback doesn't delete anything — it copies an old snapshot forward as a new revision (packages/somm-core/src/somm_core/repository.py:402, packages/somm-core/src/somm_core/repository.py:579).
• Prompts. Prompt versions are immutable; labels are the mutable pointers, resolved at call time — including deterministic weighted labels for splitting traffic between versions (packages/somm/src/somm/client.py:1846, packages/somm/src/somm/client.py:1880).
• Samples and datasets. Opt-in captured samples can be promoted idempotently into durable golden datasets, graded with deterministic structural and text comparisons, and linked to structured evaluation receipts (packages/somm-core/src/somm_core/repository.py:757, packages/somm-core/src/somm_core/graders.py:28).
• Decisions. Model-choice judgments recorded through the MCP tools are written locally and best-effort mirrored to a global cross-project repository, so future selection work can recall them (packages/somm-mcp/src/somm_mcp/server.py:360). That global database is real machine state — the test suite explicitly redirects ~/.somm/global.sqlite into temporary paths to avoid polluting it (conftest.py:18).

Privacy classes: PRIVATE workloads fail closed rather than egress

Crucially, prompt and response bodies are not stored by default at all. Bodies enter the database only through opt-in shadow sampling, which is deterministically sampled, size-capped, and forbidden for private workloads (packages/somm/src/somm/client.py:850, packages/somm-core/src/somm_core/repository.py:642). The same policy is enforced independently at several layers:

somm doctor keeps the storage layer honest over time, checking for permission drift alongside processes, cooldowns, and migrations (docs/errors/SOMM_PORT_BUSY.md:11).

Pricing without the network: the bundled cost snapshot

Synchronization is careful about ownership: a bundle refresh replaces stale seeded data but preserves manually entered prices, which stay authoritative over bundled data, and a fingerprint check skips repeat work (tests/test_pricing_bundle.py:66, tests/test_pricing_bundle.py:84). When a price is genuinely missing, the call still succeeds: cost records as zero, and paid providers emit a once-per-provider/model warning while free providers stay silent (tests/test_pricing_safety.py:87, tests/test_pricing_safety.py:101, tests/test_pricing_safety.py:115).

Plans and quotas: pacing spend across every project on the machine

Provider plans load from TOML with modes, quotas, units, windows, and soft utilization targets (tests/test_plans.py:37). Calendar-window limits flag overuse only when usage both exceeds the soft target and runs ahead of elapsed time; rolling windows use direct utilization because they lack a fixed calendar progression (tests/test_plans.py:139, tests/test_plans.py:149). The resulting status feeds routing directly:

All of this remains advisory in the fail-open sense established above: a broken or absent plan governor falls back to the original provider chain, preserving availability (tests/test_plans.py:272, tests/test_plans.py:281). The hard line stays where it was — the per-workload daily budget, checked before dispatch, which no amount of fallback can route around.