The Intelligence Loop

The previous pages showed somm as a sensor: every call lands as an immutable row in a project-local SQLite database. This page is about what happens to those rows. When the service tier runs, scheduled workers grade sampled calls, refresh model metadata, and produce recommendations — while MCP tools and the web dashboard read the same local state (notes/PLAN.md:69). Nothing in this loop requires a hosted control plane; it is the same calls.sqlite file, read and enriched in place.

flowchart TB
  app["Application calls via somm.llm()"] --> db[("calls.sqlite — calls, samples, provider health")]
  proxy["Anthropic-compatible proxy /v1/messages"] --> db
  otlp["OTLP span ingest"] --> db
  subgraph sched["Scheduler — jobs and leases stored in SQLite"]
    intel["ModelIntelWorker — daily"]
    shadow["ShadowEvalWorker — every 15 min"]
    agentw["AgentWorker — weekly"]
  end
  ext["Bundled pricing · OpenRouter · Ollama · Hugging Face"] --> intel
  db -- "captured samples" --> shadow
  gold["Gold model"] --> shadow
  intel --> mi[("model intelligence")]
  shadow --> ev[("evaluations + receipts")]
  db --> agentw
  mi --> agentw
  ev --> agentw
  agentw --> rec[("recommendations")]
  rec --> operator["Operator reviews: apply or dismiss"]
  operator -. "explicit apply only" .-> app
    

somm serve: dashboard, proxy, OTLP ingest, and the scheduler

Starting the service with somm serve --project ... brings up the local dashboard, scheduler, and workers (docs/index.html:174). Under the hood, create_app() opens the configured repository, creates or loads a service token, registers dashboard, recommendation, OTLP-ingest, and proxy routes, then applies local-security middleware (packages/somm-service/src/somm_service/app.py:1285). run_server() then runs Uvicorn plus a daemon scheduler (packages/somm-service/src/somm_service/app.py:1328). The somm-serve entry point can also run the intel, shadow, and agent stages as one-off admin commands (packages/somm-service/src/somm_service/cli.py:161).

The service adds two extra ways for telemetry to enter the ledger. The Anthropic-compatible /v1/messages proxy validates and translates each request, resolves a workload, enforces its budget before dispatch, calls LiteLLM in a thread pool, and records success or failure in the same call ledger (packages/somm-service/src/somm_service/proxy.py:269) — a rejected call creates neither spend nor telemetry (packages/somm-service/src/somm_service/proxy.py:322). OTLP JSON spans are bounded, normalized into Call records, and written without partially ingesting an over-limit batch (packages/somm-service/src/somm_service/app.py:1204). Access stays local: a generated file token plus a tightly constrained same-origin localhost path designed to resist DNS rebinding (packages/somm-service/src/somm_service/app.py:91, packages/somm-service/src/somm_service/app.py:265), with request bodies bounded both by Content-Length and incremental streaming checks (packages/somm-service/src/somm_service/http_limits.py:15).

Model intelligence: bundled pricing plus OpenRouter, Ollama, and Hugging Face signals

This intelligence is advisory, and it fails open: a missing price becomes zero cost rather than a blocked call (packages/somm-core/src/somm_core/pricing.py:240), with a once-per-provider/model warning for paid providers so the gap stays visible (tests/test_pricing_safety.py:87). Two other stances shape this layer. Signals remain distinct rather than being collapsed into an opaque composite quality score, and paid data sources stay off the default path (ROADMAP.md:237). And the docs are candid about maturity: several proposed external quality and hardware signals remain unwired, so docs/intel-sources.md:128 labels itself a scratchpad, not a committed design.

Shadow evaluation: sampled, opt-in grading with receipts

Grading itself is deliberately dependency-free: grade_response_pair() runs deterministic structural and text comparisons (packages/somm-core/src/somm_core/graders.py:28), and each result is linked to a structured evaluation receipt so a recommendation can later show exactly which graded calls back it (packages/somm-core/src/somm_core/repository.py:757).

From evidence to recommendations: the agent worker

Once a week, the AgentWorker reads everything the other stages produced — calls, evaluations, provider health, and model metadata — and converts it into deduplicated recommendations, or into parameter overrides only where a workload has explicitly opted in (packages/somm-service/src/somm_service/workers/agent.py:70). Self-healing stays recommendation-only unless the workload sets auto_heal: true (packages/somm-service/src/somm_service/workers/agent.py:84). Downstream consumers see the same evidence chain: the MCP tool somm_recommend combines open recommendations with model rankings from shadow evaluations, and when no evaluation data exists it falls back to model-intelligence candidates and prior recorded decisions (packages/somm-mcp/src/somm_mcp/server.py:117).

Recommendations expose supporting evidence and require user application rather than automatic rollout. — design decision, docs/BLUEPRINT.md:255

Propose-only by design: humans apply, somm suggests

This is the loop's defining constraint. Optimization and recommendations are propose-only by default; automatic promotion is deliberately excluded from the design (notes/GAMEPLAN-2026-07.md:131). Applying a learned recommendation is a separate, explicit operation in the library (packages/somm/src/somm/recommendations.py:133), and the operator surfaces mirror that: the dashboard's recommendation routes (packages/somm-service/src/somm_service/app.py:1285) and the MCP pair somm_apply_recommendation / somm_dismiss_recommendation (packages/somm-mcp/src/somm_mcp/server.py:197) let a human — or an agent acting with the human's tooling — accept or reject each item individually.

Durable datasets, campaigns, and prompt optimization

The loop's newest layer makes evidence reusable. Opt-in captured samples can be promoted idempotently into durable golden datasets, graded with the deterministic comparators, and linked to structured evaluation receipts (packages/somm-core/src/somm_core/repository.py:757). The operator drives this through somm eval promote-call, somm eval run, somm optimize, and somm campaign run (notes/GAMEPLAN-2026-07.md:227), or through MCP's somm_eval_promote_call (packages/somm-mcp/src/somm_mcp/server.py:509).

Schema v19 is where this shows up structurally: it evolves the original workload/prompt/call ledger into decisions, workload revisions, datasets, evaluation receipts, campaigns, and canonical model aliases (packages/somm-core/src/somm_core/version.py:7). Prompt optimization follows the same propose-only stance as everything else: a failing graded call can produce a versioned prompt proposal (packages/somm/src/somm/optimize.py:39), but optimization only ever creates a proposal label — it never promotes directly to staging or production (packages/somm/src/somm/optimize.py:49). The loop closes the way it opened: somm accumulates evidence, drafts the change, and waits for a person to say yes.