Self-hosted LLM
telemetry, routing,
intelligence loop.

Wrap your LLM calls once. Get provider fallback, streaming, cost tracking, shadow-eval, and agent-driven recommendations — all running locally on your own SQLite. Zero phone-home. Zero cloud account. Zero commercial dependencies on the hot path. 

$ pip install somm
$ python -c "
import somm
llm = somm.llm(project='my_app')
r = llm.generate('Reply: pong', workload='ping')
print(r.text, r.provider, r.cost_usd)
"
→ pong ollama 0.0
Get started Read the code

Why somm

Every LLM-using Python project grows along the same axes. You end up with tangled provider SDKs, inline retry loops, drifting prompts, and no idea which model wrote what. somm is the shared substrate you wish you'd started with.

Router across ten providers

ollama, OpenRouter, DeepSeek, Minimax, Anthropic, Gemini, OpenAI, Perplexity, plus claude/codex CLI executors and any OpenAI-compatible gateway (Groq, Together, Fireworks, vLLM, LM Studio, custom). Preference-ordered fallback overridable via SOMM_PROVIDER_ORDER, per-model cooldowns persisted to SQLite, circuit breaker. Free-tier rate limits stop being a problem.

Builds its own eval dataset

Opt a workload in to shadow-eval and a background worker samples N% of your calls, re-runs them through a gold model, grades with structural + text-similarity scorers. No hand- written evals. Budget-capped per workload. Skipped entirely on private workloads.

Agent that points at the wins

The agent worker analyzes calls + eval results + model intel and emits concrete recommendations with evidence. Not "here are some models" — specifically switch claim_extract to gemma3:27b: +45% quality, -80% latency, same cost, 47 shadow calls.

MCP for your coding agent

Ten stdio tools. Your Claude Code / Cursor / Windsurf session can query actual telemetry: "what did claim_extract cost this week?" → real answer. "Compare gemma3 and llama3 on this prompt" → real side-by-side. "Replay call X against a cheaper model" → real delta.

One contract for coding-agent harnesses

Execute one Claude Code, Codex, or OpenCode attempt through somm.harnesses. Native event streams, session resume, final output, usage, and termination reasons become one portable result contract, while your task runner keeps its queue and policy. Read the boundary and API.

Sommelier: cross-project model memory

Ask in one project: good free vision models on openrouter? Get a ranked list with capability + price reasoning. Pick one. Next week in a different project, ask the same thing — somm remembers what you picked and why, across every project on your machine. As of 0.2.2 prior decisions also weigh into the ranking: models you've used before get a small bonus; models a prior flagged as unreliable take a soft penalty, both decaying with age. Output-modality filters, meta-router exclusion, and HuggingFace pipeline_tag enrichment keep the top of the list honest.

Multimodal-native routing

Pass text + image content blocks to any provider that accepts them. The router auto-filters to vision-capable models before the network call. No surprise 400s on text-only endpoints, no bespoke branching in your code.

Loud failures, quiet success

Every non-OK outcome writes a bounded error body (status + server message + exception class) to calls.error_detail AND fires an inline on_error callback — default: a one-line stderr warning so crashes never hide in the telemetry table. Pass your own callable to forward to logging / Slack / PagerDuty, or lambda _: None to silence.

Install

# library only — fine for drop-in replacement:
$ pip install somm

# + web admin, scheduler, background workers, MCP:
$ pip install somm somm-service somm-mcp

Python 3.12+. Requires no API keys out of the box — just point somm at a local ollama and go. Every commercial provider is opt-in via its own env var (ANTHROPIC_API_KEY, OPENAI_API_KEY, OPENROUTER_API_KEY, GEMINI_API_KEY, MINIMAX_API_KEY, DEEPSEEK_API_KEY, PERPLEXITY_API_KEY).

Spin up the dashboard

$ somm serve --project my_app
→ listening on http://localhost:7878
   scheduler started (poll=30.0s)
   model_intel: refreshing (345 + 6 + 6 models cached)

Compare models on a real prompt

$ somm compare "Summarize this article in one sentence" \
    --models ollama/gemma4:e4b,openrouter/llama-3.3-70b:free,openai/gpt-4o-mini
provider    model                      lat_ms  tok_i  tok_o  cost     response
ollama      gemma4:e4b                  2140     124      18  $0.0000  A research team ...
openrouter  meta-llama/llama-3.3-70b:free 3820     124      22  $0.0000  Scientists at ...
openai      gpt-4o-mini                  920     124      20  $0.0000  Researchers find ...

Grafting into an existing project

If you already have an LLM wrapper class in your codebase, swap it with somm.compat.GenericLLMCompat. One line changes; call sites don't. 

# before:
from myproject.llm import FooLLM

# after:
from somm.compat import GenericLLMCompat as FooLLM

llm = FooLLM(project="my_app")
result = llm.generate(prompt, system=sys, max_tokens=256)
print(result.text, result.provider, result.model)  # legacy fields still work
print(result.call_id, result.cost_usd)             # somm extras

If your project calls the raw OpenAI SDK directly, there's an openai_chat_completions shim that mirrors openai.OpenAI().chat.completions.create(). See the examples directory for runnable patterns:

  • drop_in_wrapper.py — single-import swap for existing classes
  • openai_swap_in.py — OpenAI SDK replacement
  • private_workload.py — privacy_class=PRIVATE defense-in-depth

Privacy posture

somm is built for workloads you'd be uncomfortable egressing. Every defense is enforced at more than one layer — not one thing to remember, many things to forget.

  • No body capture by default. Prompt and response hashes go to SQLite; full text only lands in a separate samples table if you opt in per workload.
  • Per-workload privacy class. PRIVATE workloads never egress to any non-local provider. Enforced in the router, in the shadow-eval worker, and in the shadow_candidates SQL view.
  • File perms. SQLite files created 0600; parent dir 0700. somm doctor warns on drift.
  • Localhost-only web. The admin binds 127.0.0.1 by default. Exposing via --bind 0.0.0.0 prints a loud warning.
  • Zero beacon. The somm project learns nothing about your install or usage. There is no upstream service.

Architecture

   library (sensor) ──► local store ◄── service (brain)
       ▲                    ▲                  │
       │                    │                  ├─► shadow-eval worker
       │                    │                  ├─► model-intel worker
       │                    │                  ├─► agent worker
       │                    │                  └─► web admin
       │                    │
       └── skill (onboarding) ─── MCP (interface for coding agents)

Six packages:

somm-coreschema v19, migrations, repository, config, parse helpers
sommlibrary — SommLLM, providers, routing, coding-agent harnesses, streaming, compat shims, CLI
somm-servicestarlette web admin + HTTP API + scheduler + three workers
somm-mcpstdio MCP server with 14 tools
somm-langchainSommChatModel adapter for LangChain/LangGraph/deepagents
somm-skillonboarding templates for Claude / Codex / Cursor / Windsurf

BLUEPRINT.md is the design doc — the load-bearing forces and the data model behind somm.

Status

v0.14.0 beta / pre-1.0 — see the CHANGELOG for the release log; the test suite runs in CI on every push. somm is published for real-world dogfooding, but its public API, MCP tools, service contracts, and migration semantics are not yet declared stable. Recent releases added multimodal routing (0.1.1), the sommelier (0.2.0), Gemini + error visibility (0.2.1), sommelier quality (0.2.2), tool calling + the LangChain adapter, eval gates, the recommendation loop, OTLP ingest, authenticated service APIs, serving-performance rollups, benchmark/cache-advice CLIs, Anthropic/OpenAI-compatible proxy routes, and reusable Claude/Codex/OpenCode harness execution. MIT licensed. Upcoming work is tracked in ROADMAP.md; if you want to build your own take, the BLUEPRINT distills the six design forces.

Star on GitHub →