chip · move the code-vs-model boundary on evidence

The ladder

Six rungs. The early ones evict judgment from the model; the late ones evict governance and teaching as they stop paying for themselves. The goal is not the code end. It is the right rung, and movement runs both ways. Choose a rung to see what it evicts and what stays.

Prose skill

A document a model reads before it acts. Everything is in the model: recognition, judgment, process, and the safety gates. Frictionless to write; nothing can falsify it.

Thin skill

Typed outputs, and only recognition and interpretation left in prose: knowing when the capability is relevant, conducting the ambiguous dialogue, explaining a receipt. No cursors, no retry loops, no hidden shell state.

Hybrid chip

A deterministic envelope around one bounded judgment stage: a request schema in, a result schema out, graded by a held-out suite. This is the rung the socket measures its way toward or away from.

Deterministic chip

Every judgment is a rule a fixture can decide. The governance envelope is still here, because the behavior still has a standing relationship, state, or authority to govern.

Thin skill plus CLI

Judgment and envelope both gone. A versioned deterministic tool, and a skill that only teaches discovery and invocation. A chip retires here: its fixtures become the tool's ordinary test suite.

Plain code

No skill at all. Invoked by system wiring, not by an agent reading prose. Most enforcement-shaped behavior wants to be here, and arriving is success, not a failed decomposition.

Per behavior, one question decides the rung: can a fixture falsify it? Yes means code or evaluation material. A bounded, schema-decidable judgment means one gateway stage. Needs situated dialogue means it stays a skill, honestly. The method is skill-to-chip.md; the rubric that turns evidence into a verdict is chip.placement, below.

A socket, in one call

A socket is one typed contract with several live implementations seated behind it: deterministic code, a model at some tier, a model at a higher tier, a raw-model baseline. A strategy picks among them per call. Every call writes one receipt saying which member answered, why the earlier ones did not, and what any shadow would have said. Step through one call.

The claim arrives. The contract says: input str, output Verdict, and the reason must be short.
The code member runs first. It cannot decide this one, so it abstains with its score. An abstention is an answer, not a failure.
The abstention is the trigger. The model member asks the gateway, parses the result, and the value passes the invariants.
This call is in the 3% sample, and the daily budget has room, so the gold member runs too. It disagrees. Nothing it said reaches the caller.
The answering member is run again on the same input. It agrees with itself. That is the floor any disagreement has to clear.
One receipt lists every attempt, including the one that was replaced. The harvest line weights this input by its deferral and the disagreement.

{
  "schema": "chip.socket-receipt/v0alpha1",
  "socket": "claims.admissible",
  "strategy": "first-success",
  "attempts": [
    { "member": "code:claim-rules@sha256:…",
      "trigger": "primary", "outcome": "abstained",
      "reason": "no-entity-no-number", "score": 0.41,
      "latencyMs": 2, "costUsd": 0 },
    { "member": "model:claims-mid@sha256:…",
      "trigger": "abstained", "outcome": "value",
      "value": { "verdict": "verified",
                 "reason": "…" },
      "tuple": "ollama/qwen2.5:7b@claims-mid",
      "latencyMs": 812, "costUsd": 0.00012 }
  ],
  "answeredBy": "model:claims-mid@sha256:…",
  "shadows": [
    { "member": "model:claims-gold@sha256:…",
      "trigger": "sampled", "outcome": "value",
      "agrees": false, "costUsd": 0.0031 }
  ],
  "noise": { "member": "model:claims-mid@sha256:…",
             "outcome": "value", "agrees": true },
  "terminalReason": "answered-by-model",
  "cost": { "usd": 0.00334 },
  "harvest": "harvest:claims.admissible#srun_…"
}
// harvest weight 4 = 1 + 1 deferral
//                  + 2 disagreement

Shape per socket.md; the schema is v0alpha1 and will change.

Member: One implementation of the contract: code, model, or baseline (the raw model, kept as a standing comparator). A member returns a value, an abstention with a reason and an optional score, or a typed failure: schema or transport. Anything else it raises is recorded as error and the chain moves on.
Strategy: first-success runs the chain in order and escalates on abstention, invariant, schema, transport, or error. shadow members run on a sample under a daily budget and never answer. A canary is placed first on a fraction of calls, a policy an operator sets and never a default. The noise flag re-runs the answering member.
Trigger: Why a member ran, from a closed list: primary, abstained, invariant, schema, transport, error, budget, sampled. Two of them are about the boundary (abstained, invariant); the rest are operations. Keeping them apart is what lets a report tell a placement finding from a gateway problem.
Clean: A hook that declares what a receipt may hold. One production socket reduces a pair of headlines to two URL keys, two domains, and a score before anything is written. GitHub Scientist's clean is the precedent.

What the receipts tell you

chipsocket report receipts.jsonl folds the receipts into four numbers a socket can state about itself. It gathers nothing; it is a fold over records.

Measure	Answers
Closed by code	What share of inputs the code member answered without abstaining. The honest cost of the model is one minus this.
Escalations, by cause	The trigger histogram split into boundary (`abstained`, `invariant`) and operations (`schema`, `transport`, `error`, `budget`), with the dominant side named. Operations dominant is a gateway problem, not a placement finding.
Agreement, net of noise	Per shadow: how often it agreed with the answering member, minus that member's own self-disagreement on the same sample. Broken down by which kind of member answered, because agreeing with code on the inputs it closed is a different fact from agreeing with the model on the ones it did not.
The band	With labels: where the scoring member abstained, what was the truth, by score bucket. The lowest bucket that clears the precision you want is where code could stop abstaining.

Why the noise floor is not optional

A model member is nondeterministic at any temperature that matters, so a shadow mismatch with no same-member repeat is evidence of sampling, not of disagreement. Twitter's Diffy multicast every request to a candidate, a primary, and a second copy of the primary, and reported only differences larger than primary-versus-copy. The noise flag is that instrument, and it is why a 3% gold shadow cannot produce a confident wrong promotion.

Plug it in

You have a function that decides something, and somewhere in it a model gets asked. Declare the contract on the function. It keeps answering exactly as it did; what changes is that every call is receipted, and other members can be seated behind it.

from chipsocket import Abstain, contract, model

@contract(input=str, output=Verdict, invariants=[reason_is_short],
          sink="receipts.jsonl", harvest="harvest.jsonl")
def admissible(claim: str) -> Verdict | Abstain:
    ...    # your existing code, unchanged; return Abstain("band", score=s)
           # where it used to call the model

admissible.member(model(gateway, prompt=render, result=Verdict, profile="claims-mid"))
admissible.shadow(model(gateway, prompt=render, result=Verdict, profile="claims-gold"),
                  sample=0.03, budget_usd_daily=2.0, noise=True)

verdict = admissible(claim)    # code first; model on abstain; gold shadows 3%

The decorated function is the first code member. Its answers do not change.
The score beside an abstention is whatever ordering the code already had: a similarity, a margin, a relevance. It is what lets the band be learned instead of guessed.
gateway is any gateway(request) -> result. With somm it is SommGateway(project, workload), so cost and provenance land in the ledger you already keep. With TypeSafe it asks Jev typed questions and returns probabilities a parser can abstain on.
The shadow is sampled, budgeted in dollars per day, and never reaches the caller.

Phase 0: already in production? Change nothing.

Do not re-run the incumbent. Seat the verdict it already wrote as the primary member, at zero extra cost, and let candidates shadow it. The receipts start accumulating before anything is promoted.

from chipsocket import Socket, code, model, recorded

s = Socket("lane.verdict", output=Verdict, sink="receipts.jsonl", clean=redact)
s.member(recorded(lambda row: row.incumbent, id="incumbent", digest=TREATMENT_SHA, kind="model"))
s.shadow(code(rules), sample=1.0)                 # free, every row
s.shadow(model(gold_gateway, prompt=render, result=Verdict, profile="gold"),
         sample=0.03, budget_usd_daily=0.5, noise=True)

Then learn the band, and receipt what was learned

Once labels exist for some receipts, cuts finds the widest band at each end of one member's score where a constant answer meets a precision bar, and cut_member turns that into a code member whose digest includes the evidence it was learned from. A member re-learned from a different day is a different member on the receipts.

A same-event judge on a local 7B model, about 18,000 receipts a day. On one hand-labelled day, below 0.15 the constant answer was right 96.8% of the time against the model's 90.6%; at or above 0.75 it was right 99.2% against 98.8%. Code closes about half the population more accurately than the model it replaces, and the model is asked only where it earns the call. One day is a hypothesis: the member is seated as a shadow until a second labelled day's cuts agree.

$ chipsocket cuts receipts/ --truth labels.json --key input.id --scorer prefilter --min-n 200
{ "lo": 0.15, "hi": 0.75, "closes": 0.49, "evidence": "sha256:…", "below": {…}, "above": {…} }

from chipsocket import cut_member, cuts, load_receipts
c = cuts(load_receipts("receipts/"), truth, key=lambda r: r["input"]["id"],
         scorer="prefilter", precision=0.95, min_n=200)
s.shadow(cut_member(lambda p: p.lexical, c, negative=False, positive=True, id="lexical-cuts"))

Promotion is a rung transition and is gated the way the ladder already says: non-inferior on a held-out suite drawn from the harvest. Demotion is a circuit breaker: a promoted code member that fails an invariant on a live call goes back to shadow on that call. A member moves down on one piece of evidence and up on a suite. The runner does neither for you; it records.

When it becomes a chip

A socket has no standing observation relationship, no cross-run state, and no authority. It is called; it answers; it records. By the admission rule that makes it a tool, and that is correct. The socket is the primitive below chip.

A behavior is admissible as a chip only if both hold: it declares an ongoing observation relationship to a signal class, or owns durable cross-run state; and it produces a bounded response under an authority contract. Most things should not be chips, and the skill that decides says so first.

A chip is a socket seated in an envelope.

Promise: One bounded operational job, stated in a sentence that needs no "and".
Ports: The inputs it reads and outputs it writes, each with a versioned schema. Chips compose into linear circuits by matching ports.
Stages: Code, at most one gateway judgment, and policy. The stages are implementation; the promise is the contract.
State: Single-flight for anything with a cursor; partitioned by a declared key for map and cache shapes.
Effects and authority: Effect classes observe, synthesize, experiment, draft, promote. Effective authority is chip ∩ circuit ∩ binding ∩ host ∩ approval, and it fails closed.
Receipts: One for every run, including the quiet ones where it observed and did nothing. Taint is transitive. Targets own final deduplication.

Hosts bind and run circuits. The reference host is fab; the second host, and the proof that the contract is portable, is the socket runner itself. The spec is chip-spec.md, normative only.

The rubric

chip.placement takes an evidence record about one observed behavior and returns a verdict, a confidence, and the measures it relied on. It is pure and gathers nothing; a scanner is not portable, the rubric is. Six rules survived a hand-run over 1.1 million production calls across 113 workloads, and only those are encoded.

Admission first, scoring second. Standing versus burst is decided before anything is scored, and rules out roughly a third of a real fleet without judgment.
Time series, not aggregates. Evidence is built from daily observations; there is no constructor for a pre-summarized row. A 51-day aggregate hid a 10-day incident.
Never rank by cost. The largest observed loss was 495,130 calls that never reached a provider, invisible to any cost-ranked detector, and it cost nothing.
Repetition routes the ladder. Novel input every time is extraction and distils toward a thin skill plus a CLI; recurring input over a standing relationship is what a chip rung is for.
The failure motif separates the envelope from the judgment. Terminal envelope failure means an observer or a parser, not a judgment chip.
Undecidable is a verdict. When evidence cannot decide, the rubric says so and names what is missing.

Verdicts are a closed vocabulary: observer, mint, parser, no-standing-relationship, no-judgment-stage, undecidable. The negatives are the point. On a real fleet most subjects are not chip-shaped, and a rubric that only emits positives is an advocacy tool.

For coding agents

The lifecycle ships as eight skills in the chip-contract plugin. The same bytes ship inside the chipspec package. No service, no credentials; chip and chipsocket are commands over local packages and receipts.

# Claude Code
/plugin marketplace add lyra-forge/marketplace
/plugin install chip-contract@lyra-forge

# Codex
codex plugin marketplace add lyra-forge/marketplace
codex plugin add chip-contract@lyra-forge

chip-candidate: Is this a chip, a tool, a dialogic skill, or doctrine? Invoke before authoring anything.
chip-socket: Seat a function's code-or-model decision in a socket; phase 0; learn the cuts.
chip-author: One promise, typed ports, declared state and authority.
chip-fixtures: The suite that can fail, and later gates authority.
chip-evaluate: Run the held-out suite and record the evaluated tuple.
chip-install: Install so it actually resolves where activations run.
chip-audit: The silent failures: pinned at observe, never activated, quiet forever.
chip-outcome: Close the loop from a judgment to what happened next.

Where this sits

AgentRun is a workflow graph above the call, with Jev as a typed judge node. chip refuses graph semantics; the socket is the primitive below one call. A Jev judge is a model member here, and the two compose.
LiteLLM and Ramp ship shadow routers that run a candidate on sampled traffic and judge blind. Neither measures the noise floor.
RouteLLM and cascade routers choose a model by predicted difficulty. The socket's first member is code that abstains, and the band is read off labels.
Diffy and GitHub Scientist are the precedents for the noise floor and the clean hook.

The full survey, with the research on abstention, learning to defer, and judge reliability, is related-work.md.

Where it stands

A v0 experiment. chip 0.4.0 tracks spec 0.6.0. One reference host and one runner. Two production sockets feed receipts: the same-event judge above, and a phase-0 pilot on claim admissibility that shadows a recorded incumbent. Neither package is on PyPI yet.

The conditions under which this program should be narrowed or stopped are public: if the pilots do not beat skill-only baselines, if no second host can implement the contract without the first host's internals, if a frontier model with raw typed tools matches a chip on held-out work. A spec that cannot say how it would be proven wrong is advertising.

The code-vs-model boundary is a position that evidence should move.