evaluation

Availability is not improvement.

A projection can exist without loading, load under the wrong model or policy, and still fail to improve behavior. Spindle reports those gates separately, then evaluates paired or named-arm interventions on an exact runtime coordinate.

It is tempting to treat "the skill is installed" as "the skill works." It isn't. A skill can be present, coherent, and safely rendered and still make no difference — or make things worse. Spindle draws a hard line between the two questions and refuses to let a bind stand in for an effectiveness claim.

startup + activation

Runtime correctness

Were exact bytes available?
Did this agent load the intended realization?
Did routing and authorization match?
Did the adapter prove its runtime facts?

spindle eval

Improvement

Does the skill improve behavior?
On this task family, model, and harness?
Versus running without it?
Measured on held-out cases.

Availability is not activation. Activation is not authorization. None of them is improvement.The boundaries the evaluation contracts enforce.

The paired experiment

Every selected case runs twice: once as the baseline (skill off) and once as the variant (skill on). Comparing the same case with and without the skill is the only way to attribute a difference to the skill rather than to the case. Case order and the within-pair arm order are both shuffled from the manifest's seed — randomized to avoid ordering bias, yet perfectly reproducible.

arm One side of the pair: baseline or variant. The runner must actually invoke the skill on the variant and not on the baseline — Spindle checks.

An evaluation is a manifest. It names the skill, the runner, the frozen dimensions (model, harness), and a set of cases split into development and held_out:

# eval.toml (excerpt)
schema_version = 1
id = "diagnosing-bugs-v1"
skill = "diagnosing-bugs"
skill_path = "candidate/SKILL.md"
runner = ["python", "runner.py"]     # an argv contract, not a model client
seed = 20260711
min_held_out_cases = 10
min_improvement = 0.05

[dimensions]
harness = "isolated-executor"
model = "frozen-model-coordinate"

[[cases]]
id = "unseen-regression"
split = "held_out"
fixture = "fixtures/unseen-regression.json"

The held-out promotion gate

Development cases exist so you can iterate on fixtures and graders; they never make a run promotion-eligible. Promotion is decided only on held-out cases the skill author was not tuning against. A run may promote only if all four hold:

≥ Nheld-out cases met

0runner errors

100%complete pairs

Δ > 0variant over baseline

When min_improvement is zero, the variant mean must be strictly above baseline; otherwise it must clear the configured margin. The held-out mean is a necessary gate, not the whole decision — Spindle's own docs are explicit that you should still review case-level regressions, cost, latency, and human-correction time before you adopt. Rejected and null receipts are kept as evidence for that exact skill hash.

Named arms and evidence-driven minimalism

The paired baseline/variant format remains useful and supported. The v0.2 matrix contract adds the comparisons needed when frontier models may no longer need yesterday's instruction scaffolding:

No skill

Rebaseline the model and harness default instead of assuming the old intervention still adds value.

Invariant core

Measure the portable intent and safety residue without any tuned overlay.

Candidate overlay

Test the smallest proposed steering delta for this exact agent coordinate.

Incumbent + ablations

Compare current behavior and remove one bounded instruction at a time.

Every result reports six independent gates:

Gate	What it establishes
availability	The exact artifact was present.
activation	The intended agent actually loaded it.
routing	The correct realization reached the correct agent.
authorization	The effective tool and policy envelope matched the grant.
behavior	The task-specific grader passed with retained evidence.
adapter	The named harness build could prove the required runtime facts.

A smaller candidate qualifies only when repeated held-out pairs clear a predeclared one-sided non-inferiority bound, every run stays above a hard floor, required runtime gates pass, and the artifact reduction is material. A zero-byte overlay is valid—it is never assumed successful. Changing model, harness build, toolset, policy, package digest, active blend, or task distribution requires rebaselining.

bash

$ spindle eval matrix validate matrix.toml
$ spindle eval matrix run matrix.toml
$ spindle eval matrix freshness receipt.json \
    --coordinate model=gpt-5.6-sol --coordinate harness=codex \
    --coordinate harness_build=0.146.0 --coordinate toolset_digest=sha256:… \
    --coordinate policy_digest=sha256:… \
    --coordinate active_blend_digest=sha256:…

$ spindle eval distill classify ./candidate/review
$ spindle eval distill plan ./candidate/review --json
$ spindle eval distill stage ./candidate/review \
    --proposal sha256:… --destination ./trials/review-smaller --dry-run

Distillation classifies behavioral steering, reference knowledge, deterministic procedures, tool integrations, fixtures, and obsolete workarounds. Deterministic candidates move toward tools or Chip; rich reference material remains progressively disclosed. Staging creates a local trial revision and never auto-adopts it.

The runner is an argv contract

Spindle never calls a model provider. A manifest names an argv runner, and Spindle invokes it once per case per arm — with no shell — passing everything through environment variables. The runner may wrap an isolated executor, a benchmark harness, or a deterministic local fixture; it writes one JSON result and exits.

Variable	Meaning
`SPINDLE_EVAL_ARM`	`baseline` or `variant`
`SPINDLE_EVAL_SKILL_ENABLED`	`0` or `1` — must match the arm
`SPINDLE_EVAL_SKILL_FILE`	candidate skill path (variant); empty for baseline
`SPINDLE_EVAL_FIXTURE`	absolute path to the case fixture
`SPINDLE_EVAL_SPLIT`	`development` or `held_out`
`SPINDLE_EVAL_DIMENSIONS`	JSON of the frozen `[dimensions]`
`SPINDLE_EVAL_RESULT_PATH`	where the runner must write its JSON result

The result is small and validated. score is bounded to [0,1]; skill_invoked must match the arm; evidence must be non-empty. A timeout, a nonzero exit, a malformed result, missing evidence, or an arm mismatch is an evaluation error — and an error blocks promotion.

freeze the coordinates The runner should hold model, tools, and budgets constant across arms and declare them in [dimensions] and its own evidence — so a measured delta is attributable to the skill alone.

// the JSON the runner writes to $SPINDLE_EVAL_RESULT_PATH
{
  "score": 0.82,
  "passed": true,
  "skill_invoked": true,
  "evidence": {"grader": "exact-regression-check", "receipt": "…"},
  "metrics": {"false_positives": 0, "corrections": 1},
  "artifacts": [".../agent-output.txt"]
}

Receipts — durable, not intrusive

Each run writes a receipt that keeps input hashes, pair order, exit state, duration, stdout/stderr hashes, scores, metrics, and the promotion decision — but not full transcripts. It's enough to audit and reproduce a decision for a specific skill hash without hoarding potentially sensitive agent output.

bash

$ spindle eval validate examples/evaluation-sample/eval.toml
$ spindle eval run      examples/evaluation-sample/eval.toml
$ spindle eval show     examples/evaluation-sample/receipts/<receipt>.json

The task families it starts with

Diagnosis

Exact symptom reproduction, loop determinism, pre-edit hypothesis testing, regression sensitivity — plus completion, time, and cost.

Code review

True findings, severity, false positives, spec-vs-standards attribution, missed seeded issues, correction time — with clean controls.

Handoff

Repeated exploration, missing decisions, invalid assumptions, time to first correct action, and completion in a fresh context.

One gate, two jobs Spindle's optimizer and distillation workflow consume these evidence gates. Optimizing prose, deleting redundant steering, and evaluating behavior remain separate from adoption—and a proposed edit that drops a guardrail is rejected outright.