evaluation

Availability is not improvement.

A bind proves a skill loaded without losing guardrails or budget. It says nothing about whether the skill made the agent better. Spindle answers that separately — with a paired, held-out, reproducible experiment.

It is tempting to treat "the skill is installed" as "the skill works." It isn't. A skill can be present, coherent, and safely rendered and still make no difference — or make things worse. Spindle draws a hard line between the two questions and refuses to let a bind stand in for an effectiveness claim.

spindle bind

Availability

Can this surface load the skill?
Without losing guardrails or budget?
Coherent with the rest of the blend?
Deterministic. No model calls.

spindle eval

Improvement

Does the skill improve behavior?
On this task family, model, and harness?
Versus running without it?
Measured on held-out cases.

Rendering and binding prove a skill is available, not that it improves an agent.The boundary the evaluation harness enforces.

The paired experiment

Every selected case runs twice: once as the baseline (skill off) and once as the variant (skill on). Comparing the same case with and without the skill is the only way to attribute a difference to the skill rather than to the case. Case order and the within-pair arm order are both shuffled from the manifest's seed — randomized to avoid ordering bias, yet perfectly reproducible.

arm One side of the pair: baseline or variant. The runner must actually invoke the skill on the variant and not on the baseline — Spindle checks.

An evaluation is a manifest. It names the skill, the runner, the frozen dimensions (model, harness), and a set of cases split into development and held_out:

# eval.toml (excerpt)
schema_version = 1
id = "diagnosing-bugs-v1"
skill = "diagnosing-bugs"
skill_path = "candidate/SKILL.md"
runner = ["python", "runner.py"]     # an argv contract, not a model client
seed = 20260711
min_held_out_cases = 10
min_improvement = 0.05

[dimensions]
harness = "isolated-executor"
model = "frozen-model-coordinate"

[[cases]]
id = "unseen-regression"
split = "held_out"
fixture = "fixtures/unseen-regression.json"

The held-out promotion gate

Development cases exist so you can iterate on fixtures and graders; they never make a run promotion-eligible. Promotion is decided only on held-out cases the skill author was not tuning against. A run may promote only if all four hold:

≥ Nheld-out cases met

0runner errors

100%complete pairs

Δ > 0variant over baseline

When min_improvement is zero, the variant mean must be strictly above baseline; otherwise it must clear the configured margin. The held-out mean is a necessary gate, not the whole decision — Spindle's own docs are explicit that you should still review case-level regressions, cost, latency, and human-correction time before you adopt. Rejected and null receipts are kept as evidence for that exact skill hash.

The runner is an argv contract

Spindle never calls a model provider. A manifest names an argv runner, and Spindle invokes it once per case per arm — with no shell — passing everything through environment variables. The runner may wrap an isolated executor, a benchmark harness, or a deterministic local fixture; it writes one JSON result and exits.

Variable	Meaning
`SPINDLE_EVAL_ARM`	`baseline` or `variant`
`SPINDLE_EVAL_SKILL_ENABLED`	`0` or `1` — must match the arm
`SPINDLE_EVAL_SKILL_FILE`	candidate skill path (variant); empty for baseline
`SPINDLE_EVAL_FIXTURE`	absolute path to the case fixture
`SPINDLE_EVAL_SPLIT`	`development` or `held_out`
`SPINDLE_EVAL_DIMENSIONS`	JSON of the frozen `[dimensions]`
`SPINDLE_EVAL_RESULT_PATH`	where the runner must write its JSON result

The result is small and validated. score is bounded to [0,1]; skill_invoked must match the arm; evidence must be non-empty. A timeout, a nonzero exit, a malformed result, missing evidence, or an arm mismatch is an evaluation error — and an error blocks promotion.

freeze the coordinates The runner should hold model, tools, and budgets constant across arms and declare them in [dimensions] and its own evidence — so a measured delta is attributable to the skill alone.

// the JSON the runner writes to $SPINDLE_EVAL_RESULT_PATH
{
  "score": 0.82,
  "passed": true,
  "skill_invoked": true,
  "evidence": {"grader": "exact-regression-check", "receipt": "…"},
  "metrics": {"false_positives": 0, "corrections": 1},
  "artifacts": [".../agent-output.txt"]
}

Receipts — durable, not intrusive

Each run writes a receipt that keeps input hashes, pair order, exit state, duration, stdout/stderr hashes, scores, metrics, and the promotion decision — but not full transcripts. It's enough to audit and reproduce a decision for a specific skill hash without hoarding potentially sensitive agent output.

bash

$ spindle eval validate examples/evaluation-sample/eval.toml
$ spindle eval run      examples/evaluation-sample/eval.toml
$ spindle eval show     examples/evaluation-sample/receipts/<receipt>.json

The task families it starts with

Diagnosis

Exact symptom reproduction, loop determinism, pre-edit hypothesis testing, regression sensitivity — plus completion, time, and cost.

Code review

True findings, severity, false positives, spec-vs-standards attribution, missed seeded issues, correction time — with clean controls.

Handoff

Repeated exploration, missing decisions, invalid assumptions, time to first correct action, and completion in a fresh context.

One gate, two jobs Spindle's skill optimizer (spindle optimize) consumes these same held-out scores. Optimizing a skill's prose and evaluating its behavior therefore run through one gate — and a proposed edit that drops a guardrail is rejected outright, exactly as a bind would reject it.