The problem
The documentation burden was never the schema
Anyone who has worked with government data knows where the time goes. Not the column list — the issues. The misspelled header. The suppression sentence sitting in a rate field. The format that silently changed in 2019. The category whose meaning narrowed in 2006. The boundary that moved. The two published calculations with one name, only one of which has a trend.
Working through those is the craft. Writing them down is what almost nobody does, because there has been nowhere to put them. They end up in a paragraph in a README, in a code comment three functions deep, in a Slack thread, or in the head of the one person who has been on the beat longest. So the next project pays for the same discoveries again — and an agent handed the same files with no documentation surfaces about 30% of the known issues on its own. The rest have to be handed over.
ergo is a place to put them. It is not a catalog, a warehouse, or a validation service; it sits beside the data as one markdown file per dataset, and it is designed so that handing the list over — to a colleague, to a stranger, or to a model — is a file read rather than a conversation.
What is actually enforced
Four rules, and a validator that will not let you skip them
Every fragment below is real output, produced by running the validator when this page was built.
At rest
It's a markdown file. That's the whole trick.
A data page is one file per dataset: a document a human reads top to bottom as a
reporter's notebook, carrying fenced toml ergo blocks that a program
parses without guesswork. The blocks hold the structured facts — id, effect, type,
status, scope, the code that handles it. The prose between them holds the story, the
examples, and the numbers. Never both; a fact stated in a block is not restated in
the prose, because duplication drifts.
Parsing needs Python 3.11 and its standard library, because tomllib is
in it. There is no service to run, no database to migrate, and no dependency to
audit: a single
file you copy into your repository is the validator, the digest generator, the
exporter and the publisher. If the tool is abandoned tomorrow, what you are left with
is markdown that still reads correctly.
The distinction that does the work
An issue is a defect. A practice is a decision.
An issue is something true about the data whether or not you exist: the zeros that mean "no value", the ward numbers that were reassigned, the rate column that carries prose. A practice is a call somebody made about what may be computed from it: report the median rather than the mean, exclude the migration date, don't turn report counts into occurrence rates.
Two tests, and they disagree usefully. Deletion: would this still be true
if we deleted all our code? Yes → issue. Disagreement: could a competent
team look at the same data and rightly decide otherwise? Yes → practice, and it says
so with contested. They are separate blocks because the cardinality
differs: one defect routinely has several handlings, chosen by the question being
asked, and no single issue entry can carry two opposite ones.
Choosing where caveats live
What you're actually choosing between
These differ in kind, not in quality. Most projects need several of them, and none of the others is trying to be a registry of defects.
| Dimension | ergo data page | Catalog metadata (DCAT, schema.org) | Dataset cards (Datasheets, Croissant) | Validation tooling (Frictionless, Great Expectations) |
|---|---|---|---|---|
| What it is for | Documenting the traps in a dataset for whoever touches it next | Making a dataset discoverable and describable | Describing provenance, collection, and intended use | Asserting that today's file matches expectations |
| Where a caveat goes | A typed entry with an id, an effect, and a scope | A description string | A limitations section, in prose | A failing check, if somebody wrote one |
| Machine-readable scope | Which years, tables, columns, entities — as fields | Temporal and spatial coverage of the whole dataset | No | Per check, and only for what is executable |
| Linked to the workaround | Both ways, and the link is checked | No | No | The check is the code; the reason usually isn't recorded |
| Holds what isn't executable | Yes — definitional and institutional context is a first-class effect | Prose only | Prose only | No |
| Relationship to ergo | — | Generated from the page, never canonical. The manifest maps onto all three; the issue registry maps onto almost nothing, which is why the format exists. | ||
No composite score, no feature counts, no ranking. A dataset that is thoroughly described and completely undocumented is exactly the situation ergo was written for, and a scoring table would hide it.
Before you adopt it
The honest case against
The strongest objection is that a registry rots. Somebody writes twelve issues in a burst of enthusiasm, the dataset changes, nobody updates the page, and two years later a reader trusts a scope that is no longer true — which is worse than no documentation, because it was believed.
That objection lands, and the honest answer is partial. The round trip catches the
half that involves code: delete a workaround, or rename an issue without touching the
anchor, and check fails in CI. Validation records make an untested
caveat visibly untested, and unknowns makes silence say something.
What none of that does is notice that the publisher changed something nobody looked
at. This format does not detect an issue for you. It is a place to put one, and a
mechanism that makes some kinds of rot loud.
Thin adoption evidence
The format is proven against two projects: the NJ DOE datasets behind njschooldata, and a second adopting project's multi-publisher source-contract corpus. That is enough to know it survives contact with real government data. It is not enough to know it works across a hundred datasets, or for a team that did not write it.
No effectiveness measurement
Nothing here shows that a project with data pages makes fewer mistakes than one without. The benchmark figure above measures how much agents miss unaided; it is the reason handing issues over matters, not evidence that this is how you should hand them over.
The spec is draft, and it has moved
Version 0.4, and the changelog records
breaking renames — bite became pitfall one release ago.
Pages declare the format version they conform to and the validator still accepts the
older ones, but the format is not frozen and should not be treated as if it were.
It documents your relationship with a dataset, not the dataset
Two newsrooms will write different pages about the same census product, disagree
about what is core, and both be right, because their questions differ.
Directories cluster pages by subject and deliberately refuse to merge or rank them.
If what you want is one authoritative description of a dataset, this is not that, and
does not believe that exists.
Where this stands