Models and data: what changed, what didn't, and what to watch

If you are about to point a model at a dataset you have not used before, you want to know which parts of that job you can hand over and which parts you still have to check yourself. Public benchmarks answer this better than intuition does — but only with the dates, the models and the exact task attached, because all three change the answer.

The same benchmark, the same questions: 14.55% then 89.95%

DABstep is 450 data-analysis questions built by the payments company Adyen from its own internal documentation and files — tasks where answering correctly means reading a manual, not just writing a query. Its questions come in an easy tier and a hard tier.

In April 2025 the best configuration its authors tested — OpenAI's o4-mini, prompted to reason, in a ten-step loop that let it alternate between thinking and running code — scored 14.55% on the hard tier. Ten months later the top entry on the validated leaderboard scored 89.95% on the same questions.

Validated matters. The board has a checked tier and an open one, and the open tier holds several undisclosed entries claiming 100%, which makes it useless as evidence. And that 89.95% is now frozen: DABstep closed validation to new submissions pending a second version, so it cannot move however much systems improve. A live benchmark and a live leaderboard are different things.

It is also not one model. The entry is NVIDIA's KGMON Data Explorer, in which Claude Opus 4.5/4.6 writes helper functions offline, Haiku 4.5 runs them, and Opus or Sonnet 4.6 reviews the output. Almost every score in this piece belongs to a scaffold — the harness of prompts, tools and retry logic wrapped around one or more models — and changing the scaffold can matter as much as changing the model. Every figure below therefore names both, with the date it was set.

One more marker, used throughout. Model generations turn over roughly twice a year, so a score is partly a statement about when it was set. CURRENT means the model is still at or near the frontier as of August 2026; PRIOR means a newer generation of the same family exists; OBSOLETE means two or more generations behind. Where a headline is held by an obsolete model, that is worth noticing on its own.

Eight benchmarks, and the job each one tests

"Can models do data analysis" is too coarse to have an answer. These eight test genuinely different work, and a system strong at one can be weak at the next.

Where they sit in the actual work

A newsroom data project runs through roughly eight stages. Walking them in order shows where the measurements are — and the answer is that they bunch heavily in the middle, where a question already exists and the data is already in hand.

1. Find — which dataset answers this, and where is it published?

The newest category and the weakest results. CoDA-Bench drops an agent into a sandbox of about 980 files and asks it to locate what it needs; the best system reaches 83.0 on discovery and 61.1 end-to-end, at roughly $0.39 a task. KramaBench runs a similar test across eight real-world pipelines and tops out at 62.81 against a human baseline of 76.75. LakeQA is the closest to a journalist's situation — a 9.5 TB lake that includes data.gov — and the best score is 32.87% exact match. Every one of these appeared in 2026. This stage was essentially unmeasured a year ago.

2. Acquire — get the bytes

Largely solved, and quickly. ELT-Bench's extract-and-load stage went from 37% to 96% on a model upgrade alone, and its authors now describe it that way. This is the stage most amenable to delegation.

3. Understand — read the codebook

The most heavily measured stage, and the one this piece is mostly about. BIRD supplies a domain note that can be withheld; DABstep mounts a manual the agent must consult; LongDA gives real federal codebooks averaging 263,500 tokens; DCA-Bench asks whether a defect can be found with no one pointing at it. Their results are the subject of the next two sections.

4. Clean — types, encodings, missing-value codes

Two findings that pull in opposite directions. PrepBench scores 54.9% on writing preparation code — rising to 85.3% when the task's ambiguity is removed, which says most of the failure is unclear requirements rather than unclear code. And a 2022 result that still stands: error detection on a hospital dataset went from 0.0 to 99.1 F1 given ten worked examples (Can Foundation Models Wrangle Your Data?). Cleaning responds enormously to a small number of demonstrations.

5. Join — link it to something else

Split sharply. Matching entities is close to saturated: one set of 755,540 record pairs is answered at 98.95 F1 by GPT-4o with no scaffold. Matching schemas is nearly as good — MaDI-Bench reports perfect schema-match F1 in three of five domains. But integrating them into one coherent table falls to 84.87, and picking the right join key from a schema alone scores 35.5 on BEAVER. Knowing two columns refer to the same thing is solved; knowing which columns should be joined is not.

6. Analyse — compute the number

Well covered and improving. Spider 2.0 measures SQL against real warehouses; TableBench and MMTU cover table reasoning at scale. It is also where the effect of the harness around the model is measured most directly — on DataAgentBench, a controlled comparison holding the model fixed found a scaffold worth seven points.

7. Validate — is the conclusion actually supported?

Almost empty, and what exists scores badly. P-Bench asks not whether code ran but whether the resulting p-value is valid — whether the test suits the data, the assumptions hold, the correction was applied. The best score on its hard split is 33.0%. StatQA reaches 64.83% on a more forgiving version. Both are recent; nothing older exists to compare against.

8. Publish — what may honestly be said

Nothing measures this. No benchmark scores whether a model respects a suppression rule, distinguishes a sentinel code from a real value, applies a survey weight correctly, or reports uncertainty a reader could act on. LongDA comes nearest and explicitly places variance estimation out of scope.

What has moved since early 2026

Read the model column and one row stands out. BIRD's headline — the best score anyone has posted with the domain notes supplied — belongs to GPT-4o, a model two generations obsolete, set in December 2025 and unbeaten by fifteen entries since. On that benchmark, newer models have not translated into better answers at all.

The broader pattern: work that amounts to writing correct code against a described system has improved sharply, sometimes on a model upgrade alone. ELT-Bench's extract-and-load stage went from a third to nearly all of the way with no change but a newer Claude, and its authors now describe that stage as largely solved. Work that requires understanding an unfamiliar system has barely moved: the transformation stage of the same benchmark, where you decide what the numbers mean, sits at 22.66%.

Which part of the job models are worst at

BEAVER is the most informative benchmark here because it scores five sub-skills separately on the same tasks. Its warehouses are private company systems, so nothing in any model's training data describes them — which makes the profile below a clean read on what a model can work out from a schema alone, with no prior familiarity to fall back on.

The order is the finding, and it runs in one direction: from what a schema tells you to what only a person can. Picking the right table is largely solvable from column names. Picking the right join key is much harder, because a schema rarely says which of several plausible columns actually relates two tables. And knowing what a value means in the business that produced it is where models do worst, at 20.4 — a third of their score on table retrieval.

That bottom row is the whole argument of the next section. Domain knowledge is not a skill a model can be better at; it is information it either has or does not.

Other benchmarks show the same shape from different angles. On government open data, the largest single error category is failing to apply a condition the question never stated — a date range, a status code, or which rows are totals rather than members — at 32.4% of one system's errors (Gemini 2.5 Flash with its Answer Agent scaffold, 2026-07). On TableBench, models score above 90 on fact-checking and numerical reasoning while sitting at 61.4 on open-ended data analysis.

What handing over documentation is worth

The practical question is not how models score, but how much better they do when you give them what you know. Seven benchmarks run that comparison directly, and the answer ranges from decisive to almost nothing — which is the first thing worth noticing. What you hand over matters far more than that you hand something over.

The row to read first is the human one. Given BIRD's evidence note, human data engineers and database students scored 92.96; without it, 72.37. They lose more than twenty points to the same missing information. Whatever else these numbers show, the knowledge is not a crutch for weak models — it is a requirement of the task.

LongDA is the most relevant result published. Its questions come from real US federal survey publications and are answered against the real microdata with the real codebooks — the situation anyone working with government statistics is actually in. The best system, GPT-5 under the authors' ReAct-based scaffold with a documentation-search tool, reaches 69.16%. Delete only that documentation-search tool, changing nothing else, and it falls to 58.30% — eleven points, from the ability to look something up. The benchmark requires choices like selecting the correct survey weight from among several, which no amount of general knowledge supplies.

DCA-Bench shows the same effect as a gradient. Asked to find known defects with no hints, its best configuration (DeepSeek-R1 as curator) found 29.86%; given the most specific hint, the same model with retrieval found 78.28%.

ELT-Bench ran the blunt version: withhold the dataset documentation entirely. Denied it, one agent completed a single task and another completed none.

BIRD supplies the counterweight. Its external-knowledge annotation was worth +20.01 points in 2023. An independent re-measurement in 2026 — by a group with no stake in the benchmark, running six systems under the same conditions — found gains of 8.35, 10.82, 10.96, 11.28, 12.84 and 20.86 points. Only the oldest pipeline reached the original figure; every modern one landed between 8 and 13.

At the other end sits ScienceAgentBench, where an expert note added 1.9 points — a reminder that supplying something guarantees nothing. The spread from +1.9 to +48.4 is not noise; it tracks how closely what was handed over matched what the task actually required.

So the premium for supplying published knowledge about a public dataset is real and appears to be shrinking. The premium for supplying documentation the model cannot otherwise reach — LongDA's eleven points, measured this year on federal codebooks — is not shrinking, because nothing has absorbed it.

What the winning systems actually do

Every score in this piece belongs to a scaffold, and the papers that ablate their own scaffolds show which parts carry the weight. The pattern that comes out is narrower than "use agents", and most of it is available to one person with a terminal.

Look at the files before planning anything. The single best-measured component in this literature is DS-STAR's file profiler — a step that reads each available file and describes what is in it before any planning begins. Removing it costs 18.3 points. Its verifier is worth 10.6 and its router 5.3; the whole scaffold adds 32.5 points over the bare model, at $0.23 a task against $0.09.

Profile the columns rather than trusting the documentation. This is the surprise. AskData, the system still holding BIRD's top score on a two-generations-obsolete model, generates its own column descriptions by sampling values — and that beats BIRD's human-written metadata, 61.2 against 59.6. Looking at the data told it more than the documentation did. Note what that does and does not say: it is an argument for profiling, not against documentation, since the same benchmark's domain notes are worth 14 points on top.

Simple beats elaborate more often than not. Mini-SWE-agent is 190 lines, has bash and nothing else — no retrieval, no plugin system — and tops CoDA-Bench at 60% of the cost of the elaborate alternative.

Cost is not proportional to score. On DABstep's own baselines, o1 scored 11.11% for $435 while DeepSeek R1 scored 11.04% for $3. On ScienceAgentBench, a self-debugging loop beat a much heavier agent framework by 10.8 points at one-seventeenth the cost.

And the things that did not work

The negative results are more useful than the positive ones, because they are the techniques a reasonable person would try first.

• Retrieval over background material. On DCA-Bench, adding retrieval over curation reference material moved the best no-hint score from 29.86 to 29.41 — slightly down.
• Self-reflection without a test to fail. Reflexion-style loops without an external signal scored below having no scaffold at all (0.60 → 0.52). A reflection pass with nothing to check against mostly produces confident revision.
• Answering five times and voting. Self-consistency bought +0.13 points for five times the tokens, at p = 0.86 — indistinguishable from noise.
• Adding more agents. Multi-agent configurations degraded monotonically as agents were added.
• Compressing the context. Zero measured benefit for 24 to 94 percent more cost.

Set the failures beside the successes and one line separates them. The retrieval that worked — LongDA's document search, worth eleven points — went after a specific fact about the artifact in front of it: which variable, which weight, what this code means in this survey. The retrieval that failed went after general knowledge about the genre of work. Reference material about data curation does not tell you what is wrong with this file. The codebook does.

The same line explains why file profiling and column sampling win: both fetch specific facts about the artifact, cheaply, before any plan is made. What consistently fails is asking the model to think harder about what it already has.

One caution about all of the above. The system holding DABstep's top score publishes a repository in which the two files containing its actual helper functions are empty. Several leaderboard entries name no model and no method. The techniques that can be described here are the ones whose authors chose to describe them.

What nobody is measuring

Stage 8 is worth restating because it is the stage where a wrong answer does the most damage. No benchmark scores whether a model respects a suppression rule, tells a numeric sentinel code from a real value, applies a survey weight correctly as a scored axis, or produces design-based standard errors. These are the errors most likely to yield a confident, publishable, wrong number — and there is no evidence in either direction about how often models make them.

The gap is not that nobody thought of it. LongDA requires choosing the correct survey weight among several and its authors put variance estimation explicitly out of scope; DataGovBench's largest error category is a filter the question never stated. The pieces are visible at the edges of benchmarks built for something else. Nothing yet scores them directly.

The large caveat: some of this is the benchmarks failing

Two independent audits this year found that a substantial share of what gets recorded as model failure is benchmark error.

A CIDR 2026 paper re-examined the ground truth of the leading text-to-SQL benchmarks and reported annotation error rates of 52.8% in BIRD's Mini-Dev split and 66.1% in Spider 2.0-Snow. On corrected data one system rose from 62% to 81% and moved from fourth place to first. Separately, ELT-Bench-Verified re-checked its own benchmark and found that 82.7% of failed tasks contained an error attributable to the benchmark rather than the agent; fixing them moved the transformation score from 22.66% to 32.51% without touching the system.

This does not invalidate the direction of any finding above, but it should make you distrust small differences and any leaderboard ranking decided by a few points.

Two more things to hold lightly

Nearly every figure here is a benchmark's authors reporting results on the benchmark they built. That is normal, and it means independent corroboration is rare — the BIRD re-measurement above is the exception, not the rule. BEAVER cannot be independently re-measured at all: its warehouses are private, which is exactly what makes it a good test and what makes it unauditable.

And benchmarks that stay in circulation are disproportionately the ones models have not solved. The ones that got solved stopped being cited. Any survey of live benchmarks over-represents unsolved problems, so the picture here is probably gloomier than the truth.

What to watch

Whether the published-knowledge premium keeps shrinking. BIRD's with-and-without gap went from +20.01 to a modern-scaffold band of 8–13 in three years. If it reaches parity, models will have absorbed what published documentation used to supply.

Whether anyone measures the unreachable kind. LongDA is the first benchmark to price documentation access on real official data. One result is not a trend, and until there are more, claims about that category — including this piece's — rest on a single measurement.

Whether the jagged profile flattens. If domain knowledge climbs from 20.4 toward the 70 that table retrieval already reaches, the argument that unfamiliar systems require handed-over context weakens sharply.

What follows for the work

On the current evidence, the parts of a pipeline that amount to writing correct code against a described system are reasonable to delegate and getting more so. The parts that depend on knowing what a column means, which rows are totals, which year broke comparability, which weight to apply, or why a value is suppressed rather than missing are where systems fail most, where supplying what you know buys the most, and where nobody has measured how often things go wrong.

The check worth keeping is therefore not "did it write good code" — increasingly it does — but "did it know what it was looking at." And the measured techniques point the same way: profile the files before planning, sample the columns rather than trusting their names, and fetch the specific fact the task turns on. What does not help is asking the model to reflect harder on what it already has.

Since that specific fact usually exists — written down by the publisher, somewhere the download does not point to — going to find it before the first query costs less than discovering its absence in a published number.