Reproducibility is the credibility currency

Benchmark methodology

Every score is tied to a frozen suite, deterministic local scoring, a canonical result document, and a command another maintainer can run. Model outputs are not deterministic, and the public-suite contamination limit is stated rather than hidden.

Scoring contract

Metric definitions

Metrics are macro-averaged so every suite member has equal weight and are rounded to six decimal places. Failed and skipped members remain in the denominator and receive zero for every component.

Deterministic pass rate: A recipe passes deterministically when it validates and executes. Both an exact passed result and an executed-but-different partial result count; every failure and skip stays in the denominator.
Exact output match: Exact output match compares the canonical tidy-CSV SHA-256 digests. A member is binary: it scores 1 only when the digests are identical, and 0 otherwise.
Cell match rate: Columns are aligned by their resolved output names and rows by source order. A cell participates in the union when its row and named column exist in either table. Duplicate rows remain distinct because row positions are not deduplicated or fuzzily matched.
Value-address F1: Set-based F1 is computed over sheet!R1C1 value-cell source addresses. Duplicate selections do not create extra true positives. Two empty traces score 1.
Header accuracy: Value cells are aligned by source address in source order, then the full resolved header assignment is compared: output name, direction, selected header source, scalar value, and missing/ambiguous state. Header declaration order is not significant. Extra produced value cells are already penalized by address F1; an empty reference scores 1 only when production is also empty.

Non-passing results may carry no more than 20 mismatch rows. That sample explains a score; it is not a substitute for the full canonical table used by the metrics.

Provisional ranking

Composite formula

Gold results currently use composite_version: "provisional-1". The provisional formula is:

0.30 × deterministic pass rate
+ 0.30 × exact output match
+ 0.15 × cell match rate
+ 0.15 × value-address F1
+ 0.10 × header accuracy

The formula is explicitly provisional pending coordination with tidycell benchmarking. Result documents retain every component and the formula version so a later version cannot silently rewrite a committed score. API cost is record-only and never enters the score.

Workspace context

Model-picker benchmark feed

npm run bench:feed projects the latest validated result per model and suite into the curated OpenRouter registry. A picker rank compares only curated models that have a latest result in that same suite; it does not rank unlisted models or combine different suite versions. Composite ties are ordered deterministically by model ID.

The feed records the suite ID, canonical result-document SHA-256 digest, run finish time, component metrics, complete reported cost per solved sheet when available, and bounded per-tag pass rates. Models without committed results have no benchmark block, so an empty production results directory adds no placeholder context to the picker.

Frozen evidence

Suite composition

`gold-2026.07`

Frozen 2026-07-19T00:00:00.000Z; 30 members; 14 marked as published in the Data Bank. The immutable suite digest is sha256:4b134d7688bd9d115c6a0bb49b6ee63e83d770801998afad5b695015b5e90e63.

Members by source kind
Source kind	Members
`bank_asset`	12
`conformance`	18

Members by difficulty
Difficulty	Members
`easy` One compact table with direct header relationships and little structural ambiguity.	7
`medium` Several dimensions, fills, sparse regions, or repeated blocks require non-trivial recipe structure.	16
`hard` Multiple tables, many dimensions, large outputs, or interacting structural features make the case failure-prone.	7

Members by publisher
Publisher	Members
`abs`	25
`aihw`	2
`synthetic`	3

Members by structure feature tag
Feature tag	Definition	Members
`filled_spans`	A header value is propagated across blank cells by an explicit or directional fill rule.	8
`footnote_contamination`	Footnotes or explanatory cells occur inside or adjacent to the observation region and must not become observations.	3
`hierarchical_headers`	Observations are described by two or more nested header levels on at least one axis.	25
`merged_cells`	Merged workbook cells carry structural labels that apply to more than one observation.	8
`metadata_sheet`	The source workbook includes a separate metadata or explanatory worksheet relevant to interpretation.	4
`multi_table`	The source worksheet contains more than one logical observation table, whether or not the pinned recipe selects all of them.	5
`single_table`	The evaluated recipe emits one logical observation table.	27
`sparse_data`	The candidate observation rectangle contains intentional blanks that must be filtered or retained consistently.	13
`stacked_blocks`	Repeated table blocks are arranged vertically and share a related schema.	15
`time_series`	A calendar or reporting-period dimension repeats observations across time.	21
`transposed_headers`	A structural heading axis is oriented opposite to the conventional row-label or column-label layout.	1
`unit_rows`	Units are encoded in dedicated header rows or cells and must be attached to observations.	4

Known limitation

Contamination caveat

Bank members are published in the Data Bank and models may have trained on them; there is no held-out slice in v1.

Every suite member records published_in_bank so this exposure remains inspectable. The board therefore reports reproducible performance on a public frozen suite, not an uncontaminated estimate of generalisation.

Audit trail

Run provenance and revision policy

Every displayed result records:

suite ID and immutable suite digest;
model ID and the temperature, maximum tokens, and response format actually sent;
prompt mode, prompt version, and pipeline version;
UTC start and finish timestamps;
global call cap, attempts-per-member cap, and pacing;
runner name/version, environment, transport, invocation, and deterministic components; and
a declaration that model output is not deterministic.

Each member receives one initial strict-JSON generation attempt and at most two structured revision attempts. Validation, execution, and revision use the same shared generation loop as the workspace. Every transport call, including revisions, counts against the run cap; transport failures are recorded honestly and are not retried by a separate rate-limit policy.

Maintainer-local command

Reproduce a run

Check out the commit containing the suite, stage the private bank workbooks named by their pinned ingest records, provide the OpenRouter key only in the process environment, and run:

npm run bench:run -- --suite gold-2026.07 --model <provider/model>

The command refuses CI, verifies every available workbook and reference digest before the first model call, writes resumable state only under benchmarks/.runs/, and never commits. A maintainer reviews the canonical result before deliberately copying one small document into benchmarks/results/ and running npm run bench:validate.

Trace these claims to the committed contracts and commands: benchmarks/suites/, benchmarks/vocabulary/structure-features.json, src/lib/benchmarks/metrics.ts, src/lib/benchmarks/result-schema.ts, scripts/benchmarks/run.ts, and docs/benchmarks.md.