Static data API

TidyBank publishes approved catalog data as immutable static files. No API key is required.

Base URL: https://data.tidybank.net/

Registry and asset paths below are relative to that origin. Consumers should store the versioned asset ID and verify the published checksums rather than scraping the website.

Registry endpoints

All registry paths point to files on data.tidybank.net; they are never duplicated under the website origin. Registry generation is deterministic and the committed integrity lock is checked in CI. concepts.json receives the same canonical-JSON SHA-256 lock treatment as the asset, publication, and dataset registries; only generated_at is excluded from canonical hashing where that field exists.

datasets.json derives table_name by replacing every / and - in the asset ID with _. For example, abs/australian-industry/2023-24/table-7 becomes abs_australian_industry_2023_24_table_7. Generation fails rather than publishing if two asset IDs produce the same SQL identifier. version is the release slug.

Discovery endpoints

These machine-readable documents are published on the website origin and generated from the same cached catalog projection as the static pages:

DCAT datasets are ordered by immutable asset ID, distributions preserve the recorded CSV/JSON/Parquet order, and themes are ordered by committed topic slug. The executable Zod contract in src/lib/site/dcat.ts is checked against the approved asset set during every build and in CI. A backwards-incompatible DCAT shape or meaning change requires a documented profile/version decision just as registry contract changes require a schema-version bump.

Atom entry IDs combine the immutable asset URL with the approval decision and its append-only decided_at value, so an unchanged decision retains the same identifier across builds. A later re-approval produces a separate Updated dataset entry. The feed-level updated value is the newest included decision timestamp; an empty feed uses the documented fixed epoch rather than a build-time timestamp. Every HTML page links the feed with rel="alternate".

The DCAT document, sitemap family, robots file, and Atom feed are deterministic projections rather than immutable versioned asset endpoints: their bytes may change when approved catalog or approval-record content changes. They do not carry the registry integrity lock or a self-checksum. Consumers that need byte integrity should follow the SHA-256 values on DCAT distributions or in each asset’s metadata.json; CI instead regenerates these discovery documents, validates their contracts and catalog reconciliation, and verifies byte-identical output for unchanged inputs.

Per-asset endpoints

Replace <asset-id> with the four-segment ID <publisher>/<publication>/<release>/<table>.

Stability and immutability

A versioned release path such as data/abs/australian-industry/2023-24/table-7/tidy.parquet is immutable once published. Corrections that change data or its recipe receive a new versioned release or an explicitly recorded replacement; consumers must not expect bytes at a versioned URL to change silently.

Website routes of the form /data/<publisher>/<publication>/latest/<table> are mutable static HTML aliases. Each alias contains a canonical link, an immediate meta refresh, and a visible link to the latest approved versioned asset page; it does not promise an HTTP 3xx response or an API redirect. The target may move when a release is approved. Registries and reproducible analyses should use the canonical versioned asset ID, never a /latest/ alias.

Registry documents and per-asset manifest envelopes (metadata.json, schema.json, preview.json, semantics.json, report.json, and source.url.json) carry schema_version; backwards-incompatible field or meaning changes require a schema-version bump and a documented migration. recipe.json follows RecipeV01’s own version field. tidy.json is row data described by the asset’s schema.json, not a versioned manifest envelope. Additive fields follow the applicable contract’s version policy, and strict consumers should validate against the linked schema they support.

Checksums

metadata.json records sha256 and byte length for each generated distribution. source.url.json records the original workbook SHA-256. A checksum is written as sha256:<64 lowercase hexadecimal characters>. Verify downloaded bytes before processing or caching them. A source checksum mismatch means the agency workbook changed after ingest; it is an availability signal, not permission to accept unreviewed bytes.

License and attribution

The seeded ABS assets are published under Creative Commons Attribution 4.0 (CC-BY-4.0). Reusers must credit the Australian Bureau of Statistics, identify TidyBank’s transformation, link the licence, and indicate changes. License terms are per asset: always read metadata.json fields source.license.spdx_or_label and source.license.url before reuse. A future asset may carry different obligations.

Runnable example

This approved seeded asset is used by the site and test corpus:

abs/australian-industry/2023-24/table-7

Fetch and inspect its metadata and CSV:

BASE='https://data.tidybank.net/data/abs/australian-industry/2023-24/table-7'
curl --fail --location "$BASE/metadata.json" | python -m json.tool
curl --fail --location --output table-7.csv "$BASE/tidy.csv"

Read the same CSV with pandas:

python - <<'PY'
import pandas as pd

url = "https://data.tidybank.net/data/abs/australian-industry/2023-24/table-7/tidy.csv"
frame = pd.read_csv(url)
print(frame.head())
print(frame.shape)
PY

Query the Parquet distribution directly with DuckDB:

duckdb -c "INSTALL httpfs; LOAD httpfs; SELECT * FROM read_parquet('https://data.tidybank.net/data/abs/australian-industry/2023-24/table-7/tidy.parquet') LIMIT 5;"

For production use, fetch metadata.json, verify the selected distribution’s byte length and SHA-256, and retain the immutable asset ID plus the applicable contract version (schema_version, or RecipeV01 version) with derived results.