Datasets¶
A dataset is one table that others read, keyed by its name under datasets. Its entry says
what one row is (grain), what each column means (columns), which vendor releases it reflects (dates_asof), what
to do and not do with it, and the dated facts a reader needs. Every key is optional.
Start with five keys: purpose, grain, columns, dates_asof and facts. The rest is for when you need it.
Example¶
datasets:
Rents:
purpose: Quarterly asking rent per market and sector.
grain: [market_id, sector, period_yq]
profile:
period: period_yq
entities: [market_id]
coverage_by: [sector]
dates_asof:
vendor_a:
constant: "2026-09-30"
cadence: quarterly
columns:
market_id:
description: market code (text)
sector:
description: property sector
period_yq:
description: calendar quarter
rent:
description: asking rent
currency: USD
per: year
as_of_sources: [vendor_a]
do_not: [join on market names]
not_exported:
- what: lease-level rows
where: Leases
why: licence review pending
removed_columns:
- name: rent_old
removed: 2026-10-29
use_instead: rent
license: CC-BY-4.0
aliases: [Rent]
Fields explained¶
Grouped by job.
- Identity.
purposeis the one line an agent reads first;descriptionis longer text.grainlists the columns that identify one row; it is the most useful key in the file, because it tells an agent what a sum means and which joins are safe.aliaseslists former names of the dataset that still work. - Columns.
columnssays what each column is and its unit. See Columns. - Profile. The
profileblock (period,entities,coverage_by,series) shapes the coverage profile shown to agents. See Profile fields below. - Releases.
dates_asofsays where each vendor source's as-of date comes from: a fixed date, a column, a setting of the pipeline that builds the dataset, or an upstream dataset. See As-of dates. - Guidance.
use_casesare recipes (Use cases).do_notlists hazards: rules that are always true for this data, such as "do not join on market names"; a plain string applies to the whole dataset, thetext+columnsform names the columns it is about.factsare dated statements (Facts). Use a fact when something was learned or decided at a point in time and might change; usedo_notfor a standing rule. - What is missing.
removed_columnslists columns that used to exist and what replaces them.not_exportedsays what is deliberately not shared, and where to get it. - Commercial.
licenseis the licence text.pricinglists plans (Pricing). - Other.
checksnames other datasets whose checks must pass before this one is trusted, for example a validation table:checks: [RentsValidation].displaylists files made for people from this dataset (for example a spreadsheet), each with one line of description.identity_only_paramsis specific to the first implementation's pipeline engine (see the table).
Profile fields¶
The profile block holds four keys that only shape the profile shown to agents: period, entities,
coverage_by and series. Nothing checks them against any rule, beyond each named column being in the data. A tool
generates the profile from the data, so nobody types counts that go stale. You can leave the block out.
grain: [market_id, sector, period_yq]
profile:
period: period_yq
entities: [market_id]
coverage_by: [sector] # optional
series: [market_id, sector] # optional; default = grain minus period
The profile lines below follow the card of the first implementation; they are illustrative.
period: the time column. The profile reports the first and last period.
Coverage (G): period_yq 2010Q1..2026Q3.
series: the columns that pick out one time series, meaning one line on a chart. You rarely write it: when
grain and period are declared and period is in the grain, series defaults to the grain minus period
([market_id, sector] above). Write it only when one series is keyed by fewer columns than that. The tool counts the
series, how many reach the latest period, and how many have gaps.
Coverage (G): period_yq 2010Q1..2026Q3; 110 series by market_id x sector; 104 reach the latest period; 6 have gaps.
coverage_by (optional): a category column with few values to break coverage down by. For each value the
profile shows the first and last period, the row count and the number of distinct entities. It is not a uniqueness
rule. Do not use the period column; that would list every quarter.
By sector (G): industrial 2012Q1..2026Q3, 3,410 rows, 55 market_id; office 2010Q1..2026Q3, 4,120 rows, 55 market_id.
entities: the columns naming what the rows are about. The profile counts their distinct values, overall and per
coverage_by value (as in the line above).
Entities (G): 55 market_id.
How they differ: grain is what makes a row unique; series is the grain minus time; entities are the things you
would count ("55 markets"); coverage_by is how you would slice the counts.
Writing period, series, coverage_by or entities directly on the dataset, outside profile, is an unknown
key (error); a checker should say to move it under profile.
Definitions¶
"In the data" means among the columns of the dataset's data as stored (see Errors and warnings).
| Key | Type | Required | Priority | Check | Description |
|---|---|---|---|---|---|
purpose |
string | no | Important | prose lint | One line, shown first. |
description |
string | no | Useful | prose lint | Longer text. |
grain |
list of column names | no | Critical | each column must be in the data (error); hint if none is declared | The columns that identify one row. |
profile |
object | no | Important | unknown keys inside are errors | What shapes the profile shown to agents. Profile only. |
profile.period |
column name | no | Important | must be in the data (error) | The time column. |
profile.entities |
list of column names | no | Useful | must be in the data (error) | Columns naming what the rows are about; distinct values are counted. |
profile.coverage_by |
list of column names | no | Important | must be in the data (error) | A low-cardinality category to break coverage down by. |
profile.series |
list of column names | no | Useful | must be in the data (error) | Columns that identify one time series. Default: grain minus profile.period, when both are declared and the period is in the grain. |
columns |
map of name to Column | no | Critical | - | Column declarations. |
use_cases |
list of UseCase | no | Important | see use cases | Recipes. |
do_not |
list of string or object (text, columns) |
no | Critical | error if a named column is not declared on the dataset | Hazards: what not to do with this data. A string applies to the whole dataset. |
do_not[].text |
string | yes (object form) | Critical | prose lint | The hazard. |
do_not[].columns |
list of column names | no | Critical | error if not declared | Columns the hazard is about. |
not_exported |
list of string or object (what, where, why) |
no | Useful | prose lint | What is deliberately not shared. |
not_exported[].what |
string | yes (object form) | Useful | - | The omitted data. |
not_exported[].where |
string | no | Useful | - | Where to get it. |
not_exported[].why |
string | no | Useful | - | Why it is not shared. |
removed_columns |
list of object | no | Important | error if a removed column is still in the data (a tool may report it as a warning while publishing) | Columns that used to exist. Do not reuse the name for a new meaning; see Schema changes. |
removed_columns[].name |
string | yes | Important | as above | The removed column. |
removed_columns[].removed |
date | yes | Important | - | When it was removed. |
removed_columns[].use_instead |
string | no | Important | - | The column that replaces it. |
removed_columns[].note |
string | no | Important | prose lint | Free note. |
dates_asof |
map of source name to string or AsOf | no | Critical | see as-of | Where each vendor source's as-of date comes from: a fixed date, a column, a setting of the pipeline that builds the dataset, or an upstream dataset. See As-of dates. |
license |
string | no | None | - | Licence of this dataset. |
pricing |
list of PricingPlan, or [] |
no | None | see pricing | Plans for this dataset. [] means none, even if the project has some; absent means the project's. |
facts |
list of Fact | no | by severity | see facts | Dated facts. |
aliases |
list of string or object (name, until) |
no | Important | error unless each name uses only letters, digits and _ and does not start with a digit |
Former names of this dataset. |
aliases[].name |
string | yes (object form) | Important | as above | The former name. |
aliases[].until |
date | no | Important | - | Until when it still works. |
checks |
list of string | no | None | error if a name is not a dataset (or a <project>/<Dataset> id) |
Other datasets whose checks must pass before this one is trusted. |
display |
map of string to string | no | None | prose lint | Files made for people from this dataset, each with one line of description. |
identity_only_params |
list of string | no | None | - | Implementation-specific: settings of the pipeline run that name a variant but do not change the data. An implementation may check the names exist. |