Skip to content

Datasets

A dataset is one table that others read, keyed by its name under datasets. Its entry says what one row is (grain), what each column means (columns), which vendor releases it reflects (dates_asof), what to do and not do with it, and the dated facts a reader needs. Every key is optional.

Start with five keys: purpose, grain, columns, dates_asof and facts. The rest is for when you need it.

Example

datasets:
  Rents:
    purpose: Quarterly asking rent per market and sector.
    grain: [market_id, sector, period_yq]
    profile:
      period: period_yq
      entities: [market_id]
      coverage_by: [sector]
    dates_asof:
      vendor_a:
        constant: "2026-09-30"
        cadence: quarterly
    columns:
      market_id:
        description: market code (text)
      sector:
        description: property sector
      period_yq:
        description: calendar quarter
      rent:
        description: asking rent
        currency: USD
        per: year
        as_of_sources: [vendor_a]
    do_not: [join on market names]
    not_exported:
    - what: lease-level rows
      where: Leases
      why: licence review pending
    removed_columns:
    - name: rent_old
      removed: 2026-10-29
      use_instead: rent
    license: CC-BY-4.0
    aliases: [Rent]

Fields explained

Grouped by job.

  • Identity. purpose is the one line an agent reads first; description is longer text. grain lists the columns that identify one row; it is the most useful key in the file, because it tells an agent what a sum means and which joins are safe. aliases lists former names of the dataset that still work.
  • Columns. columns says what each column is and its unit. See Columns.
  • Profile. The profile block (period, entities, coverage_by, series) shapes the coverage profile shown to agents. See Profile fields below.
  • Releases. dates_asof says where each vendor source's as-of date comes from: a fixed date, a column, a setting of the pipeline that builds the dataset, or an upstream dataset. See As-of dates.
  • Guidance. use_cases are recipes (Use cases). do_not lists hazards: rules that are always true for this data, such as "do not join on market names"; a plain string applies to the whole dataset, the text + columns form names the columns it is about. facts are dated statements (Facts). Use a fact when something was learned or decided at a point in time and might change; use do_not for a standing rule.
  • What is missing. removed_columns lists columns that used to exist and what replaces them. not_exported says what is deliberately not shared, and where to get it.
  • Commercial. license is the licence text. pricing lists plans (Pricing).
  • Other. checks names other datasets whose checks must pass before this one is trusted, for example a validation table: checks: [RentsValidation]. display lists files made for people from this dataset (for example a spreadsheet), each with one line of description. identity_only_params is specific to the first implementation's pipeline engine (see the table).

Profile fields

The profile block holds four keys that only shape the profile shown to agents: period, entities, coverage_by and series. Nothing checks them against any rule, beyond each named column being in the data. A tool generates the profile from the data, so nobody types counts that go stale. You can leave the block out.

grain: [market_id, sector, period_yq]
profile:
  period: period_yq
  entities: [market_id]
  coverage_by: [sector]     # optional
  series: [market_id, sector]     # optional; default = grain minus period

The profile lines below follow the card of the first implementation; they are illustrative.

period: the time column. The profile reports the first and last period.

Coverage (G): period_yq 2010Q1..2026Q3.

series: the columns that pick out one time series, meaning one line on a chart. You rarely write it: when grain and period are declared and period is in the grain, series defaults to the grain minus period ([market_id, sector] above). Write it only when one series is keyed by fewer columns than that. The tool counts the series, how many reach the latest period, and how many have gaps.

Coverage (G): period_yq 2010Q1..2026Q3; 110 series by market_id x sector; 104 reach the latest period; 6 have gaps.

coverage_by (optional): a category column with few values to break coverage down by. For each value the profile shows the first and last period, the row count and the number of distinct entities. It is not a uniqueness rule. Do not use the period column; that would list every quarter.

By sector (G): industrial 2012Q1..2026Q3, 3,410 rows, 55 market_id; office 2010Q1..2026Q3, 4,120 rows, 55 market_id.

entities: the columns naming what the rows are about. The profile counts their distinct values, overall and per coverage_by value (as in the line above).

Entities (G): 55 market_id.

How they differ: grain is what makes a row unique; series is the grain minus time; entities are the things you would count ("55 markets"); coverage_by is how you would slice the counts.

Writing period, series, coverage_by or entities directly on the dataset, outside profile, is an unknown key (error); a checker should say to move it under profile.

Definitions

"In the data" means among the columns of the dataset's data as stored (see Errors and warnings).

Key Type Required Priority Check Description
purpose string no Important prose lint One line, shown first.
description string no Useful prose lint Longer text.
grain list of column names no Critical each column must be in the data (error); hint if none is declared The columns that identify one row.
profile object no Important unknown keys inside are errors What shapes the profile shown to agents. Profile only.
profile.period column name no Important must be in the data (error) The time column.
profile.entities list of column names no Useful must be in the data (error) Columns naming what the rows are about; distinct values are counted.
profile.coverage_by list of column names no Important must be in the data (error) A low-cardinality category to break coverage down by.
profile.series list of column names no Useful must be in the data (error) Columns that identify one time series. Default: grain minus profile.period, when both are declared and the period is in the grain.
columns map of name to Column no Critical - Column declarations.
use_cases list of UseCase no Important see use cases Recipes.
do_not list of string or object (text, columns) no Critical error if a named column is not declared on the dataset Hazards: what not to do with this data. A string applies to the whole dataset.
do_not[].text string yes (object form) Critical prose lint The hazard.
do_not[].columns list of column names no Critical error if not declared Columns the hazard is about.
not_exported list of string or object (what, where, why) no Useful prose lint What is deliberately not shared.
not_exported[].what string yes (object form) Useful - The omitted data.
not_exported[].where string no Useful - Where to get it.
not_exported[].why string no Useful - Why it is not shared.
removed_columns list of object no Important error if a removed column is still in the data (a tool may report it as a warning while publishing) Columns that used to exist. Do not reuse the name for a new meaning; see Schema changes.
removed_columns[].name string yes Important as above The removed column.
removed_columns[].removed date yes Important - When it was removed.
removed_columns[].use_instead string no Important - The column that replaces it.
removed_columns[].note string no Important prose lint Free note.
dates_asof map of source name to string or AsOf no Critical see as-of Where each vendor source's as-of date comes from: a fixed date, a column, a setting of the pipeline that builds the dataset, or an upstream dataset. See As-of dates.
license string no None - Licence of this dataset.
pricing list of PricingPlan, or [] no None see pricing Plans for this dataset. [] means none, even if the project has some; absent means the project's.
facts list of Fact no by severity see facts Dated facts.
aliases list of string or object (name, until) no Important error unless each name uses only letters, digits and _ and does not start with a digit Former names of this dataset.
aliases[].name string yes (object form) Important as above The former name.
aliases[].until date no Important - Until when it still works.
checks list of string no None error if a name is not a dataset (or a <project>/<Dataset> id) Other datasets whose checks must pass before this one is trusted.
display map of string to string no None prose lint Files made for people from this dataset, each with one line of description.
identity_only_params list of string no None - Implementation-specific: settings of the pipeline run that name a variant but do not change the data. An implementation may check the names exist.