Skip to content

Schema changes

Data changes shape over time. Columns are added, renamed and removed; a unit changes; a dataset is split. This page says what a producer declares for each change, using the fields of the other pages, and what a reader of the data (a person or an agent) does about it. It is guidance; the only field it adds is project removed_datasets.

Three kinds of change

  • Additive. A new column, new rows, a new category. Nothing that worked stops working. Example: Rents gains a vacancy column.
  • Loud. A rename, a removal, a retired dataset, a column that turns from a number into text. Code that reads the old shape fails, so someone notices. The fix is mechanical when the producer names the replacement.
  • Silent. A unit or scale change, a new definition, a new grain, a restatement, renumbered keys. Code keeps running and the numbers are wrong. These do the most harm and need the most care.

A column is its name

There are no column ids. A column is identified by its name, and a rename keeps the old name in the column's aliases: a citation or lookup by the old name reads the new column, and a citation keeps its build and its value, so it never silently changes meaning.

Unit, scale or meaning changed. Recommended: give the column a new name, list the old name in removed_columns with use_instead set to the new name, and add a dated fact on the new column saying what changed. Acceptable: keep the name, set the new unit, and add a dated blocker fact with columns ("from 2026-11 rent_growth is in %, was decimal"). A new name is safer: old code then fails loudly instead of reading the wrong scale.

Do not reuse a name for a different meaning. A name in a dataset's removed_columns, or in any of its columns' aliases, should not come back as a current column of that dataset. Old citations and alias lookups would then read the wrong column. Tools flag it (an error in a check; a tool may report it as a warning while publishing). If the same meaning truly comes back, delete the removed_columns entry.

What to declare

Change What the producer declares Breaks? What a reader does
Column added Nothing new; describe it (a unit if numeric). No Use it if it replaces a workaround.
Column removed removed_columns entry with removed and use_instead. Loud Switch to the replacement.
Column renamed Rename the key; the old name in the column's aliases. Loud Use the new name in code.
Type class change (number, text, date) Say so in the description; a caution fact if codes may lose leading zeros. Loud or silent Re-check joins and filters.
A column's type changed The type is filled from the data; add a dated fact when the change is breaking. Widening: no. Narrowing or a change of kind: loud or silent Re-check parsing, joins and filters on that column.
Unit or scale change Recommended: new name, old one in removed_columns, a dated fact. Acceptable: same name, new unit, a dated blocker fact. Silent Check the unit and the facts before quoting.
Meaning change, same name and type Recommended: new name as above. Acceptable: same name and a dated blocker fact. Silent Read facts dated after your last read.
Grain change (key columns or what one row means) Update grain; a dated blocker fact; a new dataset name if one row now means something else. Silent Re-check every sum and join.
Recoded key values A blocker fact; a mapping dataset and a relationship. Silent Re-join through the mapping.
New category values Nothing; optionally an info fact. No, unless the list is hard-coded Do not hard-code categories.
Blanks appear A fact saying what a blank means. Silent Say how blanks were treated.
Dataset renamed Dataset aliases with until. Loud without the alias Update the import.
Dataset split Project removed_datasets entry with use_instead and a note on how to split. Loud Filter the replacement as the note says.
Datasets merged The old names as aliases of the merged dataset; a fact naming the column that tells them apart. Loud without the alias Filter on that column.
Dataset retired Project removed_datasets entry. Loud Move to use_instead.
Project renamed or retired Project aliases or retired_names. Loud without the alias Update the import.
New as-of source or vendor file format A new project.sources key and dates_asof; a dated fact. Never reuse a source name. Silent if mixed Check which source each number came from.
Backfill or restatement A dated fact; dates_asof moves. Silent Re-check the figures you quoted.

A column's type changed. A tool records the new type from the data. Widening (integer to number, date to timestamp) is not breaking: every old value still reads. Any other change narrows the type or changes its kind (number to integer, number to string, date to string) and is breaking for readers. A reader re-checks parsing, joins and filters on that column, and rounding or truncation if it narrowed.

What a tool detects from the builds

A producer does not have to notice every change by hand. A tool that keeps the column types and a short profile of each column (type, minimum, maximum, count of blanks) for every published build can compare a new build with the previous one and flag a change the declarations do not yet explain:

Signal (new build vs previous) Likely change What the tool asks for
A column is gone Removal or rename removed_columns with use_instead, or the old name in the new column's aliases
A column's type changed Type change: widening, narrowing or a change of kind (see above) Nothing if widening; a dated fact if breaking
Minimum and maximum both moved by about x100 or x1000 (or /100, /1000), same sign Unit or scale change (decimal to percent, units to thousands) A new name (recommended) or a dated blocker fact
A declared column had no blanks and now has some Blanks appear A fact saying what a blank means
A name in removed_columns or in aliases is a current column again A reused name Rename it, or delete the removed_columns entry if the meaning truly came back

Rules that keep the signals quiet when they should be:

  • No scale signal on key or grain columns: renumbered ids are not a unit change. No scale signal when a minimum or maximum is zero: one outlier would decide it.
  • A dated fact on the column since the previous build answers the signal: the tool does not repeat it.
  • Compare only with the previous published build, never with an experiment build.
  • The signal names the column and the change, never the data values: it can travel to readers (through a hub or a notice) without the data.

Facts are the changelog

Write one dated fact per breaking change: as_of is the date of the change, severity follows the harm (silent: blocker; loud with a replacement: caution; additive: info or none), columns names the columns, and expires is set once readers have moved. A tool can then list what changed since a date from the facts plus removed_columns, the aliases and removed_datasets.

How long old names work

An alias (of a dataset or a column) with until stops working after that date; the old name then fails with a message naming the new one. An alias without until keeps working. removed_columns and removed_datasets never resolve: they only explain.

For readers of the data

  • Before quoting a number, read the facts dated after your last read of the dataset.
  • A key or grain change means re-checking every sum and join that uses the dataset.
  • Never carry a workaround past a notice or fact that supersedes it.