Schema changes¶
Data changes shape over time. Columns are added, renamed and removed; a unit changes; a dataset is split. This page
says what a producer declares for each change, using the fields of the other pages, and what a reader of the data
(a person or an agent) does about it. It is guidance; the only field it adds is project
removed_datasets.
Three kinds of change¶
- Additive. A new column, new rows, a new category. Nothing that worked stops working. Example:
Rentsgains avacancycolumn. - Loud. A rename, a removal, a retired dataset, a column that turns from a number into text. Code that reads the old shape fails, so someone notices. The fix is mechanical when the producer names the replacement.
- Silent. A unit or scale change, a new definition, a new grain, a restatement, renumbered keys. Code keeps running and the numbers are wrong. These do the most harm and need the most care.
A column is its name¶
There are no column ids. A column is identified by its name, and a rename keeps the old name in the column's
aliases: a citation or lookup by the old name reads the new column, and a citation keeps its build
and its value, so it never silently changes meaning.
Unit, scale or meaning changed. Recommended: give the column a new name, list the old name in
removed_columns with use_instead set to the new name, and add a dated fact on the new
column saying what changed. Acceptable: keep the name, set the new unit, and add a dated blocker fact with
columns ("from 2026-11 rent_growth is in %, was decimal"). A new name is safer: old code then fails loudly instead
of reading the wrong scale.
Do not reuse a name for a different meaning. A name in a dataset's removed_columns, or in any of its columns'
aliases, should not come back as a current column of that dataset. Old citations and alias lookups would then read
the wrong column. Tools flag it (an error in a check; a tool may report it as a warning while publishing). If the
same meaning truly comes back, delete the removed_columns entry.
What to declare¶
| Change | What the producer declares | Breaks? | What a reader does |
|---|---|---|---|
| Column added | Nothing new; describe it (a unit if numeric). | No | Use it if it replaces a workaround. |
| Column removed | removed_columns entry with removed and use_instead. |
Loud | Switch to the replacement. |
| Column renamed | Rename the key; the old name in the column's aliases. |
Loud | Use the new name in code. |
| Type class change (number, text, date) | Say so in the description; a caution fact if codes may lose leading zeros. |
Loud or silent | Re-check joins and filters. |
| A column's type changed | The type is filled from the data; add a dated fact when the change is breaking. | Widening: no. Narrowing or a change of kind: loud or silent | Re-check parsing, joins and filters on that column. |
| Unit or scale change | Recommended: new name, old one in removed_columns, a dated fact. Acceptable: same name, new unit, a dated blocker fact. |
Silent | Check the unit and the facts before quoting. |
| Meaning change, same name and type | Recommended: new name as above. Acceptable: same name and a dated blocker fact. |
Silent | Read facts dated after your last read. |
| Grain change (key columns or what one row means) | Update grain; a dated blocker fact; a new dataset name if one row now means something else. |
Silent | Re-check every sum and join. |
| Recoded key values | A blocker fact; a mapping dataset and a relationship. |
Silent | Re-join through the mapping. |
| New category values | Nothing; optionally an info fact. |
No, unless the list is hard-coded | Do not hard-code categories. |
| Blanks appear | A fact saying what a blank means. | Silent | Say how blanks were treated. |
| Dataset renamed | Dataset aliases with until. |
Loud without the alias | Update the import. |
| Dataset split | Project removed_datasets entry with use_instead and a note on how to split. |
Loud | Filter the replacement as the note says. |
| Datasets merged | The old names as aliases of the merged dataset; a fact naming the column that tells them apart. |
Loud without the alias | Filter on that column. |
| Dataset retired | Project removed_datasets entry. |
Loud | Move to use_instead. |
| Project renamed or retired | Project aliases or retired_names. |
Loud without the alias | Update the import. |
| New as-of source or vendor file format | A new project.sources key and dates_asof; a dated fact. Never reuse a source name. |
Silent if mixed | Check which source each number came from. |
| Backfill or restatement | A dated fact; dates_asof moves. |
Silent | Re-check the figures you quoted. |
A column's type changed. A tool records the new type from the data. Widening (integer to number, date to
timestamp) is not breaking: every old value still reads. Any other change narrows the type or changes its kind
(number to integer, number to string, date to string) and is breaking for readers. A reader re-checks parsing,
joins and filters on that column, and rounding or truncation if it narrowed.
What a tool detects from the builds¶
A producer does not have to notice every change by hand. A tool that keeps the column types and a short profile of each column (type, minimum, maximum, count of blanks) for every published build can compare a new build with the previous one and flag a change the declarations do not yet explain:
| Signal (new build vs previous) | Likely change | What the tool asks for |
|---|---|---|
| A column is gone | Removal or rename | removed_columns with use_instead, or the old name in the new column's aliases |
| A column's type changed | Type change: widening, narrowing or a change of kind (see above) | Nothing if widening; a dated fact if breaking |
| Minimum and maximum both moved by about x100 or x1000 (or /100, /1000), same sign | Unit or scale change (decimal to percent, units to thousands) | A new name (recommended) or a dated blocker fact |
| A declared column had no blanks and now has some | Blanks appear | A fact saying what a blank means |
A name in removed_columns or in aliases is a current column again |
A reused name | Rename it, or delete the removed_columns entry if the meaning truly came back |
Rules that keep the signals quiet when they should be:
- No scale signal on key or grain columns: renumbered ids are not a unit change. No scale signal when a minimum or maximum is zero: one outlier would decide it.
- A dated fact on the column since the previous build answers the signal: the tool does not repeat it.
- Compare only with the previous published build, never with an experiment build.
- The signal names the column and the change, never the data values: it can travel to readers (through a hub or a notice) without the data.
Facts are the changelog¶
Write one dated fact per breaking change: as_of is the date of the change, severity follows the harm (silent:
blocker; loud with a replacement: caution; additive: info or none), columns names the columns, and expires
is set once readers have moved. A tool can then list what changed since a date from the facts plus
removed_columns, the aliases and removed_datasets.
How long old names work¶
An alias (of a dataset or a column) with until stops working after that date; the old name then fails with a
message naming the new one. An alias without until keeps working. removed_columns and removed_datasets never resolve: they only explain.
For readers of the data¶
- Before quoting a number, read the facts dated after your last read of the dataset.
- A key or grain change means re-checking every sum and join that uses the dataset.
- Never carry a workaround past a notice or fact that supersedes it.