Skip to content

Common failures

These failure types come from real analysis projects that use licensed market data with AI coding agents. The projects are anonymised and every example on this page is synthetic. We reviewed what those projects had to write down to keep agents correct, and grouped it. Each entry gives a one-line description, a small example, what goes wrong when an agent meets it, and the part of the spec that addresses it, or open if no field does yet.

Two observations cut across the list. The same fact is often written in several places, and the copies disagree. And in the projects we reviewed only a small share of documentation lines carried a date, so most statements cannot be checked for staleness. Entries 10 and 13 return to both.

The first five are the ones we would fix first: they cause wrong numbers that look plausible.

1. Units and scale

What it is. A number's unit, scale or period is not stated, or is stated in the wrong place.

Example. Rents.rent_growth holds 0.012. The vendor's table of contents labels the column "Percent", but the values are decimals. Another column, rent, is annual in office and monthly in a residential dataset.

What goes wrong. The agent reports 0.012 percent instead of 1.2 percent, or compares monthly and annual rents across sectors. The numbers look plausible, so nothing flags them.

Addressed by. Columns: unit (UCUM), or currency plus per. Scale and kind (level, change, rate) are open; today write them in the description. A unit change: see Schema changes.

2. As-of dates and restatements

What it is. A source's as-of date is unknown, differs between sources, or the vendor revises history on a re-pull.

Example. vendor_a re-delivers the whole history each quarter and about 3% of old rows change slightly. A re-pull also drops some rows that are newly suppressed.

What goes wrong. An older release is called "latest", two releases are mixed in one table, or "the market moved" is reported when history was revised. A rerun silently replaces the numbers a finished report used.

Addressed by. As-of dates: dates_asof per source. A restatement rule can be written as a dated fact. A per-release file and a revision-policy block are open. See also Schema changes.

3. Grain and keys

What it is. What one row is, and which columns are unique.

Example. Rents has one row per market_id, sector and period_yq. A project table lists "master" projects that already include their child projects, so summing both counts everything twice.

What goes wrong. The agent sums a table that already contains its own subtotals, or treats a non-unique column as a key. Totals inflate and the query still runs.

Addressed by. Datasets: grain. A tool can test that the grain is unique; that test is not in the spec yet. A grain change: Schema changes.

4. Join rules

What it is. Which columns join two datasets, and what repeats when they do.

Example. A map table repeats market_id once per extra vendor code. Codes such as 04567 are text; the agent casts them to numbers and the join silently loses rows. Joining on market names fails for markets spelled two ways.

What goes wrong. Rows repeat, totals inflate, or rows vanish without an error. An agent that joins on names instead of codes produces a result that looks right for most markets.

Addressed by. Relationships (which columns, which side has many rows) and do_not on Datasets ("join on market names"). Mapping tables as a declared construct are open.

5. Which dataset, run or parameters to use

What it is. Several datasets, builds or parameter sets exist, and only one is the one consumers should read.

Example. The pipeline that builds a dataset has a parameter regularize that defaults to False, but the production run uses True. Another project copies the production parameters by hand into its own settings with a comment.

What goes wrong. The agent reads the default, an experiment run, or an out-of-date copy, and reports numbers from a build nobody intended to publish. Producers and consumers each assume the other knows.

Addressed by. The dataset purpose (Datasets) and recipes in Use cases say which dataset serves which question. Marking the production run, and recording the parameters of each build, is open in the file (tooling can record builds).

6. History spans and gaps

What it is. Where each series starts and ends, and which periods are missing inside that span.

Example. In office, history starts in 1979 for most markets but in 1991 for one, and several markets skip quarters and come back.

What goes wrong. A rolling window over a gapped series silently skips periods or misaligns, and a "since 1980" statistic quietly covers fewer markets earlier on. A series that is unusable before a certain date is used anyway.

Addressed by. Datasets: the profile block (period, series, coverage_by) tells a tool what to profile; spans and gaps are then generated from the data (Profile fields). A known floor can be written as a fact.

7. Meaning of derived metrics

What it is. What a computed rank, score, rating or percentile means, including its sign and direction.

Example. A column rank_improvement equals previous rank minus current rank, the opposite sign of an upstream rank_delta. A percentile runs from 0 to 1 where 1 is best, oriented by the signal's direction.

What goes wrong. The agent flips a sign, compares ranks across sectors with different sizes, or reads a backtest as a record of past published ratings.

Addressed by. Columns: description, use_instead. Long semantics belong in a dated fact. A structured metric formula is open.

8. Vendor method quirks and methodology

What it is. Behaviour of a data source or a method that is not obvious from the schema.

Example. vendor_b returns HTTP 200 when it throttles you, with an error field in the body. A misspelled field in a filter is silently ignored. A flag that looks optional is in fact required to get the intended series. Two vendors' "rent" series are not the same quantity.

What goes wrong. The agent treats an error as success, builds on a partial pull, or compares two series that only share a name. Method traps (a shared denominator, appraisal smoothing, a benchmark's weighting) produce confident but wrong interpretation.

Addressed by. Facts with a probe that checks the quirk still holds, pitfall in Use cases, and do_not on Datasets.

9. Undocumented decisions

What it is. A choice a person made that changes what the right answer is, kept in a chat or a head.

Example. "Stay on the previous release of vendor_a until the next quarter is reviewed." "Treat Metro A and Metro B as separate markets." A change to a market's classification takes effect from the next quarter, not this one.

What goes wrong. The agent re-opens a settled choice, or applies the change a quarter early. Nobody can tell whether the decision still stands.

Addressed by. Facts: status including decision-open, an expiry, and a date.

10. The same fact copied and drifting

What it is. One fact about shared data is written in several projects, and the copies disagree.

Example. Three project notes each state the history floor for office. One says 1979, one says 1984, and one is a year out of date. Several shared instruction files disagree with each other.

What goes wrong. The agent follows whichever copy it read first. Fixing one copy does not fix the others, so the disagreement grows.

Addressed by. A fact lives once, next to the data, with a stable id that others cite (Facts). Delivery of that fact to each consumer is tooling, not part of the file.

11. Renames that break consumers

What it is. A column, dataset or project is renamed or removed and the things that read it are not told.

Example. A shared project is renamed. Four projects import it by its old name and now fail at import. A column rent_old is replaced by rent.

What goes wrong. Consumers break, or worse, keep reading a stale copy under the old name.

Addressed by. Datasets: aliases, removed_columns with use_instead. Project: aliases, retired_names and removed_datasets. What to declare for each change: Schema changes. Notifying every consumer is tooling.

12. Display values versus pipeline values

What it is. A published sheet, report or export shows different numbers than the pipeline holds.

Example. A spreadsheet export shows each series' last row, not a latest-quarter snapshot. A narrative report prints quarterly figures while the workbook prints annualised ones.

What goes wrong. The agent quotes a figure from the display and cross-checks it against the pipeline, finds a mismatch, and either "corrects" the right number or trusts the wrong one.

Addressed by. Open. Today write the difference as a fact on the dataset that produces the display.

13. Undated statements that go stale

What it is. A statement has no date, status or expiry, so nobody can tell if it still holds.

Example. "The panel is balanced from 1995." No date, no scope, no check. Two years later new data breaks it.

What goes wrong. An agent treats an expired rule as current, or discards a still-valid one because it cannot tell which is which.

Addressed by. Facts: as_of, severity, status, optional expiry and a probe.

14. Licensing, usage limits and provenance

What it is. Who may see the data, whether it may go to outside AI services, what a request costs, and where each value came from.

Example. A dataset is licensed for internal use only. A scraper stores reviewer names. A source has a monthly download quota. A date was appended to a raw file by hand.

What goes wrong. In the projects we reviewed, none recorded licensing or usage limits at all. An agent can describe a licensed series in a client deliverable, exceed a quota, or trust a value whose origin nobody recorded.

Addressed by. Project and Datasets: a license text; Pricing: plans. Structured agent-use terms, per-request cost and a provenance record per value are open.