Field note · pipeline anatomy
The scrape gate
Every morning an AI agent scrapes housing listings across the Ogden metro and commits the result straight to this site. Nobody reviews it first. Models are good at reading messy listing pages and bad at being exact, so the work is split: the model only reads, code owns every rule about the data, and a gate refuses any run that looks like corruption rather than a market move. This is the anatomy of that split, with every scrape since Jun 7 replayed through the gate.
- Last scrape
- Oct 2
- shipped by GitHub Actions
- Seen this week
- 1,175
- 188 rentals · 987 for sale
- Scrapes replayed
- 75
- 22 would have corrupted the data
- Days scraped, always-on
- 40 of 42
- vs 36 of 76 on a laptop
01 · The split
The model reads. Code decides.
Rent comes from a parser that reads the listing JSON rent.com embeds in its pages, the same way every time. For-sale homes are still read by the model, from Zillow and Redfin pages that change shape without notice. Either way the model hands over a flat file of rows, and that is all it is trusted to produce.
Everything after that is code. The merge owns the keys, the price history and each listing’s lifecycle: a listing no scrape has seen for 7 days ages to missing. The gate decides whether the result may ship: it runs in the push script before anything is committed, and again in CI on every push to main.
Where does the model stop and the code start?
The teal box is the only place the model works. Everything else is plain code, and a red gate means nothing is committed.
02 · The replay
Reliable isn’t the same as right
Until August the scrape ran on a laptop: a launchd job, a Cowork task, sometimes by hand. It shipped on 36 of 76 days, and once went 13 days without a run. Since Aug 22 it has run on GitHub Actions from a fresh clone, and shipped on 40 of 42 days.
That fixed reliability, not correctness. Replayed through the gate, 5 of the 41 runs Actions shipped would have been refused for corrupting the data, and 27 more for a quieter fault covered below. On Sep 20 one run added 158 duplicates: houses the catalogue already had, back under a second address string. Nothing noticed.
Which daily scrapes would the gate have refused?
78 runs, Jun 7 to Oct 2, 2026: 22 would have corrupted the data (17 on the laptop, 5 on Actions), and 36 more were refused only for stale “active” labels. Before the dashed line each column replays the commit that shipped against the one before it; after it, the gate ran live and a refused run shows as a day without a scrape. Hover a column for its rules.
03 · The checks
Refuse corruption, not the market
A price cut, a listing that disappears, a slow week: that is the market, and the gate lets it through. It refuses what only a broken run produces: history rewritten, listings deleted, a live count that drops more than 15% overnight among listings seen that week (almost always a scrape that missed pages), or a house gaining a second key.
It blocks regressions, not the backlog. A problem already in yesterday’s file only warns, so the gate could ship on day one without first cleaning up every old mistake.
The push that prompted it: on Sep 27 a scheduled task found its checkout parked on a branch whose catalogue was frozen at Sep 20. Merging there would have pushed a week-old file over today’s. The agent stopped on its own. The gate turns that luck into a rule.
Which checks would have fired, and how often?
Bars count replayed runs; the number after the dot is listings caught. A filled dot means the week-old push below trips that check too.
What would the week-old checkout have shipped?
The Sep 20 catalogue validated as if pushed over the Sep 27 one: the replay stages the push the agent declined to make.
04 · The quiet one
When “active” stopped meaning on the market
Replaying the history surfaced a failure no single run causes. The merge only aged a listing when a run said it had covered the whole source. The for-sale scrape never reaches every page, so no run ever said so, and nothing aged. By Sep 26, 1,173 of the 2,160 homes marked active hadn’t been seen by any scrape for a week.
The fix is a rule, not a cleanup: a listing no scrape has seen for 7 days ages to missing, even on a partial run, and the gate now refuses any file that still calls one active. The next scrape applies it. Why a week: since the move to Actions, 21 of 25,492 gaps between sightings of a live home ran past 7 days, and a listing aged by mistake comes back the next time it’s seen.
How many “active” listings had no scrape seen for a week?
Active listings whose last sighting was 7+ days before the catalogue’s own date, after each scrape. The rentals include listings left over from apartments.com, which stopped being a source.
When did a scrape last see the homes marked active today?
For-sale homes marked active as of Sep 26. Only the top row is a claim a scrape backs up; the week rule ages the rest.
05 · The backlog
What the gate can’t fix
The gate stops new damage. It doesn’t repair old damage. 367 homes in the for-sale catalogue are listed more than once: the same house scraped from both Zillow and Redfin, or relabelled from South Ogden to Ogden. That inflates the Home Finder’s live count by about 356. The fix is a one-time dedupe pass, then keying each listing on the site’s own listing ID instead of its address.
The map has the same kind of gap. The merge gives every new listing its city’s center as a placeholder, and nothing replaces it: all 1,721 homes first seen since Jun 30 sit on one of seven points, which puts 83% of the for-sale map on a city center. Geocoding each address once and caching it closes that. Then a placeholder becomes a refusal too.
How many homes are listed more than once?
Properties with two or more keys after each scrape: same street, unit, ZIP and bedrooms under different address strings.
What share of map pins sit on a city’s center?
Share of every listing ever tracked (live or not) whose coordinates are exactly one of the seven city centers.
The receipt
Every push that ships appends a row to a committed receipt, housing_gate.json, and this page reads it at build time. Right now: last gate run Oct 2, 2026 on GitHub Actions: rent passed, buy unchanged. When the gate refuses, nothing ships, the day shows up as a gap, and the freshness check files an issue that morning.
The split carries past housing. Any team feeding AI-extracted data into numbers people act on, like pipeline, pricing or inventory, can run it the same way: let the model read, make code own the rules, and put a gate between the model and the push.
The scrape runs on Claude inside a GitHub Actions job. The merge, the gate and the replay are plain Node scripts in the site repo, and the listings come from public listing pages.