Unattended AI scrape → gated catalogue
Ogden Home Finder
A living catalogue of Ogden-metro housing. Every morning an AI agent scrapes 2BR rentals and for-sale homes; code merges them into a price-tracked catalogue; a gate refuses any run that looks like corruption before it ships. The pipeline’s anatomy is in The Scrape Gate.
Live · rent and buy tracked daily, every push gated
The decision moment
Housing hunt across a metro means tab overload. The decision you actually want: here are the few worth touring, with the price trend visible.
The constraint
The scraper is a model running unattended, and nobody reviews its output before it goes live. Models read messy pages well and are bad at being exact, so it can be trusted to read, never to decide what the data is.
The numbers
What shipped
- Daily scrape on GitHub Actions from a fresh clone: the agent runs two deterministic scrapers (rent.com, Redfin) over a 10-mile circle, and every map pin is the one the listing site sends
- A merge script that owns the keys, the append-only price history, and each listing’s active → missing → removed lifecycle
- A pre-push validation gate (deleted listings, rewritten history, a >15% active-count drop, new duplicate keys), also run in CI and replayed over every scrape since June
- Static page with stats, a map, a sortable table and per-listing price sparklines; Rent / Buy toggle over two catalogues
What changed
- The model’s job shrank to extraction. Keys, history and the decision to ship moved into code.
- Moving the schedule from a laptop to GitHub Actions ended the missed days; the gate is what stops the bad runs.
- The backlog is published, not hidden: duplicate listings across Zillow and Redfin are measured live on the field note, and so were the city-center map pins until each listing started carrying its source’s own coordinates.
This is the lens I bring to client work. Find the moment of decision, surface the constraint, make the next step obvious — then ship it in production, evaluated.