Build notes
Listing & Campaign Intelligence: how it came together.
Develop · Build
How it came together.
- 01
Five stages, each allowed to stop the next
Collect from several sources at once, validate what came back against schema and duplicates, score every listing, re-verify the biggest movers, then report. The stages are gated rather than sequential: validation can refuse to hand data to scoring, and scoring can refuse to hand a week to the report. A pipeline that always produces output is a pipeline that will eventually produce confident nonsense.
Data Pipelines - 02
Re-fetch the movers, because a ranking should reproduce
The listings that moved most are the ones a reader will act on, and they are also the ones most likely to be an artefact of when the page was fetched. So the top movers get a second independent fetch and the two results are compared. Anything that will not reproduce is marked unverified rather than quietly reported, which turns the interesting numbers into the ones carrying the most evidence instead of the least.
Backend Engineering - 03
It sends the alert, not the report
When the quality gates fail, the job does not send a degraded report with a caveat at the top. It sends an alert saying the week is not trustworthy and why. That is the whole design position: a weekly report that is wrong once is worse than one that is missing once, because the wrong one gets believed and quoted in a meeting. The same guard covers an empty run, so a collection that silently returns nothing raises an alarm rather than an empty PDF.
Research - 04
Built to be diagnosed weeks later
Unattended jobs fail while nobody is watching, so the failure has to be legible afterwards to somebody reading a log rather than watching it run. That principle got tested the hard way: the job stopped for two months, and the logs it left behind were 116 bytes containing nothing but a start line, because of how the output was being captured rather than anything wrong with the work. Three genuine defects were sitting behind that silence, invisible until the logging was fixed. The lesson was not the bugs. It was that a job which cannot explain its own death buys you nothing by dying quietly.
Backend Engineering