Three Instruments, All Lying: Debugging the Metrics Behind FameCake

The question was boring. “It’s Sunday afternoon and there are no bookings. Is something broken?”

The product is FameCake: a barber advertising this afternoon’s empty chairs, a pub pushing tonight’s quiz, someone putting their mate’s face up for a birthday. Anyone can get an ad onto a real digital billboard from their phone, from a fiver, without a media buyer. The answer to the question was boring too: no, it was Sunday. Sunday mornings are always slow, and this one was within a booking or two of the last three. Nothing was on fire.

But getting to that answer meant looking at the instruments, and the instruments turned out to be lying. Three of them, in three different ways, none of them broken, all of them wrong.

Instrument One: The Metric That Could Not Have Been Otherwise

We had killed our Meta campaigns a few days earlier. The evidence was clean: money spent, zero attributable bookings. Kill it.

Then I went looking for where the traffic had gone, and found this:

// AttributionCapture.tsx
if (marketingGranted) stashAdAttribution(searchParams);

Ad-click attribution (the fbclid that ties a booking back to an ad) is only captured under marketing consent. Marketing consent defaults to false. Our customers are in the UK, so they get the strict consent regime: nothing is recorded until they click Accept.

So “the ads drove zero conversions” was not a finding. It was a tautology. The instrument was structurally incapable of recording the thing we were asking it about, and it reported zero with exactly the confidence it would have reported zero if the ads were printing money.

Absence of evidence is not evidence of absence.

— Carl Sagan

The aphorism is usually deployed as a warning about humility, and it gets waved around a bit too freely. But this is the literal case it was built for: we had no evidence of conversions because our instrument could not produce that evidence, and we treated the empty log as proof the ads were dead. We then spent the empty log. We killed the campaigns.

Worse than Sagan’s version, actually. He is talking about not having looked hard enough. We had looked, with a detector wired so that the needle could never move.

The gating is correct, by the way. A click ID is a tracking identifier, and consent-gating it is almost certainly a legal requirement. The bug was never in the code. The bug was that a decision got made on the output of an instrument nobody had checked the range of.

Instrument Two: The Source of Truth With an Expiry Date

The second one is worse, because it faces the customer.

Our screens report plays into an hourly stats map on the screen document. That map is the only record of which ad played where. A background processor prunes it at 30 days, which is sensible: it’s a fat document and you don’t want a year of hourly buckets in it.

Bookings, meanwhile, can run for a year. We have cycles over 100 days in production right now.

So the play count on a customer’s booking was computed live, at read time, from a source that deletes itself. Open a booking older than a month and it reported zero plays for an ad that had run perfectly, thousands of times. Not an error, not a warning: a confident, plausible zero.

A cache had quietly become the source of truth, and the only symptom was a number that got quieter with age.

The fix looks obvious in hindsight: make the count durable. Roll the plays out of the pruning cache and onto the booking as they happen, so the raw stats only need to survive hours instead of forever. Each cycle carries a running total plus a watermark (the last hour bucket folded in), and the displayed number is that total plus whatever the live stats still hold beyond it.

What is not obvious is how many ways that goes wrong.

The Fix Was Worse Than the Bug

My first version materialised the count when a booking finished. Clean, event-driven, no extra machinery.

It is also catastrophically wrong for any booking that outlives the retention window. A 106-day booking reaches the finish line with 76 days of its history already deleted. My code would dutifully total up the surviving month and stamp that as the authoritative, permanent count. Not a zero, which at least looks like an absence. A plausible, specific, wrong number, frozen forever, indistinguishable from a real one.

I shipped that. It went to production. It sat there for two hours before I caught it.

That is the part worth sitting with. The original bug lost data. My fix would have fabricated data, which is strictly worse, and it did so in the course of fixing a data-integrity bug. The failure mode of caring about correctness is a confident, well-tested, wrong answer.

What the Second Model Caught

Last time I wrote about a model reviewing code and returning ten findings, of which two were real. High recall, low precision, poor calibration.

This time the arc ran the other way. I reviewed my own change, found four issues, fixed them, shipped. Then I handed the diff to Codex with the domain constraints spelled out (buckets are atomic, cycles roll mid-hour, the retention window is 30 days) and told it to attack.

It came back with nine, and the serious ones were real:

• A single watermark shared across screens. Screens in one booking have their own play windows. A screen whose first play landed in the bucket another screen’s watermark stopped at would resume an hour late and lose those plays permanently. Roughly one session in six.
• An hour bucket claimed by two cycles. Buckets are atomic, but a cycle can roll mid-hour. The bucket spanning the boundary got counted by the cycle that ended in it and the cycle that started in it. Double counted, durably.
• The screen-removal path. Untouched by my change, still reading the pruned source. Swap a screen on a long booking and the customer’s total halves. Re-add it and it double counts.

None of those are style nits. All of them put a wrong number in front of a paying customer. I had read that diff carefully, twice, and missed every one.

The difference from last time is not that the model got smarter in a week. The task was different: a bounded diff, the invariants stated explicitly, and an instruction to attack rather than assess. Review is a calibration problem, and calibration improves enormously when you hand over the thing the reviewer is meant to be calibrated against.

Instrument Three: The Claim Nobody Had Measured

The last one is the smallest, and the one I like most.

The site says “live in 60 seconds”. It has said that for a long time. It is a good line, and nobody had ever checked it.

So I checked. Across 727 real paid bookings, payment to the ad physically appearing on a screen:

Only 41% land inside 60 seconds. The claim was true less than half the time, which makes it a claim you would rather not have to defend.

The measured number is better anyway, because the interesting comparison was never against a stopwatch. It is against an industry where you book a billboard through an insertion order and a lead time of ten to fifteen business days, and where programmatic (the thing that was supposed to fix that) is still only about 7% of global spend. Against weeks, “median 72 seconds, 90% inside five minutes” is not a faster version of the same product. It is a different product, because a pub deciding at 4pm to advertise tonight cannot exist in a world with a three-week lead time.

I wrote that argument up properly on the product blog: Booking a Billboard Takes Weeks. We Measured Ours at 72 Seconds.

What This Doesn’t Prove

• I got lucky on the timing. The fabricating fix was in production for two hours, and the sweep that would have corrupted every long-running booking had not fired yet. A day later and this post has a very different ending.
• The Codex review was not free. It also produced findings I rejected, and reading it properly took longer than the fix. The value came from the setup (small diff, explicit invariants, adversarial instruction), not from pointing a model at a repo.
• None of this is exotic. Retention windows, consent gates and unverified marketing copy are not edge cases. They are the most boring infrastructure in the building, which is exactly why nobody looks at them.
• The numbers will drift. A 72-second median is only true until it isn’t, and a hardcoded stat on a landing page is a lie with a delay fuse. It has to be recomputed, not remembered.

The Through Line

None of these three instruments was broken. Every one of them was derived: a number computed at read time from a source that had quietly moved underneath it. Consent gating changed what attribution could see. A retention policy changed what a play count could count. Time changed whether a marketing claim was true.

Derived data has an expiry date and nobody writes it on the tin. The half life gets set by an unrelated policy three files away, decided years ago for good reasons that had nothing to do with the number you are now reading off a dashboard.

The practical version, which I believe more strongly than I did on Saturday: if a number could only ever come out one way, it is not a measurement. Before acting on an instrument, work out what it would show if the thing it measures were false. If you cannot describe that, you are not looking at data. You are looking at a decoration that happens to be numeric.

It took a boring question about a slow Sunday to find that out three times in one day.