Frozen before results: why we pre-registered our benchmark

Every phone-validation vendor publishes an accuracy number. One of them claims 99.97%. None of them publishes a dataset, a methodology, or anything you could reproduce. You are asked to take the most important number in the purchase decision on faith — from the party selling it.

We're building a validation API too, so our incentives are exactly as suspect as theirs. Which is why we're doing the one thing an incumbent can't do retroactively: freezing our benchmark methodology in public, before running a single query.

The problem with running the test first

The classic way to rig a benchmark isn't fabricating numbers — it's much quieter. You run the test, look at the results, and then decide the rules: which categories count, how partial answers score, whether "no data" counts against a vendor. Every one of those choices can be defended in isolation. Chosen after the results are known, they let you tune the outcome without ever touching the data.

Science has a name for the fix: pre-registration. You publish your method — hypotheses, dataset, scoring — before you collect data. Once results exist, the rules provably predate them.

What we froze

The full document is public, but the rules that matter most:

• Abstain is not wrong. If a vendor returns "unknown," that answer is excluded from the accuracy denominator and reported as an abstention rate instead. Not knowing is honest; being wrong is not. We refuse to let silence be scored as error — or as accuracy.
• Partial credit exists in exactly three pre-named cases — right line-type family but wrong subtype, MVNO host network instead of MVNO brand, one-port-stale carrier. Nothing else earns half marks, no matter how sympathetic it looks later.
• Validity means allocated, not answered. A real number with no active subscriber is still a valid number. Vendors that call quiet numbers "invalid" get marked wrong for it.
• The headline metric is fixed now: the unweighted mean of validity, line-type, and carrier accuracy, with 95% bootstrap confidence intervals. No new aggregate gets invented after we've seen who it flatters.
• Every vendor sees its own results 14 days before publication and can dispute them. Corrections happen in a public errata log — never as silent edits.
• Our own API gets tested under identical rules from the first re-run after launch, and the scores go up win or lose.

The freeze itself

When the methodology publishes to our public repository, the repo is tagged and the document's SHA-256 hash is posted on our site. From that moment, anyone can re-hash the file and verify that not one word changed after results came back. The methodology is drafted now and freezes with the repository — the current draft hash is already on our methodology page.

The dataset follows the same discipline: 1,000 numbers that are synthetic, owned by us, or contributed with written consent. No scraped lists, no purchased data, and no calls or texts ever placed — API queries only.

Why bother

Because "trust us" is the exact product defect we're competing against. A benchmark you could quietly rig is just marketing with extra steps. A benchmark you provably can't rig — even though you run it, even though you compete in the market it measures — is evidence.

If you want the results the moment they publish, the waitlist is on the front page. If you're one of the vendors: the dispute window is real, and we'd genuinely rather publish your corrections than our mistakes.