How to read a validation vendor's accuracy claim

Every vendor in this category publishes an accuracy number. Almost none publishes the method that produced it. That asymmetry is the actual state of the market, and it is why a percentage on a homepage carries so little information.

These are the questions that separate a measurement from a claim. They work on us as well as on everyone else, which is the point of writing them down.

1. What was the dataset, and who assembled it?#

Accuracy against what? A benchmark needs numbers whose truth is known independently of the vendors being tested, and how that ground truth was established is the single most load-bearing detail in the whole exercise.

If a vendor will not describe the dataset, the number describes nothing you can reason about.

2. Were the scoring rules fixed before the results existed?#

This is the question most published figures cannot survive.

If the rules were written after the answers came back, the benchmark measures which scoring rule flattered the sponsor, not which vendor was accurate. There is no way to detect this after the fact from a percentage — which is exactly why the timing has to be provable rather than asserted.

Boundstone's methodology for benchmark № 001 was published, tagged and checksummed on 25 July 2026, before a single vendor was queried, at sha256 24fb254e…ee34969d. You can verify that hash against the file yourself. That does not make the results good; it makes them un-editable, which is a different and more useful property.

3. Which question was being scored?#

"Accuracy" is at least three questions wearing one word:

• Validity — is this a well-formed, allocated number?
• Line type — is the designated type right?
• Carrier or liveness — is this who serves it, and does it answer?

A vendor scoring only the first and calling it accuracy is not lying, but it is answering the easiest question available. Ask which one the number refers to, and be suspicious when the answer is "all of them" without a breakdown.

4. Did every vendor see its results before publication?#

A benchmark that publishes without giving the measured parties a chance to dispute the method is a press release with a table in it.

№ 001 notifies every vendor and holds a 14-day dispute window before publication. That window exists to catch the case where we ran a query wrong — which is a real risk in any benchmark, including a careful one.

5. Is the vendor running it scored under the same rules?#

If a benchmark exists to sell you something, the sponsor's own column is the one to read hardest.

The only version of this that means anything is the sponsor being scored under the identical protocol, published whether or not the answer flatters them. We have committed to publishing our own results win or lose. You should treat that commitment as worth exactly what it is until the results exist — which is why the page reserves the space rather than filling it early.

6. What does the vendor say it did not measure?#

Every honest benchmark has a boundary. Ours is declared in a public errata file rather than discovered by a reader — including the places where a rule turned out to be weaker than intended after the file was frozen.

A benchmark with no stated limitations is not a benchmark without limitations. It is one that has not told you about them.

What we are not claiming here#

We have not published accuracy numbers, and nothing on this page should be read as one. Until benchmark № 001 publishes its results, the honest position is that we have a frozen method, a stated protocol and a reserved space in the table — and no measured figure to show you.

That is a worse marketing position than a percentage. It is a considerably better one to be checkable on, and it is the only position consistent with an API that names what it did not check on every response.

The short version#

• Ask for the method before the number. An unpublished method makes a percentage a claim about a measurement, not a measurement.
• Rules fixed before results, provably. The one property you cannot verify after the fact unless it was checksummed in advance.
• "Accuracy" is at least three questions. Validity, line type and carrier or liveness are different difficulties — insist on the breakdown.
• A dispute window is a feature. Publishing without letting measured vendors object is a press release with a table in it.
• Read the sponsor's own column hardest. Identical protocol, published win or lose, or the benchmark is an ad.
• No stated limitations means none disclosed, not none present. Look for the errata, not the absence of one.