Pre-registration · version 1.0 · published 2026-09-29
Pre-registered protocol: a negative-control benchmark for username checkers
Published before any benchmark request is sent and before any result is examined. The text above the amendments line is frozen at publication. The timestamp is this page's publication date together with an independent web-archive snapshot taken on the day of publication; amendments are numbered and dated below.
What this document is for
FootprintIQ runs open-source username checkers (WhatsMyName, Maigret, GoSearch) and labels each hit with a measured per-site observed match rate. An observed match rate is not a false-positive rate: it is the share of production searches on which a checker reported a match for a site, and it says nothing about whether those matches were right. We have said so publicly since September 2026, and we withdrew earlier copy that implied otherwise. This protocol describes the experiment that will produce an actual false-positive measurement, and it is published first so that the analysis cannot be adjusted after the results are in.
What this will and will not establish
The benchmark submits usernames that, to a very high probability, exist on no site. A checker that reports a match on such a username has produced a false positive. The result is therefore a per-site, per-checker false-positive rate on known-absent inputs — equivalently, specificity. It does not measure sensitivity (how many real accounts a checker misses), it does not measure precision on real searches (what share of a real customer's hits are wrong, which depends on the base rate of real accounts), and it does not say anything about whether a matched account belongs to the person searched. Those require labelled positive data, which this benchmark does not contain. No number from this benchmark will be presented as a probability that a given finding is genuine.
1. Hypotheses, fixed in advance
- H1. False-positive rates on known-absent usernames vary between sites by more than an order of magnitude within the same checker.
- H2. For a given site, checkers with different detection rules produce materially different false-positive rates.
- H3. A site's false-positive rate on known-absent inputs is predicted by its observed match rate in production telemetry, with sites at or above the current routing threshold (observed match rate ≥ 0.50) showing false-positive rates above 0.90.
H3 is the one that matters commercially: it tests whether the production baseline, which the product already shows customers, is a reasonable proxy for the quantity customers actually care about. If H3 fails, the product's "elevated baseline" labelling is not doing what it is meant to do, and we will say so.
2. Corpus
Size. 1,000 usernames: five strata of 200.
Strata. Random strings alone would only exercise the code path for implausible input; several sites treat short or dictionary-shaped handles differently.
| Stratum | Shape | Generation | Purpose |
|---|
| A | High-entropy control | 12 lowercase alphanumerics from a CSPRNG | The cleanest "cannot exist" case. |
| B | Common-name shape | Forename + surname + 3–4 digits; names from fixed lists, digits from a CSPRNG | The shape most real searches take. |
| C | Word–number | Dictionary word + 4–6 digits | A common real-handle shape. |
| D | Separator | Two dictionary tokens joined by _ or . plus 2–4 digits | Tests sites that normalise separators. |
| E | Short handle | 6 lowercase alphanumerics from a CSPRNG | Tests length-sensitive sites. |
Generation. Each username is generated by a script committed to the benchmark repository before the first request, with the CSPRNG seed recorded in the run manifest. Strata B–D draw from fixed, committed word and name lists.
Absence check. Every generated username is checked against FootprintIQ's production findings and against the corpus of usernames ever submitted to FootprintIQ; any collision is discarded and regenerated, and the discard count is reported. Absence on the internet at large cannot be verified — this is why the result is described as a false-positive rate on known-absent inputs with the residual risk stated, not as a true false-positive rate.
Per-site eligibility. Each site definition carries character and length constraints. A username that violates a site's constraints is ineligible for that site and is excluded from that site's denominator, not counted as a negative. Eligibility is computed from the site definitions before any request is sent and is recorded in the manifest.
3. Checkers and versions
The three checkers currently run by the product: WhatsMyName, Maigret, GoSearch. Sherlock is not included; the product stopped running it in June 2026.
Before the first request, the run manifest records for each checker: the exact source version (commit hash or release tag), the site-definition file and its hash (for WhatsMyName, wmn-data.json), the runtime version, and the container image digest. The manifest is published with the results. A run against any other versions is a different run and is reported as such.
Site identity. Results are keyed on the site's canonical identifier from its checker's definition file, not on a display name or a string parsed from HTML. This is a deliberate departure from FootprintIQ's production telemetry, whose site key was found in September 2026 to both merge distinct sites and split single ones; H3 is tested only on the subset of sites where the production key can be mapped unambiguously to a canonical id, and the size of that subset is reported.
4. Request procedure
- Each (checker, site, username) triple is attempted at most three times: one initial attempt and up to two retries on transport error or timeout, with a 30-second per-request timeout and exponential backoff (5 s, 25 s).
- Requests run from a fixed egress with a fixed user agent, recorded in the manifest. No residential proxies, no rotating identities.
- The full corpus is run against every site in a randomised order, with a per-site concurrency of 1 and a minimum inter-request interval of 2 seconds per host, so that rate-limiting is the site's decision and not an artefact of our load.
- Every response is stored raw: HTTP status, final URL after redirects, response headers, and the first 64 KB of the body, hashed. Storage is append-only and separate from the production database. Nothing is filtered before storage.
5. Classification, fixed in advance
Each attempted triple receives exactly one label:
- Positive — the checker's own detection logic, unmodified, reports the username as found.
- Negative — the checker's own detection logic reports it as not found.
- Indeterminate — all three attempts ended in transport error or timeout; or the final response is a challenge page (CAPTCHA, JavaScript challenge, WAF block), identified by a fixed list of signatures committed before the run; or the response is a rate-limit status (429, or 403 with a rate-limit signature); or the response redirected off the target host.
The checkers' detection logic is used exactly as shipped. We do not "fix" a checker's rule for a site, even where we can see it is wrong: the benchmark measures the tool as a user of the tool would experience it.
Per-site false-positive rate = positives ÷ (positives + negatives), over eligible usernames. Indeterminates are excluded from the denominator and their count and share are reported per site. A site with fewer than 100 determinate results is reported but flagged as under-powered and is excluded from the H1–H3 tests.
Rates are reported with 95% Wilson intervals. No p-values are reported for H1 and H2, which are descriptive; H3 is tested with Spearman's rank correlation between production observed match rate and benchmark false-positive rate on the mapped subset, with the coefficient and interval reported whatever they are.
6. Repeat schedule
Three full runs, seven days apart, on the same corpus and the same pinned versions. A site's headline rate is the pooled rate across the three runs; the per-run rates are published alongside so that between-run drift is visible. After the third run the pinned versions are unfrozen and the benchmark is re-run against current versions as a separate, later publication.
7. Stopping, exclusion and amendment rules
- A run is abandoned and restarted from the beginning only for infrastructure failure on our side (egress loss, storage failure). An abandoned run is reported as abandoned with its reason.
- No site is excluded after the run for any reason other than the under-powered rule in section 5. In particular, a site is not excluded because its result is embarrassing to the checker, to the site, or to FootprintIQ.
- No stratum, hypothesis, threshold or classification rule in this document changes after the first request is sent. If a defect in the protocol is discovered mid-run, the run completes as specified, is published as specified, and a numbered amendment describing the defect and the corrected protocol is published before any re-run.
8. What will be published
- This document, unchanged, with its archive snapshot.
- The run manifest for each run (versions, hashes, seed, egress, timestamps).
- The corpus, with stratum labels.
- The per-site, per-checker, per-run counts (positive / negative / indeterminate / ineligible) and pooled rates with intervals, as a CSV under an open licence.
- The H1–H3 results, including a null or negative result for H3.
- The raw response store, on request, to researchers, with the site body hashes public.
9. What FootprintIQ will do with the result
If H3 holds, the product's per-site baseline labelling stays as it is and the benchmark false-positive rates are shown alongside the observed match rate, labelled as what they are.
If H3 fails, the product stops describing observed match rate as a baseline for anything and the labelling is redesigned around the benchmark result. That commitment is made here, before the result is known, because the alternative — deciding after the fact which measurement to believe — is the error this protocol exists to prevent.
In neither case will any number from this benchmark be presented to a customer as the probability that a particular finding is theirs.
10. Conflicts of interest
FootprintIQ sells username scans built on these checkers. A result showing the checkers produce few false positives is commercially convenient; a result showing they produce many is not, but is the more likely one on the production telemetry to date. The pre-registration, the raw response store and the open per-site data are the controls on that conflict. The benchmark is run and analysed by FootprintIQ's founder; there is no independent analyst, and this is stated as a limitation rather than dressed up.
11. Timeline
Corpus generation and manifest: on or before 13 October 2026. Run 1: 20 October 2026. Runs 2 and 3: 27 October and 3 November 2026. Results: within 14 days of run 3. If any date slips, the slip and its reason are recorded as an amendment below rather than the date being edited.
Amendments
Version 1.0. The text above this line is not edited after publication. Amendments are numbered and dated here.
None.
Questions about this protocol: [email protected]. The correction notice this page was promised on is at /research.