Validation
An estimate of traffic you can’t see can’t be compared with reality: reality, after all, is what you can’t see. What you can do is build a fake site where everything is known, hide the part without consent from the estimator and check whether it finds it.
How an estimate is checked
A generator simulates every single visitor: they arrive from a channel, accept, decline or ignore the banner, come back a certain number of times, buy or don’t. Two things come out of it:
- an export identical to GA4’s, with only the visits of those who accepted;
- the right answer: the total traffic, which the estimator doesn’t see.
The generator works per visitor and the estimator works on totals, with different formulas: if they used the same ones, the test would be circular. Each scenario is repeated 500 times with different data. What is measured is how often the true value falls within the stated range.
Calibration: is “8 times out of 10” true?
In these tests the estimator’s assumptions are right on average: the true behaviors are drawn from the same ranges the estimator assumes. Here the coverage must hold: about 80 times out of 100 for the “likely” range, 95 for “almost certainly”. With 500 runs, the margin of the measurement itself is about ±4 points on the first number and ±2 on the second.
| Code | Scenario | Sessions | Key events | Transactions | Revenue | Channels: sessions |
|---|---|---|---|---|---|---|
| A1 | Average site: 50,000 sessions, 60% consent given as an estimate | 79 / 95 | 79 / 94 | 81 / 96 | 81 / 95 | 79 / 95 |
| A2 | Tiny channels: 8,000 sessions, channels with 0–5 conversions | 78 / 95 | 80 / 95 | 82 / 97 | 79 / 96 | 77 / 93 |
| A3 | Small site: 2,000 sessions | 76 / 95 | 81 / 94 | 81 / 97 | 75 / 93 | 73 / 91 |
| A4 | Very low consent: 20% | 78 / 95 | 79 / 95 | 80 / 94 | 79 / 94 | 78 / 94 |
| A5 | Very high consent: 92%, known precisely | 80 / 95 | 79 / 95 | 81 / 96 | 81 / 95 | 77 / 94 |
| A6 | Average site with real orders, audience “mixed / I don’t know” | 77 / 95 | 78 / 93 | 79 / 95 | 80 / 94 | 78 / 95 |
It holds in every scenario, including channels with very few conversions and extreme consent rates. The very small site (2,000 sessions) is borderline on revenue and, for single channels, on sessions.
| Code | Scenario | Central estimate error |
|---|---|---|
| A1 | Average site: 50,000 sessions, 60% consent given as an estimate | +1% / +1% |
| A2 | Tiny channels: 8,000 sessions, channels with 0–5 conversions | +0% / +1% |
| A3 | Small site: 2,000 sessions | +1% / +0% |
| A4 | Very low consent: 20% | +2% / +3% |
| A5 | Very high consent: 92%, known precisely | +0% / +0% |
| A6 | Average site with real orders, audience “mixed / I don’t know” | +1% / +0% |
Robustness: what a wrong assumption costs
Here one assumption is deliberately wrong. Coverage drops, and that’s unavoidable: no range protects against a false assumption. The point is to know by how much.
| Code | Scenario | Sessions | Key events | Transactions | Revenue | Channels: sessions |
|---|---|---|---|---|---|---|
| B6 | True “how well they convert” 1.0 instead of 0.4 | 78 / 95 | 54 / 89 | 41 / 79 | 44 / 82 | 79 / 95 |
| B7 | True “how well they convert” 0.15 instead of 0.4 | 79 / 95 | 44 / 88 | 34 / 72 | 35 / 74 | 79 / 95 |
| B8a | True “how well they convert” 1.0, with real orders | 78 / 95 | 100 / 100 | 100 / 100 | 100 / 100 | 79 / 94 |
| B8b | True “how well they convert” 0.15, with real orders | 79 / 95 | 100 / 100 | 100 / 100 | 100 / 100 | 79 / 95 |
| B9 | Consent very different across channels (30% to 85%) | 79 / 94 | 77 / 94 | 76 / 94 | 72 / 91 | 30 / 44 |
| B10a | “Per banner” rate declared “per visitor” | 57 / 96 | 75 / 93 | 71 / 93 | 72 / 93 | 71 / 93 |
| B10b | “Per banner” rate declared correctly | 99 / 100 | 85 / 97 | 89 / 98 | 88 / 98 | 95 / 100 |
| B11a | Rate excluding visitors who ignore the banner, declared complete | 58 / 83 | 77 / 93 | 74 / 92 | 75 / 92 | 64 / 87 |
| B11b | Rate excluding visitors who ignore the banner, declared as such | 94 / 99 | 84 / 97 | 88 / 97 | 86 / 97 | 90 / 99 |
| B12a | Banner change mid-period, declared | 89 / 98 | 82 / 96 | 85 / 96 | 83 / 96 | 85 / 97 |
| B12b | Banner change mid-period, not declared | 87 / 98 | 82 / 96 | 85 / 96 | 83 / 96 | 85 / 98 |
| B14 | Very irregular key events (clustered) | 78 / 95 | 77 / 94 | 81 / 95 | 81 / 95 | 78 / 95 |
| B16a | Consent differs between mobile and desktop, overall rate only | 76 / 94 | 78 / 94 | 80 / 95 | 80 / 94 | 80 / 95 |
| B16d | As B16a, overall rate declared precise | 77 / 94 | 78 / 95 | 81 / 94 | 80 / 94 | 80 / 95 |
| B16b | As B16a, with rates by device | 80 / 96 | 80 / 95 | 80 / 95 | 80 / 94 | 81 / 96 |
| B16c | Consent differs, same visits across devices (control) | 79 / 95 | 79 / 95 | 81 / 95 | 81 / 95 | 80 / 95 |
| Code | Scenario | Central estimate error |
|---|---|---|
| B6 | True “how well they convert” 1.0 instead of 0.4 | +1% / −20% |
| B7 | True “how well they convert” 0.15 instead of 0.4 | +1% / +13% |
| B8a | True “how well they convert” 1.0, with real orders | +1% / −0% |
| B8b | True “how well they convert” 0.15, with real orders | +1% / +0% |
| B9 | Consent very different across channels (30% to 85%) | +0% / +5% |
| B10a | “Per banner” rate declared “per visitor” | +19% / +10% |
| B10b | “Per banner” rate declared correctly | +6% / +4% |
| B11a | Rate excluding visitors who ignore the banner, declared complete | −12% / −5% |
| B11b | Rate excluding visitors who ignore the banner, declared as such | +3% / +2% |
| B12a | Banner change mid-period, declared | +0% / +1% |
| B12b | Banner change mid-period, not declared | +1% / +1% |
| B14 | Very irregular key events (clustered) | +1% / +0% |
| B16a | Consent differs between mobile and desktop, overall rate only | +4% / +3% |
| B16d | As B16a, overall rate declared precise | +4% / +2% |
| B16b | As B16a, with rates by device | +0% / +1% |
| B16c | Consent differs, same visits across devices (control) | +1% / +1% |
What these rows say:
- Wrong “how well they convert” (1.0 or 0.15 instead of 0.4): sessions stay correct, transactions are off by 13–20% and the range contains the true value about one time in three. With real orders the error disappears.
- Consent very different across channels: totals hold, but the session range for a single channel contains the true value only 30 times out of 100.
- Rate declared wrongly: a “per banner” rate declared “per visitor” inflates sessions by 19%; a rate that excludes visitors who ignore the banner, declared complete, reduces them by 12%. Declared correctly, the estimate is correct and cautious again.
- Consent differs between mobile and desktop: with differences like those tested (50% consent on mobile and 75% on desktop, one and a half times the visits on desktop), with the overall rate alone sessions come out about 4% too high, even with the rate declared precise, but coverage changes little (76–77 out of 100 instead of 79); with bigger differences the error grows. With rates by device the error disappears; with a similar number of visits across devices there is no error even without them.
- Undeclared banner change and irregular key events: no harm to the totals.
Ad blockers and real orders
With real orders, the result depends on the share of orders GA4 misses for reasons other than consent, that is, on the audience type you choose.
| Code | Scenario | Sessions | Key events | Transactions | Revenue | Channels: sessions |
|---|---|---|---|---|---|---|
| B13 | Real orders, true other losses 20%, audience “mixed / I don’t know” | 79 / 95 | 49 / 96 | 66 / 100 | 62 / 100 | 79 / 95 |
| B13b | Real orders, true other losses 20%, audience declared “consumer” | 79 / 95 | 23 / 56 | 0 / 32 | 0 / 33 | 79 / 95 |
| B15a | 12% ad blockers who decline more often, without real orders | 86 / 97 | 82 / 96 | 85 / 96 | 84 / 95 | 84 / 97 |
| B15b | 12% ad blockers who decline more often, with real orders | 85 / 97 | 91 / 99 | 100 / 100 | 100 / 100 | 84 / 97 |
| B15c | Technical audience, 35% ad blockers, declared “technical”, with real orders | 76 / 95 | 95 / 100 | 100 / 100 | 100 / 100 | 77 / 95 |
| Code | Scenario | Central estimate error |
|---|---|---|
| B13 | Real orders, true other losses 20%, audience “mixed / I don’t know” | +1% / +14% |
| B13b | Real orders, true other losses 20%, audience declared “consumer” | +1% / +16% |
| B15a | 12% ad blockers who decline more often, without real orders | −2% / −1% |
| B15b | 12% ad blockers who decline more often, with real orders | −2% / −1% |
| B15c | Technical audience, 35% ad blockers, declared “technical”, with real orders | −9% / +1% |
- True share 20%, answer “mixed / I don’t know”: the true value always stays within the “almost certainly” range, but the central estimate of transactions is 14% too high.
- True share 20%, wrong answer “consumer”: the estimate is 16% too high and the true value falls outside the range two times out of three. The audience answer matters.
- Ad blockers that decline more often and partly hide the banner: with real orders, transactions stay correct. Sessions come out slightly below actual traffic (by 2% with 12% of visitors blocking GA, by 9% with 35%): the estimator doesn’t estimate visitors GA4 wouldn’t see even with consent.
What to take away
- When the assumptions are reasonable, the ranges keep their promise.
- Total sessions are the most robust estimate; they depend on the consent rate and how it’s measured.
- Orders and revenue depend on a weak assumption (“how well they convert”): real orders replace it, but bring in the audience question.
- The channel breakdown should be read as a pointer, not a measurement.
How to rerun the tests
The generator, scenarios and results are in the publicly available source code, in the GitHub repository (opens in a new tab). To rerun the tests, download the repository, install the dependencies with npm ci and run npm run validazione: it reruns everything from scratch (about half an hour) and regenerates both the tables on this page and the VALIDAZIONE.md file. An automated test also checks that the calculation engine always returns the same numbers, digit for digit, for the same data and assumptions.