Skip to content
ENIT

Validation

An estimate of traffic you can’t see can’t be compared with reality: reality, after all, is what you can’t see. What you can do is build a fake site where everything is known, hide the part without consent from the estimator and check whether it finds it.

How an estimate is checked

A generator simulates every single visitor: they arrive from a channel, accept, decline or ignore the banner, come back a certain number of times, buy or don’t. Two things come out of it:

  • an export identical to GA4’s, with only the visits of those who accepted;
  • the right answer: the total traffic, which the estimator doesn’t see.

The generator works per visitor and the estimator works on totals, with different formulas: if they used the same ones, the test would be circular. Each scenario is repeated 500 times with different data. What is measured is how often the true value falls within the stated range.

Calibration: is “8 times out of 10” true?

In these tests the estimator’s assumptions are right on average: the true behaviors are drawn from the same ranges the estimator assumes. Here the coverage must hold: about 80 times out of 100 for the “likely” range, 95 for “almost certainly”. With 500 runs, the margin of the measurement itself is about ±4 points on the first number and ±2 on the second.

How many times out of 100 the true value falls in the “8 out of 10” / “19 out of 20” range. It should be close to 80 / 95. 500 runs per scenario, 1000 simulations per estimate.
CodeScenarioSessionsKey eventsTransactionsRevenueChannels: sessions
A1Average site: 50,000 sessions, 60% consent given as an estimate79 / 9579 / 9481 / 9681 / 9579 / 95
A2Tiny channels: 8,000 sessions, channels with 0–5 conversions78 / 9580 / 9582 / 9779 / 9677 / 93
A3Small site: 2,000 sessions76 / 9581 / 9481 / 9775 / 9373 / 91
A4Very low consent: 20%78 / 9579 / 9580 / 9479 / 9478 / 94
A5Very high consent: 92%, known precisely80 / 9579 / 9581 / 9681 / 9577 / 94
A6Average site with real orders, audience “mixed / I don’t know”77 / 9578 / 9379 / 9580 / 9478 / 95

It holds in every scenario, including channels with very few conversions and extreme consent rates. The very small site (2,000 sessions) is borderline on revenue and, for single channels, on sessions.

Median error of the central estimate on the total: sessions / transactions. 500 runs per scenario, 1000 simulations per estimate.
CodeScenarioCentral estimate error
A1Average site: 50,000 sessions, 60% consent given as an estimate+1% / +1%
A2Tiny channels: 8,000 sessions, channels with 0–5 conversions+0% / +1%
A3Small site: 2,000 sessions+1% / +0%
A4Very low consent: 20%+2% / +3%
A5Very high consent: 92%, known precisely+0% / +0%
A6Average site with real orders, audience “mixed / I don’t know”+1% / +0%

Robustness: what a wrong assumption costs

Here one assumption is deliberately wrong. Coverage drops, and that’s unavoidable: no range protects against a false assumption. The point is to know by how much.

How many times out of 100 the true value falls in the “8 out of 10” / “19 out of 20” range. It should be close to 80 / 95. 500 runs per scenario, 1000 simulations per estimate.
CodeScenarioSessionsKey eventsTransactionsRevenueChannels: sessions
B6True “how well they convert” 1.0 instead of 0.478 / 9554 / 8941 / 7944 / 8279 / 95
B7True “how well they convert” 0.15 instead of 0.479 / 9544 / 8834 / 7235 / 7479 / 95
B8aTrue “how well they convert” 1.0, with real orders78 / 95100 / 100100 / 100100 / 10079 / 94
B8bTrue “how well they convert” 0.15, with real orders79 / 95100 / 100100 / 100100 / 10079 / 95
B9Consent very different across channels (30% to 85%)79 / 9477 / 9476 / 9472 / 9130 / 44
B10a“Per banner” rate declared “per visitor”57 / 9675 / 9371 / 9372 / 9371 / 93
B10b“Per banner” rate declared correctly99 / 10085 / 9789 / 9888 / 9895 / 100
B11aRate excluding visitors who ignore the banner, declared complete58 / 8377 / 9374 / 9275 / 9264 / 87
B11bRate excluding visitors who ignore the banner, declared as such94 / 9984 / 9788 / 9786 / 9790 / 99
B12aBanner change mid-period, declared89 / 9882 / 9685 / 9683 / 9685 / 97
B12bBanner change mid-period, not declared87 / 9882 / 9685 / 9683 / 9685 / 98
B14Very irregular key events (clustered)78 / 9577 / 9481 / 9581 / 9578 / 95
B16aConsent differs between mobile and desktop, overall rate only76 / 9478 / 9480 / 9580 / 9480 / 95
B16dAs B16a, overall rate declared precise77 / 9478 / 9581 / 9480 / 9480 / 95
B16bAs B16a, with rates by device80 / 9680 / 9580 / 9580 / 9481 / 96
B16cConsent differs, same visits across devices (control)79 / 9579 / 9581 / 9581 / 9580 / 95
Median error of the central estimate on the total: sessions / transactions. 500 runs per scenario, 1000 simulations per estimate.
CodeScenarioCentral estimate error
B6True “how well they convert” 1.0 instead of 0.4+1% / −20%
B7True “how well they convert” 0.15 instead of 0.4+1% / +13%
B8aTrue “how well they convert” 1.0, with real orders+1% / −0%
B8bTrue “how well they convert” 0.15, with real orders+1% / +0%
B9Consent very different across channels (30% to 85%)+0% / +5%
B10a“Per banner” rate declared “per visitor”+19% / +10%
B10b“Per banner” rate declared correctly+6% / +4%
B11aRate excluding visitors who ignore the banner, declared complete−12% / −5%
B11bRate excluding visitors who ignore the banner, declared as such+3% / +2%
B12aBanner change mid-period, declared+0% / +1%
B12bBanner change mid-period, not declared+1% / +1%
B14Very irregular key events (clustered)+1% / +0%
B16aConsent differs between mobile and desktop, overall rate only+4% / +3%
B16dAs B16a, overall rate declared precise+4% / +2%
B16bAs B16a, with rates by device+0% / +1%
B16cConsent differs, same visits across devices (control)+1% / +1%

What these rows say:

  • Wrong “how well they convert” (1.0 or 0.15 instead of 0.4): sessions stay correct, transactions are off by 13–20% and the range contains the true value about one time in three. With real orders the error disappears.
  • Consent very different across channels: totals hold, but the session range for a single channel contains the true value only 30 times out of 100.
  • Rate declared wrongly: a “per banner” rate declared “per visitor” inflates sessions by 19%; a rate that excludes visitors who ignore the banner, declared complete, reduces them by 12%. Declared correctly, the estimate is correct and cautious again.
  • Consent differs between mobile and desktop: with differences like those tested (50% consent on mobile and 75% on desktop, one and a half times the visits on desktop), with the overall rate alone sessions come out about 4% too high, even with the rate declared precise, but coverage changes little (76–77 out of 100 instead of 79); with bigger differences the error grows. With rates by device the error disappears; with a similar number of visits across devices there is no error even without them.
  • Undeclared banner change and irregular key events: no harm to the totals.

Ad blockers and real orders

With real orders, the result depends on the share of orders GA4 misses for reasons other than consent, that is, on the audience type you choose.

How many times out of 100 the true value falls in the “8 out of 10” / “19 out of 20” range. It should be close to 80 / 95. 500 runs per scenario, 1000 simulations per estimate.
CodeScenarioSessionsKey eventsTransactionsRevenueChannels: sessions
B13Real orders, true other losses 20%, audience “mixed / I don’t know”79 / 9549 / 9666 / 10062 / 10079 / 95
B13bReal orders, true other losses 20%, audience declared “consumer”79 / 9523 / 560 / 320 / 3379 / 95
B15a12% ad blockers who decline more often, without real orders86 / 9782 / 9685 / 9684 / 9584 / 97
B15b12% ad blockers who decline more often, with real orders85 / 9791 / 99100 / 100100 / 10084 / 97
B15cTechnical audience, 35% ad blockers, declared “technical”, with real orders76 / 9595 / 100100 / 100100 / 10077 / 95
Median error of the central estimate on the total: sessions / transactions. 500 runs per scenario, 1000 simulations per estimate.
CodeScenarioCentral estimate error
B13Real orders, true other losses 20%, audience “mixed / I don’t know”+1% / +14%
B13bReal orders, true other losses 20%, audience declared “consumer”+1% / +16%
B15a12% ad blockers who decline more often, without real orders−2% / −1%
B15b12% ad blockers who decline more often, with real orders−2% / −1%
B15cTechnical audience, 35% ad blockers, declared “technical”, with real orders−9% / +1%
  • True share 20%, answer “mixed / I don’t know”: the true value always stays within the “almost certainly” range, but the central estimate of transactions is 14% too high.
  • True share 20%, wrong answer “consumer”: the estimate is 16% too high and the true value falls outside the range two times out of three. The audience answer matters.
  • Ad blockers that decline more often and partly hide the banner: with real orders, transactions stay correct. Sessions come out slightly below actual traffic (by 2% with 12% of visitors blocking GA, by 9% with 35%): the estimator doesn’t estimate visitors GA4 wouldn’t see even with consent.

What to take away

  1. When the assumptions are reasonable, the ranges keep their promise.
  2. Total sessions are the most robust estimate; they depend on the consent rate and how it’s measured.
  3. Orders and revenue depend on a weak assumption (“how well they convert”): real orders replace it, but bring in the audience question.
  4. The channel breakdown should be read as a pointer, not a measurement.

How to rerun the tests

The generator, scenarios and results are in the publicly available source code, in the GitHub repository (opens in a new tab). To rerun the tests, download the repository, install the dependencies with npm ci and run npm run validazione: it reruns everything from scratch (about half an hour) and regenerates both the tables on this page and the VALIDAZIONE.md file. An automated test also checks that the calculation engine always returns the same numbers, digit for digit, for the same data and assumptions.

Start