Quantified Fake‑Door: Exact Sample Sizes & Decision Rules for No‑Code Prebuild Tests

A fake‑door test reduces a product decision to a binary signal: someone saw the CTA and either converted (clicked, signed up, paid intent) or didn’t. That makes the primary metric a proportion (conversion rate). Design your test so the observed event maps to the future value you care about — e.g., a paid intent click is a stronger signal than an email capture.

Before you run any numbers, write a one‑line decision rule: “If conversion ≥ X% with N users and sustained across our two priority segments, we build MVP A; otherwise we kill it.” That rule forces you to quantify business tradeoffs (cost to build, expected LTV, acceptable risk) and select a meaningful Minimum Detectable Effect (MDE).

For a binary outcome compare the observed conversion rate to your baseline (or compare two buckets) using the standard two‑proportion z‑test sample‑size formula. For each variant the per‑group sample size n is approximately: n = ((Z_{1−α/2}√(2p(1−p)) + Z_{1−β}√(p1(1−p1)+p2(1−p2)))^2) / (p1−p2)^2. In practice most teams specify baseline p (current conversion), desired MDE (absolute or relative), α (commonly 0.05) and power 1−β (commonly 0.8).

If that formula looks heavy, use any reputable online calculator that implements the two‑proportion test (Evan Miller style calculators, Statsig, or other sample‑size tools). Plug in your baseline and the smallest uplift you’d act on (your MDE) — this gives the minimal per‑variant sample. If your traffic can’t reach that sample in a reasonable time, either increase the MDE you’ll accept or reframe the test (e.g., run a more targeted paid campaign to accelerate traffic).

Statistical acceptance alone (p < 0.05) is not enough. Use a two‑part rule: (A) a statistical test that meets your α/power requirements, and (B) a business threshold that maps to value. Example rule: “We require at least 80% power to detect our MDE, and observed conversion ≥ target conversion (baseline × multiplier) across both priority segments — otherwise fail.” That prevents small but statistically significant lifts with no business impact from triggering builds.

Add operational constraints: require the effect to persist over a holdout window (e.g., at least one full acquisition cycle or 7–14 days) and check primary segments separately (top acquisition channel and target persona). If the signal is statistically significant overall but driven by one fringe segment, don’t build for the entire market.

Segment early and pre‑register. Define 2–4 priority segments before the test (e.g., organic vs paid, power users vs new users, country A vs B). Treat segment checks like mini decision gates: if overall significance passes but fails for both priority segments, downgrade the signal and require a follow‑up test targeted to the winning segment.

Adjust for multiple comparisons: every extra segment or metric you check inflates false positive risk. Use Bonferroni or Benjamini‑Hochberg corrections if you perform many independent tests, or focus on a small, pre‑specified set of segments to keep statistical interpretation simple.

Instrument each fake‑door using a strict UTM scheme and a single canonical KPI so results are unambiguous. Example UTM pattern: utm_source=fakedoor&utm_medium=banner|email|paid&utm_campaign=feature‑name_v1&utm_term=segment. Capture the canonical KPI (e.g., click‑to‑intent rate or paid intent actions) and a secondary KPI for quality (e.g., email open, trial activation).

Report a one‑page results snapshot for each test showing: sample sizes by variant and segment, raw conversions, conversion rates with 95% CIs, p‑value (or posterior probability if using Bayesian), business value mapping (expected customers × LTV), and the final green/amber/red verdict per the pre‑registered decision rule. Store templates and past results in a lightweight tracker (AppWispr users can keep these templates in their analysis folder).