Signal Budget: A Privacy‑First Telemetry Plan for No‑Backend Micro‑Experiments

Micro‑experiments only need a handful of well‑named, queryable events. Aim for 6–10 event types across acquisition, engagement, and outcome (for example: experiment_impression, experiment_click, signup_started, signup_completed, payment_attempted, demo_requested). Use a consistent naming prefix (experiment_) so you can filter quickly in GA4 or BigQuery.

Record only the parameters that are necessary for analysis. For most price/demo experiments that means: experiment_id, cohort (variant), timestamp (event_date), and a coarse outcome flag (success=true/false). Avoid collecting free‑text or identifiable fields. When you must collect an identifier for deduplication, use a hashed, short‑lived client id and limit retention to the window of the micro‑experiment.

Bullets:

- Keep event names queryable and prefixed (e.g., experiment_signup_completed). - Only surface parameters you will actually analyze; drop raw URLs, free text, and PII. - Use hashed ephemeral IDs for dedupe when necessary; prefer aggregated counts instead of user‑level storage.

Decide sampling upfront and apply it deterministically on the client (e.g., hash(user_id) % 100 < 10 for 10% sampling). For early micro‑experiments, a 5–20% uniform sample is often sufficient to observe directional signals while containing event volume and cost. If the experiment targets low‑volume audiences, raise the sample to preserve detectability.

Retention should match your decision horizon. For a 7‑day micro‑experiment, keep raw events for 7–14 days and roll up daily aggregates afterward. This minimizes storage and privacy footprint while preserving the ability to re‑run queries across the test window. If you export to BigQuery, be mindful of daily table size and query cost — limiting raw retention reduces repeated scanning.

Bullets:

- Use deterministic client sampling to keep variant balance. - For 7‑day experiments: raw event retention 7–14 days, aggregated metrics retained longer. - Prefer uniform sampling; avoid complex stratified sampling unless you need it.

Start with simple, low‑cost aggregation queries that operate on the exported GA4 events table. Use event_name and event_params to count conversions by variant and date. Example to count daily signups per variant (replace dataset.table with your GA4 export): SELECT PARSE_DATE('%Y%m%d', event_date) AS date, (SELECT value.string_value FROM UNNEST(event_params) WHERE key='variant') AS variant, COUNTIF(event_name='signup_completed') AS signups, COUNT(DISTINCT user_pseudo_id) AS unique_users FROM `project.dataset.events_*` WHERE _TABLE_SUFFIX BETWEEN '20260101' AND '20260107' GROUP BY date, variant ORDER BY date, variant;

You can compute conversion rates and simple confidence intervals without pulling user‑level exports by aggregating counts then applying standard binomial CI formulas in SQL or in your notebook. When using GA4’s BigQuery export, follow schema guides to select event_params safely (avoid heavy joins on repeated fields). The GA Developers BigQuery queries and Google Cloud docs are a useful reference for schema and transformations.

Bullets:

- Replace table names and date ranges in the query above for your export. - Aggregate first, then compute rates — avoids scanning user‑level fields repeatedly. - Use COUNTIF and COUNT(DISTINCT user_pseudo_id) for quick conversion metrics.

For very small, rapid tests (7 days), treat decisions as directional rather than definitive. Use simple thresholds to decide whether to iterate, pause, or scale: - If your variant shows a ≥2× effect on primary outcome with at least 20 absolute events (e.g., signups ≥20) in 7 days, treat it as a strong signal to iterate or scale. - If you observe a consistent 10–30% relative lift with 50+ events and the lift is stable across days, consider prolonged testing for statistical confidence. - If event counts are below 10–20 for primary outcomes, the experiment is inconclusive — either expand the audience or accept the early stop.

For founder workflows, use this 7‑day decision rubric: Day 1–2: sanity checks (event flow, sampling balance). Day 3–5: directional read (daily aggregation, compare variant ratios). Day 6: evaluate counts vs thresholds above. Day 7: decide — scale/iterate/pivot based on threshold outcomes. If you need formal hypothesis testing, compute required sample sizes using standard formulas for proportions or means (NIST and classical power calculations are reliable references).

Bullets:

- Strong signal: ≥2× effect and ≥20 events — iterate/scale. - Consider confirmatory testing if 50+ events and 10–30% lift. - Inconclusive if <20 events — expand audience or collect longer.

Apply privacy‑first defaults: collect minimal parameters, hash or avoid identifiers, apply short retention, and document the experiment’s purpose and retention policy. Where possible, use local randomized responses or client‑side aggregation to reduce exposure. For organizations that need stronger guarantees, consult differential privacy guidance — NIST and W3C provide practical evaluation checklists for when to consider DP and how to reason about privacy budgets.

Operationally, maintain an experiment registry (experiment_id, owners, start_date, retention_end_date, sample_rate). Link this registry to your telemetry so every event can be traced to an experiment and a documented deletion time. That registry plus short raw retention reduces compliance risk and keeps BigQuery costs predictable.

Bullets:

- Minimal parameters, hashed ephemeral ids, short retention (7–14 days raw). - Maintain an experiment registry and enforce deletion/aggregation windows. - Consult DP guidance before adding additional user‑level signals.