Data Analysis — Free Sample

1. Your team's onboarding email gets a 14% click-through rate among users who clicked the welcome banner — substantially higher than the 6% rate among the general user base. Marketing concludes the email is highly effective. What's the most-defensible critique of this conclusion?

The comparison conditions on a downstream behavior (clicked the welcome banner), creating selection bias — engaged users who clicked the banner are also more likely to engage with the email regardless of email quality. The right comparison is randomized assignment of the email itself, not conditioning on banner clickers. The 14% rate is statistically significant given the sample size, so the conclusion is supported — selection bias is a theoretical concern but not a practical one here. The email is probably effective for engaged users specifically; the conclusion needs narrower scoping but isn't wrong. Click-through rate is the wrong metric; conversion rate or revenue per user would be more meaningful.

2. An A/B test runs for 4 days and shows the variant beating control by 3% with p=0.03. Marketing wants to ship the variant. The team's standard practice is 14-day tests. What's the most-aligned-with-good-practice take?

Don't ship yet. Early-positive results are a known artifact of optional-stopping and noise; the 14-day-test convention exists because day-of-week effects, novelty effects, and statistical fluctuation produce false positives at short durations. Either run the full 14 days or pre-register a shorter-duration analysis with appropriate alpha adjustment. Ship — p=0.03 clears the team's significance threshold and the lift is meaningful; waiting risks losing the gain. Run a sequential analysis with appropriate stopping rules to make the early-shipping decision defensible. Cancel the test and redesign — 4 days isn't enough data regardless of significance level.

3. An analysis shows that customers who complete the onboarding tutorial have 40% higher 30-day retention than customers who skip it. The team concludes the onboarding tutorial improves retention. What's the most-defensible critique?

Tutorial completion is correlated with motivation, intent-to-use, and time-investment — all of which independently predict retention. Without a randomized intervention, the 40% gap is consistent with selection (motivated users both complete tutorials and retain) rather than causation (tutorials cause retention). The 40% gap is too large to be selection alone; some causal effect from the tutorial is plausible even if the magnitude is uncertain. The conclusion is supported by the data; the team should ship a tutorial-completion-mandatory experiment to maximize retention. The analysis should control for user segment and acquisition channel; with appropriate controls, the causal effect can be estimated.

4. You need to communicate the results of an analysis showing that revenue grew 15% year-over-year, but the growth came almost entirely from one large customer's expansion (the rest of the customer base shrank slightly). Which presentation choice is most-aligned with good analytical practice?

Lead with the headline 15% growth, then break down the composition (one customer driving most of it; underlying base shrinking). Show both the aggregate and decomposed views so stakeholders can see the headline AND the underlying pattern. Lead with the underlying-base shrinkage as the primary finding; the 15% headline is misleading. Lead with the 15% headline; the composition detail is too granular for the executive audience and risks confusing the message. Don't present the 15% number; concentration in one customer makes the aggregate meaningless.

5. A dashboard shows weekly active users (WAU) trending up 8% month-over-month, but daily active users (DAU) are flat. Stakeholders are asking what's going on. Which interpretation is most-likely-correct?

User engagement frequency is decreasing — fewer users are returning daily, but those who do return are spread across more weeks. The WAU/DAU ratio (a 'stickiness' proxy) is decreasing, which usually indicates engagement-quality decline even when WAU appears healthy. The product is acquiring more occasional users — WAU growth without DAU growth means the user base is broadening but not deepening. This is a viable growth pattern depending on the product. There's a measurement issue with one of the metrics; DAU and WAU should move together. Investigate the instrumentation. The product is healthy — WAU is the primary engagement metric and 8% month-over-month growth is strong; DAU flatness is noise.

Your directional indicator

This 5-question sample gives a rough indication only. For a valid, verified Skills Passport credential, take the full 40-question assessment — it takes about 30 minutes.

Coming soon

Full assessment launching soon