Customize this training

Get a certificate — $34.29$12

Statistics and A/B Testing for Data Analysts

Statistics and A/B Testing for Data Analysts

Describe data, build and read confidence intervals, run hypothesis tests without misusing p-values, design A/B tests with the right sample size, and report results honestly.

What you will learn

  • Summarise a dataset with the right measures of centre and spread and spot skew and outliers before any test.
  • Explain sampling variation, standard error and the main sources of sampling bias.
  • Calculate and interpret confidence intervals for a mean, a proportion and a difference.
  • Choose and run a hypothesis test and interpret the p-value, errors and power correctly.
  • Distinguish correlation from causation and read a simple regression, including its limits.
  • Design an A/B test: unit, metrics, randomisation, sample size, duration and checks such as sample ratio mismatch.
  • Report experiment results honestly: effect size, interval, multiple comparisons, peeking and a clear decision.

Modules

  1. Describing data before testing it
  2. Samples, distributions and standard error
  3. Confidence intervals
  4. Hypothesis tests and p-values
  5. Correlation, causation and simple regression
  6. Designing an A/B test
  7. Reading and reporting results honestly

Course facts

Estimated study time
about 60 minutes
Final assessment
42 questions · Pass mark 80%
Available in
English, Français, العربية
Price
Free. Optional verifiable certificate: $12
Last updated
Published by
AMAADOR ACADEMY

Who this course is for

Junior and working data analysts, BI and product analysts, marketers, operations and HR analysts, and anyone who has to judge whether a difference in the numbers is real. Basic spreadsheet skills are enough; the code examples use Python and R but every idea also works in Excel.

Prerequisites

School-level arithmetic and percentages. Some experience with a spreadsheet or a data tool helps. No programming is required.

Course content

1. Describing data before testing it

Every statistical choice starts with the type of variable. Categorical variables are labels (country, plan, device). Numeric variables are quantities (order value, load time). A yes/no outcome such as "converted" is categorical, and its average is a proportion. Mixing the types up is the most common reason an analyst picks the wrong test later.

To describe the centre of a numeric variable you have three tools. The mean adds everything up and divides by the count. The median is the middle value once the data are sorted. The mode is the most frequent value.

Take five page-load times in seconds: 12, 13, 14, 15 and 90. The mean is 144 / 5 = 28.8. The median is 14. One slow session dragged the mean to a value that describes nobody. The median still tells the truth about a typical visit.

QuestionUse
Typical value, data roughly symmetricMean
Typical value, skewed data (income, time, order value)Median
Most common category or valueMode
Total impact (revenue, hours)Mean times count

2. Samples, distributions and standard error

You almost never see the whole population. You see a sample, and a different sample would give a slightly different answer. That wobble is sampling variation, and all of inference is about measuring it.

The standard error (SE) is the typical size of that wobble for a statistic such as a mean or a proportion. Do not confuse it with the standard deviation, which describes spread among individuals.

StatisticStandard errorExample
MeanSD / square root of nSD 20, n = 100 gives 2
Proportionsquare root of p(1 - p) / np = 0.5, n = 400 gives 0.025

Two consequences matter every week. First, the SE shrinks with the square root of n: to halve your uncertainty you need four times the data. Second, a small sample can show a big difference by luck alone.

The central limit theorem explains why this toolbox works: with enough observations, the distribution of sample means is close to a normal (bell) curve even when the raw data are skewed. About 95% of sample means fall within roughly two SEs of the true mean. For very skewed metrics such as revenue per user, "enough" can mean thousands of users, so check with a bootstrap (resampling your own data) when in doubt.

3. Confidence intervals

A single number such as "13% conversion" hides how sure you are. A confidence interval (CI) adds a range: the values of the true figure that are reasonably compatible with your data. The general recipe is estimate ± critical value × standard error, where the critical value is about 1.96 for a 95% interval based on the normal curve.

Proportion. 52 of 400 visitors signed up, so p = 0.13. SE = square root of 0.13 × 0.87 / 400 = 0.0168. The margin is 1.96 × 0.0168 = 0.033. The 95% CI is 9.7% to 16.3%. Same data, honest answer: "between about 10% and 16%".

Mean. 25 support calls last 42.0 minutes on average with an SD of 10. SE = 10 / 5 = 2. With 24 degrees of freedom the t critical value is 2.064, so the margin is 4.1 minutes and the CI is 37.9 to 46.1 minutes. For small samples use the t distribution, not 1.96.

Say

  • "We are 95% confident the true rate lies between 9.7% and 16.3%."
  • "If we repeated the study many times, about 95% of such intervals would contain the true value."

4. Hypothesis tests and p-values

A hypothesis test asks one narrow question: if there were truly no difference, how surprising would my data be? The assumption of no difference is the null hypothesis (H0). The claim you hope to support is the alternative (H1).

The p-value is the probability of seeing a result at least as extreme as yours if H0 were true. Take A at 5.0% (200 of 4,000) and B at 6.0% (240 of 4,000). The pooled rate is 5.5%, the SE under H0 is 0.51 points, so z = 1.0 / 0.51 = 1.96 and the two-sided p-value is about 0.0498. A result at p below the chosen significance level alpha (usually 0.05) is called statistically significant.

Two errors are possible:

H0 is true (no effect)H0 is false (real effect)
You reject H0Type I error, false positive, probability alphaCorrect detection
You keep H0CorrectType II error, miss, probability beta

Power = 1 - beta, the chance of detecting a real effect of a given size. The usual target is 80%. With 4,000 users per group and a true lift of 1 point, power is only about 50%: a coin flip whether you will see it. You will fix this with sample size in module 6.

5. Correlation, causation and simple regression

The correlation coefficient r runs from -1 to +1 and measures how closely two numeric variables move along a straight line. Near 0 means no linear relationship; near +1 or -1 means a strong one. Pearson's r (scipy.stats.pearsonr, cor() in R, CORREL in Excel) assumes a roughly linear link and is sensitive to outliers; Spearman's rank correlation (scipy.stats.spearmanr) copes better with skew and curves.

When A and B are correlated, there are four explanations: A causes B, B causes A, a third factor C (a confounder) drives both, or it is chance. Ice-cream sales and drownings rise together in summer because of the weather, not because ice cream is dangerous.

Four explanations for one correlation. Data alone rarely tell you which.

Case: in autumn 1973 the University of California, Berkeley admitted about 44% of 8,442 male applicants and about 35% of 4,321 female applicants, which looked like bias against women. Bickel, Hammel and O'Connell (Science, 1975) split the data by department: women had applied more to departments that admit few applicants, and within departments the gap was small or reversed. The aggregate pattern was reversed by a hidden grouping variable. This is Simpson's paradox.

6. Designing an A/B test

An A/B test (a randomised controlled experiment) splits users at random into a control (A) and a variant (B), shows each a different experience, and compares an outcome. Because chance decides who sees what, the groups are alike on everything except the change, so a difference can be read as an effect of the change. This is the only everyday tool that gives you causation.

  1. Write the hypothesis — "Showing delivery cost earlier will raise checkout completion for new visitors".
  2. Choose the randomisation unit — usually the user, not the page view, so one person never sees both versions. Keep it stable across visits.
  3. Fix the primary metric — one number that decides the test, such as checkout completion rate.
  4. Add guardrail metrics — things that must not get worse: page speed, refund rate, error rate, revenue per user.
  5. Choose alpha, power and the smallest effect worth detecting and write it all down before launch.
Metric typeRoleExample
PrimaryDecides ship or notCheckout completion rate
SecondaryExplains whyAdd-to-cart rate
GuardrailProtects against harmSupport tickets, load time

7. Reading and reporting results honestly

Each test at alpha = 0.05 has a 5% chance of a false alarm when nothing is going on. Run many and the alarms add up. With 20 independent metrics and no real effect, the chance of at least one "significant" result is 1 - 0.95^20, which is about 64%. Slice the same experiment by country, device and plan and you can almost always find a winner.

Typical case: a test of a new homepage shows no lift overall. The analyst then checks 12 segments and reports that "returning visitors on Android in Spain" improved with p = 0.03. With 12 looks, one or two such results are expected by chance. The honest label is "exploratory hypothesis to retest".

The remedies are simple.

  • Pre-register one primary metric. Everything else is secondary or exploratory, and you say so.
  • Correct for many tests when you must. Bonferroni divides alpha by the number of tests (0.05 / 20 = 0.0025); it is simple and strict. Holm is uniformly more powerful. The Benjamini-Hochberg method controls the false discovery rate and suits large screens. In Python: statsmodels.stats.multitest.multipletests or scipy.stats.false_discovery_control; in R: p.adjust().
  • Replicate any surprising segment finding with a fresh test.

The full lesson continues in the interactive course on this page.

Sources and references

Educational course on statistical practice. It is not professional advice for regulated work such as clinical trials, financial risk models or legal evidence, where a qualified statistician and the applicable rules must be used. Tool names and functions are described as documented by their publishers in 2026; check the current documentation for your version. Worked examples use rounded, illustrative numbers.

All courses · AI & Data - Free courses, certificate from $7 | AMAADOR ACADEMY