Statistics

Why Statistics Matters in Data Science

Every dataset is a sample drawn from an unknown reality. Statistics is the rigorous language for reasoning about that gap — telling us when a pattern is real, when it's noise, and how confident we should be in any claim. Models without statistical grounding are just curve-fitting; statistics is what makes data science a science.

• Quantify uncertainty — confidence intervals, standard errors, p-values
• Test hypotheses — separate signal from random variation
• Estimate effects — measure how much X moves Y
• Design experiments — A/B tests, sample size, power analysis
• Validate models — residual analysis, goodness-of-fit, calibration

1. Descriptive Statistics

Summarize the shape of data before modeling it. Always look at the data before running any test.

• Central tendency — mean (sensitive to outliers), median (robust), mode
• Spread — variance, standard deviation, IQR, range
• Shape — skewness (asymmetry), kurtosis (tail-heaviness)
• Position — percentiles, quartiles, z-scores

import numpy as np
import pandas as pd

x = pd.Series([12, 15, 14, 10, 18, 22, 11, 13, 14, 9, 100])  class=class="str">"com"># note the outlier

print(class="str">"mean    :", x.mean())          class=class="str">"com"># pulled up by 100
print(class="str">"median  :", x.median())        class=class="str">"com"># robust
print(class="str">"std     :", x.std(ddof=1))
print(class="str">"IQR     :", x.quantile(0.75) - x.quantile(0.25))
print(class="str">"skew    :", x.skew())
print(class="str">"describe:\n", x.describe())

Mean vs Median — when to use which

• Symmetric, no outliers → mean (more efficient)
• Skewed or has outliers → median (more robust)
• Income, prices, latencies → almost always median + percentiles

2. Probability Distributions

A distribution describes how likely each outcome is. Recognizing the right distribution unlocks the right test.

• Normal (Gaussian) — heights, measurement errors, sample means (CLT)
• Binomial — number of successes in n trials (clicks, conversions)
• Poisson — count of rare events per interval (arrivals, defects)
• Exponential — time between Poisson events (waiting times)
• Log-normal — multiplicative processes (incomes, file sizes)
• Uniform — equal probability across a range

The Central Limit Theorem (CLT)

The sample mean of any distribution (with finite variance) approaches a normal distribution as n grows. This is why so many tests assume normality of the sampling distribution, not the data itself. Rule of thumb: n ≥ 30 is usually enough unless the data is wildly skewed.

3. Inferential Statistics

Use a sample to make claims about the population.

Standard Error & Confidence Intervals

The standard error of the mean is SE = s / √n. A 95% confidence interval for the mean is roughly x̄ ± 1.96 · SE. Interpretation: if we repeated the experiment many times, 95% of such intervals would contain the true mean — not "there's a 95% chance the true mean is here."

Hypothesis Testing — the framework

1. State H₀ (null, "no effect") and H₁ (alternative)
2. Choose a test statistic and significance level α (usually 0.05)
3. Compute the p-value — probability of seeing data this extreme if H₀ were true
4. If p < α → reject H₀; otherwise fail to reject (never "accept" H₀)

Errors

• Type I (α) — false positive: rejecting a true H₀
• Type II (β) — false negative: failing to detect a real effect
• Power = 1 − β — probability of detecting a real effect (aim for ≥ 0.80)

4. Common Statistical Tests

from scipy import stats
import numpy as np

class=class="str">"com"># Two-sample t-test: does a new landing page increase time-on-site?
control = np.array([42, 38, 45, 39, 41, 44, 40, 37, 43, 39])
variant = np.array([48, 52, 47, 50, 49, 53, 46, 51, 49, 50])

t, p = stats.ttest_ind(variant, control, equal_var=False)  class=class="str">"com"># Welchclass="str">'s t-test
print(f"t = {t:.3f}, p = {p:.4f}")

class="com"># Cohen's d (effect size) — magnitude, not just significance
pooled_sd = np.sqrt((variant.var(ddof=1) + control.var(ddof=1)) / 2)
d = (variant.mean() - control.mean()) / pooled_sd
print(fclass="str">"Cohen's d = {d:.2f}")  class=class="str">"com"># 0.2 small, 0.5 medium, 0.8 large

5. P-values: What They Are and Aren't

• Is: P(data this extreme | H₀ is true)
• Is not: P(H₀ is true | data) — that's a Bayesian posterior
• Is not: a measure of effect size — a tiny effect with huge n can have p < 0.001 yet be practically meaningless
• Always pair p-value with effect size (Cohen's d, lift %, odds ratio) and a confidence interval

6. Correlation vs Causation

corr(x, y) ≠ 0 means they move together — it does not mean x causes y. Confounders, reverse causality, and selection bias are everywhere.

• Pearson r — linear, [-1, 1], sensitive to outliers
• Spearman ρ — rank-based, monotonic, robust
• To prove causation — randomized experiments, or causal inference (instrumental variables, diff-in-diff, propensity scores)

7. Regression — Modeling Relationships

Regression estimates how Y changes with X, controlling for other variables. It's the workhorse of statistical analysis.

import statsmodels.api as sm
import pandas as pd

df = pd.read_csv(class="str">"sales.csv")
X = df[[class="str">"ad_spend", class="str">"price", class="str">"is_holiday"]]
X = sm.add_constant(X)  class=class="str">"com"># adds intercept
y = df[class="str">"revenue"]

model = sm.OLS(y, X).fit()
print(model.summary())
class=class="str">"com"># Key things to read:
class=class="str">"com">#   coef         — effect of a 1-unit change in X on Y
class=class="str">"com">#   P>|t|        — p-value for each coefficient
class=class="str">"com">#   R-squared    — % of variance in Y explained
class=class="str">"com">#   [0.025, 0.975] — 95% CI for each coefficient

Assumptions to check (otherwise inference is invalid)

• Linearity — Y is roughly linear in X (plot residuals)
• Independence — observations don't influence each other
• Homoscedasticity — constant residual variance
• Normality of residuals — Q-Q plot, Shapiro test
• No severe multicollinearity — check VIF (< 5 is fine)

8. A/B Testing — Statistics in the Wild

1. Define the metric — primary KPI, guardrail metrics
2. Compute sample size — based on baseline rate, MDE, α, power
3. Randomize — assignment must be independent of user traits
4. Run for full cycles — avoid weekday/seasonality bias; don't peek
5. Analyze — z-test for proportions or t-test for continuous
6. Report — lift %, confidence interval, p-value, practical significance

from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportions_ztest

class=class="str">"com"># Sample size for detecting a 2pp lift over a 10% baseline (80% power, α=0.05)
from statsmodels.stats.proportion import proportion_effectsize
effect = proportion_effectsize(0.12, 0.10)
n = NormalIndPower().solve_power(effect_size=effect, alpha=0.05, power=0.8)
print(fclass="str">"Need ~{int(n)} users per arm")

class=class="str">"com"># After the test
successes = [120, 150]   class=class="str">"com"># control, variant
trials    = [1000, 1000]
z, p = proportions_ztest(successes, trials)
print(fclass="str">"z = {z:.3f}, p = {p:.4f}")

Common A/B Pitfalls

• Peeking — checking results daily inflates false positives. Use sequential tests if you must.
• Multiple comparisons — testing 20 metrics at α=0.05 → ~64% chance of a false positive. Apply Bonferroni or FDR.
• Sample ratio mismatch (SRM) — if 50/50 split is actually 52/48, randomization is broken.
• Novelty / primacy effects — short tests can mislead; let the effect stabilize.

9. Bayesian Thinking

Frequentist stats answer "how surprising is this data under H₀?" Bayesian stats answer "given this data, what do I now believe?" — combining a prior with the likelihood to produce a posterior distribution over the unknown.

P(θ | data) ∝ P(data | θ) · P(θ)

• Posterior — updated belief after seeing data
• Credible interval — direct probability statement: "95% chance θ is in [a, b]"
• Use cases — small samples, sequential testing, incorporating prior knowledge, multi-armed bandits

10. Resampling: Bootstrap & Permutation

When formulas are intractable or assumptions don't hold, simulate. These methods are often the most honest answer.

import numpy as np

rng = np.random.default_rng(42)
data = np.array([4.1, 3.9, 4.5, 3.8, 4.2, 4.0, 4.4, 3.7, 4.3, 4.1])

class=class="str">"com"># Bootstrap 95% CI for the mean — no normality assumption
boot_means = [rng.choice(data, size=len(data), replace=True).mean()
              for _ in range(10_000)]
ci = np.percentile(boot_means, [2.5, 97.5])
print(fclass="str">"Mean = {data.mean():.3f}, 95% bootstrap CI = {ci}")

11. Sampling Bias — the Silent Killer

• Selection bias — sample doesn't reflect the population (survey only online users)
• Survivorship bias — only seeing the winners (only active customers, only published studies)
• Confirmation bias — running tests until you get the result you want
• Simpson's paradox — a trend in groups reverses when groups are combined; always segment

12. Statistical Workflow Checklist

1. Visualize the raw data (histogram, boxplot, scatter)
2. Check assumptions before any test
3. State H₀, H₁, α, and required power before looking at results
4. Report effect size + confidence interval, not just p-value
5. Correct for multiple comparisons
6. Sanity-check with a non-parametric or bootstrap version
7. Communicate uncertainty in plain language to stakeholders

Recommended Resources

• Books — "Practical Statistics for Data Scientists" (Bruce), "Statistical Rethinking" (McElreath), "Think Stats" (Downey)
• Python — scipy.stats, statsmodels, pingouin, pymc
• Practice — re-analyze public datasets, replicate paper results, run your own A/B tests on side projects