Data Science

What is Data Science?

Data Science is the discipline of turning raw data into actionable knowledge. It sits at the intersection of statistics, computer science, and domain expertise — using mathematical rigor and engineering practice to answer questions, surface patterns, and build predictive systems.

• Descriptive — what happened? (dashboards, reports)
• Diagnostic — why did it happen? (root-cause, segmentation)
• Predictive — what will happen? (forecasting, ML models)
• Prescriptive — what should we do? (optimization, A/B testing)

The Data Science Lifecycle

1. Problem Framing — translate a business question into a measurable hypothesis with a success metric.
2. Data Acquisition — pull from databases (SQL), APIs, files, or streaming sources.
3. Data Cleaning — handle nulls, duplicates, outliers, and type coercion. Often 60–80% of the work.
4. Exploratory Data Analysis (EDA) — distributions, correlations, segment behavior, anomalies.
5. Feature Engineering — derive predictive signals (ratios, lags, encodings, embeddings).
6. Modeling — statistical inference or machine learning, validated with cross-validation.
7. Communication — visualize, narrate, and recommend a decision to stakeholders.
8. Deployment & Monitoring — ship the model/insight and watch for drift.

Core Skills

1. Statistics & Probability

• Descriptive statistics: mean, median, variance, percentiles
• Distributions: Normal, Binomial, Poisson, Exponential
• Hypothesis testing: t-test, chi-square, ANOVA, p-values
• Confidence intervals & bootstrapping
• Bayesian reasoning & conditional probability
• A/B testing — power analysis, sample size, MDE

2. Programming

• Python — pandas, NumPy, scikit-learn, matplotlib, seaborn
• SQL — joins, window functions, CTEs for data extraction
• R — statistical computing & tidyverse (optional but common in research)
• Version control with Git, reproducible notebooks (Jupyter, Quarto)

3. Data Wrangling Example

import pandas as pd

class=class="str">"com"># Load and inspect
df = pd.read_csv(class="str">"sales.csv")
print(df.shape, df.dtypes)
print(df.isna().sum())

class=class="str">"com"># Clean
df = df.drop_duplicates()
df[class="str">"order_date"] = pd.to_datetime(df[class="str">"order_date"])
df[class="str">"revenue"] = df[class="str">"revenue"].fillna(0)

class=class="str">"com"># Feature engineering
df[class="str">"month"] = df[class="str">"order_date"].dt.to_period(class="str">"M")
df[class="str">"aov"] = df[class="str">"revenue"] / df[class="str">"units"].replace(0, pd.NA)

class=class="str">"com"># Aggregate insight
monthly = (
    df.groupby([class="str">"month", class="str">"region"], as_index=False)
      .agg(revenue=(class="str">"revenue", class="str">"sum"),
           orders=(class="str">"order_id", class="str">"nunique"),
           aov=(class="str">"aov", class="str">"mean"))
)
print(monthly.head())

4. Visualization & Storytelling

• Choose the right chart: bar (compare), line (trend), scatter (relationship)
• Reduce ink — remove gridlines, redundant legends, 3D effects
• Lead with the insight, support with the chart, end with the recommendation
• Tools: matplotlib, seaborn, Plotly, Power BI, Tableau

Statistical Foundations in Practice

from scipy import stats
import numpy as np

control = np.array([12.1, 11.8, 12.5, 12.0, 11.9, 12.3])
variant = np.array([12.9, 13.1, 12.7, 13.4, 12.8, 13.0])

class=class="str">"com"># Two-sample t-test
t_stat, p_value = stats.ttest_ind(variant, control)
print(fclass="str">"t = {t_stat:.3f}, p = {p_value:.4f}")

class=class="str">"com"># 95% confidence interval for the lift
diff = variant.mean() - control.mean()
se = np.sqrt(variant.var(ddof=1)/len(variant) + control.var(ddof=1)/len(control))
ci = (diff - 1.96*se, diff + 1.96*se)
print(fclass="str">"Lift = {diff:.2f}, 95% CI = {ci}")

Data Science vs Adjacent Roles

• Data Analyst — focuses on descriptive + diagnostic, dashboards, SQL.
• Data Scientist — adds predictive modeling, experimentation, statistical inference.
• ML Engineer — productionizes models, owns infra, latency, scale.
• Data Engineer — builds pipelines, warehouses, and the data foundation.

The Modern Toolkit

• Languages — Python, SQL, R
• Notebooks — Jupyter, Google Colab, VS Code, Quarto
• Libraries — pandas, NumPy, scikit-learn, statsmodels, XGBoost, PyTorch
• Visualization — matplotlib, seaborn, Plotly, Tableau, Power BI
• Infrastructure — Git, Docker, Airflow, dbt, Snowflake / BigQuery
• Collaboration — GitHub, MLflow, Weights & Biases

Best Practices

• Reproducibility — pin dependencies, seed random states, version data.
• Validate assumptions — never trust a model without sanity-checking the data.
• Beware leakage — never let test data influence training (or feature engineering).
• Communicate uncertainty — report confidence intervals, not just point estimates.
• Iterate fast — a simple baseline beats a complex model that ships late.
• Ethics & bias — audit datasets and predictions for fairness and representation.

Recommended Learning Path

1. Master SQL & Python fundamentals
2. Learn statistics and probability deeply
3. Practice EDA on real datasets (Kaggle, UCI, public APIs)
4. Build end-to-end projects with clear narratives
5. Layer in machine learning with scikit-learn
6. Learn experimentation (A/B testing) and causal inference
7. Deploy a model and monitor it in production