{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Lesson 19: Experimentation and Causal Inference for Promotions\n",
        "\n",
        "**CS230 inspiration:** ML strategy, case studies, and decision-focused statistical rigor.\n",
        "\n",
        "**Demand forecasting focus:** Estimate whether a promotion caused incremental demand instead of merely predicting promoted demand.\n",
        "\n",
        "**Learning objectives**\n",
        "\n",
        "- Separate predictive lift from causal lift.\n",
        "- Estimate treatment effects with randomized tests and difference-in-differences logic.\n",
        "- Use variance reduction and minimum detectable effect calculations.\n",
        "\n",
        "**Senior-readiness checkpoint:** You can tell a stakeholder whether a promotion should be repeated, not just whether the model predicted it.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Code Cell 1 Explanation\n",
        "\n",
        "**What this cell does:** This setup cell imports the shared scientific Python stack and the curriculum helpers used by later cells. It also establishes the deterministic seed convention so repeated runs are comparable.\n",
        "\n",
        "**Why it matters:** A restart-and-run notebook should make its dependencies explicit before any modeling work begins. For forecasting, reproducibility is especially important because small data or seed changes can change validation conclusions.\n",
        "\n",
        "**Key operations to notice:** confirm that shared helper imports match the lesson's code path; check that `SEED` is defined before any generated data or model training\n",
        "\n",
        "**What to inspect:** Check that imports are grouped by purpose: general analysis packages, forecasting data helpers, metrics, plotting utilities, and any lesson-specific libraries.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "from demand_forecasting_curriculum.data import (\n",
        "    DEFAULT_FEATURE_COLUMNS,\n",
        "    generate_daily_demand,\n",
        "    make_sequence_dataset,\n",
        "    make_supervised_table,\n",
        "    time_series_split,\n",
        ")\n",
        "from demand_forecasting_curriculum.metrics import (\n",
        "    interval_coverage,\n",
        "    mase,\n",
        "    pinball_loss,\n",
        "    smape,\n",
        "    wmape,\n",
        ")\n",
        "from demand_forecasting_curriculum.plotting import plot_predictions, plot_series\n",
        "\n",
        "SEED = 42\n",
        "np.random.seed(SEED)\n",
        "\n",
        "import math\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Hiring-Manager Signal\n",
        "\n",
        "Forecasting promotions is not the same as measuring promotion incrementality. Hiring teams notice candidates who can separate correlation from causal evidence and can communicate experiment limits without hiding behind model accuracy.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Code Cell 2 Explanation\n",
        "\n",
        "**What this cell does:** This cell advances **Experimentation and Causal Inference for Promotions** by working on synthetic demand generation, causal inference. It creates or updates `rng`, `df`, `df`, `cutoff`, `units`, `treated_units` and prepares evidence that later cells use rather than hiding state in prose.\n",
        "\n",
        "**Why it matters:** The goal is tied to this lesson's forecasting focus: Estimate whether a promotion caused incremental demand instead of merely predicting promoted demand. Each object is kept inspectable so the learner can connect code, assumptions, and modeling consequences.\n",
        "\n",
        "**Key operations to notice:** creates a synthetic retail panel with known demand drivers\n",
        "\n",
        "**What to inspect:** Before moving on, inspect shapes, date ranges, metric values, segment behavior, or generated tables. If a value looks surprisingly good, surprisingly bad, or unavailable at forecast time, treat that as a modeling clue.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "rng = np.random.default_rng(SEED)\n",
        "df = generate_daily_demand(n_stores=8, n_items=10, periods=420, seed=SEED)\n",
        "df = df.assign(unit=lambda x: x[\"store_id\"].astype(str) + \"_\" + x[\"item_id\"].astype(str))\n",
        "\n",
        "cutoff = pd.Timestamp(\"2021-10-01\")\n",
        "units = np.array(sorted(df[\"unit\"].unique()))\n",
        "treated_units = set(rng.choice(units, size=len(units) // 2, replace=False))\n",
        "\n",
        "experiment = df.assign(\n",
        "    treated=lambda x: x[\"unit\"].isin(treated_units).astype(int),\n",
        "    post=lambda x: (x[\"date\"] >= cutoff).astype(int),\n",
        ")\n",
        "treatment_lift = 0.08\n",
        "noise = rng.normal(0, 2.0, len(experiment))\n",
        "experiment[\"experiment_demand\"] = np.where(\n",
        "    (experiment[\"treated\"] == 1) & (experiment[\"post\"] == 1),\n",
        "    experiment[\"demand\"] * (1 + treatment_lift) + noise,\n",
        "    experiment[\"demand\"] + noise,\n",
        ")\n",
        "experiment[\"experiment_demand\"] = experiment[\"experiment_demand\"].clip(lower=0)\n",
        "experiment.head()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Code Cell 3 Explanation\n",
        "\n",
        "**What this cell does:** This cell advances **Experimentation and Causal Inference for Promotions** by working on causal inference. It creates or updates `summary`, `diff_in_diff` and prepares evidence that later cells use rather than hiding state in prose.\n",
        "\n",
        "**Why it matters:** The goal is tied to this lesson's forecasting focus: Estimate whether a promotion caused incremental demand instead of merely predicting promoted demand. Each object is kept inspectable so the learner can connect code, assumptions, and modeling consequences.\n",
        "\n",
        "**Key operations to notice:** follow the assignments, final displayed object, and any checks in the cell because those are the evidence the next cell depends on\n",
        "\n",
        "**What to inspect:** Before moving on, inspect shapes, date ranges, metric values, segment behavior, or generated tables. If a value looks surprisingly good, surprisingly bad, or unavailable at forecast time, treat that as a modeling clue.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "summary = (\n",
        "    experiment.groupby([\"treated\", \"post\"])[\"experiment_demand\"]\n",
        "    .mean()\n",
        "    .unstack()\n",
        "    .rename(columns={0: \"pre\", 1: \"post\"})\n",
        ")\n",
        "summary[\"post_minus_pre\"] = summary[\"post\"] - summary[\"pre\"]\n",
        "diff_in_diff = summary.loc[1, \"post_minus_pre\"] - summary.loc[0, \"post_minus_pre\"]\n",
        "\n",
        "summary, diff_in_diff\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Code Cell 4 Explanation\n",
        "\n",
        "**What this cell does:** This cell advances **Experimentation and Causal Inference for Promotions** by working on causal inference. It creates or updates `y`, `treated`, `post`, `design`, `did_table` and prepares evidence that later cells use rather than hiding state in prose.\n",
        "\n",
        "**Why it matters:** The goal is tied to this lesson's forecasting focus: Estimate whether a promotion caused incremental demand instead of merely predicting promoted demand. Each object is kept inspectable so the learner can connect code, assumptions, and modeling consequences.\n",
        "\n",
        "**Key operations to notice:** fits a simple regression estimate for causal diagnostics; creates an inspectable table for assumptions, metrics, or decisions\n",
        "\n",
        "**What to inspect:** Before moving on, inspect shapes, date ranges, metric values, segment behavior, or generated tables. If a value looks surprisingly good, surprisingly bad, or unavailable at forecast time, treat that as a modeling clue.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "y = experiment[\"experiment_demand\"].to_numpy()\n",
        "treated = experiment[\"treated\"].to_numpy()\n",
        "post = experiment[\"post\"].to_numpy()\n",
        "design = np.column_stack(\n",
        "    [\n",
        "        np.ones(len(experiment)),\n",
        "        treated,\n",
        "        post,\n",
        "        treated * post,\n",
        "    ]\n",
        ")\n",
        "beta, *_ = np.linalg.lstsq(design, y, rcond=None)\n",
        "did_table = pd.DataFrame(\n",
        "    {\n",
        "        \"term\": [\"intercept\", \"treated\", \"post\", \"treated_x_post\"],\n",
        "        \"estimate\": beta,\n",
        "    }\n",
        ")\n",
        "did_table\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Code Cell 5 Explanation\n",
        "\n",
        "**What this cell does:** This cell advances **Experimentation and Causal Inference for Promotions** by working on causal inference. It creates or updates `pre_unit_mean`, `post_rows`, `raw_variance`, `theta`, `cuped_variance`, `n_per_group` and prepares evidence that later cells use rather than hiding state in prose.\n",
        "\n",
        "**Why it matters:** The goal is tied to this lesson's forecasting focus: Estimate whether a promotion caused incremental demand instead of merely predicting promoted demand. Each object is kept inspectable so the learner can connect code, assumptions, and modeling consequences.\n",
        "\n",
        "**Key operations to notice:** follow the assignments, final displayed object, and any checks in the cell because those are the evidence the next cell depends on\n",
        "\n",
        "**What to inspect:** Before moving on, inspect shapes, date ranges, metric values, segment behavior, or generated tables. If a value looks surprisingly good, surprisingly bad, or unavailable at forecast time, treat that as a modeling clue.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "pre_unit_mean = (\n",
        "    experiment.query(\"post == 0\")\n",
        "    .groupby(\"unit\")[\"experiment_demand\"]\n",
        "    .mean()\n",
        "    .rename(\"pre_mean\")\n",
        ")\n",
        "post_rows = experiment.query(\"post == 1\").join(pre_unit_mean, on=\"unit\")\n",
        "\n",
        "raw_variance = post_rows[\"experiment_demand\"].var()\n",
        "theta = np.cov(post_rows[\"experiment_demand\"], post_rows[\"pre_mean\"])[0, 1] / post_rows[\"pre_mean\"].var()\n",
        "post_rows[\"cuped_outcome\"] = post_rows[\"experiment_demand\"] - theta * (\n",
        "    post_rows[\"pre_mean\"] - post_rows[\"pre_mean\"].mean()\n",
        ")\n",
        "cuped_variance = post_rows[\"cuped_outcome\"].var()\n",
        "\n",
        "n_per_group = post_rows.groupby(\"treated\").size().min()\n",
        "z_alpha = 1.96\n",
        "z_power = 0.84\n",
        "mde_raw = (z_alpha + z_power) * math.sqrt(2 * raw_variance / n_per_group)\n",
        "mde_cuped = (z_alpha + z_power) * math.sqrt(2 * cuped_variance / n_per_group)\n",
        "\n",
        "{\n",
        "    \"raw_variance\": raw_variance,\n",
        "    \"cuped_variance\": cuped_variance,\n",
        "    \"variance_reduction\": 1 - cuped_variance / raw_variance,\n",
        "    \"mde_raw_units\": mde_raw,\n",
        "    \"mde_cuped_units\": mde_cuped,\n",
        "}\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Lesson Recap\n",
        "\n",
        "In this lesson, we used 5 runnable code cells to work through **Experimentation and Causal Inference for Promotions**. The practical focus was: Estimate whether a promotion caused incremental demand instead of merely predicting promoted demand.\n",
        "\n",
        "We did this because deep learning for demand forecasting is not only about choosing an architecture. The learner needs to understand the data contract, the target and horizon, the validation path, the metric, and the operational decision supported by the forecast.\n",
        "\n",
        "The core takeaways were:\n",
        "\n",
        "- Separate predictive lift from causal lift.\n",
        "- Estimate treatment effects with randomized tests and difference-in-differences logic.\n",
        "- Use variance reduction and minimum detectable effect calculations.\n",
        "\n",
        "By the end, you should be able to explain not just what the code produced, but why each step was necessary and what could go wrong if the same step were skipped or performed with leakage.\n",
        "\n",
        "**Senior-readiness reminder:** You can tell a stakeholder whether a promotion should be repeated, not just whether the model predicted it.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Exercises\n",
        "\n",
        "- Create a sample-ratio-mismatch check for treated vs control units.\n",
        "- Estimate treatment effects separately for high-volume and low-volume items.\n",
        "- Write a recommendation that distinguishes statistical evidence, business impact, and rollout risk.\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python (deep-learning-demand)",
      "language": "python",
      "name": "deep-learning-demand"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.12"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}