{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Lesson 09: Attention and Transformers\n",
        "\n",
        "**CS230 inspiration:** Attention mechanisms and transformer networks.\n",
        "\n",
        "**Demand forecasting focus:** Build a compact transformer encoder that attends across historical demand context.\n",
        "\n",
        "**Learning objectives**\n",
        "\n",
        "- Understand self-attention as learned interactions across time steps.\n",
        "- Add positional information for order-aware sequence modeling.\n",
        "- Evaluate a transformer as a forecasting architecture, not just an NLP tool.\n",
        "\n",
        "**Senior-readiness checkpoint:** You can decide when transformer complexity is justified for forecasting data scale and latency.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Code Cell 1 Explanation\n",
        "\n",
        "**What this cell does:** This setup cell imports the shared scientific Python stack and the curriculum helpers used by later cells. It also establishes the deterministic seed convention so repeated runs are comparable.\n",
        "\n",
        "**Why it matters:** A restart-and-run notebook should make its dependencies explicit before any modeling work begins. For forecasting, reproducibility is especially important because small data or seed changes can change validation conclusions.\n",
        "\n",
        "**Key operations to notice:** confirm that shared helper imports match the lesson's code path; check that `SEED` is defined before any generated data or model training\n",
        "\n",
        "**What to inspect:** Check that imports are grouped by purpose: general analysis packages, forecasting data helpers, metrics, plotting utilities, and any lesson-specific libraries.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "from demand_forecasting_curriculum.data import (\n",
        "    DEFAULT_FEATURE_COLUMNS,\n",
        "    generate_daily_demand,\n",
        "    make_sequence_dataset,\n",
        "    make_supervised_table,\n",
        "    time_series_split,\n",
        ")\n",
        "from demand_forecasting_curriculum.metrics import (\n",
        "    interval_coverage,\n",
        "    mase,\n",
        "    pinball_loss,\n",
        "    smape,\n",
        "    wmape,\n",
        ")\n",
        "from demand_forecasting_curriculum.plotting import plot_predictions, plot_series\n",
        "\n",
        "SEED = 42\n",
        "np.random.seed(SEED)\n",
        "\n",
        "import torch\n",
        "from torch import nn\n",
        "from torch.utils.data import DataLoader, TensorDataset\n",
        "from demand_forecasting_curriculum.modeling import set_seed\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Code Cell 2 Explanation\n",
        "\n",
        "**What this cell does:** This cell advances **Attention and Transformers** by working on synthetic demand generation, sequence-window construction. It creates or updates `df`, `X`, `y`, `meta`, `target_start`, `train_idx` and prepares evidence that later cells use rather than hiding state in prose.\n",
        "\n",
        "**Why it matters:** The goal is tied to this lesson's forecasting focus: Build a compact transformer encoder that attends across historical demand context. Each object is kept inspectable so the learner can connect code, assumptions, and modeling consequences.\n",
        "\n",
        "**Key operations to notice:** creates a synthetic retail panel with known demand drivers; builds lookback windows for sequence models; batches examples for neural-network training; packages tensors into a PyTorch dataset\n",
        "\n",
        "**What to inspect:** Before moving on, inspect shapes, date ranges, metric values, segment behavior, or generated tables. If a value looks surprisingly good, surprisingly bad, or unavailable at forecast time, treat that as a modeling clue.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "set_seed(SEED)\n",
        "df = generate_daily_demand(n_stores=4, n_items=6, periods=520, seed=SEED)\n",
        "X, y, meta = make_sequence_dataset(df, lookback=56, horizon=14)\n",
        "target_start = pd.to_datetime(meta[\"target_start\"])\n",
        "train_idx = target_start < target_start.quantile(0.70)\n",
        "valid_idx = (target_start >= target_start.quantile(0.70)) & (target_start < target_start.quantile(0.85))\n",
        "\n",
        "x_mean = X[train_idx].mean(axis=(0, 1), keepdims=True)\n",
        "x_std = np.where(X[train_idx].std(axis=(0, 1), keepdims=True) == 0, 1, X[train_idx].std(axis=(0, 1), keepdims=True))\n",
        "X_scaled = (X - x_mean) / x_std\n",
        "\n",
        "train_loader = DataLoader(\n",
        "    TensorDataset(torch.tensor(X_scaled[train_idx]).float(), torch.tensor(np.log1p(y[train_idx])).float()),\n",
        "    batch_size=128,\n",
        "    shuffle=True,\n",
        ")\n",
        "valid_loader = DataLoader(\n",
        "    TensorDataset(torch.tensor(X_scaled[valid_idx]).float(), torch.tensor(np.log1p(y[valid_idx])).float()),\n",
        "    batch_size=256,\n",
        ")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Code Cell 3 Explanation\n",
        "\n",
        "**What this cell does:** This cell advances **Attention and Transformers** by working on model definition. It creates or updates `model` and prepares evidence that later cells use rather than hiding state in prose.\n",
        "\n",
        "**Why it matters:** The goal is tied to this lesson's forecasting focus: Build a compact transformer encoder that attends across historical demand context. Each object is kept inspectable so the learner can connect code, assumptions, and modeling consequences.\n",
        "\n",
        "**Key operations to notice:** models interactions across the lookback window with attention\n",
        "\n",
        "**What to inspect:** Before moving on, inspect shapes, date ranges, metric values, segment behavior, or generated tables. If a value looks surprisingly good, surprisingly bad, or unavailable at forecast time, treat that as a modeling clue.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "class TransformerForecaster(nn.Module):\n",
        "    def __init__(self, n_features, d_model, horizon, lookback, n_heads=4):\n",
        "        super().__init__()\n",
        "        self.input_projection = nn.Linear(n_features, d_model)\n",
        "        self.position = nn.Parameter(torch.zeros(1, lookback, d_model))\n",
        "        layer = nn.TransformerEncoderLayer(\n",
        "            d_model=d_model,\n",
        "            nhead=n_heads,\n",
        "            dim_feedforward=4 * d_model,\n",
        "            dropout=0.1,\n",
        "            batch_first=True,\n",
        "        )\n",
        "        self.encoder = nn.TransformerEncoder(layer, num_layers=2)\n",
        "        self.head = nn.Sequential(nn.LayerNorm(d_model), nn.Linear(d_model, horizon))\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = self.input_projection(x) + self.position[:, : x.shape[1]]\n",
        "        encoded = self.encoder(x)\n",
        "        return self.head(encoded[:, -1])\n",
        "\n",
        "model = TransformerForecaster(X.shape[-1], d_model=64, horizon=y.shape[-1], lookback=X.shape[1])\n",
        "model(next(iter(train_loader))[0]).shape\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Code Cell 4 Explanation\n",
        "\n",
        "**What this cell does:** This cell advances **Attention and Transformers** by working on forecast metrics, model training. It creates or updates `optimizer`, `loss_fn`, `pred`, `actual` and prepares evidence that later cells use rather than hiding state in prose.\n",
        "\n",
        "**Why it matters:** The goal is tied to this lesson's forecasting focus: Build a compact transformer encoder that attends across historical demand context. Each object is kept inspectable so the learner can connect code, assumptions, and modeling consequences.\n",
        "\n",
        "**Key operations to notice:** evaluates scale-weighted forecast error; updates model weights after backpropagation; computes gradients for model parameters; runs inference without tracking gradients\n",
        "\n",
        "**What to inspect:** Before moving on, inspect shapes, date ranges, metric values, segment behavior, or generated tables. If a value looks surprisingly good, surprisingly bad, or unavailable at forecast time, treat that as a modeling clue.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "optimizer = torch.optim.AdamW(model.parameters(), lr=8e-4, weight_decay=1e-4)\n",
        "loss_fn = nn.L1Loss()\n",
        "\n",
        "for epoch in range(3):\n",
        "    model.train()\n",
        "    for xb, yb in train_loader:\n",
        "        optimizer.zero_grad(set_to_none=True)\n",
        "        loss = loss_fn(model(xb), yb)\n",
        "        loss.backward()\n",
        "        optimizer.step()\n",
        "\n",
        "model.eval()\n",
        "with torch.no_grad():\n",
        "    preds, actuals = [], []\n",
        "    for xb, yb in valid_loader:\n",
        "        preds.append(model(xb).numpy())\n",
        "        actuals.append(yb.numpy())\n",
        "\n",
        "pred = np.expm1(np.vstack(preds))\n",
        "actual = np.expm1(np.vstack(actuals))\n",
        "{\"valid_wmape_all_horizons\": wmape(actual.ravel(), pred.ravel())}\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Lesson Recap\n",
        "\n",
        "In this lesson, we used 4 runnable code cells to work through **Attention and Transformers**. The practical focus was: Build a compact transformer encoder that attends across historical demand context.\n",
        "\n",
        "We did this because deep learning for demand forecasting is not only about choosing an architecture. The learner needs to understand the data contract, the target and horizon, the validation path, the metric, and the operational decision supported by the forecast.\n",
        "\n",
        "The core takeaways were:\n",
        "\n",
        "- Understand self-attention as learned interactions across time steps.\n",
        "- Add positional information for order-aware sequence modeling.\n",
        "- Evaluate a transformer as a forecasting architecture, not just an NLP tool.\n",
        "\n",
        "By the end, you should be able to explain not just what the code produced, but why each step was necessary and what could go wrong if the same step were skipped or performed with leakage.\n",
        "\n",
        "**Senior-readiness reminder:** You can decide when transformer complexity is justified for forecasting data scale and latency.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Exercises\n",
        "\n",
        "- Increase `lookback` to 112 days and watch memory/training cost.\n",
        "- Change `n_heads` and explain the divisibility constraint with `d_model`.\n",
        "- Compare the transformer's operational cost to the CNN and GRU notebooks.\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python (deep-learning-demand)",
      "language": "python",
      "name": "deep-learning-demand"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.12"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}