{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Lesson 12: Hyperparameter Tuning and Experiments\n",
        "\n",
        "**CS230 inspiration:** Hyperparameter tuning, batch normalization, programming frameworks, and systematic experimentation.\n",
        "\n",
        "**Demand forecasting focus:** Run a small, auditable tuning loop and record enough metadata to reproduce decisions.\n",
        "\n",
        "**Learning objectives**\n",
        "\n",
        "- Design a compact search space.\n",
        "- Record model, data, and metric metadata.\n",
        "- Choose a model using validation data and reserve test data for final estimation.\n",
        "\n",
        "**Senior-readiness checkpoint:** You can explain the difference between experimentation discipline and leaderboard chasing.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Code Cell 1 Explanation\n",
        "\n",
        "**What this cell does:** This setup cell imports the shared scientific Python stack and the curriculum helpers used by later cells. It also establishes the deterministic seed convention so repeated runs are comparable.\n",
        "\n",
        "**Why it matters:** A restart-and-run notebook should make its dependencies explicit before any modeling work begins. For forecasting, reproducibility is especially important because small data or seed changes can change validation conclusions.\n",
        "\n",
        "**Key operations to notice:** confirm that shared helper imports match the lesson's code path; check that `SEED` is defined before any generated data or model training\n",
        "\n",
        "**What to inspect:** Check that imports are grouped by purpose: general analysis packages, forecasting data helpers, metrics, plotting utilities, and any lesson-specific libraries.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "from demand_forecasting_curriculum.data import (\n",
        "    DEFAULT_FEATURE_COLUMNS,\n",
        "    generate_daily_demand,\n",
        "    make_sequence_dataset,\n",
        "    make_supervised_table,\n",
        "    time_series_split,\n",
        ")\n",
        "from demand_forecasting_curriculum.metrics import (\n",
        "    interval_coverage,\n",
        "    mase,\n",
        "    pinball_loss,\n",
        "    smape,\n",
        "    wmape,\n",
        ")\n",
        "from demand_forecasting_curriculum.plotting import plot_predictions, plot_series\n",
        "\n",
        "SEED = 42\n",
        "np.random.seed(SEED)\n",
        "\n",
        "import torch\n",
        "from demand_forecasting_curriculum.modeling import (\n",
        "    MLPRegressor,\n",
        "    Standardizer,\n",
        "    make_regression_loader,\n",
        "    predict_regression_model,\n",
        "    set_seed,\n",
        "    train_regression_model,\n",
        ")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Code Cell 2 Explanation\n",
        "\n",
        "**What this cell does:** This cell advances **Hyperparameter Tuning and Experiments** by working on synthetic demand generation, time-based splitting, supervised feature construction. It creates or updates `df`, `supervised`, `train`, `valid`, `test`, `feature_cols` and prepares evidence that later cells use rather than hiding state in prose.\n",
        "\n",
        "**Why it matters:** The goal is tied to this lesson's forecasting focus: Run a small, auditable tuning loop and record enough metadata to reproduce decisions. Each object is kept inspectable so the learner can connect code, assumptions, and modeling consequences.\n",
        "\n",
        "**Key operations to notice:** creates a synthetic retail panel with known demand drivers; turns time series rows into forecast-origin training examples; keeps train, validation, and test windows ordered by date; fits scaling parameters on training data only; wraps NumPy arrays in PyTorch DataLoaders\n",
        "\n",
        "**What to inspect:** Before moving on, inspect shapes, date ranges, metric values, segment behavior, or generated tables. If a value looks surprisingly good, surprisingly bad, or unavailable at forecast time, treat that as a modeling clue.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "set_seed(SEED)\n",
        "df = generate_daily_demand(n_stores=4, n_items=6, periods=500, seed=SEED)\n",
        "supervised = make_supervised_table(df, horizon=7)\n",
        "train, valid, test = time_series_split(supervised, valid_days=60, test_days=60)\n",
        "\n",
        "feature_cols = [\"store_id\", \"item_id\", *DEFAULT_FEATURE_COLUMNS]\n",
        "scaler = Standardizer.fit(train[feature_cols].to_numpy(dtype=np.float32))\n",
        "X_train = scaler.transform(train[feature_cols].to_numpy(dtype=np.float32)).astype(np.float32)\n",
        "X_valid = scaler.transform(valid[feature_cols].to_numpy(dtype=np.float32)).astype(np.float32)\n",
        "y_train = np.log1p(train[\"y\"].to_numpy(dtype=np.float32))\n",
        "y_valid = np.log1p(valid[\"y\"].to_numpy(dtype=np.float32))\n",
        "train_loader = make_regression_loader(X_train, y_train, batch_size=256, shuffle=True)\n",
        "valid_loader = make_regression_loader(X_valid, y_valid, batch_size=512, shuffle=False)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Code Cell 3 Explanation\n",
        "\n",
        "**What this cell does:** This cell advances **Hyperparameter Tuning and Experiments** by working on forecast metrics, model training, model definition, Ralph-style iteration. It creates or updates `search_space`, `runs`, `experiment_log` and prepares evidence that later cells use rather than hiding state in prose.\n",
        "\n",
        "**Why it matters:** The goal is tied to this lesson's forecasting focus: Run a small, auditable tuning loop and record enough metadata to reproduce decisions. Each object is kept inspectable so the learner can connect code, assumptions, and modeling consequences.\n",
        "\n",
        "**Key operations to notice:** instantiates a feed-forward demand forecasting model; runs the shared PyTorch training loop; generates validation or test predictions from a trained model; evaluates scale-weighted forecast error; keeps a reproducible history of loop results; creates an inspectable table for assumptions, metrics, or decisions\n",
        "\n",
        "**What to inspect:** Before moving on, inspect shapes, date ranges, metric values, segment behavior, or generated tables. If a value looks surprisingly good, surprisingly bad, or unavailable at forecast time, treat that as a modeling clue.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "search_space = [\n",
        "    {\"hidden\": (64, 32), \"dropout\": 0.0, \"lr\": 1e-3, \"weight_decay\": 0.0},\n",
        "    {\"hidden\": (96, 48), \"dropout\": 0.05, \"lr\": 1e-3, \"weight_decay\": 1e-4},\n",
        "    {\"hidden\": (128, 64), \"dropout\": 0.10, \"lr\": 7e-4, \"weight_decay\": 1e-4},\n",
        "    {\"hidden\": (128, 64, 32), \"dropout\": 0.15, \"lr\": 7e-4, \"weight_decay\": 1e-3},\n",
        "]\n",
        "\n",
        "runs = []\n",
        "for run_id, params in enumerate(search_space, start=1):\n",
        "    set_seed(SEED)\n",
        "    model = MLPRegressor(X_train.shape[1], hidden_sizes=params[\"hidden\"], dropout=params[\"dropout\"])\n",
        "    history = train_regression_model(\n",
        "        model,\n",
        "        train_loader,\n",
        "        valid_loader,\n",
        "        epochs=5,\n",
        "        learning_rate=params[\"lr\"],\n",
        "        weight_decay=params[\"weight_decay\"],\n",
        "    )\n",
        "    pred = np.expm1(predict_regression_model(model, X_valid))\n",
        "    actual = np.expm1(y_valid)\n",
        "    runs.append(\n",
        "        {\n",
        "            \"run_id\": run_id,\n",
        "            **params,\n",
        "            \"valid_wmape\": wmape(actual, pred),\n",
        "            \"valid_mae_log\": history[-1][\"valid_mae\"],\n",
        "            \"data_seed\": SEED,\n",
        "            \"horizon\": 7,\n",
        "        }\n",
        "    )\n",
        "\n",
        "experiment_log = pd.DataFrame(runs).sort_values(\"valid_wmape\")\n",
        "experiment_log\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Code Cell 4 Explanation\n",
        "\n",
        "**What this cell does:** This cell advances **Hyperparameter Tuning and Experiments** by working on Ralph-style iteration. It creates or updates `best` and prepares evidence that later cells use rather than hiding state in prose.\n",
        "\n",
        "**Why it matters:** The goal is tied to this lesson's forecasting focus: Run a small, auditable tuning loop and record enough metadata to reproduce decisions. Each object is kept inspectable so the learner can connect code, assumptions, and modeling consequences.\n",
        "\n",
        "**Key operations to notice:** keeps a reproducible history of loop results\n",
        "\n",
        "**What to inspect:** Before moving on, inspect shapes, date ranges, metric values, segment behavior, or generated tables. If a value looks surprisingly good, surprisingly bad, or unavailable at forecast time, treat that as a modeling clue.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "best = experiment_log.iloc[0].to_dict()\n",
        "best\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Lesson Recap\n",
        "\n",
        "In this lesson, we used 4 runnable code cells to work through **Hyperparameter Tuning and Experiments**. The practical focus was: Run a small, auditable tuning loop and record enough metadata to reproduce decisions.\n",
        "\n",
        "We did this because deep learning for demand forecasting is not only about choosing an architecture. The learner needs to understand the data contract, the target and horizon, the validation path, the metric, and the operational decision supported by the forecast.\n",
        "\n",
        "The core takeaways were:\n",
        "\n",
        "- Design a compact search space.\n",
        "- Record model, data, and metric metadata.\n",
        "- Choose a model using validation data and reserve test data for final estimation.\n",
        "\n",
        "By the end, you should be able to explain not just what the code produced, but why each step was necessary and what could go wrong if the same step were skipped or performed with leakage.\n",
        "\n",
        "**Senior-readiness reminder:** You can explain the difference between experimentation discipline and leaderboard chasing.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Exercises\n",
        "\n",
        "- Add batch size to the search space and decide how many runs are affordable.\n",
        "- Create an `experiments/` CSV log, but do not commit huge artifacts.\n",
        "- Explain when you would move from manual search to Optuna, Ray Tune, or a managed platform.\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python (deep-learning-demand)",
      "language": "python",
      "name": "deep-learning-demand"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.12"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}