local_2_AnomalyDetection_workshop.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e80693de-5d45-4ca6-9eec-8f7a8239d9ca",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "**Before You Start**: Make sure that you ran `local_0_setup_workshop.ipynb`!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d78f1f2d-57fc-4ec5-b341-5830fd24970a",
   "metadata": {},
   "source": [
    "# ECG Arrhythmia detection\n",
    "\n",
    "This project aims to demonstrate the Anomaly Detection module implemented in Darts\n",
    "### Dataset\n",
    "The MIT-BIH Supraventricular Arrhythmia Database (SVDB) contains 2 channels, and 78 half-hour ECG recordings obtained from 47 objects between 1975-1979. To keep the data lean, we only stored data of patient 842.\n",
    "\n",
    "### Task\n",
    "Develop an anomaly detection model to identity arrhythmia in the ECG signal."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9360ba4-5400-46c5-b5db-6f40417dca12",
   "metadata": {},
   "source": [
    "## Task #1\n",
    "### Load data of a patient into a darts timeseries object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f981579-3229-47fb-b889-654c77bef494",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from darts import TimeSeries\n",
    "\n",
    "# Load data into darts TimeSeries object\n",
    "fpath = os.path.join(\"data\", \"anomaly_detection\", \"842.test.csv\")\n",
    "timeseries = TimeSeries.from_csv(fpath, time_col='timestamp')\n",
    "ts_ecg = timeseries[['ECG1','ECG2']]\n",
    "ts_anomaly = timeseries['is_anomaly']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2978f8d9-8e7d-49c9-8bd4-aa1653076353",
   "metadata": {},
   "source": [
    "### Visualize signal and anomaly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60fec587-43fa-43b3-a6ae-0f4cf3bd4a27",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "fig,ax = plt.subplots(1,1, figsize=(10,5))\n",
    "ts_ecg['ECG1'].plot(ax=ax,label='ECG1', lw=0.5)\n",
    "(ts_anomaly-2).plot(ax=ax,label='is_anomaly',color='r', lw=0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45b19862-a47d-4cae-8df8-0bfd92d64b80",
   "metadata": {},
   "source": [
    "## Task #2\n",
    "### Identify a region of ~15000 datapoints with anomalies (otherwise training takes a while) and visualize it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9664934-39ef-4517-9ae9-31f5bd8191e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Identify a subset for demonstration\n",
    "start, end = 15000, 30000\n",
    "# Create subset time series ecg and anomaly object\n",
    "\n",
    "\"\"\"\n",
    "Extract a subset from `ts_ecg` and `ts_anomaly` using the start and end points from above:\n",
    "\n",
    "Hint: you can slice TimeSeries like this:  `series[idx_left:idx_right]`\n",
    "\"\"\"\n",
    "ts_ecg_subset= # TO FILL\n",
    "ts_anomaly_subset = # TO FILL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4270cd9-3563-4ca0-891f-d40375c8b201",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualize the subset\n",
    "fig,ax = plt.subplots(figsize=(10, 5))\n",
    "ts_ecg_subset['ECG1'].plot(label='ECG1', lw=1.)\n",
    "((ts_anomaly_subset/2)-1.5).plot(label='is_anomaly', color='r')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6142af44-0be7-46b0-97e4-87ef6fbb5597",
   "metadata": {},
   "source": [
    "## Task #3\n",
    "### Create training and test sets (e.g., 10k/5k)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff6cdea3-6c0d-499b-ace6-d07d9002545d",
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Extract a train and test set from `ts_ecg_subset` and `ts_anomaly_subset`.\n",
    "\n",
    "The train set should end at index 10'000, and test set at index 15'000\n",
    "\"\"\"\n",
    "train_end, test_end = # TO FILL \n",
    "ts_ecg_train = # TO FILL (extract from the `*_subset` series)\n",
    "ts_ecg_test =  # TO FILL\n",
    "ts_anomaly_test = # TO FILL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9482f567-0a82-4632-bbec-8c175439e051",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualize the train / test set as well as the test set anomalies\n",
    "fig,ax = plt.subplots(figsize=(10, 5))\n",
    "ts_ecg_train['ECG1'].plot(label='train', lw=1.)\n",
    "ts_ecg_test['ECG1'].plot(label='test', lw=1.)\n",
    "((ts_anomaly_test/2)-1.5).plot(label='is_anomaly', color='r')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3f0da52-003d-49b3-bd08-d2528fe3fc19",
   "metadata": {},
   "source": [
    "## Task #4\n",
    "### Assess data properties such as periodicity and identify most common period."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea0b3c3e-66b7-4dc9-b36d-2de90c4ff8a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "from darts.utils.statistics import plot_acf\n",
    "\n",
    "# Visualise signal auto correlation to identify most common periodicity\n",
    "plot_acf(ts=ts_ecg_subset['ECG1'], max_lag=220, m=92)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "22f843f1-eeab-429b-82a4-57b0afe996e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Identified most common period\n",
    "period = 92"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c3a3760-aa66-43b9-95be-1ca13a4dac96",
   "metadata": {},
   "source": [
    "## Task #5\n",
    "### Develop an anomaly detection model step by step by (see figure below):\n",
    "1. Create a forecasting model based on the train timeseries ECG data\n",
    "2. Compute historical forecasting for the test timeseries ECG data\n",
    "3. Compute anomaly scores using 2 different scores based on the forecasted and actual ECG signal\n",
    "\n",
    "<img src=\"images/ad_inside_anomaly_model.png\" alt=\"Image\" width=\"60%\" height=\"60%\">\n",
    "\n",
    "Links:\n",
    "- Forecasting models: https://unit8co.github.io/darts/generated_api/darts.models.forecasting.html\n",
    "- Scorers: https://unit8co.github.io/darts/generated_api/darts.ad.scorers.html?highlight=scorer#"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5932fdb4-f567-4d23-883a-78a1a2ab0cb8",
   "metadata": {},
   "source": [
    "#### Create a Forecasting model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf9d557d-0cf4-46a3-806f-6157477b0255",
   "metadata": {},
   "outputs": [],
   "source": [
    "from darts.models import LinearRegressionModel\n",
    "\n",
    "\"\"\"\n",
    "Create a `LinearRegressionModel` and fit it on `ts_ecg_train`\n",
    "\n",
    "Setup the model like this:\n",
    "- use the last `period` steps of the target series as model input (hint: `lags`) \n",
    "- train the model to predict one step at a time (hint: `output_chunk_length`)\n",
    "\n",
    "Documentation: https://unit8co.github.io/darts/generated_api/darts.models.forecasting.linear_regression_model.html#linear-regression-model\n",
    "\"\"\"\n",
    "\n",
    "# create the model\n",
    "forecasting_model = # TO FILL\n",
    "\n",
    "# train the forecasting model on the training dataset `ts_ecg_train`\n",
    "forecasting_model.fit(\n",
    "    # TO FILL\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "347b76a4-abd2-44c3-99e5-18eea5b847e1",
   "metadata": {},
   "source": [
    "Okay and let's compute the historical forecasts and the residuals of the forecasts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f262a52-f0c6-42d0-9e88-7540e9da9a6f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# historical predictions\n",
    "ts_ecg_test_predicted = forecasting_model.historical_forecasts(ts_ecg_test, retrain=False)\n",
    "\n",
    "# residuals of these predictions = (y_true - y_pred)\n",
    "ts_ecg_residuals = forecasting_model.residuals(ts_ecg_test, historical_forecasts=ts_ecg_test_predicted)\n",
    "\n",
    "# Visualization of predicted and actual signal\n",
    "fig, (ax1, ax2) = plt.subplots(nrows=2, figsize=(10, 10))\n",
    "ts_ecg_test_predicted['ECG1'].plot(ax=ax1, label='ECG1_predicted')\n",
    "ts_ecg_test['ECG1'].plot(ax=ax1, color='r', label='ECG1_test')\n",
    "ts_ecg_residuals['ECG1'].plot(ax=ax2, color='b', label='Difference')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b410090-28be-4718-a81c-78e2263f4db6",
   "metadata": {},
   "source": [
    "#### Use a NormScorer for scoring"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "87b4f794-0ea3-4f3c-b067-75333c1db361",
   "metadata": {},
   "outputs": [],
   "source": [
    "from darts.ad.scorers import NormScorer\n",
    "from darts.ad.utils import eval_metric_from_scores\n",
    "\n",
    "\"\"\"\n",
    "Create a NormScorer and compute the anomaly scores on the test set (between the actual values and \n",
    "historical forecasts).\n",
    "\n",
    "The scorer should use:\n",
    "- a norm order of `1`\n",
    "- it should compare components jointly for scoring (hint parameter: `component_wise`)\n",
    "\n",
    "Documentation:\n",
    "- Scorer(): https://unit8co.github.io/darts/generated_api/darts.ad.scorers.norm_scorer.html\n",
    "- Scorer.score_from_prediction(): https://unit8co.github.io/darts/generated_api/darts.ad.scorers.norm_scorer.html#darts.ad.scorers.norm_scorer.NormScorer.score_from_prediction\n",
    "\"\"\"\n",
    "# create the scorer\n",
    "scorer = NormScorer(\n",
    "    ord= # TO FILL\n",
    "    component_wise= # TO FILL\n",
    ")\n",
    "\n",
    "# compute the anomaly scores between the actual test series and the historical forecasts for that series\n",
    "scores = scorer.score_from_prediction(\n",
    "    series= # TO FILL\n",
    "    pred_series= # TO FILL\n",
    ")\n",
    "scores.plot(label='Anomaly Score')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4bdfed06-91bc-4de3-9911-63408896b086",
   "metadata": {},
   "outputs": [],
   "source": [
    "from darts.ad.utils import eval_metric_from_scores\n",
    "\n",
    "\"\"\"\n",
    "Evaluate the computed anomaly score using utility methods in darts\n",
    "\"\"\"\n",
    "eval_metric_from_scores(\n",
    "    pred_scores= # TO FILL (hint: the predicted scores), \n",
    "    anomalies= # TO FILL (hint: the actual anomalies), \n",
    "    metric= # TO FILL (e.g. 'AUC_ROC')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9310e4e-2bee-4de4-8698-1d382aacfe7e",
   "metadata": {},
   "source": [
    "#### Use fittable KMeansScorer for scoring\n",
    "The Norm scorer calculates the norm between the predicted and actual time series point-wise. Since predicting the peaks of the ECG signals is challenging for the forecasting model, the biggest differences between the actual and predicted values are mostly found at the peak locations.\n",
    "\n",
    "To overcome the issue from point-wise comparisons, we'll use a windowing approach. This extracts vectors/chunks from an input series by applying a sliding window of width `window` to it. Currently, the scorers that support windowing are `KMeansScorer`, `PyODScorer` and `WasserSteinScorer`. They are also trainable, meaning that we have to fit them first on a set of window vectors/chunks. You want the training set to be anomaly-free!\n",
    "\n",
    "To fit the scorers, you can either:\n",
    "- 1) fit it only on some actual time series with `scorer.fit()`, or\n",
    "- 2) fit it on the difference between some actual series and model forecasts for those series with `scorer.fit_from_prediction()`.\n",
    "\n",
    "The scoring on new data also has two options (pick the same option as done when fitting): \n",
    "- 1) score on some new time series with `scorer.score()`, or\n",
    "- 2) score on the difference between some new series and model forecasts for those series with `scorer.score_from_prediction()`.\n",
    " \n",
    "In option 2), the scorers simply compute some difference function (e.g. absolute differnce, ...) to reduce the actual series and forecasts into one series. After that, internally it works exactly the same as option 1).\n",
    "\n",
    "Let's look at `KMeansScorer` as an example. It fits `k` centroids on the windowed anomaly-free input data. To score how anomalous a new series is, it computes the closest centroid distance for each window of that series. If there were any anomalies, then the distances should be larger than any of the anomaly free trainig windows.\n",
    "\n",
    "The figure below illustrates how the KMeanScorer works when applied directly to a time series with option 1) (or option 2) after taking the difference).\n",
    "\n",
    "#### Training & Scoring\n",
    "<img src=\"images/kmeansscorer.png\" alt=\"Image\" width=\"70%\" height=\"70%\">\n",
    "\n",
    "<img src=\"images/ad_windowing.png\" alt=\"Image\" width=\"70%\" height=\"70%\">\n",
    "\n",
    "In the example below, we will use the previously developed forecasting model to create historical predictions for the train dataset and train the KMeanScorer on the absolute difference between the actual training and forecasted training datasets (default).\n",
    "\n",
    "The difference function can be changed with parameter `diff_fn`. It can be any of Darts [\"per time step\" metrics](https://unit8co.github.io/darts/generated_api/darts.metrics.html).  By default, it uses `darts.metrics.ae` (absolute difference)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9970d0d7-9473-41d8-af2d-689e23823bb1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# First we compute historical forecasts on the train dataset to later train the scorer\n",
    "ts_ecg_train_predicted = forecasting_model.historical_forecasts(ts_ecg_train, retrain=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "45d03c19-be0a-4741-ab31-bb255702d612",
   "metadata": {},
   "outputs": [],
   "source": [
    "from darts.ad.scorers import KMeansScorer\n",
    "\"\"\"\n",
    "Create a KMeanScorer, and train it on the train set (on the actual series and historical forecasts).\n",
    "\n",
    "The scorer should use a window size of `50`.\n",
    "\n",
    "Documentation:\n",
    "- KMeanScorer: https://unit8co.github.io/darts/generated_api/darts.ad.scorers.kmeans_scorer.html\n",
    "\"\"\"\n",
    "\n",
    "# create the scorer\n",
    "scorer = KMeansScorer(\n",
    "    window= # TO FILL\n",
    "    component_wise=False,\n",
    "    random_state=42,\n",
    ")\n",
    "\n",
    "# train the scorer on the train set\n",
    "scorer.fit_from_prediction(\n",
    "    series= # TO FILL\n",
    "    pred_series= # TO FILL\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0147a0e2-b482-439d-8226-af45511059fa",
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Compute the scores on the test set (on the actual series and historical forecasts).\n",
    "\"\"\"\n",
    "\n",
    "# score on the test set\n",
    "scores = scorer.score_from_prediction(\n",
    "    series= # TO FILL\n",
    "    pred_series= # TO FILL\n",
    ")\n",
    "\n",
    "# plot the results\n",
    "scores.plot(label='Anomaly Score')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee0d6e9e-4d46-4694-b514-098b560b635c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# compute a metric for the scores\n",
    "eval_metric_from_scores(\n",
    "    pred_scores=scores,\n",
    "    anomalies=ts_anomaly_test,\n",
    "    window=2*period,\n",
    "    metric='AUC_ROC'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5c643e9-9cf8-4a8c-8c5d-30b0829b59da",
   "metadata": {},
   "source": [
    "## Task #6\n",
    "### Develop the anomaly detection models by using the Forecasting Anomaly Model via dedicated Darts API interface\n",
    "This exercise aims to illustrate the power of the darts anomaly detection module by hiding all of the previously made steps under the hood into one dedicated AnomalyModel and corresponding APIs\n",
    "\n",
    "We'll use the already pretrained forecasting model, but you can also give an un-trained model and call `ForecastingAnomalyModel.fit()` with `allow_model_training=True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e23939c1-8328-4332-9848-e1f70ec3218b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from darts.ad.anomaly_model.forecasting_am import ForecastingAnomalyModel\n",
    "from darts.ad.scorers import NormScorer, KMeansScorer\n",
    "\n",
    "# Instantiate the anomaly model with: one forecasting model, and one or more scorers (and corresponding parameters)\n",
    "anomaly_model = ForecastingAnomalyModel(\n",
    "    model=forecasting_model,\n",
    "    scorer=[\n",
    "         NormScorer(ord=1),\n",
    "         KMeansScorer(k=50, window=2*period, component_wise=False, random_state=42)\n",
    "    ],\n",
    ")\n",
    "\n",
    "\"\"\"\n",
    "Fit the Forecasting Anomaly Model on the train set `ts_ecg_train`.\n",
    "\"\"\"\n",
    "\n",
    "# fit the forecasting anomaly model\n",
    "anomaly_model.fit(\n",
    "    series= # TO FILL\n",
    "    allow_model_training=False  # (we use a pre-trained model)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c289ddd-9096-419c-af45-416113857a69",
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Compute the scores the test set. Use the actual test target series `ts_ecg_test`.\n",
    "\"\"\"\n",
    "\n",
    "# compute scores on the test set and return historical forecasts in one step\n",
    "anomaly_scores, predictions = anomaly_model.score(\n",
    "    series= # TO FILL, \n",
    "    return_model_prediction=True  # (also return historical forecasts)\n",
    ")\n",
    "\n",
    "# plot the actual test series, and historical forecasts\n",
    "fig, ax plt.subplots()\n",
    "ts_ecg_test['ECG1'].plot(label='test', lw=1.)\n",
    "predictions['ECG1'].plot(label='prediction', lw=1.)\n",
    "plt.show()\n",
    "\n",
    "# plot the scores from each scorer\n",
    "fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(10, 5))\n",
    "anomaly_scores[0].plot(ax=ax1, label=\"NormScorer\") # indeces corresponding to the scorers\n",
    "anomaly_scores[1].plot(ax=ax2, label=\"KMeansScorer\") # indeces corresponding to the scorers\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb2c90a6-1340-4802-9185-fd168ca76889",
   "metadata": {},
   "source": [
    "### Leverage the inbuilt darts visualization tool to evaluate and show anomalies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c9a0f46-5ac4-44c6-8288-84440f3f387f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualize and evaluate detection of anomalies\n",
    "anomaly_model.show_anomalies(\n",
    "    series=ts_ecg_test,\n",
    "    anomalies=ts_anomaly_test,\n",
    "    metric=\"AUC_ROC\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da20c142-3112-4958-9e97-c0c7c90900b1",
   "metadata": {},
   "source": [
    "## Task #7\n",
    "### Use a detector to binarize the anomaly scores\n",
    "\n",
    "Currently there are two types of Detectors in Darts:\n",
    "- QuantileDetector: flag points as anomalous if they are outside upper and lower quantile bound\n",
    "- ThresholdDetector: flag points as anomalous if they are above/below a maximum/minimum threshold value\n",
    "\n",
    "Link:\n",
    "- https://unit8co.github.io/darts/generated_api/darts.ad.detectors.html?highlight=detector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a08cfd60-8c51-4777-9f3c-6577ae3cd3df",
   "metadata": {},
   "outputs": [],
   "source": [
    "from darts.ad.detectors import QuantileDetector, ThresholdDetector\n",
    "\n",
    "# Instantiate a detector\n",
    "detector = QuantileDetector(low_quantile=0, high_quantile=0.70)\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "(detector.fit_detect(anomaly_scores[0])-0).plot(lw=1., label='NormScorer - detected_anomaly')\n",
    "(detector.fit_detect(anomaly_scores[1])-2).plot(lw=1., label='KMeanScorer - detected_anomaly')\n",
    "(ts_anomaly_test-4).plot(color='r', lw=1, label='is_anomaly')\n",
    "ax.legend(bbox_to_anchor=(1.05, 1), loc='upper left')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12dbc014-37e1-4a10-a876-fb7f07a56ba3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiate a detector\n",
    "detector =  ThresholdDetector(low_threshold=0, high_threshold=0.42)\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "(detector.detect(anomaly_scores[0])).plot(lw=1, label='NormScorer - detected_anomaly')\n",
    "#(detector.detect(anomaly_scores[1])-2).plot(lw=1, label='KMeanScorer - detected_anomaly')\n",
    "(ts_anomaly_test-2).plot(color='r', lw=1, label='is_anomaly')\n",
    "ax.legend(bbox_to_anchor=(1.05, 1), loc='upper left')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}