mlp.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Homework-2: MLP for MNIST Classification\n",
    "\n",
    "### In this homework, you need to\n",
    "- #### implement SGD optimizer (`./optimizer.py`)\n",
    "- #### implement forward and backward for FCLayer (`layers/fc_layer.py`)\n",
    "- #### implement forward and backward for SigmoidLayer (`layers/sigmoid_layer.py`)\n",
    "- #### implement forward and backward for ReLULayer (`layers/relu_layer.py`)\n",
    "- #### implement EuclideanLossLayer (`criterion/euclidean_loss.py`)\n",
    "- #### implement SoftmaxCrossEntropyLossLayer (`criterion/softmax_cross_entropy.py`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import tensorflow.compat.v1 as tf\n",
    "tf.disable_eager_execution()\n",
    "\n",
    "from network import Network\n",
    "from solver import train, test\n",
    "from plot import plot_loss_and_acc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load MNIST Dataset\n",
    "We use tensorflow tools to load dataset for convenience."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def decode_image(image):\n",
    "    # Normalize from [0, 255.] to [0., 1.0], and then subtract by the mean value\n",
    "    image = tf.cast(image, tf.float32)\n",
    "    image = tf.reshape(image, [784])\n",
    "    image = image / 255.0\n",
    "    image = image - tf.reduce_mean(image)\n",
    "    return image\n",
    "\n",
    "def decode_label(label):\n",
    "    # Encode label with one-hot encoding\n",
    "    return tf.one_hot(label, depth=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Data Preprocessing\n",
    "x_train = tf.data.Dataset.from_tensor_slices(x_train).map(decode_image)\n",
    "y_train = tf.data.Dataset.from_tensor_slices(y_train).map(decode_label)\n",
    "data_train = tf.data.Dataset.zip((x_train, y_train))\n",
    "\n",
    "x_test = tf.data.Dataset.from_tensor_slices(x_test).map(decode_image)\n",
    "y_test = tf.data.Dataset.from_tensor_slices(y_test).map(decode_label)\n",
    "data_test = tf.data.Dataset.zip((x_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set Hyerparameters\n",
    "You can modify hyerparameters by yourself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 100\n",
    "max_epoch = 20\n",
    "init_std = 0.01\n",
    "\n",
    "learning_rate_SGD = 0.001\n",
    "weight_decay = 0.1\n",
    "\n",
    "disp_freq = 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. MLP with Euclidean Loss\n",
    "In part-1, you need to train a MLP with **Euclidean Loss**.  \n",
    "**Sigmoid Activation Function** and **ReLU Activation Function** will be used respectively.\n",
    "### TODO\n",
    "Before executing the following code, you should complete **./optimizer.py** and **criterion/euclidean_loss.py**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from criterion import EuclideanLossLayer\n",
    "from optimizer import SGD\n",
    "\n",
    "criterion = EuclideanLossLayer()\n",
    "\n",
    "sgd = SGD(learning_rate_SGD, weight_decay)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1 MLP with Euclidean Loss and Sigmoid Activation Function\n",
    "Build and train a MLP contraining one hidden layer with 128 units using Sigmoid activation function and Euclidean loss function.\n",
    "\n",
    "### TODO\n",
    "Before executing the following code, you should complete **layers/fc_layer.py** and **layers/sigmoid_layer.py**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from layers import FCLayer, SigmoidLayer\n",
    "\n",
    "sigmoidMLP = Network()\n",
    "# Build MLP with FCLayer and SigmoidLayer\n",
    "# 128 is the number of hidden units, you can change by your own\n",
    "sigmoidMLP.add(FCLayer(784, 128))\n",
    "sigmoidMLP.add(SigmoidLayer())\n",
    "sigmoidMLP.add(FCLayer(128, 10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Users/youssefbr/Downloads/homework-2/homework2-mlp/solver.py:15: DatasetV1.make_one_shot_iterator (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "This is a deprecated API that should only be used in TF 1 graph mode and legacy TF 2 graph mode available through `tf.compat.v1`. In all other situations -- namely, eager mode and inside `tf.function` -- you can consume dataset elements using `for elem in dataset: ...` or by explicitly creating iterator via `iterator = iter(dataset)` and fetching its elements via `values = next(iterator)`. Furthermore, this API is not available in TF 2. During the transition from TF 1 to TF 2 you can use `tf.compat.v1.data.make_one_shot_iterator(dataset)` to create a TF 1 graph mode style iterator for a dataset created through TF 2 APIs. Note that this should be a transient state of your code base as there are in general no guarantees about the interoperability of TF 1 and TF 2 code.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-08-07 17:36:08.275838: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch [0][20]\t Batch [0][550]\t Training Loss 1.5899\t Accuracy 0.1200\n",
      "Epoch [0][20]\t Batch [50][550]\t Training Loss 1.5299\t Accuracy 0.1208\n",
      "Epoch [0][20]\t Batch [100][550]\t Training Loss 1.5214\t Accuracy 0.1211\n",
      "Epoch [0][20]\t Batch [150][550]\t Training Loss 1.5262\t Accuracy 0.1202\n",
      "Epoch [0][20]\t Batch [200][550]\t Training Loss 1.5207\t Accuracy 0.1212\n",
      "Epoch [0][20]\t Batch [250][550]\t Training Loss 1.5223\t Accuracy 0.1210\n",
      "Epoch [0][20]\t Batch [300][550]\t Training Loss 1.5208\t Accuracy 0.1222\n",
      "Epoch [0][20]\t Batch [350][550]\t Training Loss 1.5184\t Accuracy 0.1218\n",
      "Epoch [0][20]\t Batch [400][550]\t Training Loss 1.5178\t Accuracy 0.1216\n",
      "Epoch [0][20]\t Batch [450][550]\t Training Loss 1.5173\t Accuracy 0.1222\n",
      "Epoch [0][20]\t Batch [500][550]\t Training Loss 1.5182\t Accuracy 0.1217\n",
      "\n",
      "Epoch [0]\t Average training loss 1.5172\t Average training accuracy 0.1220\n",
      "Epoch [0]\t Average validation loss 1.5089\t Average validation accuracy 0.1190\n",
      "\n",
      "Epoch [1][20]\t Batch [0][550]\t Training Loss 1.5899\t Accuracy 0.1200\n",
      "Epoch [1][20]\t Batch [50][550]\t Training Loss 1.5299\t Accuracy 0.1208\n",
      "Epoch [1][20]\t Batch [100][550]\t Training Loss 1.5214\t Accuracy 0.1211\n",
      "Epoch [1][20]\t Batch [150][550]\t Training Loss 1.5262\t Accuracy 0.1202\n",
      "Epoch [1][20]\t Batch [200][550]\t Training Loss 1.5207\t Accuracy 0.1212\n",
      "Epoch [1][20]\t Batch [250][550]\t Training Loss 1.5223\t Accuracy 0.1210\n",
      "Epoch [1][20]\t Batch [300][550]\t Training Loss 1.5208\t Accuracy 0.1222\n",
      "Epoch [1][20]\t Batch [350][550]\t Training Loss 1.5184\t Accuracy 0.1218\n",
      "Epoch [1][20]\t Batch [400][550]\t Training Loss 1.5178\t Accuracy 0.1216\n",
      "Epoch [1][20]\t Batch [450][550]\t Training Loss 1.5173\t Accuracy 0.1222\n",
      "Epoch [1][20]\t Batch [500][550]\t Training Loss 1.5182\t Accuracy 0.1217\n",
      "\n",
      "Epoch [1]\t Average training loss 1.5172\t Average training accuracy 0.1220\n",
      "Epoch [1]\t Average validation loss 1.5089\t Average validation accuracy 0.1190\n",
      "\n",
      "Epoch [2][20]\t Batch [0][550]\t Training Loss 1.5899\t Accuracy 0.1200\n",
      "Epoch [2][20]\t Batch [50][550]\t Training Loss 1.5299\t Accuracy 0.1208\n",
      "Epoch [2][20]\t Batch [100][550]\t Training Loss 1.5214\t Accuracy 0.1211\n",
      "Epoch [2][20]\t Batch [150][550]\t Training Loss 1.5262\t Accuracy 0.1202\n",
      "Epoch [2][20]\t Batch [200][550]\t Training Loss 1.5207\t Accuracy 0.1212\n",
      "Epoch [2][20]\t Batch [250][550]\t Training Loss 1.5223\t Accuracy 0.1210\n",
      "Epoch [2][20]\t Batch [300][550]\t Training Loss 1.5208\t Accuracy 0.1222\n",
      "Epoch [2][20]\t Batch [350][550]\t Training Loss 1.5184\t Accuracy 0.1218\n",
      "Epoch [2][20]\t Batch [400][550]\t Training Loss 1.5178\t Accuracy 0.1216\n",
      "Epoch [2][20]\t Batch [450][550]\t Training Loss 1.5173\t Accuracy 0.1222\n",
      "Epoch [2][20]\t Batch [500][550]\t Training Loss 1.5182\t Accuracy 0.1217\n",
      "\n",
      "Epoch [2]\t Average training loss 1.5172\t Average training accuracy 0.1220\n",
      "Epoch [2]\t Average validation loss 1.5089\t Average validation accuracy 0.1190\n",
      "\n",
      "Epoch [3][20]\t Batch [0][550]\t Training Loss 1.5899\t Accuracy 0.1200\n",
      "Epoch [3][20]\t Batch [50][550]\t Training Loss 1.5299\t Accuracy 0.1208\n",
      "Epoch [3][20]\t Batch [100][550]\t Training Loss 1.5214\t Accuracy 0.1211\n",
      "Epoch [3][20]\t Batch [150][550]\t Training Loss 1.5262\t Accuracy 0.1202\n",
      "Epoch [3][20]\t Batch [200][550]\t Training Loss 1.5207\t Accuracy 0.1212\n",
      "Epoch [3][20]\t Batch [250][550]\t Training Loss 1.5223\t Accuracy 0.1210\n",
      "Epoch [3][20]\t Batch [300][550]\t Training Loss 1.5208\t Accuracy 0.1222\n",
      "Epoch [3][20]\t Batch [350][550]\t Training Loss 1.5184\t Accuracy 0.1218\n",
      "Epoch [3][20]\t Batch [400][550]\t Training Loss 1.5178\t Accuracy 0.1216\n",
      "Epoch [3][20]\t Batch [450][550]\t Training Loss 1.5173\t Accuracy 0.1222\n",
      "Epoch [3][20]\t Batch [500][550]\t Training Loss 1.5182\t Accuracy 0.1217\n",
      "\n",
      "Epoch [3]\t Average training loss 1.5172\t Average training accuracy 0.1220\n",
      "Epoch [3]\t Average validation loss 1.5089\t Average validation accuracy 0.1190\n",
      "\n",
      "Epoch [4][20]\t Batch [0][550]\t Training Loss 1.5899\t Accuracy 0.1200\n",
      "Epoch [4][20]\t Batch [50][550]\t Training Loss 1.5299\t Accuracy 0.1208\n",
      "Epoch [4][20]\t Batch [100][550]\t Training Loss 1.5214\t Accuracy 0.1211\n",
      "Epoch [4][20]\t Batch [150][550]\t Training Loss 1.5262\t Accuracy 0.1202\n",
      "Epoch [4][20]\t Batch [200][550]\t Training Loss 1.5207\t Accuracy 0.1212\n",
      "Epoch [4][20]\t Batch [250][550]\t Training Loss 1.5223\t Accuracy 0.1210\n",
      "Epoch [4][20]\t Batch [300][550]\t Training Loss 1.5208\t Accuracy 0.1222\n",
      "Epoch [4][20]\t Batch [350][550]\t Training Loss 1.5184\t Accuracy 0.1218\n",
      "Epoch [4][20]\t Batch [400][550]\t Training Loss 1.5178\t Accuracy 0.1216\n",
      "Epoch [4][20]\t Batch [450][550]\t Training Loss 1.5173\t Accuracy 0.1222\n",
      "Epoch [4][20]\t Batch [500][550]\t Training Loss 1.5182\t Accuracy 0.1217\n",
      "\n",
      "Epoch [4]\t Average training loss 1.5172\t Average training accuracy 0.1220\n",
      "Epoch [4]\t Average validation loss 1.5089\t Average validation accuracy 0.1190\n",
      "\n",
      "Epoch [5][20]\t Batch [0][550]\t Training Loss 1.5899\t Accuracy 0.1200\n",
      "Epoch [5][20]\t Batch [50][550]\t Training Loss 1.5299\t Accuracy 0.1208\n",
      "Epoch [5][20]\t Batch [100][550]\t Training Loss 1.5214\t Accuracy 0.1211\n",
      "Epoch [5][20]\t Batch [150][550]\t Training Loss 1.5262\t Accuracy 0.1202\n",
      "Epoch [5][20]\t Batch [200][550]\t Training Loss 1.5207\t Accuracy 0.1212\n",
      "Epoch [5][20]\t Batch [250][550]\t Training Loss 1.5223\t Accuracy 0.1210\n",
      "Epoch [5][20]\t Batch [300][550]\t Training Loss 1.5208\t Accuracy 0.1222\n",
      "Epoch [5][20]\t Batch [350][550]\t Training Loss 1.5184\t Accuracy 0.1218\n",
      "Epoch [5][20]\t Batch [400][550]\t Training Loss 1.5178\t Accuracy 0.1216\n",
      "Epoch [5][20]\t Batch [450][550]\t Training Loss 1.5173\t Accuracy 0.1222\n",
      "Epoch [5][20]\t Batch [500][550]\t Training Loss 1.5182\t Accuracy 0.1217\n",
      "\n",
      "Epoch [5]\t Average training loss 1.5172\t Average training accuracy 0.1220\n",
      "Epoch [5]\t Average validation loss 1.5089\t Average validation accuracy 0.1190\n",
      "\n",
      "Epoch [6][20]\t Batch [0][550]\t Training Loss 1.5899\t Accuracy 0.1200\n",
      "Epoch [6][20]\t Batch [50][550]\t Training Loss 1.5299\t Accuracy 0.1208\n",
      "Epoch [6][20]\t Batch [100][550]\t Training Loss 1.5214\t Accuracy 0.1211\n",
      "Epoch [6][20]\t Batch [150][550]\t Training Loss 1.5262\t Accuracy 0.1202\n",
      "Epoch [6][20]\t Batch [200][550]\t Training Loss 1.5207\t Accuracy 0.1212\n",
      "Epoch [6][20]\t Batch [250][550]\t Training Loss 1.5223\t Accuracy 0.1210\n",
      "Epoch [6][20]\t Batch [300][550]\t Training Loss 1.5208\t Accuracy 0.1222\n",
      "Epoch [6][20]\t Batch [350][550]\t Training Loss 1.5184\t Accuracy 0.1218\n",
      "Epoch [6][20]\t Batch [400][550]\t Training Loss 1.5178\t Accuracy 0.1216\n",
      "Epoch [6][20]\t Batch [450][550]\t Training Loss 1.5173\t Accuracy 0.1222\n",
      "Epoch [6][20]\t Batch [500][550]\t Training Loss 1.5182\t Accuracy 0.1217\n",
      "\n",
      "Epoch [6]\t Average training loss 1.5172\t Average training accuracy 0.1220\n",
      "Epoch [6]\t Average validation loss 1.5089\t Average validation accuracy 0.1190\n",
      "\n",
      "Epoch [7][20]\t Batch [0][550]\t Training Loss 1.5899\t Accuracy 0.1200\n",
      "Epoch [7][20]\t Batch [50][550]\t Training Loss 1.5299\t Accuracy 0.1208\n",
      "Epoch [7][20]\t Batch [100][550]\t Training Loss 1.5214\t Accuracy 0.1211\n",
      "Epoch [7][20]\t Batch [150][550]\t Training Loss 1.5262\t Accuracy 0.1202\n",
      "Epoch [7][20]\t Batch [200][550]\t Training Loss 1.5207\t Accuracy 0.1212\n",
      "Epoch [7][20]\t Batch [250][550]\t Training Loss 1.5223\t Accuracy 0.1210\n",
      "Epoch [7][20]\t Batch [300][550]\t Training Loss 1.5208\t Accuracy 0.1222\n",
      "Epoch [7][20]\t Batch [350][550]\t Training Loss 1.5184\t Accuracy 0.1218\n",
      "Epoch [7][20]\t Batch [400][550]\t Training Loss 1.5178\t Accuracy 0.1216\n",
      "Epoch [7][20]\t Batch [450][550]\t Training Loss 1.5173\t Accuracy 0.1222\n",
      "Epoch [7][20]\t Batch [500][550]\t Training Loss 1.5182\t Accuracy 0.1217\n",
      "\n",
      "Epoch [7]\t Average training loss 1.5172\t Average training accuracy 0.1220\n",
      "Epoch [7]\t Average validation loss 1.5089\t Average validation accuracy 0.1190\n",
      "\n",
      "Epoch [8][20]\t Batch [0][550]\t Training Loss 1.5899\t Accuracy 0.1200\n",
      "Epoch [8][20]\t Batch [50][550]\t Training Loss 1.5299\t Accuracy 0.1208\n",
      "Epoch [8][20]\t Batch [100][550]\t Training Loss 1.5214\t Accuracy 0.1211\n",
      "Epoch [8][20]\t Batch [150][550]\t Training Loss 1.5262\t Accuracy 0.1202\n",
      "Epoch [8][20]\t Batch [200][550]\t Training Loss 1.5207\t Accuracy 0.1212\n",
      "Epoch [8][20]\t Batch [250][550]\t Training Loss 1.5223\t Accuracy 0.1210\n",
      "Epoch [8][20]\t Batch [300][550]\t Training Loss 1.5208\t Accuracy 0.1222\n",
      "Epoch [8][20]\t Batch [350][550]\t Training Loss 1.5184\t Accuracy 0.1218\n",
      "Epoch [8][20]\t Batch [400][550]\t Training Loss 1.5178\t Accuracy 0.1216\n",
      "Epoch [8][20]\t Batch [450][550]\t Training Loss 1.5173\t Accuracy 0.1222\n",
      "Epoch [8][20]\t Batch [500][550]\t Training Loss 1.5182\t Accuracy 0.1217\n",
      "\n",
      "Epoch [8]\t Average training loss 1.5172\t Average training accuracy 0.1220\n",
      "Epoch [8]\t Average validation loss 1.5089\t Average validation accuracy 0.1190\n",
      "\n",
      "Epoch [9][20]\t Batch [0][550]\t Training Loss 1.5899\t Accuracy 0.1200\n",
      "Epoch [9][20]\t Batch [50][550]\t Training Loss 1.5299\t Accuracy 0.1208\n",
      "Epoch [9][20]\t Batch [100][550]\t Training Loss 1.5214\t Accuracy 0.1211\n",
      "Epoch [9][20]\t Batch [150][550]\t Training Loss 1.5262\t Accuracy 0.1202\n",
      "Epoch [9][20]\t Batch [200][550]\t Training Loss 1.5207\t Accuracy 0.1212\n",
      "Epoch [9][20]\t Batch [250][550]\t Training Loss 1.5223\t Accuracy 0.1210\n",
      "Epoch [9][20]\t Batch [300][550]\t Training Loss 1.5208\t Accuracy 0.1222\n",
      "Epoch [9][20]\t Batch [350][550]\t Training Loss 1.5184\t Accuracy 0.1218\n",
      "Epoch [9][20]\t Batch [400][550]\t Training Loss 1.5178\t Accuracy 0.1216\n",
      "Epoch [9][20]\t Batch [450][550]\t Training Loss 1.5173\t Accuracy 0.1222\n",
      "Epoch [9][20]\t Batch [500][550]\t Training Loss 1.5182\t Accuracy 0.1217\n",
      "\n",
      "Epoch [9]\t Average training loss 1.5172\t Average training accuracy 0.1220\n",
      "Epoch [9]\t Average validation loss 1.5089\t Average validation accuracy 0.1190\n",
      "\n",
      "Epoch [10][20]\t Batch [0][550]\t Training Loss 1.5899\t Accuracy 0.1200\n",
      "Epoch [10][20]\t Batch [50][550]\t Training Loss 1.5299\t Accuracy 0.1208\n",
      "Epoch [10][20]\t Batch [100][550]\t Training Loss 1.5214\t Accuracy 0.1211\n",
      "Epoch [10][20]\t Batch [150][550]\t Training Loss 1.5262\t Accuracy 0.1202\n",
      "Epoch [10][20]\t Batch [200][550]\t Training Loss 1.5207\t Accuracy 0.1212\n",
      "Epoch [10][20]\t Batch [250][550]\t Training Loss 1.5223\t Accuracy 0.1210\n",
      "Epoch [10][20]\t Batch [300][550]\t Training Loss 1.5208\t Accuracy 0.1222\n",
      "Epoch [10][20]\t Batch [350][550]\t Training Loss 1.5184\t Accuracy 0.1218\n",
      "Epoch [10][20]\t Batch [400][550]\t Training Loss 1.5178\t Accuracy 0.1216\n",
      "Epoch [10][20]\t Batch [450][550]\t Training Loss 1.5173\t Accuracy 0.1222\n",
      "Epoch [10][20]\t Batch [500][550]\t Training Loss 1.5182\t Accuracy 0.1217\n",
      "\n",
      "Epoch [10]\t Average training loss 1.5172\t Average training accuracy 0.1220\n",
      "Epoch [10]\t Average validation loss 1.5089\t Average validation accuracy 0.1190\n",
      "\n",
      "Epoch [11][20]\t Batch [0][550]\t Training Loss 1.5899\t Accuracy 0.1200\n",
      "Epoch [11][20]\t Batch [50][550]\t Training Loss 1.5299\t Accuracy 0.1208\n",
      "Epoch [11][20]\t Batch [100][550]\t Training Loss 1.5214\t Accuracy 0.1211\n",
      "Epoch [11][20]\t Batch [150][550]\t Training Loss 1.5262\t Accuracy 0.1202\n",
      "Epoch [11][20]\t Batch [200][550]\t Training Loss 1.5207\t Accuracy 0.1212\n",
      "Epoch [11][20]\t Batch [250][550]\t Training Loss 1.5223\t Accuracy 0.1210\n",
      "Epoch [11][20]\t Batch [300][550]\t Training Loss 1.5208\t Accuracy 0.1222\n",
      "Epoch [11][20]\t Batch [350][550]\t Training Loss 1.5184\t Accuracy 0.1218\n",
      "Epoch [11][20]\t Batch [400][550]\t Training Loss 1.5178\t Accuracy 0.1216\n",
      "Epoch [11][20]\t Batch [450][550]\t Training Loss 1.5173\t Accuracy 0.1222\n",
      "Epoch [11][20]\t Batch [500][550]\t Training Loss 1.5182\t Accuracy 0.1217\n",
      "\n",
      "Epoch [11]\t Average training loss 1.5172\t Average training accuracy 0.1220\n",
      "Epoch [11]\t Average validation loss 1.5089\t Average validation accuracy 0.1190\n",
      "\n",
      "Epoch [12][20]\t Batch [0][550]\t Training Loss 1.5899\t Accuracy 0.1200\n",
      "Epoch [12][20]\t Batch [50][550]\t Training Loss 1.5299\t Accuracy 0.1208\n",
      "Epoch [12][20]\t Batch [100][550]\t Training Loss 1.5214\t Accuracy 0.1211\n",
      "Epoch [12][20]\t Batch [150][550]\t Training Loss 1.5262\t Accuracy 0.1202\n",
      "Epoch [12][20]\t Batch [200][550]\t Training Loss 1.5207\t Accuracy 0.1212\n",
      "Epoch [12][20]\t Batch [250][550]\t Training Loss 1.5223\t Accuracy 0.1210\n",
      "Epoch [12][20]\t Batch [300][550]\t Training Loss 1.5208\t Accuracy 0.1222\n",
      "Epoch [12][20]\t Batch [350][550]\t Training Loss 1.5184\t Accuracy 0.1218\n",
      "Epoch [12][20]\t Batch [400][550]\t Training Loss 1.5178\t Accuracy 0.1216\n",
      "Epoch [12][20]\t Batch [450][550]\t Training Loss 1.5173\t Accuracy 0.1222\n",
      "Epoch [12][20]\t Batch [500][550]\t Training Loss 1.5182\t Accuracy 0.1217\n",
      "\n",
      "Epoch [12]\t Average training loss 1.5172\t Average training accuracy 0.1220\n",
      "Epoch [12]\t Average validation loss 1.5089\t Average validation accuracy 0.1190\n",
      "\n",
      "Epoch [13][20]\t Batch [0][550]\t Training Loss 1.5899\t Accuracy 0.1200\n",
      "Epoch [13][20]\t Batch [50][550]\t Training Loss 1.5299\t Accuracy 0.1208\n",
      "Epoch [13][20]\t Batch [100][550]\t Training Loss 1.5214\t Accuracy 0.1211\n",
      "Epoch [13][20]\t Batch [150][550]\t Training Loss 1.5262\t Accuracy 0.1202\n",
      "Epoch [13][20]\t Batch [200][550]\t Training Loss 1.5207\t Accuracy 0.1212\n",
      "Epoch [13][20]\t Batch [250][550]\t Training Loss 1.5223\t Accuracy 0.1210\n",
      "Epoch [13][20]\t Batch [300][550]\t Training Loss 1.5208\t Accuracy 0.1222\n",
      "Epoch [13][20]\t Batch [350][550]\t Training Loss 1.5184\t Accuracy 0.1218\n",
      "Epoch [13][20]\t Batch [400][550]\t Training Loss 1.5178\t Accuracy 0.1216\n",
      "Epoch [13][20]\t Batch [450][550]\t Training Loss 1.5173\t Accuracy 0.1222\n",
      "Epoch [13][20]\t Batch [500][550]\t Training Loss 1.5182\t Accuracy 0.1217\n",
      "\n",
      "Epoch [13]\t Average training loss 1.5172\t Average training accuracy 0.1220\n",
      "Epoch [13]\t Average validation loss 1.5089\t Average validation accuracy 0.1190\n",
      "\n",
      "Epoch [14][20]\t Batch [0][550]\t Training Loss 1.5899\t Accuracy 0.1200\n",
      "Epoch [14][20]\t Batch [50][550]\t Training Loss 1.5299\t Accuracy 0.1208\n",
      "Epoch [14][20]\t Batch [100][550]\t Training Loss 1.5214\t Accuracy 0.1211\n",
      "Epoch [14][20]\t Batch [150][550]\t Training Loss 1.5262\t Accuracy 0.1202\n",
      "Epoch [14][20]\t Batch [200][550]\t Training Loss 1.5207\t Accuracy 0.1212\n",
      "Epoch [14][20]\t Batch [250][550]\t Training Loss 1.5223\t Accuracy 0.1210\n",
      "Epoch [14][20]\t Batch [300][550]\t Training Loss 1.5208\t Accuracy 0.1222\n",
      "Epoch [14][20]\t Batch [350][550]\t Training Loss 1.5184\t Accuracy 0.1218\n",
      "Epoch [14][20]\t Batch [400][550]\t Training Loss 1.5178\t Accuracy 0.1216\n",
      "Epoch [14][20]\t Batch [450][550]\t Training Loss 1.5173\t Accuracy 0.1222\n",
      "Epoch [14][20]\t Batch [500][550]\t Training Loss 1.5182\t Accuracy 0.1217\n",
      "\n",
      "Epoch [14]\t Average training loss 1.5172\t Average training accuracy 0.1220\n",
      "Epoch [14]\t Average validation loss 1.5089\t Average validation accuracy 0.1190\n",
      "\n",
      "Epoch [15][20]\t Batch [0][550]\t Training Loss 1.5899\t Accuracy 0.1200\n",
      "Epoch [15][20]\t Batch [50][550]\t Training Loss 1.5299\t Accuracy 0.1208\n",
      "Epoch [15][20]\t Batch [100][550]\t Training Loss 1.5214\t Accuracy 0.1211\n",
      "Epoch [15][20]\t Batch [150][550]\t Training Loss 1.5262\t Accuracy 0.1202\n",
      "Epoch [15][20]\t Batch [200][550]\t Training Loss 1.5207\t Accuracy 0.1212\n",
      "Epoch [15][20]\t Batch [250][550]\t Training Loss 1.5223\t Accuracy 0.1210\n",
      "Epoch [15][20]\t Batch [300][550]\t Training Loss 1.5208\t Accuracy 0.1222\n",
      "Epoch [15][20]\t Batch [350][550]\t Training Loss 1.5184\t Accuracy 0.1218\n",
      "Epoch [15][20]\t Batch [400][550]\t Training Loss 1.5178\t Accuracy 0.1216\n",
      "Epoch [15][20]\t Batch [450][550]\t Training Loss 1.5173\t Accuracy 0.1222\n",
      "Epoch [15][20]\t Batch [500][550]\t Training Loss 1.5182\t Accuracy 0.1217\n",
      "\n",
      "Epoch [15]\t Average training loss 1.5172\t Average training accuracy 0.1220\n",
      "Epoch [15]\t Average validation loss 1.5089\t Average validation accuracy 0.1190\n",
      "\n",
      "Epoch [16][20]\t Batch [0][550]\t Training Loss 1.5899\t Accuracy 0.1200\n",
      "Epoch [16][20]\t Batch [50][550]\t Training Loss 1.5299\t Accuracy 0.1208\n",
      "Epoch [16][20]\t Batch [100][550]\t Training Loss 1.5214\t Accuracy 0.1211\n",
      "Epoch [16][20]\t Batch [150][550]\t Training Loss 1.5262\t Accuracy 0.1202\n",
      "Epoch [16][20]\t Batch [200][550]\t Training Loss 1.5207\t Accuracy 0.1212\n",
      "Epoch [16][20]\t Batch [250][550]\t Training Loss 1.5223\t Accuracy 0.1210\n",
      "Epoch [16][20]\t Batch [300][550]\t Training Loss 1.5208\t Accuracy 0.1222\n",
      "Epoch [16][20]\t Batch [350][550]\t Training Loss 1.5184\t Accuracy 0.1218\n",
      "Epoch [16][20]\t Batch [400][550]\t Training Loss 1.5178\t Accuracy 0.1216\n",
      "Epoch [16][20]\t Batch [450][550]\t Training Loss 1.5173\t Accuracy 0.1222\n",
      "Epoch [16][20]\t Batch [500][550]\t Training Loss 1.5182\t Accuracy 0.1217\n",
      "\n",
      "Epoch [16]\t Average training loss 1.5172\t Average training accuracy 0.1220\n",
      "Epoch [16]\t Average validation loss 1.5089\t Average validation accuracy 0.1190\n",
      "\n",
      "Epoch [17][20]\t Batch [0][550]\t Training Loss 1.5899\t Accuracy 0.1200\n",
      "Epoch [17][20]\t Batch [50][550]\t Training Loss 1.5299\t Accuracy 0.1208\n",
      "Epoch [17][20]\t Batch [100][550]\t Training Loss 1.5214\t Accuracy 0.1211\n",
      "Epoch [17][20]\t Batch [150][550]\t Training Loss 1.5262\t Accuracy 0.1202\n",
      "Epoch [17][20]\t Batch [200][550]\t Training Loss 1.5207\t Accuracy 0.1212\n",
      "Epoch [17][20]\t Batch [250][550]\t Training Loss 1.5223\t Accuracy 0.1210\n",
      "Epoch [17][20]\t Batch [300][550]\t Training Loss 1.5208\t Accuracy 0.1222\n",
      "Epoch [17][20]\t Batch [350][550]\t Training Loss 1.5184\t Accuracy 0.1218\n",
      "Epoch [17][20]\t Batch [400][550]\t Training Loss 1.5178\t Accuracy 0.1216\n",
      "Epoch [17][20]\t Batch [450][550]\t Training Loss 1.5173\t Accuracy 0.1222\n",
      "Epoch [17][20]\t Batch [500][550]\t Training Loss 1.5182\t Accuracy 0.1217\n",
      "\n",
      "Epoch [17]\t Average training loss 1.5172\t Average training accuracy 0.1220\n",
      "Epoch [17]\t Average validation loss 1.5089\t Average validation accuracy 0.1190\n",
      "\n",
      "Epoch [18][20]\t Batch [0][550]\t Training Loss 1.5899\t Accuracy 0.1200\n",
      "Epoch [18][20]\t Batch [50][550]\t Training Loss 1.5299\t Accuracy 0.1208\n",
      "Epoch [18][20]\t Batch [100][550]\t Training Loss 1.5214\t Accuracy 0.1211\n",
      "Epoch [18][20]\t Batch [150][550]\t Training Loss 1.5262\t Accuracy 0.1202\n",
      "Epoch [18][20]\t Batch [200][550]\t Training Loss 1.5207\t Accuracy 0.1212\n",
      "Epoch [18][20]\t Batch [250][550]\t Training Loss 1.5223\t Accuracy 0.1210\n",
      "Epoch [18][20]\t Batch [300][550]\t Training Loss 1.5208\t Accuracy 0.1222\n",
      "Epoch [18][20]\t Batch [350][550]\t Training Loss 1.5184\t Accuracy 0.1218\n",
      "Epoch [18][20]\t Batch [400][550]\t Training Loss 1.5178\t Accuracy 0.1216\n",
      "Epoch [18][20]\t Batch [450][550]\t Training Loss 1.5173\t Accuracy 0.1222\n",
      "Epoch [18][20]\t Batch [500][550]\t Training Loss 1.5182\t Accuracy 0.1217\n",
      "\n",
      "Epoch [18]\t Average training loss 1.5172\t Average training accuracy 0.1220\n",
      "Epoch [18]\t Average validation loss 1.5089\t Average validation accuracy 0.1190\n",
      "\n",
      "Epoch [19][20]\t Batch [0][550]\t Training Loss 1.5899\t Accuracy 0.1200\n",
      "Epoch [19][20]\t Batch [50][550]\t Training Loss 1.5299\t Accuracy 0.1208\n",
      "Epoch [19][20]\t Batch [100][550]\t Training Loss 1.5214\t Accuracy 0.1211\n",
      "Epoch [19][20]\t Batch [150][550]\t Training Loss 1.5262\t Accuracy 0.1202\n",
      "Epoch [19][20]\t Batch [200][550]\t Training Loss 1.5207\t Accuracy 0.1212\n",
      "Epoch [19][20]\t Batch [250][550]\t Training Loss 1.5223\t Accuracy 0.1210\n",
      "Epoch [19][20]\t Batch [300][550]\t Training Loss 1.5208\t Accuracy 0.1222\n",
      "Epoch [19][20]\t Batch [350][550]\t Training Loss 1.5184\t Accuracy 0.1218\n",
      "Epoch [19][20]\t Batch [400][550]\t Training Loss 1.5178\t Accuracy 0.1216\n",
      "Epoch [19][20]\t Batch [450][550]\t Training Loss 1.5173\t Accuracy 0.1222\n",
      "Epoch [19][20]\t Batch [500][550]\t Training Loss 1.5182\t Accuracy 0.1217\n",
      "\n",
      "Epoch [19]\t Average training loss 1.5172\t Average training accuracy 0.1220\n",
      "Epoch [19]\t Average validation loss 1.5089\t Average validation accuracy 0.1190\n",
      "\n"
     ]
    }
   ],
   "source": [
    "sigmoidMLP, sigmoid_loss, sigmoid_acc = train(sigmoidMLP, criterion, sgd, data_train, max_epoch, batch_size, disp_freq)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Testing...\n",
      "The test accuracy is 0.1184.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "test(sigmoidMLP, criterion, data_test, batch_size, disp_freq)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2 MLP with Euclidean Loss and ReLU Activation Function\n",
    "Build and train a MLP contraining one hidden layer with 128 units using ReLU activation function and Euclidean loss function.\n",
    "\n",
    "### TODO\n",
    "Before executing the following code, you should complete **layers/relu_layer.py**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from layers import ReLULayer\n",
    "\n",
    "reluMLP = Network()\n",
    "# TODO build ReLUMLP with FCLayer and ReLULayer\n",
    "reluMLP.add(FCLayer(784, 128))\n",
    "reluMLP.add(ReLULayer())\n",
    "reluMLP.add(FCLayer(128, 10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m/Users/youssefbr/Downloads/homework-2/homework2-mlp/homework2-mlp.ipynb Cell 17'\u001b[0m in \u001b[0;36m<cell line: 1>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> <a href='vscode-notebook-cell:/Users/youssefbr/Downloads/homework-2/homework2-mlp/homework2-mlp.ipynb#ch0000016?line=0'>1</a>\u001b[0m reluMLP, relu_loss, relu_acc \u001b[39m=\u001b[39m train(reluMLP, criterion, sgd, data_train, max_epoch, batch_size, disp_freq)\n",
      "File \u001b[0;32m~/Downloads/homework-2/homework2-mlp/solver.py:35\u001b[0m, in \u001b[0;36mtrain\u001b[0;34m(model, criterion, optimizer, dataset, max_epoch, batch_size, disp_freq)\u001b[0m\n\u001b[1;32m     33\u001b[0m \u001b[39m# Training process\u001b[39;00m\n\u001b[1;32m     34\u001b[0m \u001b[39mfor\u001b[39;00m epoch \u001b[39min\u001b[39;00m \u001b[39mrange\u001b[39m(max_epoch):\n\u001b[0;32m---> 35\u001b[0m \tbatch_train_loss, batch_train_acc \u001b[39m=\u001b[39m train_one_epoch(model, criterion, optimizer, train_get_next,\n\u001b[1;32m     36\u001b[0m \t                                                    max_epoch, batch_size, disp_freq, epoch, sess)\n\u001b[1;32m     37\u001b[0m \tbatch_val_loss, batch_val_acc \u001b[39m=\u001b[39m validate(model, criterion, valid_get_next, batch_size, sess)\n\u001b[1;32m     39\u001b[0m \tavg_train_acc\u001b[39m.\u001b[39mappend(np\u001b[39m.\u001b[39mmean(batch_train_acc))\n",
      "File \u001b[0;32m~/Downloads/homework-2/homework2-mlp/solver.py:71\u001b[0m, in \u001b[0;36mtrain_one_epoch\u001b[0;34m(model, criterion, optimizer, data_get_next, max_epoch, batch_size, disp_freq, epoch, sess)\u001b[0m\n\u001b[1;32m     69\u001b[0m \u001b[39m# Backward pass\u001b[39;00m\n\u001b[1;32m     70\u001b[0m delta \u001b[39m=\u001b[39m criterion\u001b[39m.\u001b[39mbackward()\n\u001b[0;32m---> 71\u001b[0m model\u001b[39m.\u001b[39;49mbackward(delta)\n\u001b[1;32m     73\u001b[0m \u001b[39m# Update weights, see optimize.py\u001b[39;00m\n\u001b[1;32m     74\u001b[0m optimizer\u001b[39m.\u001b[39mstep(model)\n",
      "File \u001b[0;32m~/Downloads/homework-2/homework2-mlp/network.py:21\u001b[0m, in \u001b[0;36mNetwork.backward\u001b[0;34m(self, delta)\u001b[0m\n\u001b[1;32m     18\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39mbackward\u001b[39m(\u001b[39mself\u001b[39m, delta):\n\u001b[1;32m     19\u001b[0m \t\u001b[39m# backward layer by layer\u001b[39;00m\n\u001b[1;32m     20\u001b[0m \t\u001b[39mfor\u001b[39;00m i \u001b[39min\u001b[39;00m \u001b[39mreversed\u001b[39m(\u001b[39mrange\u001b[39m(\u001b[39mself\u001b[39m\u001b[39m.\u001b[39mnumLayer)): \u001b[39m# reversed\u001b[39;00m\n\u001b[0;32m---> 21\u001b[0m \t\tdelta \u001b[39m=\u001b[39m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mlayerList[i]\u001b[39m.\u001b[39;49mbackward(delta)\n",
      "File \u001b[0;32m~/Downloads/homework-2/homework2-mlp/layers/relu_layer.py:45\u001b[0m, in \u001b[0;36mReLULayer.backward\u001b[0;34m(self, delta)\u001b[0m\n\u001b[1;32m     38\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39mbackward\u001b[39m(\u001b[39mself\u001b[39m, delta):\n\u001b[1;32m     39\u001b[0m \n\u001b[1;32m     40\u001b[0m \t\u001b[39m############################################################################\u001b[39;00m\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m     43\u001b[0m \n\u001b[1;32m     44\u001b[0m \t\u001b[39m#return np.dot(self.out, np.dot(np.transpose(self.relu(self.out)), delta))\u001b[39;00m\n\u001b[0;32m---> 45\u001b[0m \t\u001b[39mreturn\u001b[39;00m delta \u001b[39m*\u001b[39m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mreluP(\u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mout)\n",
      "File \u001b[0;32m~/Downloads/homework-2/homework2-mlp/layers/relu_layer.py:12\u001b[0m, in \u001b[0;36mReLULayer.reluP\u001b[0;34m(z)\u001b[0m\n\u001b[1;32m     10\u001b[0m \u001b[39m@staticmethod\u001b[39m\n\u001b[1;32m     11\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39mreluP\u001b[39m(z):\n\u001b[0;32m---> 12\u001b[0m \t\u001b[39mif\u001b[39;00m(z \u001b[39m>\u001b[39m \u001b[39m0\u001b[39m):\n\u001b[1;32m     13\u001b[0m \t\t\u001b[39mreturn\u001b[39;00m \u001b[39m1\u001b[39m\n\u001b[1;32m     14\u001b[0m \t\u001b[39melse\u001b[39;00m:\n",
      "\u001b[0;31mValueError\u001b[0m: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()"
     ]
    }
   ],
   "source": [
    "reluMLP, relu_loss, relu_acc = train(reluMLP, criterion, sgd, data_train, max_epoch, batch_size, disp_freq)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Testing...\n",
      "The test accuracy is 0.0980.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "test(reluMLP, criterion, data_test, batch_size, disp_freq)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEJCAYAAACDscAcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAZNklEQVR4nO3df7jWdZ3n8ecbDoiCvxIsBAScsBFFEY9IUymamZKC7lToZVPqFGmLZbPTqLXXZjldTZPbzrjbpcMqlmujO2s/hhxntLXCtvXXAc1CJFmlPEIruP4ifyD63j/uL8zxeID7I+d77ht4Pq7rvs79/f2+73Of+3U+3x+fb2QmkiQ1a1CrC5Ak7VgMDklSEYNDklTE4JAkFTE4JElFDA5JUpFagyMiTo6IFRGxMiIu6WP6H0bEXRHxckT8ecmykqTWiLqu44iIwcCvgfcB3cB9wFmZ+VCPefYHxgOnA09n5hXNLitJao2OGtc9HViZmY8CRMRNwBxg85d/Zj4JPBkRHyhdti8jR47MCRMm9NsLkKSd3ZIlS9Zl5qiSZeoMjjHA4z2Gu4Fj+nvZiJgHzAM48MAD6erqKq9UknZREfGb0mXqPMYRfYxrdr9Y08tm5oLM7MzMzlGjikJTkvQm1Bkc3cC4HsNjgdUDsKwkqUZ1Bsd9wKSImBgRQ4EzgUUDsKwkqUa1HePIzI0RMR+4DRgMLMzMZRFxfjX96oh4G9AF7AW8FhEXAZMz87m+lq2rVkk7pldeeYXu7m5eeumlVpfS9oYNG8bYsWMZMmTIdq+rttNxW6GzszM9OC7tOh577DH23HNP9ttvPyL6OjQqgMzkqaee4vnnn2fixImvmxYRSzKzs2R9XjkuaYf10ksvGRpNiAj222+/fmuZGRySdmiGRnP6830yOCRJRQwOSdpOX/nKVzj00EM5/PDDmTp1Kvfccw8f//jHeeihentJmjVrFs8888wbxl922WVcccUVtW23zivHJaltdP7lj1i3fsMbxo8cMZSuf/++N73eu+66i1tuuYWlS5ey2267sW7dOjZs2MA111yzPeU25dZbb619G32xxSFpl9BXaGxtfLPWrFnDyJEj2W233QAYOXIkBxxwADNnztzcBdK1117LwQcfzMyZM/nEJz7B/PnzATjnnHO44IILOP744znooINYvHgx5513HocccgjnnHPO5m3ceOONTJkyhcMOO4yLL7548/gJEyawbt06oNHqecc73sGJJ57IihUrtus1bYstDkk7hS/9cBkPrX7uTS079+/u6nP85AP24ounHbrVZU866SS+/OUvc/DBB3PiiScyd+5cjjvuuM3TV69ezeWXX87SpUvZc889OeGEEzjiiCM2T3/66af58Y9/zKJFizjttNP4+c9/zjXXXMPRRx/NAw88wP7778/FF1/MkiVL2HfffTnppJP4wQ9+wOmnn755HUuWLOGmm27i/vvvZ+PGjUybNo2jjjrqTb0XzbDFIUnbYcSIESxZsoQFCxYwatQo5s6dy7e+9a3N0++9916OO+443vKWtzBkyBA+9KEPvW750047jYhgypQpvPWtb2XKlCkMGjSIQw89lFWrVnHfffcxc+ZMRo0aRUdHB2effTZ33nnn69bxs5/9jDPOOIM99tiDvfbai9mzZ9f6mm1xSNopbKtlMOGSf9ritP/+yXdu17YHDx7MzJkzmTlzJlOmTOHb3/725mnbush60y6uQYMGbX6+aXjjxo10dDT3NT2QpyXb4pCk7bBixQoeeeSRzcMPPPAA48eP3zw8ffp0Fi9ezNNPP83GjRv57ne/W7T+Y445hsWLF7Nu3TpeffVVbrzxxtftCgM49thj+f73v8+LL77I888/zw9/+MPte1HbYItD0i5h5IihWzyranusX7+eCy+8kGeeeYaOjg7e/va3s2DBAj74wQ8CMGbMGD7/+c9zzDHHcMABBzB58mT23nvvptc/evRovvrVr3L88ceTmcyaNYs5c+a8bp5p06Yxd+5cpk6dyvjx43nPe96zXa9pW+yrStIOa/ny5RxyyCGtLmOb1q9fz4gRI9i4cSNnnHEG5513HmecccaA19HX+2VfVZLUhi677DKmTp3KYYcdxsSJE193RtSOyF1VklSzOq/ibgVbHJKkIgaHJKmIwSFJKmJwSJKKGBySNABGjBjR6hL6jWdVSdo1fH0S/P7JN44fvj987pE3jn8TMpPMZNCgnft/8p371UnSJn2FxtbGN2nVqlUccsghfOpTn2LatGlcfvnlHH300Rx++OF88YtffMP8P/3pTzn11FM3D8+fP/91nSLuCGxxSNo5/PMl8Ltfvrllr/tA3+PfNgVO+attLr5ixQquu+46Tj/9dG6++WbuvfdeMpPZs2dz5513cuyxx765utqULQ5J2k7jx49nxowZ3H777dx+++0ceeSRTJs2jYcffvh1HSDuLGxxSNo5bKtlcNlWOhY8d8tdrjdj+PDhQOMYx6WXXsonP/nJLc7b0dHBa6+9tnn4pZde2q5tt4ItDknqJ+9///tZuHAh69evB+CJJ57gySdffwxl/PjxPPTQQ7z88ss8++yz3HHHHa0odbvY4pC0axi+/5bPquonJ510EsuXL+ed72zcGGrEiBHccMMN7L//v25j3LhxfPjDH+bwww9n0qRJHHnkkf22/YFit+qSdlg7Srfq7cJu1SVJLWFwSJKKGBySdmg70+72OvXn+2RwSNphDRs2jKeeesrw2IbM5KmnnmLYsGH9sj7PqpK0wxo7dizd3d2sXbu21aW0vWHDhjF27Nh+WZfBIWmHNWTIECZOnNjqMnY57qqSJBWpNTgi4uSIWBERKyPikj6mR0RcWU1/MCKm9Zj22YhYFhG/iogbI6J/ds5JkrZLbcEREYOBbwKnAJOBsyJicq/ZTgEmVY95wFXVsmOATwOdmXkYMBg4s65aJUnNq7PFMR1YmZmPZuYG4CZgTq955gDXZ8PdwD4RMbqa1gHsHhEdwB7A6hprlSQ1qc7gGAM83mO4uxq3zXky8wngCuC3wBrg2cy8vcZaJUlNqjM4oo9xvU+27nOeiNiXRmtkInAAMDwiPtLnRiLmRURXRHR5Sp4k1a/O4OgGxvUYHssbdzdtaZ4Tgccyc21mvgJ8D/ijvjaSmQsyszMzO0eNGtVvxUuS+lZncNwHTIqIiRExlMbB7UW95lkEfLQ6u2oGjV1Sa2jsopoREXtERADvBZbXWKskqUm1XQCYmRsjYj5wG42zohZm5rKIOL+afjVwKzALWAm8AJxbTbsnIm4GlgIbgfuBBXXVKklqnvfjkKRdmPfjkCTVzuCQJBUxOCRJRQwOSVIRg0OSVMTgkCQVMTgkSUUMDklSEYNDklTE4JAkFTE4JElFDA5JUhGDQ5JUxOCQJBUxOCRJRQwOSVIRg0OSVMTgkCQVMTgkSUUMDklSEYNDklTE4JAkFTE4JElFDA5JUhGDQ5JUxOCQJBUxOCRJRQwOSVIRg0OSVMTgkCQVMTgkSUUMDklSEYNDklTE4JAkFTE4JElFag2OiDg5IlZExMqIuKSP6RERV1bTH4yIaT2m7RMRN0fEwxGxPCLeWWetkqTm1BYcETEY+CZwCjAZOCsiJvea7RRgUvWYB1zVY9rfAv+SmX8IHAEsr6tWSVLz6mxxTAdWZuajmbkBuAmY02ueOcD12XA3sE9EjI6IvYBjgWsBMnNDZj5TY62SpCbVGRxjgMd7DHdX45qZ5yBgLXBdRNwfEddExPC+NhIR8yKiKyK61q5d23/VS5L6VGdwRB/jssl5OoBpwFWZeSTwe+ANx0gAMnNBZnZmZueoUaO2p15JUhPqDI5uYFyP4bHA6ibn6Qa6M/OeavzNNIJEktRidQbHfcCkiJgYEUOBM4FFveZZBHy0OrtqBvBsZq7JzN8Bj0fEO6r53gs8VGOtkqQmddS14szcGBHzgduAwcDCzFwWEedX068GbgVmASuBF4Bze6ziQuA7Veg82muaJKlFIrP3YYcdV2dnZ3Z1dbW6DEnaYUTEkszsLFnGK8clSUUMDklSEYNDklTE4JAkFTE4JElFDA5JUhGDQ5JUpKngiIjhETGoen5wRMyOiCH1liZJakfNtjjuBIZFxBjgDhpXcX+rrqIkSe2r2eCIzHwB+DfAf87MM2jcnEmStItpOjiqW7eeDfxTNa62fq4kSe2r2eC4CLgU+H7VUeFBwE9qq0qS1LaaajVk5mJgMUB1kHxdZn66zsIkSe2p2bOq/j4i9qpu3/oQsCIiPldvaZKkdtTsrqrJmfkccDqNe2gcCPxJXUVJktpXs8ExpLpu43TgHzPzFd54/3BJ0i6g2eD4O2AVMBy4MyLGA8/VVZQkqX01e3D8SuDKHqN+ExHH11OSJKmdNXtwfO+I+EZEdFWP/0ij9SFJ2sU0u6tqIfA88OHq8RxwXV1FSZLaV7NXf/9BZv5xj+EvRcQDNdQjSWpzzbY4XoyId28aiIh3AS/WU5IkqZ012+I4H7g+Ivauhp8GPlZPSZKkdtbsWVW/AI6IiL2q4eci4iLgwRprkyS1oaI7AGbmc9UV5AB/VkM9kqQ2tz23jo1+q0KStMPYnuCwyxFJ2gVt9RhHRDxP3wERwO61VCRJamtbDY7M3HOgCpEk7Ri2Z1eVJGkXZHBIkooYHJKkIgaHJKmIwSFJKlJrcETEyRGxIiJWRsQlfUyPiLiymv5gREzrNX1wRNwfEbfUWackqXm1BUdEDAa+CZwCTAbOiojJvWY7BZhUPeYBV/Wa/hlgeV01SpLK1dnimA6szMxHM3MDcBMwp9c8c4Drs+FuYJ+IGA0QEWOBDwDX1FijJKlQncExBni8x3B3Na7Zef4G+Avgta1tJCLmbbql7dq1a7erYEnSttUZHH11gti7+5I+54mIU4EnM3PJtjaSmQsyszMzO0eNGvVm6pQkFagzOLqBcT2GxwKrm5znXcDsiFhFYxfXCRFxQ32lSpKaVWdw3AdMioiJETEUOBNY1GueRcBHq7OrZgDPZuaazLw0M8dm5oRquR9n5kdqrFWS1KRmbx1bLDM3RsR84DZgMLAwM5dFxPnV9KuBW4FZwErgBeDcuuqRJPWPyNx5bqvR2dmZXV1drS5DknYYEbEkMztLlvHKcUlSEYNDklTE4JAkFTE4JElFDA5JUhGDQ5JUxOCQJBUxOCRJRQwOSVIRg0OSVMTgkCQVMTgkSUUMDklSEYNDklTE4JAkFTE4JElFDA5JUhGDQ5JUxOCQJBUxOCRJRQwOSVIRg0OSVMTgkCQVMTgkSUUMDklSEYNDklTE4JAkFTE4JElFDA5JUhGDQ5JUxOCQJBUxOCRJRQwOSVIRg0OSVKTW4IiIkyNiRUSsjIhL+pgeEXFlNf3BiJhWjR8XET+JiOURsSwiPlNnnZKk5tUWHBExGPgmcAowGTgrIib3mu0UYFL1mAdcVY3fCPy7zDwEmAH82z6WlSS1QJ0tjunAysx8NDM3ADcBc3rNMwe4PhvuBvaJiNGZuSYzlwJk5vPAcmBMjbVKkppUZ3CMAR7vMdzNG7/8tzlPREwAjgTu6f8SJUml6gyO6GNclswTESOA7wIXZeZzfW4kYl5EdEVE19q1a990sZKk5tQZHN3AuB7DY4HVzc4TEUNohMZ3MvN7W9pIZi7IzM7M7Bw1alS/FC5J2rI6g+M+YFJETIyIocCZwKJe8ywCPlqdXTUDeDYz10REANcCyzPzGzXWKEkq1FHXijNzY0TMB24DBgMLM3NZRJxfTb8auBWYBawEXgDOrRZ/F/AnwC8j4oFq3Ocz89a66pUkNScyex922HF1dnZmV1dXq8uQpB1GRCzJzM6SZbxyXJJUxOCQJBUxOCRJRQwOSVIRg0OSVMTgkCQVMTgkSUUMDklSEYNDklTE4JAkFTE4JElFDA5JUhGDQ5JUxOCQJBUxOCRJRQwOSVIRg0OSVMTgkCQVMTgkSUUMDklSEYNDklTE4JAkFTE4JElFDA5JUhGDQ5JUxOCQJBUxOCRJRQwOSVIRg0OSVMTgkCQVMTgkSUUMDklSEYNDklTE4JAkFelodQGttu6yAxnJs28cz96MvOy3u9Q62qGG/lhHO9TQLutohxraZR3tUEO7rKPn8keNHnRUUxvtodYWR0ScHBErImJlRFzSx/SIiCur6Q9GxLRml+0vfb35Wxu/M6+jHWroj3W0Qw3tso52qKFd1tEONbTLOkq21ZfIzO1awRZXHDEY+DXwPqAbuA84KzMf6jHPLOBCYBZwDPC3mXlMM8v2pbOzM7u6usoKvWzvLU5aNnRKU6s4dMMvd4p1tEMN/bGOdqihXdbRDjW0yzraoYZ2WUfP5TsXrKdr9avR1IYrdbY4pgMrM/PRzNwA3ATM6TXPHOD6bLgb2CciRje5rCSpBeo8xjEGeLzHcDeNVsW25hnT5LIARMQ8YF41+HJE/KqkyK3t31uy5udLdqV1tEMN/bGOdqihXdbRDjW0yzraoYZ2WUfP5Vc981ozm3ydOoOjr6ZP7/1iW5qnmWUbIzMXAAsAIqIrMztLiuxv7VBDu9RhDe1VRzvU0C51tEMN7VJHRBTu3683OLqBcT2GxwKrm5xnaBPLSpJaoM5jHPcBkyJiYkQMBc4EFvWaZxHw0ersqhnAs5m5psllJUktUFuLIzM3RsR84DZgMLAwM5dFxPnV9KuBW2mcUbUSeAE4d2vLNrHZBf3/Soq1Qw3QHnVYw79qhzraoQZojzraoQZojzqKa6jtdFxJ0s7JLkckSUUMDklSkZ0iOAaqe5Jt1DAuIn4SEcsjYllEfKYVdVS1DI6I+yPilhbWsE9E3BwRD1fvyTtbUMNnq9/FryLixogYNkDbXRgRT/a8pigi3hIRP4qIR6qf+7aghq9Xv48HI+L7EbFPnTVsqY4e0/48IjIiRraihoi4sPreWBYRf11nDVuqIyKmRsTdEfFARHRFxPSaa+jze6r485mZO/SDxsHz/wMcROM03l8Ak1tQx2hgWvV8Txpdpgx4HdX2/wz4e+CWFv5evg18vHo+FNhngLc/BngM2L0a/gfgnAHa9rHANOBXPcb9NXBJ9fwS4GstqOEkoKN6/rW6a9hSHdX4cTROfvkNMLIF78XxwP8EdquG92/R5+J24JTq+SzgpzXX0Of3VOnnc2docbRF9ySZuSYzl1bPnweW0/jyGlARMRb4AHDNQG+7Rw170fgjuRYgMzdk5jMtKKUD2D0iOoA9GKBrgTLzTuD/9Ro9h0aYUv08faBryMzbM3NjNXg3jeujarWF9wLgPwF/wRYu7B2AGi4A/iozX67mebJFdSSwV/V8b2r+jG7le6ro87kzBMeWui1pmYiYABwJ3NOCzf8NjT/I8n4E+s9BwFrgumqX2TURMXwgC8jMJ4ArgN8Ca2hcI3T7QNbQy1uzcY0S1c/9W1gLwHnAP7diwxExG3giM3/Riu1XDgbeExH3RMTiiDi6RXVcBHw9Ih6n8Xm9dKA23Ot7qujzuTMER9PdkwyEiBgBfBe4KDOfG+Btnwo8mZlN9XdTow4aTfKrMvNI4Pc0mr8DptpHOweYCBwADI+IjwxkDe0qIr4AbAS+04Jt7wF8AfgPA73tXjqAfYEZwOeAf4iIoh5i+8kFwGczcxzwWapWet2293tqZwiOZro2GRARMYTGL+M7mfm9FpTwLmB2RKyiscvuhIi4oQV1dAPdmbmpxXUzjSAZSCcCj2Xm2sx8Bfge8EcDXENP/7fq+ZnqZ+27RvoSER8DTgXOzmqH9gD7Axph/ovqczoWWBoRbxvgOrqB72XDvTRa6LUepN+Cj9H4bAL8Dxq73mu1he+pos/nzhAcbdE9SfXfyrXA8sz8xkBvHyAzL83MsZk5gcb78OPMHPD/sjPzd8DjEfGOatR7ga3eS6UGvwVmRMQe1e/mvTT257bKIhpfElQ//3GgC4iIk4GLgdmZ+cJAbx8gM3+Zmftn5oTqc9pN42Dt7wa4lB8AJwBExME0TuBYN8A1QOOf3OOq5ycAj9S5sa18T5V9Pus+k2AgHjTORvg1jbOrvtCiGt5NYxfZg8AD1WNWC9+TmbT2rKqpQFf1fvwA2LcFNXwJeBj4FfDfqM6gGYDt3kjjuMorNL4Y/xTYD7iDxhfDHcBbWlDDShrHAzd9Pq9uxXvRa/oq6j+rqq/3YihwQ/XZWAqc0KLPxbuBJTTOBr0HOKrmGvr8nir9fNrliCSpyM6wq0qSNIAMDklSEYNDklTE4JAkFTE4JElFDA6pQES8WvVkuunRb1fER8SEvnqRldpNbbeOlXZSL2bm1FYXIbWSLQ6pH0TEqoj4WkTcWz3eXo0fHxF3VPfAuCMiDqzGv7W6J8Yvqsem7lAGR8R/re6VcHtE7N6yFyVtgcEhldm9166quT2mPZeZ04H/QqOXYqrn12fm4TQ6FbyyGn8lsDgzj6DRj9eyavwk4JuZeSjwDPDHtb4a6U3wynGpQESsz8wRfYxfRaPbikerTuR+l5n7RcQ6YHRmvlKNX5OZIyNiLTA2q/tBVOuYAPwoMydVwxcDQzLzLwfgpUlNs8Uh9Z/cwvMtzdOXl3s8fxWPQ6oNGRxS/5nb4+dd1fP/TaOnYoCzgf9VPb+Dxr0YNt0jftNd4KS2538zUpndI+KBHsP/kpmbTsndLSLuofEP2VnVuE8DCyPiczTuinhuNf4zwIKI+FMaLYsLaPScKrU9j3FI/aA6xtGZma24p4M0oNxVJUkqYotDklTEFockqYjBIUkqYnBIkooYHJKkIgaHJKnI/wdFc//BH/+/2gAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZUAAAEGCAYAAACtqQjWAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAfd0lEQVR4nO3dfbhVdZ338fcHDoiCKAooAgNU+IBiRzyiTaX4REIKcpWh0+RjoRaWzdiIdV8T5TSZWd2X93jJoGJ2ZzilUdRNqUOp3d0+cA6RikSQQ3oE5cFHMsAD3/uPtQ6zOWzO2fuw1t77HD6v69rXXuu31u+3vmuz2d+zfmut31JEYGZmloUe1Q7AzMy6DycVMzPLjJOKmZllxknFzMwy46RiZmaZqat2AJUwcODAGDlyZLXDMDPrUpqamjZGxKBy6uwTSWXkyJE0NjZWOwwzsy5F0p/LrePuLzMzy4yTipmZZcZJxczMMrNPnFMxs33LO++8Q3NzM1u2bKl2KF1Cnz59GDZsGL169drrtpxUzKzbaW5u5sADD2TkyJFIqnY4NS0i2LRpE83NzYwaNWqv23P3l5l1O1u2bOHQQw91QimBJA499NDMjuqcVMysW3JCKV2Wn5WTipmZZcZJxcwsJ1/72tc49thjOf7446mvr+fJJ5/kk5/8JM8991yu2508eTKvv/76buWzZ8/mlltuyXXbPlFvZvu0hn95mI2bt+1WPrBfbxr/x9mdbvfxxx/n5z//OUuXLmW//fZj48aNbNu2jTvvvHNvwi3JokWLct/GnvhIxcz2acUSSnvlpVq3bh0DBw5kv/32A2DgwIEcccQRTJgwYeewUXfddRdHHnkkEyZM4FOf+hQzZ84E4NJLL+Xqq6/m9NNP513vehePPvool19+OccccwyXXnrpzm3Mnz+fsWPHctxxx3H99dfvLB85ciQbN24EkqOlo446irPOOouVK1fu1T6VwkcqZtatfeVny3lu7Zudqjv93x8vWj7miP58+bxj2607ceJEvvrVr3LkkUdy1llnMX36dE477bSdy9euXcuNN97I0qVLOfDAAznjjDN473vfu3P5a6+9xq9+9SsWLlzIeeedx29/+1vuvPNOTjrpJJYtW8bgwYO5/vrraWpqYsCAAUycOJGf/OQnnH/++TvbaGpq4r777uN3v/sdLS0tjBs3jhNPPLFTn0WpfKRiZpaDfv360dTUxNy5cxk0aBDTp0/nu9/97s7lTz31FKeddhqHHHIIvXr14oILLtil/nnnnYckxo4dy2GHHcbYsWPp0aMHxx57LGvWrGHJkiVMmDCBQYMGUVdXx8c//nEee+yxXdr4zW9+w7Rp0zjggAPo378/U6ZMyX2/faRiZt1aR0cUI2f9nz0u+48r37dX2+7ZsycTJkxgwoQJjB07lnvuuWfnsohot25rt1mPHj12TrfOt7S0UFdX2s93pS+t9pGKmVkOVq5cyapVq3bOL1u2jBEjRuycHz9+PI8++iivvfYaLS0tPPDAA2W1f/LJJ/Poo4+yceNGtm/fzvz583fpXgM49dRTWbBgAX/961956623+NnPfrZ3O1UCH6mY2T5tYL/ee7z6a29s3ryZa665htdff526ujre8573MHfuXD760Y8CMHToUL74xS9y8sknc8QRRzBmzBgOOuigktsfMmQIX//61zn99NOJCCZPnszUqVN3WWfcuHFMnz6d+vp6RowYwQc/+MG92qdSqKNDsO6goaEh/JAus33HihUrOOaYY6odRoc2b95Mv379aGlpYdq0aVx++eVMmzatKrEU+8wkNUVEQzntuPvLzKxKZs+eTX19PccddxyjRo3a5cqtrsrdX2ZmVZL33e3V4CMVMzPLTK5JRdI5klZKWi1pVpHlR0t6XNJWSdcVlB8laVnB601J16bLZkt6qWDZ5Dz3wczMSpdb95eknsBtwNlAM7BE0sKIKBxJ7VXgs8D5hXUjYiVQX9DOS8CCglW+ExHd77jRzKyLy/NIZTywOiKej4htwH3ALte7RcT6iFgCvNNOO2cCf4qIP+cXqpmZZSHPpDIUeLFgvjktK9eFwPw2ZTMlPS1pnqQBxSpJmiGpUVLjhg0bOrFZM7P89evXr9ohZCrPq7+KjQ1Q1k0xknoDU4AbCopvB25M27oR+BZw+W4bipgLzIXkPpVytmtm+5Bvjoa/rN+9vO9g+MKq3cs7ISKICHr06P7XRuW5h83A8IL5YcDaMtuYBCyNiFdaCyLilYjYHhE7gDtIutnMzDqnWEJpr7xEa9as4ZhjjuHTn/4048aN48Ybb+Skk07i+OOP58tf/vJu6z/yyCOce+65O+dnzpy5ywCUXUWeRypLgNGSRpGcaL8Q+Lsy27iINl1fkoZExLp0dhrw7N4Gambd2C9mwcvPdK7u3R8uXn74WJh0U4fVV65cyd13383555/P/fffz1NPPUVEMGXKFB577DFOPfXUzsVVw3JLKhHRImkm8CDQE5gXEcslXZUunyPpcKAR6A/sSC8bHhMRb0o6gOTKsSvbNH2zpHqS7q81RZabmdWEESNGcMopp3Ddddfx0EMPccIJJwDJ8CyrVq1yUilXRCwCFrUpm1Mw/TJJt1ixum8DhxYp/0TGYZpZd9bREcXsdgZxvGzPw+KXom/fvkByTuWGG27gyiv3/DdwXV0dO3bs2Dm/ZcuWvdp2tXT/s0ZmZlX2oQ99iHnz5rF582YAXnrpJdav3/WczYgRI3juuefYunUrb7zxBosXL65GqHvNY3+Z2b6t7+A9X/2VkYkTJ7JixQre977koV/9+vXj+9//PoMH//c2hg8fzsc+9jGOP/54Ro8evbOrrKvx0Pdm1u10laHva4mHvjczs5rjpGJmZplxUjGzbmlf6NrPSpaflZOKmXU7ffr0YdOmTU4sJYgINm3aRJ8+fTJpz1d/mVm3M2zYMJqbm/FgsqXp06cPw4YVvWWwbE4qZtbt9OrVi1GjRlU7jH2Su7/MzCwzTipmZpYZJxUzM8uMk4qZmWXGScXMzDLjpGJmZplxUjEzs8w4qZiZWWacVMzMLDNOKmZmlplck4qkcyStlLRa0qwiy4+W9LikrZKua7NsjaRnJC2T1FhQfoikhyWtSt8H5LkPZmZWutySiqSewG3AJGAMcJGkMW1WexX4LHDLHpo5PSLq2zx5bBawOCJGA4vTeTMzqwF5HqmMB1ZHxPMRsQ24D5hauEJErI+IJcA7ZbQ7Fbgnnb4HOD+DWM3MLAN5JpWhwIsF881pWakCeEhSk6QZBeWHRcQ6gPR9cLHKkmZIapTU6OGvzcwqI8+koiJl5Twx5/0RMY6k++wzkk4tZ+MRMTciGiKiYdCgQeVUNTOzTsozqTQDwwvmhwFrS60cEWvT9/XAApLuNIBXJA0BSN/XZxKtmZnttTyTyhJgtKRRknoDFwILS6koqa+kA1ungYnAs+nihcAl6fQlwE8zjdrMzDottyc/RkSLpJnAg0BPYF5ELJd0Vbp8jqTDgUagP7BD0rUkV4oNBBZIao3xBxHxy7Tpm4AfSroCeAG4IK99MDOz8iiinNMcXVNDQ0M0NjZ2vKKZme0kqanNLR0d8h31ZmaWGScVMzPLjJOKmZllxknFzMwy46RiZmaZcVIxM7PMOKmYmVlmnFTMzCwzTipmZpYZJxUzM8uMk4qZmWXGScXMzDLjpGJmZplxUjEzs8w4qZiZWWacVMzMLDNOKmZmlhknFTMzy0yuSUXSOZJWSlotaVaR5UdLelzSVknXFZQPl/RrSSskLZf0uYJlsyW9JGlZ+pqc5z6YmVnp6vJqWFJP4DbgbKAZWCJpYUQ8V7Daq8BngfPbVG8B/jEilko6EGiS9HBB3e9ExC15xW5mZp2T55HKeGB1RDwfEduA+4CphStExPqIWAK806Z8XUQsTaffAlYAQ3OM1czMMpBnUhkKvFgw30wnEoOkkcAJwJMFxTMlPS1pnqQBe6g3Q1KjpMYNGzaUu1kzM+uEPJOKipRFWQ1I/YAHgGsj4s20+Hbg3UA9sA74VrG6ETE3IhoiomHQoEHlbNbMzDopz6TSDAwvmB8GrC21sqReJAnl3oj4cWt5RLwSEdsjYgdwB0k3m5mZ1YA8k8oSYLSkUZJ6AxcCC0upKEnAXcCKiPh2m2VDCmanAc9mFK+Zme2l3K7+iogWSTOBB4GewLyIWC7pqnT5HEmHA41Af2CHpGuBMcDxwCeAZyQtS5v8YkQsAm6WVE/SlbYGuDKvfTAzs/IooqzTHF1SQ0NDNDY2VjsMM7MuRVJTRDSUU8d31JuZWWacVMzMLDNOKmZmlpkOk4qkcyU5+ZiZWYdKSRYXAqsk3SzpmLwDMjOzrqvDpBIRf08yTMqfgLvTUYVnpAM9mpmZ7VRSt1Y6RMoDJINCDiG56XCppGtyjM3MzLqYUs6pnCdpAfAroBcwPiImAe8Frmu3spmZ7VNKuaP+ApLnlzxWWBgRb0u6PJ+wzMysKyolqXyZZDRgACTtDxwWEWsiYnFukZmZWZdTyjmVHwE7Cua3p2VmZma7KCWp1KVPbgQgne6dX0hmZtZVlZJUNkia0jojaSqwMb+QzMysqyrlnMpVwL2S/o3kaY4vAhfnGpWZmXVJHSaViPgTcEr6aF9FxFv5h2VmZl1RSQ/pkvRh4FigT/JQRoiIr+YYl5mZdUGl3Pw4B5gOXEPS/XUBMCLnuMzMrAsq5UT930bExcBrEfEV4H3A8HzDMjOzrqiUpLIlfX9b0hHAO8Co/EIyM7OuqpSk8jNJBwPfBJYCa4D5pTQu6RxJKyWtljSryPKj01GPt0q6rpS6kg6R9LCkVen7gFJiMTOz/LWbVNKHcy2OiNcj4gGScylHR8Q/d9SwpJ7AbcAkYAxwkaQxbVZ7FfgscEsZdWelMY0GFqfzZmZWA9q9+isidkj6Fsl5FCJiK7C1xLbHA6sj4nkASfcBU4HnCtpfD6xPry4rte5UYEK63j3AI8D1JcZUlo2z/4aBvLF7OQcxcPYLudfvTm3UQgy10kYtxFArbdRCDLXSRi3E0LaNE4f0OLGkSgVK6f56SNJH1HotcemGktwo2ao5LdvbuodFxDqA9H1wsQbSB4k1SmrcsGFDWYG3KvaP01551vW7Uxu1EEOttFELMdRKG7UQQ620UQsxlLtuMaXcp/IPQF+gRdIWksuKIyL6d1CvWBKKEuPam7rJyhFzgbkADQ0NZdUtxfJ//UCH6xy7l/W7Uxu1EEOttFELMdRKG7UQQ620UQsxdNRGKUq5o76zjw1uZtdLj4cBazOo+4qkIRGxTtIQYH0n4zMzs4wpov0/4iWdWqy87UO7itSrA/4InAm8BCwB/i4ilhdZdzawOSJu6aiupG8CmyLipvSqsEMi4p/ai6WhoSEaGxvb3c+iZh/UzrISDhH3tn53aqMWYqiVNmohhlppoxZiqJU2aiGGNm00zN1M49rtZZ36KKX76wsF031ITqI3AWe0VykiWiTNBB4EegLz0qRwVbp8jqTDgUagP7BD0rXAmIh4s1jdtOmbgB9KugJ4geQOfzMzqwEdnqiPiPMKXmcDxwGvlNJ4RCyKiCMj4t0R8bW0bE5EzEmnX46IYRHRPyIOTqff3FPdtHxTRJwZEaPT91c7s+Ol2EjxrL+n8qzrd6c2aiGGWmmjFmKolTZqIYZaaaMWYih33WI67P7arUJyFdjTETF2r7ZcQZ3u/jIz24dJaoqIhnLqdNj9Jel/8d9XXvUA6oHflx2dmZl1e6WcUyn8E78FmB8Rv80pHjMz68JKSSr3A1siYjskQ6hIOiAi3s43NDMz62pKuaN+MbB/wfz+wH/mE46ZmXVlpSSVPhGxuXUmnT4gv5DMzKyrKiWp/EXSuNYZSScCf80vJDMz66pKOadyLfAjSa3DpAwhebywmZnZLkoZ+2uJpKOBo0gGevxDRLyTe2RmZtbldNj9JekzQN+IeDYingH6Sfp0/qGZmVlXU8o5lU9FxOutMxHxGvCp3CIyM7Muq5Sk0qPwAV3po3575xeSmZl1VaWcqH+QZFTgOSTDtVwF/CLXqMzMrEsqJalcD8wAriY5Uf87kivAzMzMdlHK0Pc7gCeA54EGkgdnrcg5LjMz64L2eKQi6UjgQuAiYBPwHwARcXplQjMzs66mve6vPwC/Ac6LiNUAkj5fkajMzKxLaq/76yPAy8CvJd0h6UyScypmZmZF7TGpRMSCiJgOHA08AnweOEzS7ZImltK4pHMkrZS0WtKsIssl6dZ0+dOtY4xJOkrSsoLXm+nz65E0W9JLBcsml7/bZmaWh1JO1P8lIu6NiHOBYcAyYLcE0VZ6P8ttwCRgDHCRpDFtVpsEjE5fM4Db022ujIj6iKgHTgTeBhYU1PtO6/KIWNRRLGZmVhml3Py4U0S8GhH/HhFnlLD6eGB1RDwfEduA+4CpbdaZCnwvEk8AB0tqe7nymcCfIuLP5cRqZmaVV1ZSKdNQ4MWC+ea0rNx1LgTmtymbmXaXzZM0oNjGJc2Q1CipccOGDeVHb2ZmZcszqRQ7qR/lrCOpNzAF+FHB8tuBdwP1wDrgW8U2HhFzI6IhIhoGDRpURthmZtZZeSaVZmB4wfwwYG2Z60wClkbEK60FEfFKRGxPb8q8g6SbzczMakCeSWUJMFrSqPSI40JgYZt1FgIXp1eBnQK8ERHrCpZfRJuurzbnXKYBz2YfupmZdUYpY391SkS0SJpJMiBlT2BeRCyXdFW6fA6wCJgMrCa5wuuy1vqSDgDOBq5s0/TNkupJusnWFFluZmZVooi2pzm6n4aGhmhsbKx2GGZmXYqkpohoKKdOnt1fZma2j3FSMTOzzDipmJlZZpxUzMwsM04qZmaWGScVMzPLjJOKmZllxknFzMwy46RiZmaZcVIxM7PMOKmYmVlmnFTMzCwzTipmZpYZJxUzM8uMk4qZmWXGScXMzDLjpGJmZplxUjEzs8w4qZiZWWZyTSqSzpG0UtJqSbOKLJekW9PlT0saV7BsjaRnJC2T1FhQfoikhyWtSt8H5LkPZmZWutySiqSewG3AJGAMcJGkMW1WmwSMTl8zgNvbLD89IuojoqGgbBawOCJGA4vTeTMzqwF5HqmMB1ZHxPMRsQ24D5jaZp2pwPci8QRwsKQhHbQ7Fbgnnb4HOD/DmM3MbC/kmVSGAi8WzDenZaWuE8BDkpokzShY57CIWAeQvg8utnFJMyQ1SmrcsGHDXuyGmZmVKs+koiJlUcY674+IcSRdZJ+RdGo5G4+IuRHREBENgwYNKqeqmZl1Up5JpRkYXjA/DFhb6joR0fq+HlhA0p0G8EprF1n6vj7zyM3MrFPyTCpLgNGSRknqDVwILGyzzkLg4vQqsFOANyJinaS+kg4EkNQXmAg8W1DnknT6EuCnOe6DmZmVoS6vhiOiRdJM4EGgJzAvIpZLuipdPgdYBEwGVgNvA5el1Q8DFkhqjfEHEfHLdNlNwA8lXQG8AFyQ1z6YmVl5FNH2NEf309DQEI2NjR2vaGZmO0lqanNLR4d8R72ZmWXGScXMzDLjpGJmZplxUjEzs8w4qZiZWWacVMzMLDNOKmZmlhknFTMzy4yTipmZZcZJxczMMuOkYmZmmXFSMTOzzDipmJlZZpxUzMwsM04qZmaWGScVMzPLjJOKmZllxknFzMwyk2tSkXSOpJWSVkuaVWS5JN2aLn9a0ri0fLikX0taIWm5pM8V1Jkt6SVJy9LX5Dz3wczMSleXV8OSegK3AWcDzcASSQsj4rmC1SYBo9PXycDt6XsL8I8RsVTSgUCTpIcL6n4nIm7JK3YzM+ucPI9UxgOrI+L5iNgG3AdMbbPOVOB7kXgCOFjSkIhYFxFLASLiLWAFMDTHWM3MLAN5JpWhwIsF883snhg6XEfSSOAE4MmC4plpd9k8SQOKbVzSDEmNkho3bNjQyV0wM7Ny5JlUVKQsyllHUj/gAeDaiHgzLb4deDdQD6wDvlVs4xExNyIaIqJh0KBBZYZuZmadkWdSaQaGF8wPA9aWuo6kXiQJ5d6I+HHrChHxSkRsj4gdwB0k3WxmZlYD8kwqS4DRkkZJ6g1cCCxss85C4OL0KrBTgDciYp0kAXcBKyLi24UVJA0pmJ0GPJvfLpiZWTlyu/orIlokzQQeBHoC8yJiuaSr0uVzgEXAZGA18DZwWVr9/cAngGckLUvLvhgRi4CbJdWTdJOtAa7Max/MzKw8imh7mqP7aWhoiMbGxmqHYWbWpUhqioiGcur4jnozM8uMk4qZmWXGScXMzDLjpGJmZplxUjEzs8w4qZiZWWacVMzMLDNOKmZmlhknFTMzy4yTipmZZcZJxczMMuOkYmZmmXFSMTOzzDipmJlZZpxUzMwsM04qZmaWGScVMzPLjJOKmZllxknFzMwyk2tSkXSOpJWSVkuaVWS5JN2aLn9a0riO6ko6RNLDklal7wPy3AczMytdbklFUk/gNmASMAa4SNKYNqtNAkanrxnA7SXUnQUsjojRwOJ03szMakCeRyrjgdUR8XxEbAPuA6a2WWcq8L1IPAEcLGlIB3WnAvek0/cA5+e4D2ZmVoa6HNseCrxYMN8MnFzCOkM7qHtYRKwDiIh1kgYX27ikGSRHPwBbJT3bmZ3I0EBgY5VjgNqIoxZigNqIoxZigNqIoxZigNqIoxZiADiq3Ap5JhUVKYsS1ymlbrsiYi4wF0BSY0Q0lFM/a7UQQ63EUQsx1EoctRBDrcRRCzHUShy1EENrHOXWybP7qxkYXjA/DFhb4jrt1X0l7SIjfV+fYcxmZrYX8kwqS4DRkkZJ6g1cCCxss85C4OL0KrBTgDfSrq326i4ELkmnLwF+muM+mJlZGXLr/oqIFkkzgQeBnsC8iFgu6ap0+RxgETAZWA28DVzWXt206ZuAH0q6AngBuKCEcOZmt2edVgsxQG3EUQsxQG3EUQsxQG3EUQsxQG3EUQsxQCfiUERZpyrMzMz2yHfUm5lZZpxUzMwsM906qXQ0TEyFYhgu6deSVkhaLulz1YgjjaWnpN9J+nkVYzhY0v2S/pB+Ju+rQgyfT/8tnpU0X1KfCm13nqT1hfdMVWPYoT3E8c303+RpSQskHVzpGAqWXScpJA3MM4b24pB0TfrbsVzSzZWOQVK9pCckLZPUKGl8zjEU/Z3q1PczIrrli+QE/5+AdwG9gd8DY6oQxxBgXDp9IPDHasSRbv8fgB8AP6/iv8s9wCfT6d7AwRXe/lDgv4D90/kfApdWaNunAuOAZwvKbgZmpdOzgG9UKY6JQF06/Y284ygWQ1o+nOQCnT8DA6v0WZwO/CewXzo/uAoxPARMSqcnA4/kHEPR36nOfD+785FKKcPE5C4i1kXE0nT6LWAFyQ9bRUkaBnwYuLPS2y6IoT/Jf6C7ACJiW0S8XoVQ6oD9JdUBB7D7/VO5iIjHgFfbFFd82KFicUTEQxHRks4+QXJvWEVjSH0H+CfKvNk54ziuBm6KiK3pOrneC7eHGALon04fRM7f0XZ+p8r+fnbnpLKnIWCqRtJI4ATgySps/n+S/GfdUYVtt3oXsAG4O+2Gu1NS30oGEBEvAbeQXI6+juTeqIcqGUMbuww7BBQddqjCLgd+UemNSpoCvBQRv6/0tts4EvigpCclPSrppCrEcC3wTUkvknxfb6jUhtv8TpX9/ezOSWWvh3rJkqR+wAPAtRHxZoW3fS6wPiKaKrndIupIDvNvj4gTgL9Q4VGm0z7hqcAo4Aigr6S/r2QMtUzSl4AW4N4Kb/cA4EvAP1dyu3tQBwwATgG+QHJfXLHfkzxdDXw+IoYDnyc9us9bFr9T3TmplDJMTEVI6kXyD3VvRPy4CiG8H5giaQ1JN+AZkr5fhTiageaIaD1Su58kyVTSWcB/RcSGiHgH+DHwtxWOoVDNDDsk6RLgXODjkXaiV9C7SRL979Pv6TBgqaTDKxwHJN/TH0fiKZKj+9wvGmjjEpLvJsCPSLrzc7WH36myv5/dOamUMkxM7tK/cO4CVkTEtyu9fYCIuCEihkXESJLP4VcRUfG/ziPiZeBFSa0jn54JPFfhMF4ATpF0QPpvcyZJ/3G11MSwQ5LOAa4HpkTE25XefkQ8ExGDI2Jk+j1tJjlx/HKlYwF+ApwBIOlIkgtKKj1i8FrgtHT6DGBVnhtr53eq/O9nnlcUVPtFctXEH0muAvtSlWL4AEm329PAsvQ1uYqfyQSqe/VXPdCYfh4/AQZUIYavAH8AngX+N+lVPhXY7nyS8zjvkPxoXgEcSvKwuVXp+yFVimM1yTnI1u/onErH0Gb5Gipz9Vexz6I38P30+7EUOKMKMXwAaCK5avVJ4MScYyj6O9WZ76eHaTEzs8x05+4vMzOrMCcVMzPLjJOKmZllxknFzMwy46RiZmaZcVIxy4Ck7emIsq2vzEYKkDSy2Gi+ZrUot8cJm+1j/hoR9dUOwqzafKRiliNJayR9Q9JT6es9afkISYvT55cslvQ3aflh6fNMfp++WoeQ6SnpjvRZFw9J2r9qO2XWDicVs2zs36b7a3rBsjcjYjzwbySjRZNOfy8ijicZvPHWtPxW4NGIeC/JuGjL0/LRwG0RcSzwOvCRXPfGrJN8R71ZBiRtjoh+RcrXkAzz8Xw6YN/LEXGopI3AkIh4Jy1fFxEDJW0AhkX6LI+0jZHAwxExOp2/HugVEf9SgV0zK4uPVMzyF3uY3tM6xWwtmN6Oz4dajXJSMcvf9IL3x9Pp/0cyYjTAx4H/m04vJnmWBpJ6pk/LNOsy/NeOWTb2l7SsYP6XEdF6WfF+kp4k+SPuorTss8A8SV8geRrmZWn554C5kq4gOSK5mmQEW7MuwedUzHKUnlNpiIhKP4/DrCrc/WVmZpnxkYqZmWXGRypmZpYZJxUzM8uMk4qZmWXGScXMzDLjpGJmZpn5/xxPz3kE7fURAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_loss_and_acc({'Sigmoid': [sigmoid_loss, sigmoid_acc],\n",
    "                   'relu': [relu_loss, relu_acc]})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. MLP with Softmax Cross-Entropy Loss\n",
    "In part-2, you need to train a MLP with **Softmax Cross-Entropy Loss**.  \n",
    "**Sigmoid Activation Function** and **ReLU Activation Function** will be used respectively again.\n",
    "### TODO\n",
    "Before executing the following code, you should complete **criterion/softmax_cross_entropy_loss.py**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from criterion import SoftmaxCrossEntropyLossLayer\n",
    "\n",
    "criterion = SoftmaxCrossEntropyLossLayer()\n",
    "\n",
    "sgd = SGD(learning_rate_SGD, weight_decay)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1 MLP with Softmax Cross-Entropy Loss and Sigmoid Activation Function\n",
    "Build and train a MLP contraining one hidden layer with 128 units using Sigmoid activation function and Softmax cross-entropy loss function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sigmoidMLP = Network()\n",
    "# Build MLP with FCLayer and SigmoidLayer\n",
    "# 128 is the number of hidden units, you can change by your own\n",
    "sigmoidMLP.add(FCLayer(784, 128))\n",
    "sigmoidMLP.add(SigmoidLayer())\n",
    "sigmoidMLP.add(FCLayer(128, 10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "File \u001b[0;32m/opt/homebrew/Caskroom/miniforge/base/envs/tf_m1/lib/python3.8/contextlib.py:131\u001b[0m, in \u001b[0;36m_GeneratorContextManager.__exit__\u001b[0;34m(self, type, value, traceback)\u001b[0m\n\u001b[1;32m    130\u001b[0m \u001b[39mtry\u001b[39;00m:\n\u001b[0;32m--> 131\u001b[0m     \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mgen\u001b[39m.\u001b[39;49mthrow(\u001b[39mtype\u001b[39;49m, value, traceback)\n\u001b[1;32m    132\u001b[0m \u001b[39mexcept\u001b[39;00m \u001b[39mStopIteration\u001b[39;00m \u001b[39mas\u001b[39;00m exc:\n\u001b[1;32m    133\u001b[0m     \u001b[39m# Suppress StopIteration *unless* it's the same exception that\u001b[39;00m\n\u001b[1;32m    134\u001b[0m     \u001b[39m# was passed to throw().  This prevents a StopIteration\u001b[39;00m\n\u001b[1;32m    135\u001b[0m     \u001b[39m# raised inside the \"with\" statement from being suppressed.\u001b[39;00m\n",
      "File \u001b[0;32m/opt/homebrew/Caskroom/miniforge/base/envs/tf_m1/lib/python3.8/site-packages/tensorflow/python/framework/ops.py:5845\u001b[0m, in \u001b[0;36m_DefaultGraphStack.get_controller\u001b[0;34m(self, default)\u001b[0m\n\u001b[1;32m   5843\u001b[0m   \u001b[39mwith\u001b[39;00m \u001b[39msuper\u001b[39m(_DefaultGraphStack,\n\u001b[1;32m   5844\u001b[0m              \u001b[39mself\u001b[39m)\u001b[39m.\u001b[39mget_controller(default) \u001b[39mas\u001b[39;00m g, context\u001b[39m.\u001b[39mgraph_mode():\n\u001b[0;32m-> 5845\u001b[0m     \u001b[39myield\u001b[39;00m g\n\u001b[1;32m   5846\u001b[0m \u001b[39mfinally\u001b[39;00m:\n\u001b[1;32m   5847\u001b[0m   \u001b[39m# If an exception is raised here it may be hiding a related exception in\u001b[39;00m\n\u001b[1;32m   5848\u001b[0m   \u001b[39m# the try-block (just above).\u001b[39;00m\n",
      "File \u001b[0;32m~/Downloads/homework-2/homework2-mlp/solver.py:35\u001b[0m, in \u001b[0;36mtrain\u001b[0;34m(model, criterion, optimizer, dataset, max_epoch, batch_size, disp_freq)\u001b[0m\n\u001b[1;32m     34\u001b[0m \u001b[39mfor\u001b[39;00m epoch \u001b[39min\u001b[39;00m \u001b[39mrange\u001b[39m(max_epoch):\n\u001b[0;32m---> 35\u001b[0m \tbatch_train_loss, batch_train_acc \u001b[39m=\u001b[39m train_one_epoch(model, criterion, optimizer, train_get_next,\n\u001b[1;32m     36\u001b[0m \t                                                    max_epoch, batch_size, disp_freq, epoch, sess)\n\u001b[1;32m     37\u001b[0m \tbatch_val_loss, batch_val_acc \u001b[39m=\u001b[39m validate(model, criterion, valid_get_next, batch_size, sess)\n",
      "File \u001b[0;32m~/Downloads/homework-2/homework2-mlp/solver.py:71\u001b[0m, in \u001b[0;36mtrain_one_epoch\u001b[0;34m(model, criterion, optimizer, data_get_next, max_epoch, batch_size, disp_freq, epoch, sess)\u001b[0m\n\u001b[1;32m     70\u001b[0m delta \u001b[39m=\u001b[39m criterion\u001b[39m.\u001b[39mbackward()\n\u001b[0;32m---> 71\u001b[0m model\u001b[39m.\u001b[39;49mbackward(delta)\n\u001b[1;32m     73\u001b[0m \u001b[39m# Update weights, see optimize.py\u001b[39;00m\n",
      "File \u001b[0;32m~/Downloads/homework-2/homework2-mlp/network.py:21\u001b[0m, in \u001b[0;36mNetwork.backward\u001b[0;34m(self, delta)\u001b[0m\n\u001b[1;32m     20\u001b[0m \u001b[39mfor\u001b[39;00m i \u001b[39min\u001b[39;00m \u001b[39mreversed\u001b[39m(\u001b[39mrange\u001b[39m(\u001b[39mself\u001b[39m\u001b[39m.\u001b[39mnumLayer)): \u001b[39m# reversed\u001b[39;00m\n\u001b[0;32m---> 21\u001b[0m \tdelta \u001b[39m=\u001b[39m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mlayerList[i]\u001b[39m.\u001b[39;49mbackward(delta)\n",
      "File \u001b[0;32m~/Downloads/homework-2/homework2-mlp/layers/fc_layer.py:61\u001b[0m, in \u001b[0;36mFCLayer.backward\u001b[0;34m(self, delta)\u001b[0m\n\u001b[1;32m     60\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mdelta \u001b[39m=\u001b[39m delta\n\u001b[0;32m---> 61\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39merr \u001b[39m=\u001b[39m delta\u001b[39m.\u001b[39;49mdot(\u001b[39mself\u001b[39m\u001b[39m.\u001b[39mgrad_W\u001b[39m.\u001b[39mT)\n\u001b[1;32m     63\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39merr\n",
      "\u001b[0;31mAttributeError\u001b[0m: 'numpy.float64' object has no attribute 'dot'",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[1;32m/Users/youssefbr/Downloads/homework-2/homework2-mlp/homework2-mlp.ipynb Cell 18'\u001b[0m in \u001b[0;36m<cell line: 1>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> <a href='vscode-notebook-cell:/Users/youssefbr/Downloads/homework-2/homework2-mlp/homework2-mlp.ipynb#ch0000017?line=0'>1</a>\u001b[0m sigmoidMLP, sigmoid_loss, sigmoid_acc \u001b[39m=\u001b[39m train(sigmoidMLP, criterion, sgd, data_train, max_epoch, batch_size, disp_freq)\n",
      "File \u001b[0;32m~/Downloads/homework-2/homework2-mlp/solver.py:50\u001b[0m, in \u001b[0;36mtrain\u001b[0;34m(model, criterion, optimizer, dataset, max_epoch, batch_size, disp_freq)\u001b[0m\n\u001b[1;32m     45\u001b[0m \t\t\u001b[39mprint\u001b[39m(\u001b[39m'\u001b[39m\u001b[39mEpoch [\u001b[39m\u001b[39m{}\u001b[39;00m\u001b[39m]\u001b[39m\u001b[39m\\t\u001b[39;00m\u001b[39m Average training loss \u001b[39m\u001b[39m{:.4f}\u001b[39;00m\u001b[39m\\t\u001b[39;00m\u001b[39m Average training accuracy \u001b[39m\u001b[39m{:.4f}\u001b[39;00m\u001b[39m'\u001b[39m\u001b[39m.\u001b[39mformat(\n\u001b[1;32m     46\u001b[0m \t\t\tepoch, avg_train_loss[\u001b[39m-\u001b[39m\u001b[39m1\u001b[39m], avg_train_acc[\u001b[39m-\u001b[39m\u001b[39m1\u001b[39m]))\n\u001b[1;32m     48\u001b[0m \t\t\u001b[39mprint\u001b[39m(\u001b[39m'\u001b[39m\u001b[39mEpoch [\u001b[39m\u001b[39m{}\u001b[39;00m\u001b[39m]\u001b[39m\u001b[39m\\t\u001b[39;00m\u001b[39m Average validation loss \u001b[39m\u001b[39m{:.4f}\u001b[39;00m\u001b[39m\\t\u001b[39;00m\u001b[39m Average validation accuracy \u001b[39m\u001b[39m{:.4f}\u001b[39;00m\u001b[39m'\u001b[39m\u001b[39m.\u001b[39mformat(\n\u001b[1;32m     49\u001b[0m \t\t\tepoch, avg_val_loss[\u001b[39m-\u001b[39m\u001b[39m1\u001b[39m], avg_val_acc[\u001b[39m-\u001b[39m\u001b[39m1\u001b[39m]))\n\u001b[0;32m---> 50\u001b[0m \t\t\u001b[39mprint\u001b[39m()\n\u001b[1;32m     52\u001b[0m \u001b[39mreturn\u001b[39;00m model, avg_val_loss, avg_val_acc\n",
      "File \u001b[0;32m/opt/homebrew/Caskroom/miniforge/base/envs/tf_m1/lib/python3.8/site-packages/tensorflow/python/client/session.py:1652\u001b[0m, in \u001b[0;36mSession.__exit__\u001b[0;34m(self, exec_type, exec_value, exec_tb)\u001b[0m\n\u001b[1;32m   1650\u001b[0m close_thread\u001b[39m.\u001b[39mdaemon \u001b[39m=\u001b[39m \u001b[39mTrue\u001b[39;00m\n\u001b[1;32m   1651\u001b[0m close_thread\u001b[39m.\u001b[39mstart()\n\u001b[0;32m-> 1652\u001b[0m close_thread\u001b[39m.\u001b[39;49mjoin(\u001b[39m30.0\u001b[39;49m)\n\u001b[1;32m   1653\u001b[0m \u001b[39mif\u001b[39;00m close_thread\u001b[39m.\u001b[39mis_alive():\n\u001b[1;32m   1654\u001b[0m   logging\u001b[39m.\u001b[39merror(\n\u001b[1;32m   1655\u001b[0m       \u001b[39m'\u001b[39m\u001b[39mSession failed to close after 30 seconds. Continuing after this \u001b[39m\u001b[39m'\u001b[39m\n\u001b[1;32m   1656\u001b[0m       \u001b[39m'\u001b[39m\u001b[39mpoint may leave your program in an undefined state.\u001b[39m\u001b[39m'\u001b[39m)\n",
      "File \u001b[0;32m/opt/homebrew/Caskroom/miniforge/base/envs/tf_m1/lib/python3.8/threading.py:1015\u001b[0m, in \u001b[0;36mThread.join\u001b[0;34m(self, timeout)\u001b[0m\n\u001b[1;32m   1011\u001b[0m     \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_wait_for_tstate_lock()\n\u001b[1;32m   1012\u001b[0m \u001b[39melse\u001b[39;00m:\n\u001b[1;32m   1013\u001b[0m     \u001b[39m# the behavior of a negative timeout isn't documented, but\u001b[39;00m\n\u001b[1;32m   1014\u001b[0m     \u001b[39m# historically .join(timeout=x) for x<0 has acted as if timeout=0\u001b[39;00m\n\u001b[0;32m-> 1015\u001b[0m     \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_wait_for_tstate_lock(timeout\u001b[39m=\u001b[39;49m\u001b[39mmax\u001b[39;49m(timeout, \u001b[39m0\u001b[39;49m))\n",
      "File \u001b[0;32m/opt/homebrew/Caskroom/miniforge/base/envs/tf_m1/lib/python3.8/threading.py:1027\u001b[0m, in \u001b[0;36mThread._wait_for_tstate_lock\u001b[0;34m(self, block, timeout)\u001b[0m\n\u001b[1;32m   1025\u001b[0m \u001b[39mif\u001b[39;00m lock \u001b[39mis\u001b[39;00m \u001b[39mNone\u001b[39;00m:  \u001b[39m# already determined that the C code is done\u001b[39;00m\n\u001b[1;32m   1026\u001b[0m     \u001b[39massert\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_is_stopped\n\u001b[0;32m-> 1027\u001b[0m \u001b[39melif\u001b[39;00m lock\u001b[39m.\u001b[39;49macquire(block, timeout):\n\u001b[1;32m   1028\u001b[0m     lock\u001b[39m.\u001b[39mrelease()\n\u001b[1;32m   1029\u001b[0m     \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_stop()\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "sigmoidMLP, sigmoid_loss, sigmoid_acc = train(sigmoidMLP, criterion, sgd, data_train, max_epoch, batch_size, disp_freq)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Testing...\n",
      "The test accuracy is 0.0980.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "test(sigmoidMLP, criterion, data_test, batch_size, disp_freq)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 MLP with Softmax Cross-Entropy Loss and ReLU Activation Function\n",
    "Build and train a MLP contraining one hidden layer with 128 units using ReLU activation function and Softmax cross-entropy loss function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reluMLP = Network()\n",
    "# Build ReLUMLP with FCLayer and ReLULayer\n",
    "# 128 is the number of hidden units, you can change by your own\n",
    "reluMLP.add(FCLayer(784, 128))\n",
    "reluMLP.add(ReLULayer())\n",
    "reluMLP.add(FCLayer(128, 10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch [0][20]\t Batch [0][550]\t Training Loss 2.5354\t Accuracy 0.1300\n",
      "Epoch [0][20]\t Batch [50][550]\t Training Loss 2.5354\t Accuracy 0.0963\n",
      "Epoch [0][20]\t Batch [100][550]\t Training Loss 2.5354\t Accuracy 0.1001\n",
      "Epoch [0][20]\t Batch [150][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [0][20]\t Batch [200][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [0][20]\t Batch [250][550]\t Training Loss 2.5354\t Accuracy 0.0988\n",
      "Epoch [0][20]\t Batch [300][550]\t Training Loss 2.5354\t Accuracy 0.0987\n",
      "Epoch [0][20]\t Batch [350][550]\t Training Loss 2.5354\t Accuracy 0.0981\n",
      "Epoch [0][20]\t Batch [400][550]\t Training Loss 2.5354\t Accuracy 0.0982\n",
      "Epoch [0][20]\t Batch [450][550]\t Training Loss 2.5354\t Accuracy 0.0983\n",
      "Epoch [0][20]\t Batch [500][550]\t Training Loss 2.5354\t Accuracy 0.0986\n",
      "\n",
      "Epoch [0]\t Average training loss 2.5354\t Average training accuracy 0.0988\n",
      "Epoch [0]\t Average validation loss 2.5354\t Average validation accuracy 0.0978\n",
      "\n",
      "Epoch [1][20]\t Batch [0][550]\t Training Loss 2.5354\t Accuracy 0.1300\n",
      "Epoch [1][20]\t Batch [50][550]\t Training Loss 2.5354\t Accuracy 0.0963\n",
      "Epoch [1][20]\t Batch [100][550]\t Training Loss 2.5354\t Accuracy 0.1001\n",
      "Epoch [1][20]\t Batch [150][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [1][20]\t Batch [200][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [1][20]\t Batch [250][550]\t Training Loss 2.5354\t Accuracy 0.0988\n",
      "Epoch [1][20]\t Batch [300][550]\t Training Loss 2.5354\t Accuracy 0.0987\n",
      "Epoch [1][20]\t Batch [350][550]\t Training Loss 2.5354\t Accuracy 0.0981\n",
      "Epoch [1][20]\t Batch [400][550]\t Training Loss 2.5354\t Accuracy 0.0982\n",
      "Epoch [1][20]\t Batch [450][550]\t Training Loss 2.5354\t Accuracy 0.0983\n",
      "Epoch [1][20]\t Batch [500][550]\t Training Loss 2.5354\t Accuracy 0.0986\n",
      "\n",
      "Epoch [1]\t Average training loss 2.5354\t Average training accuracy 0.0988\n",
      "Epoch [1]\t Average validation loss 2.5354\t Average validation accuracy 0.0978\n",
      "\n",
      "Epoch [2][20]\t Batch [0][550]\t Training Loss 2.5354\t Accuracy 0.1300\n",
      "Epoch [2][20]\t Batch [50][550]\t Training Loss 2.5354\t Accuracy 0.0963\n",
      "Epoch [2][20]\t Batch [100][550]\t Training Loss 2.5354\t Accuracy 0.1001\n",
      "Epoch [2][20]\t Batch [150][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [2][20]\t Batch [200][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [2][20]\t Batch [250][550]\t Training Loss 2.5354\t Accuracy 0.0988\n",
      "Epoch [2][20]\t Batch [300][550]\t Training Loss 2.5354\t Accuracy 0.0987\n",
      "Epoch [2][20]\t Batch [350][550]\t Training Loss 2.5354\t Accuracy 0.0981\n",
      "Epoch [2][20]\t Batch [400][550]\t Training Loss 2.5354\t Accuracy 0.0982\n",
      "Epoch [2][20]\t Batch [450][550]\t Training Loss 2.5354\t Accuracy 0.0983\n",
      "Epoch [2][20]\t Batch [500][550]\t Training Loss 2.5354\t Accuracy 0.0986\n",
      "\n",
      "Epoch [2]\t Average training loss 2.5354\t Average training accuracy 0.0988\n",
      "Epoch [2]\t Average validation loss 2.5354\t Average validation accuracy 0.0978\n",
      "\n",
      "Epoch [3][20]\t Batch [0][550]\t Training Loss 2.5354\t Accuracy 0.1300\n",
      "Epoch [3][20]\t Batch [50][550]\t Training Loss 2.5354\t Accuracy 0.0963\n",
      "Epoch [3][20]\t Batch [100][550]\t Training Loss 2.5354\t Accuracy 0.1001\n",
      "Epoch [3][20]\t Batch [150][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [3][20]\t Batch [200][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [3][20]\t Batch [250][550]\t Training Loss 2.5354\t Accuracy 0.0988\n",
      "Epoch [3][20]\t Batch [300][550]\t Training Loss 2.5354\t Accuracy 0.0987\n",
      "Epoch [3][20]\t Batch [350][550]\t Training Loss 2.5354\t Accuracy 0.0981\n",
      "Epoch [3][20]\t Batch [400][550]\t Training Loss 2.5354\t Accuracy 0.0982\n",
      "Epoch [3][20]\t Batch [450][550]\t Training Loss 2.5354\t Accuracy 0.0983\n",
      "Epoch [3][20]\t Batch [500][550]\t Training Loss 2.5354\t Accuracy 0.0986\n",
      "\n",
      "Epoch [3]\t Average training loss 2.5354\t Average training accuracy 0.0988\n",
      "Epoch [3]\t Average validation loss 2.5354\t Average validation accuracy 0.0978\n",
      "\n",
      "Epoch [4][20]\t Batch [0][550]\t Training Loss 2.5354\t Accuracy 0.1300\n",
      "Epoch [4][20]\t Batch [50][550]\t Training Loss 2.5354\t Accuracy 0.0963\n",
      "Epoch [4][20]\t Batch [100][550]\t Training Loss 2.5354\t Accuracy 0.1001\n",
      "Epoch [4][20]\t Batch [150][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [4][20]\t Batch [200][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [4][20]\t Batch [250][550]\t Training Loss 2.5354\t Accuracy 0.0988\n",
      "Epoch [4][20]\t Batch [300][550]\t Training Loss 2.5354\t Accuracy 0.0987\n",
      "Epoch [4][20]\t Batch [350][550]\t Training Loss 2.5354\t Accuracy 0.0981\n",
      "Epoch [4][20]\t Batch [400][550]\t Training Loss 2.5354\t Accuracy 0.0982\n",
      "Epoch [4][20]\t Batch [450][550]\t Training Loss 2.5354\t Accuracy 0.0983\n",
      "Epoch [4][20]\t Batch [500][550]\t Training Loss 2.5354\t Accuracy 0.0986\n",
      "\n",
      "Epoch [4]\t Average training loss 2.5354\t Average training accuracy 0.0988\n",
      "Epoch [4]\t Average validation loss 2.5354\t Average validation accuracy 0.0978\n",
      "\n",
      "Epoch [5][20]\t Batch [0][550]\t Training Loss 2.5354\t Accuracy 0.1300\n",
      "Epoch [5][20]\t Batch [50][550]\t Training Loss 2.5354\t Accuracy 0.0963\n",
      "Epoch [5][20]\t Batch [100][550]\t Training Loss 2.5354\t Accuracy 0.1001\n",
      "Epoch [5][20]\t Batch [150][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [5][20]\t Batch [200][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [5][20]\t Batch [250][550]\t Training Loss 2.5354\t Accuracy 0.0988\n",
      "Epoch [5][20]\t Batch [300][550]\t Training Loss 2.5354\t Accuracy 0.0987\n",
      "Epoch [5][20]\t Batch [350][550]\t Training Loss 2.5354\t Accuracy 0.0981\n",
      "Epoch [5][20]\t Batch [400][550]\t Training Loss 2.5354\t Accuracy 0.0982\n",
      "Epoch [5][20]\t Batch [450][550]\t Training Loss 2.5354\t Accuracy 0.0983\n",
      "Epoch [5][20]\t Batch [500][550]\t Training Loss 2.5354\t Accuracy 0.0986\n",
      "\n",
      "Epoch [5]\t Average training loss 2.5354\t Average training accuracy 0.0988\n",
      "Epoch [5]\t Average validation loss 2.5354\t Average validation accuracy 0.0978\n",
      "\n",
      "Epoch [6][20]\t Batch [0][550]\t Training Loss 2.5354\t Accuracy 0.1300\n",
      "Epoch [6][20]\t Batch [50][550]\t Training Loss 2.5354\t Accuracy 0.0963\n",
      "Epoch [6][20]\t Batch [100][550]\t Training Loss 2.5354\t Accuracy 0.1001\n",
      "Epoch [6][20]\t Batch [150][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [6][20]\t Batch [200][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [6][20]\t Batch [250][550]\t Training Loss 2.5354\t Accuracy 0.0988\n",
      "Epoch [6][20]\t Batch [300][550]\t Training Loss 2.5354\t Accuracy 0.0987\n",
      "Epoch [6][20]\t Batch [350][550]\t Training Loss 2.5354\t Accuracy 0.0981\n",
      "Epoch [6][20]\t Batch [400][550]\t Training Loss 2.5354\t Accuracy 0.0982\n",
      "Epoch [6][20]\t Batch [450][550]\t Training Loss 2.5354\t Accuracy 0.0983\n",
      "Epoch [6][20]\t Batch [500][550]\t Training Loss 2.5354\t Accuracy 0.0986\n",
      "\n",
      "Epoch [6]\t Average training loss 2.5354\t Average training accuracy 0.0988\n",
      "Epoch [6]\t Average validation loss 2.5354\t Average validation accuracy 0.0978\n",
      "\n",
      "Epoch [7][20]\t Batch [0][550]\t Training Loss 2.5354\t Accuracy 0.1300\n",
      "Epoch [7][20]\t Batch [50][550]\t Training Loss 2.5354\t Accuracy 0.0963\n",
      "Epoch [7][20]\t Batch [100][550]\t Training Loss 2.5354\t Accuracy 0.1001\n",
      "Epoch [7][20]\t Batch [150][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [7][20]\t Batch [200][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [7][20]\t Batch [250][550]\t Training Loss 2.5354\t Accuracy 0.0988\n",
      "Epoch [7][20]\t Batch [300][550]\t Training Loss 2.5354\t Accuracy 0.0987\n",
      "Epoch [7][20]\t Batch [350][550]\t Training Loss 2.5354\t Accuracy 0.0981\n",
      "Epoch [7][20]\t Batch [400][550]\t Training Loss 2.5354\t Accuracy 0.0982\n",
      "Epoch [7][20]\t Batch [450][550]\t Training Loss 2.5354\t Accuracy 0.0983\n",
      "Epoch [7][20]\t Batch [500][550]\t Training Loss 2.5354\t Accuracy 0.0986\n",
      "\n",
      "Epoch [7]\t Average training loss 2.5354\t Average training accuracy 0.0988\n",
      "Epoch [7]\t Average validation loss 2.5354\t Average validation accuracy 0.0978\n",
      "\n",
      "Epoch [8][20]\t Batch [0][550]\t Training Loss 2.5354\t Accuracy 0.1300\n",
      "Epoch [8][20]\t Batch [50][550]\t Training Loss 2.5354\t Accuracy 0.0963\n",
      "Epoch [8][20]\t Batch [100][550]\t Training Loss 2.5354\t Accuracy 0.1001\n",
      "Epoch [8][20]\t Batch [150][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [8][20]\t Batch [200][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [8][20]\t Batch [250][550]\t Training Loss 2.5354\t Accuracy 0.0988\n",
      "Epoch [8][20]\t Batch [300][550]\t Training Loss 2.5354\t Accuracy 0.0987\n",
      "Epoch [8][20]\t Batch [350][550]\t Training Loss 2.5354\t Accuracy 0.0981\n",
      "Epoch [8][20]\t Batch [400][550]\t Training Loss 2.5354\t Accuracy 0.0982\n",
      "Epoch [8][20]\t Batch [450][550]\t Training Loss 2.5354\t Accuracy 0.0983\n",
      "Epoch [8][20]\t Batch [500][550]\t Training Loss 2.5354\t Accuracy 0.0986\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Epoch [8]\t Average training loss 2.5354\t Average training accuracy 0.0988\n",
      "Epoch [8]\t Average validation loss 2.5354\t Average validation accuracy 0.0978\n",
      "\n",
      "Epoch [9][20]\t Batch [0][550]\t Training Loss 2.5354\t Accuracy 0.1300\n",
      "Epoch [9][20]\t Batch [50][550]\t Training Loss 2.5354\t Accuracy 0.0963\n",
      "Epoch [9][20]\t Batch [100][550]\t Training Loss 2.5354\t Accuracy 0.1001\n",
      "Epoch [9][20]\t Batch [150][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [9][20]\t Batch [200][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [9][20]\t Batch [250][550]\t Training Loss 2.5354\t Accuracy 0.0988\n",
      "Epoch [9][20]\t Batch [300][550]\t Training Loss 2.5354\t Accuracy 0.0987\n",
      "Epoch [9][20]\t Batch [350][550]\t Training Loss 2.5354\t Accuracy 0.0981\n",
      "Epoch [9][20]\t Batch [400][550]\t Training Loss 2.5354\t Accuracy 0.0982\n",
      "Epoch [9][20]\t Batch [450][550]\t Training Loss 2.5354\t Accuracy 0.0983\n",
      "Epoch [9][20]\t Batch [500][550]\t Training Loss 2.5354\t Accuracy 0.0986\n",
      "\n",
      "Epoch [9]\t Average training loss 2.5354\t Average training accuracy 0.0988\n",
      "Epoch [9]\t Average validation loss 2.5354\t Average validation accuracy 0.0978\n",
      "\n",
      "Epoch [10][20]\t Batch [0][550]\t Training Loss 2.5354\t Accuracy 0.1300\n",
      "Epoch [10][20]\t Batch [50][550]\t Training Loss 2.5354\t Accuracy 0.0963\n",
      "Epoch [10][20]\t Batch [100][550]\t Training Loss 2.5354\t Accuracy 0.1001\n",
      "Epoch [10][20]\t Batch [150][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [10][20]\t Batch [200][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [10][20]\t Batch [250][550]\t Training Loss 2.5354\t Accuracy 0.0988\n",
      "Epoch [10][20]\t Batch [300][550]\t Training Loss 2.5354\t Accuracy 0.0987\n",
      "Epoch [10][20]\t Batch [350][550]\t Training Loss 2.5354\t Accuracy 0.0981\n",
      "Epoch [10][20]\t Batch [400][550]\t Training Loss 2.5354\t Accuracy 0.0982\n",
      "Epoch [10][20]\t Batch [450][550]\t Training Loss 2.5354\t Accuracy 0.0983\n",
      "Epoch [10][20]\t Batch [500][550]\t Training Loss 2.5354\t Accuracy 0.0986\n",
      "\n",
      "Epoch [10]\t Average training loss 2.5354\t Average training accuracy 0.0988\n",
      "Epoch [10]\t Average validation loss 2.5354\t Average validation accuracy 0.0978\n",
      "\n",
      "Epoch [11][20]\t Batch [0][550]\t Training Loss 2.5354\t Accuracy 0.1300\n",
      "Epoch [11][20]\t Batch [50][550]\t Training Loss 2.5354\t Accuracy 0.0963\n",
      "Epoch [11][20]\t Batch [100][550]\t Training Loss 2.5354\t Accuracy 0.1001\n",
      "Epoch [11][20]\t Batch [150][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [11][20]\t Batch [200][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [11][20]\t Batch [250][550]\t Training Loss 2.5354\t Accuracy 0.0988\n",
      "Epoch [11][20]\t Batch [300][550]\t Training Loss 2.5354\t Accuracy 0.0987\n",
      "Epoch [11][20]\t Batch [350][550]\t Training Loss 2.5354\t Accuracy 0.0981\n",
      "Epoch [11][20]\t Batch [400][550]\t Training Loss 2.5354\t Accuracy 0.0982\n",
      "Epoch [11][20]\t Batch [450][550]\t Training Loss 2.5354\t Accuracy 0.0983\n",
      "Epoch [11][20]\t Batch [500][550]\t Training Loss 2.5354\t Accuracy 0.0986\n",
      "\n",
      "Epoch [11]\t Average training loss 2.5354\t Average training accuracy 0.0988\n",
      "Epoch [11]\t Average validation loss 2.5354\t Average validation accuracy 0.0978\n",
      "\n",
      "Epoch [12][20]\t Batch [0][550]\t Training Loss 2.5354\t Accuracy 0.1300\n",
      "Epoch [12][20]\t Batch [50][550]\t Training Loss 2.5354\t Accuracy 0.0963\n",
      "Epoch [12][20]\t Batch [100][550]\t Training Loss 2.5354\t Accuracy 0.1001\n",
      "Epoch [12][20]\t Batch [150][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [12][20]\t Batch [200][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [12][20]\t Batch [250][550]\t Training Loss 2.5354\t Accuracy 0.0988\n",
      "Epoch [12][20]\t Batch [300][550]\t Training Loss 2.5354\t Accuracy 0.0987\n",
      "Epoch [12][20]\t Batch [350][550]\t Training Loss 2.5354\t Accuracy 0.0981\n",
      "Epoch [12][20]\t Batch [400][550]\t Training Loss 2.5354\t Accuracy 0.0982\n",
      "Epoch [12][20]\t Batch [450][550]\t Training Loss 2.5354\t Accuracy 0.0983\n",
      "Epoch [12][20]\t Batch [500][550]\t Training Loss 2.5354\t Accuracy 0.0986\n",
      "\n",
      "Epoch [12]\t Average training loss 2.5354\t Average training accuracy 0.0988\n",
      "Epoch [12]\t Average validation loss 2.5354\t Average validation accuracy 0.0978\n",
      "\n",
      "Epoch [13][20]\t Batch [0][550]\t Training Loss 2.5354\t Accuracy 0.1300\n",
      "Epoch [13][20]\t Batch [50][550]\t Training Loss 2.5354\t Accuracy 0.0963\n",
      "Epoch [13][20]\t Batch [100][550]\t Training Loss 2.5354\t Accuracy 0.1001\n",
      "Epoch [13][20]\t Batch [150][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [13][20]\t Batch [200][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [13][20]\t Batch [250][550]\t Training Loss 2.5354\t Accuracy 0.0988\n",
      "Epoch [13][20]\t Batch [300][550]\t Training Loss 2.5354\t Accuracy 0.0987\n",
      "Epoch [13][20]\t Batch [350][550]\t Training Loss 2.5354\t Accuracy 0.0981\n",
      "Epoch [13][20]\t Batch [400][550]\t Training Loss 2.5354\t Accuracy 0.0982\n",
      "Epoch [13][20]\t Batch [450][550]\t Training Loss 2.5354\t Accuracy 0.0983\n",
      "Epoch [13][20]\t Batch [500][550]\t Training Loss 2.5354\t Accuracy 0.0986\n",
      "\n",
      "Epoch [13]\t Average training loss 2.5354\t Average training accuracy 0.0988\n",
      "Epoch [13]\t Average validation loss 2.5354\t Average validation accuracy 0.0978\n",
      "\n",
      "Epoch [14][20]\t Batch [0][550]\t Training Loss 2.5354\t Accuracy 0.1300\n",
      "Epoch [14][20]\t Batch [50][550]\t Training Loss 2.5354\t Accuracy 0.0963\n",
      "Epoch [14][20]\t Batch [100][550]\t Training Loss 2.5354\t Accuracy 0.1001\n",
      "Epoch [14][20]\t Batch [150][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [14][20]\t Batch [200][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [14][20]\t Batch [250][550]\t Training Loss 2.5354\t Accuracy 0.0988\n",
      "Epoch [14][20]\t Batch [300][550]\t Training Loss 2.5354\t Accuracy 0.0987\n",
      "Epoch [14][20]\t Batch [350][550]\t Training Loss 2.5354\t Accuracy 0.0981\n",
      "Epoch [14][20]\t Batch [400][550]\t Training Loss 2.5354\t Accuracy 0.0982\n",
      "Epoch [14][20]\t Batch [450][550]\t Training Loss 2.5354\t Accuracy 0.0983\n",
      "Epoch [14][20]\t Batch [500][550]\t Training Loss 2.5354\t Accuracy 0.0986\n",
      "\n",
      "Epoch [14]\t Average training loss 2.5354\t Average training accuracy 0.0988\n",
      "Epoch [14]\t Average validation loss 2.5354\t Average validation accuracy 0.0978\n",
      "\n",
      "Epoch [15][20]\t Batch [0][550]\t Training Loss 2.5354\t Accuracy 0.1300\n",
      "Epoch [15][20]\t Batch [50][550]\t Training Loss 2.5354\t Accuracy 0.0963\n",
      "Epoch [15][20]\t Batch [100][550]\t Training Loss 2.5354\t Accuracy 0.1001\n",
      "Epoch [15][20]\t Batch [150][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [15][20]\t Batch [200][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [15][20]\t Batch [250][550]\t Training Loss 2.5354\t Accuracy 0.0988\n",
      "Epoch [15][20]\t Batch [300][550]\t Training Loss 2.5354\t Accuracy 0.0987\n",
      "Epoch [15][20]\t Batch [350][550]\t Training Loss 2.5354\t Accuracy 0.0981\n",
      "Epoch [15][20]\t Batch [400][550]\t Training Loss 2.5354\t Accuracy 0.0982\n",
      "Epoch [15][20]\t Batch [450][550]\t Training Loss 2.5354\t Accuracy 0.0983\n",
      "Epoch [15][20]\t Batch [500][550]\t Training Loss 2.5354\t Accuracy 0.0986\n",
      "\n",
      "Epoch [15]\t Average training loss 2.5354\t Average training accuracy 0.0988\n",
      "Epoch [15]\t Average validation loss 2.5354\t Average validation accuracy 0.0978\n",
      "\n",
      "Epoch [16][20]\t Batch [0][550]\t Training Loss 2.5354\t Accuracy 0.1300\n",
      "Epoch [16][20]\t Batch [50][550]\t Training Loss 2.5354\t Accuracy 0.0963\n",
      "Epoch [16][20]\t Batch [100][550]\t Training Loss 2.5354\t Accuracy 0.1001\n",
      "Epoch [16][20]\t Batch [150][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [16][20]\t Batch [200][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [16][20]\t Batch [250][550]\t Training Loss 2.5354\t Accuracy 0.0988\n",
      "Epoch [16][20]\t Batch [300][550]\t Training Loss 2.5354\t Accuracy 0.0987\n",
      "Epoch [16][20]\t Batch [350][550]\t Training Loss 2.5354\t Accuracy 0.0981\n",
      "Epoch [16][20]\t Batch [400][550]\t Training Loss 2.5354\t Accuracy 0.0982\n",
      "Epoch [16][20]\t Batch [450][550]\t Training Loss 2.5354\t Accuracy 0.0983\n",
      "Epoch [16][20]\t Batch [500][550]\t Training Loss 2.5354\t Accuracy 0.0986\n",
      "\n",
      "Epoch [16]\t Average training loss 2.5354\t Average training accuracy 0.0988\n",
      "Epoch [16]\t Average validation loss 2.5354\t Average validation accuracy 0.0978\n",
      "\n",
      "Epoch [17][20]\t Batch [0][550]\t Training Loss 2.5354\t Accuracy 0.1300\n",
      "Epoch [17][20]\t Batch [50][550]\t Training Loss 2.5354\t Accuracy 0.0963\n",
      "Epoch [17][20]\t Batch [100][550]\t Training Loss 2.5354\t Accuracy 0.1001\n",
      "Epoch [17][20]\t Batch [150][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [17][20]\t Batch [200][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [17][20]\t Batch [250][550]\t Training Loss 2.5354\t Accuracy 0.0988\n",
      "Epoch [17][20]\t Batch [300][550]\t Training Loss 2.5354\t Accuracy 0.0987\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch [17][20]\t Batch [350][550]\t Training Loss 2.5354\t Accuracy 0.0981\n",
      "Epoch [17][20]\t Batch [400][550]\t Training Loss 2.5354\t Accuracy 0.0982\n",
      "Epoch [17][20]\t Batch [450][550]\t Training Loss 2.5354\t Accuracy 0.0983\n",
      "Epoch [17][20]\t Batch [500][550]\t Training Loss 2.5354\t Accuracy 0.0986\n",
      "\n",
      "Epoch [17]\t Average training loss 2.5354\t Average training accuracy 0.0988\n",
      "Epoch [17]\t Average validation loss 2.5354\t Average validation accuracy 0.0978\n",
      "\n",
      "Epoch [18][20]\t Batch [0][550]\t Training Loss 2.5354\t Accuracy 0.1300\n",
      "Epoch [18][20]\t Batch [50][550]\t Training Loss 2.5354\t Accuracy 0.0963\n",
      "Epoch [18][20]\t Batch [100][550]\t Training Loss 2.5354\t Accuracy 0.1001\n",
      "Epoch [18][20]\t Batch [150][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [18][20]\t Batch [200][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [18][20]\t Batch [250][550]\t Training Loss 2.5354\t Accuracy 0.0988\n",
      "Epoch [18][20]\t Batch [300][550]\t Training Loss 2.5354\t Accuracy 0.0987\n",
      "Epoch [18][20]\t Batch [350][550]\t Training Loss 2.5354\t Accuracy 0.0981\n",
      "Epoch [18][20]\t Batch [400][550]\t Training Loss 2.5354\t Accuracy 0.0982\n",
      "Epoch [18][20]\t Batch [450][550]\t Training Loss 2.5354\t Accuracy 0.0983\n",
      "Epoch [18][20]\t Batch [500][550]\t Training Loss 2.5354\t Accuracy 0.0986\n",
      "\n",
      "Epoch [18]\t Average training loss 2.5354\t Average training accuracy 0.0988\n",
      "Epoch [18]\t Average validation loss 2.5354\t Average validation accuracy 0.0978\n",
      "\n",
      "Epoch [19][20]\t Batch [0][550]\t Training Loss 2.5354\t Accuracy 0.1300\n",
      "Epoch [19][20]\t Batch [50][550]\t Training Loss 2.5354\t Accuracy 0.0963\n",
      "Epoch [19][20]\t Batch [100][550]\t Training Loss 2.5354\t Accuracy 0.1001\n",
      "Epoch [19][20]\t Batch [150][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [19][20]\t Batch [200][550]\t Training Loss 2.5354\t Accuracy 0.0996\n",
      "Epoch [19][20]\t Batch [250][550]\t Training Loss 2.5354\t Accuracy 0.0988\n",
      "Epoch [19][20]\t Batch [300][550]\t Training Loss 2.5354\t Accuracy 0.0987\n",
      "Epoch [19][20]\t Batch [350][550]\t Training Loss 2.5354\t Accuracy 0.0981\n",
      "Epoch [19][20]\t Batch [400][550]\t Training Loss 2.5354\t Accuracy 0.0982\n",
      "Epoch [19][20]\t Batch [450][550]\t Training Loss 2.5354\t Accuracy 0.0983\n",
      "Epoch [19][20]\t Batch [500][550]\t Training Loss 2.5354\t Accuracy 0.0986\n",
      "\n",
      "Epoch [19]\t Average training loss 2.5354\t Average training accuracy 0.0988\n",
      "Epoch [19]\t Average validation loss 2.5354\t Average validation accuracy 0.0978\n",
      "\n"
     ]
    }
   ],
   "source": [
    "reluMLP, relu_loss, relu_acc = train(reluMLP, criterion, sgd, data_train, max_epoch, batch_size, disp_freq)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Testing...\n",
      "The test accuracy is 0.0980.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "test(reluMLP, criterion, data_test, batch_size, disp_freq)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZUAAAEGCAYAAACtqQjWAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAfJUlEQVR4nO3df5xVdb3v8ddbBkUZNBRQ5NeggQ9AcMAR9ZKKP6IkBTlHw47Xq2nhtTDpeLuo596iPD0yM+v04+Qhsehqek0QzaMFlxSsa+gwjSKMpHXIkEkGrgqUpoOf+8de0HbcM+xh1tp7M7yfj8d+zFrf9f3xWcNmf+a71tprKSIwMzNLwwHlDsDMzLoPJxUzM0uNk4qZmaXGScXMzFLjpGJmZqmpKncApdCvX7+oqakpdxhmZvuU1atXb4mI/p1ps18klZqaGurr68sdhpnZPkXSHzrbxoe/zMwsNU4qZmaWGicVMzNLzX5xTsXM9i9vv/02Gzdu5M033yx3KPuEXr16MXjwYHr27NnlvpxUzKzb2bhxI3369KGmpgZJ5Q6nokUEW7duZePGjQwfPrzL/fnwl5l1O2+++SZHHHGEE0oRJHHEEUekNqvLLKlIGiLpMUlNktZKuradepMlNSZ1VuypraR5kl5O2jRKmprVPpjZvssJpXhp/q6yPPzVClwXEQ2S+gCrJS2LiHW7Kkh6H/CvwIcj4iVJA4ps+42IuDXD2M3MbC9kNlOJiOaIaEiWtwNNwKA21f4BWBwRLyX1NneirZlZRfvyl7/MmDFjGDduHLW1taxatYpPfOITrFu3bs+Nu2Dq1Km89tpr7ymfN28et96a7d/jJTlRL6kGGA+sarNpJNBT0uNAH+BfIuJHRbSdLem/APXkZjSvZhO5mXV3df+8jC073npPeb/qA6n/Hx/c636ffPJJHn74YRoaGjjooIPYsmULb731FnfccUdXwi3KI488kvkY7cn8RL2kamARMCcitrXZXAWcCHwE+BDwPyWN3EPb7wHHArVAM/D1dsadJaleUn1LS0uKe2Rm3UmhhNJRebGam5vp168fBx10EAD9+vXj6KOPZvLkybtvG7VgwQJGjhzJ5MmT+eQnP8ns2bMBuPzyy7n66qs588wzOeaYY1ixYgVXXHEFo0aN4vLLL989xj333MPYsWM5/vjjmTt37u7ympoatmzZAuRmS8cddxznnHMO69ev79I+FSPTmYqknuSSwt0RsbhAlY3Aloj4M/BnSSuBE4Dfttc2Il7J6//7wMOFxo6I+cB8gLq6Oj8z2Ww/9cWfrmXdprZ/zxZn5r89WbB89NGH8oXzx3TYdsqUKXzpS19i5MiRnHPOOcycOZMzzjhj9/ZNmzZx00030dDQQJ8+fTjrrLM44YQTdm9/9dVX+cUvfsFDDz3E+eefz69+9SvuuOMOTjrpJBobGxkwYABz585l9erV9O3blylTprBkyRIuuOCC3X2sXr2ae++9l9/85je0trYyYcIETjzxxL36XRQry6u/BCwAmiLitnaqPQicJqlK0iHAyUBTR20lDcxbnQE8l370ZmZdU11dzerVq5k/fz79+/dn5syZ/PCHP9y9/amnnuKMM87g8MMPp2fPnlx00UXvan/++ecjibFjx3LkkUcyduxYDjjgAMaMGcOGDRt4+umnmTx5Mv3796eqqopLLrmElStXvquPJ554ghkzZnDIIYdw6KGHMm3atMz3O8uZyiTgUmCNpMak7EZgKEBE3B4RTZJ+BjwLvAPcERHPSfpAobYR8Qhwi6RaIIANwFUZ7oOZ7eP2NKOouf7f2932v686tUtj9+jRg8mTJzN58mTGjh3LwoULd2+L6PgAyq7DZgcccMDu5V3rra2tVFUV9/Fd6kurs7z665cRoYgYFxG1yeuRJJncnlfvaxExOiKOj4hvdtQ22XZpRIxNtk2LiOas9sHMbG+tX7+eF154Yfd6Y2Mjw4YN270+ceJEVqxYwauvvkprayuLFi3qVP8nn3wyK1asYMuWLezcuZN77rnnXYfXAE4//XQeeOAB3njjDbZv385Pf/rTru1UEXybFjPbr/WrPrDdq7+6YseOHVxzzTW89tprVFVV8f73v5/58+dz4YUXAjBo0CBuvPFGTj75ZI4++mhGjx7NYYcdVnT/AwcO5Ctf+QpnnnkmEcHUqVOZPn36u+pMmDCBmTNnUltby7BhwzjttNO6tE/F0J6mYN1BXV1d+CFdZvuPpqYmRo0aVe4w9mjHjh1UV1fT2trKjBkzuOKKK5gxY0ZZYin0O5O0OiLqOtOP7/1lZlYm8+bNo7a2luOPP57hw4e/68qtfZUPf5mZlUnW324vB89UzMwsNU4qZmaWGicVMzNLjZOKmZmlxknFzKyMqquryx1Cqnz1l5nt3742Av68+b3lvQfA5154b/leiAgiggMO6P5/x3f/PTQz60ihhNJReZE2bNjAqFGj+NSnPsWECRO46aabOOmkkxg3bhxf+MIX3lP/8ccf57zzztu9Pnv27HfdgHJf4ZmKmXVvj14Pf1qzd21/8JHC5UeNhXNv3mPz9evX84Mf/IALLriA+++/n6eeeoqIYNq0aaxcuZLTTz997+KqYJ6pmJllZNiwYZxyyiksXbqUpUuXMn78eCZMmMDzzz//rptNdieeqZhZ97anGcW8Dm7i+PH2b4tfjN69ewO5cyo33HADV13V/pM6qqqqeOedd3avv/nmm10au1w8UzEzy9iHPvQh7rzzTnbs2AHAyy+/zObN7z5nM2zYMNatW8df//pXXn/9dZYvX16OULvMMxUz27/1HtD+1V8pmTJlCk1NTZx6au6hX9XV1dx1110MGPC3MYYMGcJHP/pRxo0bx4gRIxg/fnxq45eSb31vZt3OvnLr+0riW9+bmVnFcVIxM7PUZJZUJA2R9JikJklrJV3bTr3JkhqTOivyyj8sab2kFyVdn1d+uKRlkl5IfvbNah/MbN+1PxzaT0uav6ssZyqtwHURMQo4Bfi0pNH5FSS9D/hXYFpEjAEuSsp7AN8FzgVGAx/La3s9sDwiRgDLk3Uzs9169erF1q1bnViKEBFs3bqVXr16pdJfZld/RUQz0Jwsb5fUBAwC1uVV+wdgcUS8lNTbdQnGRODFiPg9gKR7gelJ2+nA5KTeQuBxYG5W+2Fm+57BgwezceNGWlpayh3KPqFXr14MHjw4lb5KckmxpBpgPLCqzaaRQE9JjwN9gH+JiB+RSz5/zKu3ETg5WT4ySVhERLOkgtf9SZoFzAIYOnRoOjtiZvuEnj17Mnz48HKHsV/KPKlIqgYWAXMiYluB8U8EzgYOBp6U9GtABbrq1Dw2IuYD8yF3SXFn4zYzs87LNKlI6kkuodwdEYsLVNkIbImIPwN/lrQSOCEpH5JXbzCwKVl+RdLAZJYyEOjarUTNzCw1WV79JWAB0BQRt7VT7UHgNElVkg4hd4irCXgaGCFpuKQDgYuBh5I2DwGXJcuXJX2YmVkFyHKmMgm4FFgjqTEpuxEYChARt0dEk6SfAc8C7wB3RMRzAJJmAz8HegB3RsTapI+bgfskXQm8RHLFmJmZlZ9v02JmZgX5Ni1mZlZWTipmZpYaJxUzM0uNk4qZmaXGScXMzFLjpGJmZqlxUjEzs9Q4qZiZWWqcVMzMLDVOKmZmlhonFTMzS42TipmZpcZJxczMUuOkYmZmqXFSMTOz1DipmJlZapxUzMwsNU4qZmaWmsySiqQhkh6T1CRpraRrC9SZLOl1SY3J6/NJ+XF5ZY2Stkmak2ybJ+nlvG1Ts9oHMzPrnKoM+24FrouIBkl9gNWSlkXEujb1noiI8/ILImI9UAsgqQfwMvBAXpVvRMSt2YVuZmZ7I7OZSkQ0R0RDsrwdaAIG7UVXZwO/i4g/pBmfmZmlryTnVCTVAOOBVQU2nyrpGUmPShpTYPvFwD1tymZLelbSnZL6tjPmLEn1kupbWlq6FL+ZmRUn86QiqRpYBMyJiG1tNjcAwyLiBODbwJI2bQ8EpgE/ySv+HnAsucNjzcDXC40bEfMjoi4i6vr375/CnpiZ2Z5kmlQk9SSXUO6OiMVtt0fEtojYkSw/AvSU1C+vyrlAQ0S8ktfmlYjYGRHvAN8HJma5D2ZmVrwsr/4SsABoiojb2qlzVFIPSROTeLbmVfkYbQ59SRqYtzoDeC7NuM3MbO9lefXXJOBSYI2kxqTsRmAoQETcDlwIXC2pFXgDuDgiAkDSIcAHgava9HuLpFoggA0FtpuZWZlkllQi4peA9lDnO8B32tn2F+CIAuWXphKgmZmlzt+oNzOz1DipmJlZapxUzMwsNU4qZmaWGicVMzNLjZOKmZmlxknFzMxS46RiZmapcVIxM7PUOKmYmVlqnFTMzCw1TipmZpYaJxUzM0uNk4qZmaXGScXMzFLjpGJmZqlxUjEzs9Q4qZiZWWoySyqShkh6TFKTpLWSri1QZ7Kk1yU1Jq/P523bIGlNUl6fV364pGWSXkh+9s1qH8zMrHOynKm0AtdFxCjgFODTkkYXqPdERNQmry+12XZmUl6XV3Y9sDwiRgDLk3UzM6sAmSWViGiOiIZkeTvQBAxKoevpwMJkeSFwQQp9mplZCkpyTkVSDTAeWFVg86mSnpH0qKQxeeUBLJW0WtKsvPIjI6IZcokLGNDOmLMk1Uuqb2lpSWdHzMysQ1VZDyCpGlgEzImIbW02NwDDImKHpKnAEmBEsm1SRGySNABYJun5iFhZ7LgRMR+YD1BXVxdd3Q8zM9uzTGcqknqSSyh3R8TittsjYltE7EiWHwF6SuqXrG9Kfm4GHgAmJs1ekTQw6X8gsDnLfTAzs+JlefWXgAVAU0Tc1k6do5J6SJqYxLNVUm9JfZLy3sAU4Lmk2UPAZcnyZcCDWe2DmZl1TpaHvyYBlwJrJDUmZTcCQwEi4nbgQuBqSa3AG8DFERGSjgQeSPJNFfDjiPhZ0sfNwH2SrgReAi7KcB/MzKwTFNH9TzfU1dVFfX39niuamdlukla3+UrHHvkb9WZmlhonFTMzS01RSSU5cX5AsjxS0rTkyi4zM7Pdip2prAR6SRpE7tYoHwd+mFVQZma2byo2qSgi/gL8HfDtiJgBFLqPl5mZ7ceKTiqSTgUuAf49Kcv82/hmZrZvKTapzAFuAB6IiLWSjgEeyywqMzPbJxU124iIFcAKgOSE/ZaI+EyWgVWCLfOG0o/X31vOYfSb91Lm7btTH5UQQ6X0UQkxVEoflRBDpfRRCTG07ePEgQecWFSjPMVe/fVjSYcmt0xZB6yX9LnODravKfSP01F52u27Ux+VEEOl9FEJMVRKH5UQQ6X0UQkxdLZuIUV9o15SY0TUSroEOBGYC6yOiHFdGr1E9vob9fMOa3fT2gPH7rH5mLfWdKl9d+qjEmKolD4qIYZK6aMSYqiUPiohhrZ91M3fQf2mnSqqYaLYcyo9k++lXAA8GBFvk3veiZmZ2W7FzlQ+Q2528gzwEXI3hbwrIk7LNrx0ZDFTYV4RU8Sutu9OfVRCDJXSRyXEUCl9VEIMldJHJcTQpo+9makUe6L+W8C38or+IOnMzgxkZmbdX7En6g+TdNuux/NK+jrQO+PYym4LhbN+e+Vpt+9OfVRCDJXSRyXEUCl9VEIMldJHJcTQ2bqFFHv4axG5h2QtTIouBU6IiL/r0ugl4lvfm5l13t7c+r7Yb8UfGxF/n7f+xbwHb5mZmQHFX/31hqQP7FqRNInckxrNzMx2K3am8l+BH0nadbDtVf72nHgzMzOgyJlKRDwTEScA44BxETEeOKujNpKGSHpMUpOktZKuLVBnsqTXJTUmr8/vqa2keZJezmsztVN7bGZmmenUnYYjYlve6j8C3+ygeitwXUQ0SOoDrJa0LCLWtan3RESc18m234iIWzsTu5mZZa8rjxPu8AsxEdEcEQ3J8nagCRhUTMddaWtmZuXTlaRS9G1aJNUA44FVBTafKukZSY9KGlNk29mSnpV0p6S+7Yw5a9f3alpaWooN1czMuqDDpCJpu6RtBV7bgaOLGUBSNbAImNPm8BlAAzAsOV/zbWBJEW2/BxwL1ALNwNcLjRsR8yOiLiLq+vfvX0yoZmbWRR0mlYjoExGHFnj1iYg9no9JbkK5CLg7IhYX6H9bROxIlh8hd+PKfh21jYhXImJnRLwDfB+Y2In9NTOzDHXl8FeHJAlYADRFxG3t1DkqqYekiUk8WztqK2lg3uoMct/0NzOzCpDlc+Ynkbudy5q8b9/fSO4Ox0TE7cCFwNWSWsl9mfLiiIjki5bvaZvMZm6RVEvunM4G4KoM98HMzDqhqHt/7et87y8zs87bm3t/ZXb4y8zM9j9OKmZmlhonFTMzS42TipmZpcZJxczMUuOkYmZmqXFSMTOz1DipmJlZapxUzMwsNU4qZmaWGicVMzNLjZOKmZmlxknFzMxS46RiZmapcVIxM7PUOKmYmVlqnFTMzCw1TipmZpaazJKKpCGSHpPUJGmtpGsL1Jks6XVJjcnr83nbPixpvaQXJV2fV364pGWSXkh+9s1qH8zMrHOynKm0AtdFxCjgFODTkkYXqPdERNQmry8BSOoBfBc4FxgNfCyv7fXA8ogYASxP1s3MrAJkllQiojkiGpLl7UATMKjI5hOBFyPi9xHxFnAvMD3ZNh1YmCwvBC5ILWgzM+uSkpxTkVQDjAdWFdh8qqRnJD0qaUxSNgj4Y16djfwtIR0ZEc2QS1zAgGyiNjOzzqrKegBJ1cAiYE5EbGuzuQEYFhE7JE0FlgAjABXoKjo57ixgFsDQoUM7G7aZme2FTGcqknqSSyh3R8TittsjYltE7EiWHwF6SupHbmYyJK/qYGBTsvyKpIFJ/wOBzYXGjoj5EVEXEXX9+/dPbZ/MzKx9WV79JWAB0BQRt7VT56ikHpImJvFsBZ4GRkgaLulA4GLgoaTZQ8BlyfJlwINZ7YOZmXVOloe/JgGXAmskNSZlNwJDASLiduBC4GpJrcAbwMUREUCrpNnAz4EewJ0RsTbp42bgPklXAi8BF2W4D2Zm1gnKfYZ3b3V1dVFfX1/uMMzM9imSVkdEXWfa+Bv1ZmaWGicVMzNLjZOKmZmlxknFzMxS46RiZmapcVIxM7PUOKmYmVlqnFTMzCw1TipmZpYaJxUzM0uNk4qZmaXGScXMzFLjpGJmZqlxUjEzs9Q4qZiZWWqcVMzMLDVOKmZmlhonFTMzS42TipmZpSazpCJpiKTHJDVJWivp2g7qniRpp6QLk/XjJDXmvbZJmpNsmyfp5bxtU7PaBzMz65yqDPtuBa6LiAZJfYDVkpZFxLr8SpJ6AF8Ffr6rLCLWA7V5218GHshr9o2IuDXD2M3MbC9kNlOJiOaIaEiWtwNNwKACVa8BFgGb2+nqbOB3EfGHTAI1M7PUlOSciqQaYDywqk35IGAGcHsHzS8G7mlTNlvSs5LulNS3nTFnSaqXVN/S0rL3wZuZWdEyTyqSqsnNROZExLY2m78JzI2Ine20PRCYBvwkr/h7wLHkDo81A18v1DYi5kdEXUTU9e/fv0v7YGZmxcnynAqSepJLKHdHxOICVeqAeyUB9AOmSmqNiCXJ9nOBhoh4ZVeD/GVJ3wcezih8MzPrpMySinKZYgHQFBG3FaoTEcPz6v8QeDgvoQB8jDaHviQNjIjmZHUG8FyKYZuZWRdkOVOZBFwKrJHUmJTdCAwFiIiOzqMg6RDgg8BVbTbdIqkWCGBDge1mZlYmmSWViPgloE7Uv7zN+l+AIwrUu7TLwZmZWSb8jXozM0uNk4qZmaXGScXMzFLjpGJmZqlxUjEzs9Q4qZiZWWqcVMzMLDVOKmZmlhonFTMzS42TipmZpcZJxczMUuOkYmZmqXFSMTOz1DipmJlZapxUzMwsNU4qZmaWGicVMzNLjZOKmZmlJrOkImmIpMckNUlaK+naDuqeJGmnpAvzyjZIWiOpUVJ9XvnhkpZJeiH52TerfTAzs87JcqbSClwXEaOAU4BPSxrdtpKkHsBXgZ8X6OPMiKiNiLq8suuB5RExAlierJuZWQXILKlERHNENCTL24EmYFCBqtcAi4DNRXY9HViYLC8ELuhapGZmlpaSnFORVAOMB1a1KR8EzABuL9AsgKWSVkualVd+ZEQ0Qy5xAQPaGXOWpHpJ9S0tLSnshZmZ7UnmSUVSNbmZyJyI2NZm8zeBuRGxs0DTSRExATiX3KGz0zszbkTMj4i6iKjr37//3oRuZmadVJVl55J6kksod0fE4gJV6oB7JQH0A6ZKao2IJRGxCSAiNkt6AJgIrARekTQwIpolDaT4w2ZmZpaxLK/+ErAAaIqI2wrViYjhEVETETXA/cCnImKJpN6S+iT99AamAM8lzR4CLkuWLwMezGofzMysc7KcqUwCLgXWSGpMym4EhgJERKHzKLscCTyQzGCqgB9HxM+SbTcD90m6EngJuCj90M3MbG9kllQi4peAOlH/8rzl3wMntFNvK3B2V+MzM7P0+Rv1ZmaWGicVMzNLjZOKmZmlxknFzMxS46RiZmapcVIxM7PUOKmYmVlqnFTMzCw1TipmZpYaJxUzM0uNIqLcMWRO0nZgfZnD6AdsKXMMUBlxVEIMUBlxVEIMUBlxVEIMUBlxVEIMAMdFRJ/ONMj01vcVZH2bRxKXnKT6csdQKXFUQgyVEkclxFApcVRCDJUSRyXEsCuOzrbx4S8zM0uNk4qZmaVmf0kq88sdAJURA1RGHJUQA1RGHJUQA1RGHJUQA1RGHJUQA+xFHPvFiXozMyuN/WWmYmZmJeCkYmZmqenWSUXShyWtl/SipOvLFMMQSY9JapK0VtK15YgjiaWHpN9IeriMMbxP0v2Snk9+J6eWIYbPJv8Wz0m6R1KvEo17p6TNkp7LKztc0jJJLyQ/+5Ypjq8l/ybPSnpA0vtKHUPetv8mKST1yzKGjuKQdE3y2bFW0i2ljkFSraRfS2qUVC9pYsYxFPyc2qv3Z0R0yxfQA/gdcAxwIPAMMLoMcQwEJiTLfYDfliOOZPx/BH4MPFzGf5eFwCeS5QOB95V4/EHAfwAHJ+v3AZeXaOzTgQnAc3lltwDXJ8vXA18tUxxTgKpk+atZx1EohqR8CPBz4A9AvzL9Ls4E/g9wULI+oAwxLAXOTZanAo9nHEPBz6m9eX9255nKRODFiPh9RLwF3AtML3UQEdEcEQ3J8nagidwHW0lJGgx8BLij1GPnxXAouf9ACwAi4q2IeK0MoVQBB0uqAg4BNpVi0IhYCfy/NsXTySVakp8XlCOOiFgaEa3J6q+BwaWOIfEN4L8DJbmCqJ04rgZujoi/JnU2lyGGAA5Nlg8j4/doB59TnX5/duekMgj4Y976RsrwYZ5PUg0wHlhVhuG/Se4/6ztlGHuXY4AW4AfJYbg7JPUuZQAR8TJwK/AS0Ay8HhFLSxlDG0dGRHMSWzMwoIyx7HIF8GipB5U0DXg5Ip4p9dhtjAROk7RK0gpJJ5UhhjnA1yT9kdz79YZSDdzmc6rT78/unFRUoKxs109LqgYWAXMiYluJxz4P2BwRq0s5bgFV5Kb534uI8cCfyU2pSyY5JjwdGA4cDfSW9J9LGUMlk/RPQCtwd4nHPQT4J+DzpRy3HVVAX+AU4HPAfZIKfZ5k6WrgsxExBPgsyew+a2l8TnXnpLKR3PHZXQZTosMcbUnqSe4f6u6IWFyGECYB0yRtIHcY8CxJd5Uhjo3AxojYNVO7n1ySKaVzgP+IiJaIeBtYDPynEseQ7xVJAwGSn5keaumIpMuA84BLIjmIXkLHkkv0zyTv08FAg6SjShwH5N6niyPnKXKz+8wvGmjjMnLvTYCfkDucn6l2Pqc6/f7szknlaWCEpOGSDgQuBh4qdRDJXzgLgKaIuK3U4wNExA0RMTgiasj9Hn4RESX/6zwi/gT8UdJxSdHZwLoSh/EScIqkQ5J/m7PJHT8ul4fIfYCQ/HywHEFI+jAwF5gWEX8p9fgRsSYiBkRETfI+3UjuxPGfSh0LsAQ4C0DSSHIXlJT6jsGbgDOS5bOAF7IcrIPPqc6/P7O8oqDcL3JXTfyW3FVg/1SmGD5A7rDbs0Bj8ppaxt/JZMp79VctUJ/8PpYAfcsQwxeB54HngP9FcpVPCca9h9x5nLfJfWheCRwBLCf3obEcOLxMcbxI7hzkrvfo7aWOoc32DZTm6q9Cv4sDgbuS90cDcFYZYvgAsJrcVaurgBMzjqHg59TevD99mxYzM0tNdz78ZWZmJeakYmZmqXFSMTOz1DipmJlZapxUzMwsNU4qZimQtDO5o+yuV2p3CpBUU+huvmaVqKrcAZh1E29ERG25gzArN89UzDIkaYOkr0p6Knm9PykfJml58vyS5ZKGJuVHJs8zeSZ57bqFTA9J30+edbFU0sFl2ymzDjipmKXj4DaHv2bmbdsWEROB75C7WzTJ8o8iYhy5mzd+Kyn/FrAiIk4gd1+0tUn5COC7ETEGeA34+0z3xmwv+Rv1ZimQtCMiqguUbyB3m4/fJzfs+1NEHCFpCzAwIt5Oypsjop+kFmBwJM/ySPqoAZZFxIhkfS7QMyL+uQS7ZtYpnqmYZS/aWW6vTiF/zVveic+HWoVyUjHL3sy8n08my/+X3B2jAS4BfpksLyf3LA0k9Uielmm2z/BfO2bpOFhSY976zyJi12XFB0laRe6PuI8lZZ8B7pT0OXJPw/x4Un4tMF/SleRmJFeTu4Ot2T7B51TMMpScU6mLiFI/j8OsLHz4y8zMUuOZipmZpcYzFTMzS42TipmZpcZJxczMUuOkYmZmqXFSMTOz1Px/jZgihkBl9OwAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZUAAAEGCAYAAACtqQjWAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAfd0lEQVR4nO3dfbhVdZ338fcHDoiCKAooAgNU+IBiRzyiTaX4REIKcpWh0+RjoRaWzdiIdV8T5TSZWd2X93jJoGJ2ZzilUdRNqUOp3d0+cA6RikSQQ3oE5cFHMsAD3/uPtQ6zOWzO2fuw1t77HD6v69rXXuu31u+3vmuz2d+zfmut31JEYGZmloUe1Q7AzMy6DycVMzPLjJOKmZllxknFzMwy46RiZmaZqat2AJUwcODAGDlyZLXDMDPrUpqamjZGxKBy6uwTSWXkyJE0NjZWOwwzsy5F0p/LrePuLzMzy4yTipmZZcZJxczMMrNPnFMxs33LO++8Q3NzM1u2bKl2KF1Cnz59GDZsGL169drrtpxUzKzbaW5u5sADD2TkyJFIqnY4NS0i2LRpE83NzYwaNWqv23P3l5l1O1u2bOHQQw91QimBJA499NDMjuqcVMysW3JCKV2Wn5WTipmZZcZJxcwsJ1/72tc49thjOf7446mvr+fJJ5/kk5/8JM8991yu2508eTKvv/76buWzZ8/mlltuyXXbPlFvZvu0hn95mI2bt+1WPrBfbxr/x9mdbvfxxx/n5z//OUuXLmW//fZj48aNbNu2jTvvvHNvwi3JokWLct/GnvhIxcz2acUSSnvlpVq3bh0DBw5kv/32A2DgwIEcccQRTJgwYeewUXfddRdHHnkkEyZM4FOf+hQzZ84E4NJLL+Xqq6/m9NNP513vehePPvool19+OccccwyXXnrpzm3Mnz+fsWPHctxxx3H99dfvLB85ciQbN24EkqOlo446irPOOouVK1fu1T6VwkcqZtatfeVny3lu7Zudqjv93x8vWj7miP58+bxj2607ceJEvvrVr3LkkUdy1llnMX36dE477bSdy9euXcuNN97I0qVLOfDAAznjjDN473vfu3P5a6+9xq9+9SsWLlzIeeedx29/+1vuvPNOTjrpJJYtW8bgwYO5/vrraWpqYsCAAUycOJGf/OQnnH/++TvbaGpq4r777uN3v/sdLS0tjBs3jhNPPLFTn0WpfKRiZpaDfv360dTUxNy5cxk0aBDTp0/nu9/97s7lTz31FKeddhqHHHIIvXr14oILLtil/nnnnYckxo4dy2GHHcbYsWPp0aMHxx57LGvWrGHJkiVMmDCBQYMGUVdXx8c//nEee+yxXdr4zW9+w7Rp0zjggAPo378/U6ZMyX2/faRiZt1aR0cUI2f9nz0u+48r37dX2+7ZsycTJkxgwoQJjB07lnvuuWfnsohot25rt1mPHj12TrfOt7S0UFdX2s93pS+t9pGKmVkOVq5cyapVq3bOL1u2jBEjRuycHz9+PI8++iivvfYaLS0tPPDAA2W1f/LJJ/Poo4+yceNGtm/fzvz583fpXgM49dRTWbBgAX/961956623+NnPfrZ3O1UCH6mY2T5tYL/ee7z6a29s3ryZa665htdff526ujre8573MHfuXD760Y8CMHToUL74xS9y8sknc8QRRzBmzBgOOuigktsfMmQIX//61zn99NOJCCZPnszUqVN3WWfcuHFMnz6d+vp6RowYwQc/+MG92qdSqKNDsO6goaEh/JAus33HihUrOOaYY6odRoc2b95Mv379aGlpYdq0aVx++eVMmzatKrEU+8wkNUVEQzntuPvLzKxKZs+eTX19PccddxyjRo3a5cqtrsrdX2ZmVZL33e3V4CMVMzPLTK5JRdI5klZKWi1pVpHlR0t6XNJWSdcVlB8laVnB601J16bLZkt6qWDZ5Dz3wczMSpdb95eknsBtwNlAM7BE0sKIKBxJ7VXgs8D5hXUjYiVQX9DOS8CCglW+ExHd77jRzKyLy/NIZTywOiKej4htwH3ALte7RcT6iFgCvNNOO2cCf4qIP+cXqpmZZSHPpDIUeLFgvjktK9eFwPw2ZTMlPS1pnqQBxSpJmiGpUVLjhg0bOrFZM7P89evXr9ohZCrPq7+KjQ1Q1k0xknoDU4AbCopvB25M27oR+BZw+W4bipgLzIXkPpVytmtm+5Bvjoa/rN+9vO9g+MKq3cs7ISKICHr06P7XRuW5h83A8IL5YcDaMtuYBCyNiFdaCyLilYjYHhE7gDtIutnMzDqnWEJpr7xEa9as4ZhjjuHTn/4048aN48Ybb+Skk07i+OOP58tf/vJu6z/yyCOce+65O+dnzpy5ywCUXUWeRypLgNGSRpGcaL8Q+Lsy27iINl1fkoZExLp0dhrw7N4Gambd2C9mwcvPdK7u3R8uXn74WJh0U4fVV65cyd13383555/P/fffz1NPPUVEMGXKFB577DFOPfXUzsVVw3JLKhHRImkm8CDQE5gXEcslXZUunyPpcKAR6A/sSC8bHhMRb0o6gOTKsSvbNH2zpHqS7q81RZabmdWEESNGcMopp3Ddddfx0EMPccIJJwDJ8CyrVq1yUilXRCwCFrUpm1Mw/TJJt1ixum8DhxYp/0TGYZpZd9bREcXsdgZxvGzPw+KXom/fvkByTuWGG27gyiv3/DdwXV0dO3bs2Dm/ZcuWvdp2tXT/s0ZmZlX2oQ99iHnz5rF582YAXnrpJdav3/WczYgRI3juuefYunUrb7zxBosXL65GqHvNY3+Z2b6t7+A9X/2VkYkTJ7JixQre977koV/9+vXj+9//PoMH//c2hg8fzsc+9jGOP/54Ro8evbOrrKvx0Pdm1u10laHva4mHvjczs5rjpGJmZplxUjGzbmlf6NrPSpaflZOKmXU7ffr0YdOmTU4sJYgINm3aRJ8+fTJpz1d/mVm3M2zYMJqbm/FgsqXp06cPw4YVvWWwbE4qZtbt9OrVi1GjRlU7jH2Su7/MzCwzTipmZpYZJxUzM8uMk4qZmWXGScXMzDLjpGJmZplxUjEzs8w4qZiZWWacVMzMLDNOKmZmlplck4qkcyStlLRa0qwiy4+W9LikrZKua7NsjaRnJC2T1FhQfoikhyWtSt8H5LkPZmZWutySiqSewG3AJGAMcJGkMW1WexX4LHDLHpo5PSLq2zx5bBawOCJGA4vTeTMzqwF5HqmMB1ZHxPMRsQ24D5hauEJErI+IJcA7ZbQ7Fbgnnb4HOD+DWM3MLAN5JpWhwIsF881pWakCeEhSk6QZBeWHRcQ6gPR9cLHKkmZIapTU6OGvzcwqI8+koiJl5Twx5/0RMY6k++wzkk4tZ+MRMTciGiKiYdCgQeVUNTOzTsozqTQDwwvmhwFrS60cEWvT9/XAApLuNIBXJA0BSN/XZxKtmZnttTyTyhJgtKRRknoDFwILS6koqa+kA1ungYnAs+nihcAl6fQlwE8zjdrMzDottyc/RkSLpJnAg0BPYF5ELJd0Vbp8jqTDgUagP7BD0rUkV4oNBBZIao3xBxHxy7Tpm4AfSroCeAG4IK99MDOz8iiinNMcXVNDQ0M0NjZ2vKKZme0kqanNLR0d8h31ZmaWGScVMzPLjJOKmZllxknFzMwy46RiZmaZcVIxM7PMOKmYmVlmnFTMzCwzTipmZpYZJxUzM8uMk4qZmWXGScXMzDLjpGJmZplxUjEzs8w4qZiZWWacVMzMLDNOKmZmlhknFTMzy0yuSUXSOZJWSlotaVaR5UdLelzSVknXFZQPl/RrSSskLZf0uYJlsyW9JGlZ+pqc5z6YmVnp6vJqWFJP4DbgbKAZWCJpYUQ8V7Daq8BngfPbVG8B/jEilko6EGiS9HBB3e9ExC15xW5mZp2T55HKeGB1RDwfEduA+4CphStExPqIWAK806Z8XUQsTaffAlYAQ3OM1czMMpBnUhkKvFgw30wnEoOkkcAJwJMFxTMlPS1pnqQBe6g3Q1KjpMYNGzaUu1kzM+uEPJOKipRFWQ1I/YAHgGsj4s20+Hbg3UA9sA74VrG6ETE3IhoiomHQoEHlbNbMzDopz6TSDAwvmB8GrC21sqReJAnl3oj4cWt5RLwSEdsjYgdwB0k3m5mZ1YA8k8oSYLSkUZJ6AxcCC0upKEnAXcCKiPh2m2VDCmanAc9mFK+Zme2l3K7+iogWSTOBB4GewLyIWC7pqnT5HEmHA41Af2CHpGuBMcDxwCeAZyQtS5v8YkQsAm6WVE/SlbYGuDKvfTAzs/IooqzTHF1SQ0NDNDY2VjsMM7MuRVJTRDSUU8d31JuZWWacVMzMLDNOKmZmlpkOk4qkcyU5+ZiZWYdKSRYXAqsk3SzpmLwDMjOzrqvDpBIRf08yTMqfgLvTUYVnpAM9mpmZ7VRSt1Y6RMoDJINCDiG56XCppGtyjM3MzLqYUs6pnCdpAfAroBcwPiImAe8Frmu3spmZ7VNKuaP+ApLnlzxWWBgRb0u6PJ+wzMysKyolqXyZZDRgACTtDxwWEWsiYnFukZmZWZdTyjmVHwE7Cua3p2VmZma7KCWp1KVPbgQgne6dX0hmZtZVlZJUNkia0jojaSqwMb+QzMysqyrlnMpVwL2S/o3kaY4vAhfnGpWZmXVJHSaViPgTcEr6aF9FxFv5h2VmZl1RSQ/pkvRh4FigT/JQRoiIr+YYl5mZdUGl3Pw4B5gOXEPS/XUBMCLnuMzMrAsq5UT930bExcBrEfEV4H3A8HzDMjOzrqiUpLIlfX9b0hHAO8Co/EIyM7OuqpSk8jNJBwPfBJYCa4D5pTQu6RxJKyWtljSryPKj01GPt0q6rpS6kg6R9LCkVen7gFJiMTOz/LWbVNKHcy2OiNcj4gGScylHR8Q/d9SwpJ7AbcAkYAxwkaQxbVZ7FfgscEsZdWelMY0GFqfzZmZWA9q9+isidkj6Fsl5FCJiK7C1xLbHA6sj4nkASfcBU4HnCtpfD6xPry4rte5UYEK63j3AI8D1JcZUlo2z/4aBvLF7OQcxcPYLudfvTm3UQgy10kYtxFArbdRCDLXSRi3E0LaNE4f0OLGkSgVK6f56SNJH1HotcemGktwo2ao5LdvbuodFxDqA9H1wsQbSB4k1SmrcsGFDWYG3KvaP01551vW7Uxu1EEOttFELMdRKG7UQQ620UQsxlLtuMaXcp/IPQF+gRdIWksuKIyL6d1CvWBKKEuPam7rJyhFzgbkADQ0NZdUtxfJ//UCH6xy7l/W7Uxu1EEOttFELMdRKG7UQQ620UQsxdNRGKUq5o76zjw1uZtdLj4cBazOo+4qkIRGxTtIQYH0n4zMzs4wpov0/4iWdWqy87UO7itSrA/4InAm8BCwB/i4ilhdZdzawOSJu6aiupG8CmyLipvSqsEMi4p/ai6WhoSEaGxvb3c+iZh/UzrISDhH3tn53aqMWYqiVNmohhlppoxZiqJU2aiGGNm00zN1M49rtZZ36KKX76wsF031ITqI3AWe0VykiWiTNBB4EegLz0qRwVbp8jqTDgUagP7BD0rXAmIh4s1jdtOmbgB9KugJ4geQOfzMzqwEdnqiPiPMKXmcDxwGvlNJ4RCyKiCMj4t0R8bW0bE5EzEmnX46IYRHRPyIOTqff3FPdtHxTRJwZEaPT91c7s+Ol2EjxrL+n8qzrd6c2aiGGWmmjFmKolTZqIYZaaaMWYih33WI67P7arUJyFdjTETF2r7ZcQZ3u/jIz24dJaoqIhnLqdNj9Jel/8d9XXvUA6oHflx2dmZl1e6WcUyn8E78FmB8Rv80pHjMz68JKSSr3A1siYjskQ6hIOiAi3s43NDMz62pKuaN+MbB/wfz+wH/mE46ZmXVlpSSVPhGxuXUmnT4gv5DMzKyrKiWp/EXSuNYZSScCf80vJDMz66pKOadyLfAjSa3DpAwhebywmZnZLkoZ+2uJpKOBo0gGevxDRLyTe2RmZtbldNj9JekzQN+IeDYingH6Sfp0/qGZmVlXU8o5lU9FxOutMxHxGvCp3CIyM7Muq5Sk0qPwAV3po3575xeSmZl1VaWcqH+QZFTgOSTDtVwF/CLXqMzMrEsqJalcD8wAriY5Uf87kivAzMzMdlHK0Pc7gCeA54EGkgdnrcg5LjMz64L2eKQi6UjgQuAiYBPwHwARcXplQjMzs66mve6vPwC/Ac6LiNUAkj5fkajMzKxLaq/76yPAy8CvJd0h6UyScypmZmZF7TGpRMSCiJgOHA08AnweOEzS7ZImltK4pHMkrZS0WtKsIssl6dZ0+dOtY4xJOkrSsoLXm+nz65E0W9JLBcsml7/bZmaWh1JO1P8lIu6NiHOBYcAyYLcE0VZ6P8ttwCRgDHCRpDFtVpsEjE5fM4Db022ujIj6iKgHTgTeBhYU1PtO6/KIWNRRLGZmVhml3Py4U0S8GhH/HhFnlLD6eGB1RDwfEduA+4CpbdaZCnwvEk8AB0tqe7nymcCfIuLP5cRqZmaVV1ZSKdNQ4MWC+ea0rNx1LgTmtymbmXaXzZM0oNjGJc2Q1CipccOGDeVHb2ZmZcszqRQ7qR/lrCOpNzAF+FHB8tuBdwP1wDrgW8U2HhFzI6IhIhoGDRpURthmZtZZeSaVZmB4wfwwYG2Z60wClkbEK60FEfFKRGxPb8q8g6SbzczMakCeSWUJMFrSqPSI40JgYZt1FgIXp1eBnQK8ERHrCpZfRJuurzbnXKYBz2YfupmZdUYpY391SkS0SJpJMiBlT2BeRCyXdFW6fA6wCJgMrCa5wuuy1vqSDgDOBq5s0/TNkupJusnWFFluZmZVooi2pzm6n4aGhmhsbKx2GGZmXYqkpohoKKdOnt1fZma2j3FSMTOzzDipmJlZZpxUzMwsM04qZmaWGScVMzPLjJOKmZllxknFzMwy46RiZmaZcVIxM7PMOKmYmVlmnFTMzCwzTipmZpYZJxUzM8uMk4qZmWXGScXMzDLjpGJmZplxUjEzs8w4qZiZWWZyTSqSzpG0UtJqSbOKLJekW9PlT0saV7BsjaRnJC2T1FhQfoikhyWtSt8H5LkPZmZWutySiqSewG3AJGAMcJGkMW1WmwSMTl8zgNvbLD89IuojoqGgbBawOCJGA4vTeTMzqwF5HqmMB1ZHxPMRsQ24D5jaZp2pwPci8QRwsKQhHbQ7Fbgnnb4HOD/DmM3MbC/kmVSGAi8WzDenZaWuE8BDkpokzShY57CIWAeQvg8utnFJMyQ1SmrcsGHDXuyGmZmVKs+koiJlUcY674+IcSRdZJ+RdGo5G4+IuRHREBENgwYNKqeqmZl1Up5JpRkYXjA/DFhb6joR0fq+HlhA0p0G8EprF1n6vj7zyM3MrFPyTCpLgNGSRknqDVwILGyzzkLg4vQqsFOANyJinaS+kg4EkNQXmAg8W1DnknT6EuCnOe6DmZmVoS6vhiOiRdJM4EGgJzAvIpZLuipdPgdYBEwGVgNvA5el1Q8DFkhqjfEHEfHLdNlNwA8lXQG8AFyQ1z6YmVl5FNH2NEf309DQEI2NjR2vaGZmO0lqanNLR4d8R72ZmWXGScXMzDLjpGJmZplxUjEzs8w4qZiZWWacVMzMLDNOKmZmlhknFTMzy4yTipmZZcZJxczMMuOkYmZmmXFSMTOzzDipmJlZZpxUzMwsM04qZmaWGScVMzPLjJOKmZllxknFzMwyk2tSkXSOpJWSVkuaVWS5JN2aLn9a0ri0fLikX0taIWm5pM8V1Jkt6SVJy9LX5Dz3wczMSleXV8OSegK3AWcDzcASSQsj4rmC1SYBo9PXycDt6XsL8I8RsVTSgUCTpIcL6n4nIm7JK3YzM+ucPI9UxgOrI+L5iNgG3AdMbbPOVOB7kXgCOFjSkIhYFxFLASLiLWAFMDTHWM3MLAN5JpWhwIsF883snhg6XEfSSOAE4MmC4plpd9k8SQOKbVzSDEmNkho3bNjQyV0wM7Ny5JlUVKQsyllHUj/gAeDaiHgzLb4deDdQD6wDvlVs4xExNyIaIqJh0KBBZYZuZmadkWdSaQaGF8wPA9aWuo6kXiQJ5d6I+HHrChHxSkRsj4gdwB0k3WxmZlYD8kwqS4DRkkZJ6g1cCCxss85C4OL0KrBTgDciYp0kAXcBKyLi24UVJA0pmJ0GPJvfLpiZWTlyu/orIlokzQQeBHoC8yJiuaSr0uVzgEXAZGA18DZwWVr9/cAngGckLUvLvhgRi4CbJdWTdJOtAa7Max/MzKw8imh7mqP7aWhoiMbGxmqHYWbWpUhqioiGcur4jnozM8uMk4qZmWXGScXMzDLjpGJmZplxUjEzs8w4qZiZWWacVMzMLDNOKmZmlhknFTMzy4yTipmZZcZJxczMMuOkYmZmmXFSMTOzzDipmJlZZpxUzMwsM04qZmaWGScVMzPLjJOKmZllxknFzMwyk2tSkXSOpJWSVkuaVWS5JN2aLn9a0riO6ko6RNLDklal7wPy3AczMytdbklFUk/gNmASMAa4SNKYNqtNAkanrxnA7SXUnQUsjojRwOJ03szMakCeRyrjgdUR8XxEbAPuA6a2WWcq8L1IPAEcLGlIB3WnAvek0/cA5+e4D2ZmVoa6HNseCrxYMN8MnFzCOkM7qHtYRKwDiIh1kgYX27ikGSRHPwBbJT3bmZ3I0EBgY5VjgNqIoxZigNqIoxZigNqIoxZigNqIoxZiADiq3Ap5JhUVKYsS1ymlbrsiYi4wF0BSY0Q0lFM/a7UQQ63EUQsx1EoctRBDrcRRCzHUShy1EENrHOXWybP7qxkYXjA/DFhb4jrt1X0l7SIjfV+fYcxmZrYX8kwqS4DRkkZJ6g1cCCxss85C4OL0KrBTgDfSrq326i4ELkmnLwF+muM+mJlZGXLr/oqIFkkzgQeBnsC8iFgu6ap0+RxgETAZWA28DVzWXt206ZuAH0q6AngBuKCEcOZmt2edVgsxQG3EUQsxQG3EUQsxQG3EUQsxQG3EUQsxQCfiUERZpyrMzMz2yHfUm5lZZpxUzMwsM906qXQ0TEyFYhgu6deSVkhaLulz1YgjjaWnpN9J+nkVYzhY0v2S/pB+Ju+rQgyfT/8tnpU0X1KfCm13nqT1hfdMVWPYoT3E8c303+RpSQskHVzpGAqWXScpJA3MM4b24pB0TfrbsVzSzZWOQVK9pCckLZPUKGl8zjEU/Z3q1PczIrrli+QE/5+AdwG9gd8DY6oQxxBgXDp9IPDHasSRbv8fgB8AP6/iv8s9wCfT6d7AwRXe/lDgv4D90/kfApdWaNunAuOAZwvKbgZmpdOzgG9UKY6JQF06/Y284ygWQ1o+nOQCnT8DA6v0WZwO/CewXzo/uAoxPARMSqcnA4/kHEPR36nOfD+785FKKcPE5C4i1kXE0nT6LWAFyQ9bRUkaBnwYuLPS2y6IoT/Jf6C7ACJiW0S8XoVQ6oD9JdUBB7D7/VO5iIjHgFfbFFd82KFicUTEQxHRks4+QXJvWEVjSH0H+CfKvNk54ziuBm6KiK3pOrneC7eHGALon04fRM7f0XZ+p8r+fnbnpLKnIWCqRtJI4ATgySps/n+S/GfdUYVtt3oXsAG4O+2Gu1NS30oGEBEvAbeQXI6+juTeqIcqGUMbuww7BBQddqjCLgd+UemNSpoCvBQRv6/0tts4EvigpCclPSrppCrEcC3wTUkvknxfb6jUhtv8TpX9/ezOSWWvh3rJkqR+wAPAtRHxZoW3fS6wPiKaKrndIupIDvNvj4gTgL9Q4VGm0z7hqcAo4Aigr6S/r2QMtUzSl4AW4N4Kb/cA4EvAP1dyu3tQBwwATgG+QHJfXLHfkzxdDXw+IoYDnyc9us9bFr9T3TmplDJMTEVI6kXyD3VvRPy4CiG8H5giaQ1JN+AZkr5fhTiageaIaD1Su58kyVTSWcB/RcSGiHgH+DHwtxWOoVDNDDsk6RLgXODjkXaiV9C7SRL979Pv6TBgqaTDKxwHJN/TH0fiKZKj+9wvGmjjEpLvJsCPSLrzc7WH36myv5/dOamUMkxM7tK/cO4CVkTEtyu9fYCIuCEihkXESJLP4VcRUfG/ziPiZeBFSa0jn54JPFfhMF4ATpF0QPpvcyZJ/3G11MSwQ5LOAa4HpkTE25XefkQ8ExGDI2Jk+j1tJjlx/HKlYwF+ApwBIOlIkgtKKj1i8FrgtHT6DGBVnhtr53eq/O9nnlcUVPtFctXEH0muAvtSlWL4AEm329PAsvQ1uYqfyQSqe/VXPdCYfh4/AQZUIYavAH8AngX+N+lVPhXY7nyS8zjvkPxoXgEcSvKwuVXp+yFVimM1yTnI1u/onErH0Gb5Gipz9Vexz6I38P30+7EUOKMKMXwAaCK5avVJ4MScYyj6O9WZ76eHaTEzs8x05+4vMzOrMCcVMzPLjJOKmZllxknFzMwy46RiZmaZcVIxy4Ck7emIsq2vzEYKkDSy2Gi+ZrUot8cJm+1j/hoR9dUOwqzafKRiliNJayR9Q9JT6es9afkISYvT55cslvQ3aflh6fNMfp++WoeQ6SnpjvRZFw9J2r9qO2XWDicVs2zs36b7a3rBsjcjYjzwbySjRZNOfy8ijicZvPHWtPxW4NGIeC/JuGjL0/LRwG0RcSzwOvCRXPfGrJN8R71ZBiRtjoh+RcrXkAzz8Xw6YN/LEXGopI3AkIh4Jy1fFxEDJW0AhkX6LI+0jZHAwxExOp2/HugVEf9SgV0zK4uPVMzyF3uY3tM6xWwtmN6Oz4dajXJSMcvf9IL3x9Pp/0cyYjTAx4H/m04vJnmWBpJ6pk/LNOsy/NeOWTb2l7SsYP6XEdF6WfF+kp4k+SPuorTss8A8SV8geRrmZWn554C5kq4gOSK5mmQEW7MuwedUzHKUnlNpiIhKP4/DrCrc/WVmZpnxkYqZmWXGRypmZpYZJxUzM8uMk4qZmWXGScXMzDLjpGJmZpn5/xxPz3kE7fURAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_loss_and_acc({'Sigmoid': [sigmoid_loss, sigmoid_acc],\n",
    "                   'relu': [relu_loss, relu_acc]})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ~~You have finished homework2-mlp, congratulations!~~  \n",
    "\n",
    "**Next, according to the requirements 4) of report:**\n",
    "### **You need to construct a two-hidden-layer MLP, using any activation function and loss function.**\n",
    "\n",
    "**Note: Please insert some new cells blow (using '+' bottom in the toolbar) refer to above codes. Do not modify the former code directly.**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.13 ('tf_m1')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "899be85650f4fd9fd1c61155ad1038b3bd3ce4f173c19491b42003eb0b61918c"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}