05-linear-model-and-neural-net-from-scratch.ipynb

{"cells":[{"cell_type":"markdown","metadata":{},"source":["## Introduction"]},{"cell_type":"markdown","metadata":{},"source":["In this notebook we're going to build and train a deep learning model \"from scratch\" -- by which I mean that we're not going to use any pre-built architecture, or optimizers, or data loading frameworks, etc.\n","\n","We'll be assuming you already know the basics of how a neural network works. If you don't, read this notebook first: [How does a neural net really work?\n","](https://www.kaggle.com/code/jhoward/how-does-a-neural-net-really-work). We'll be using Kaggle's [Titanic](https://www.kaggle.com/competitions/titanic/) competition in this notebook, because it's very small and simple, but also has displays many of the tricky real-life issues that we need to handle in most practical projects. (Note, however, that this competition is a small \"learner\" competition on Kaggle, so don't expect to actually see much benefits from using a neural net just yet; that will come once we try our some real competitions!)\n","\n","It's great to be able to run the same notebook on your own machine or Colab, as well as Kaggle. To allow for this, we use this code to download the data as needed when not on Kaggle (see [this notebook](https://www.kaggle.com/code/jhoward/getting-started-with-nlp-for-absolute-beginners/) for details about this technique):"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:34:17.763822Z","iopub.status.busy":"2022-05-30T22:34:17.763494Z","iopub.status.idle":"2022-05-30T22:34:17.771348Z","shell.execute_reply":"2022-05-30T22:34:17.770444Z","shell.execute_reply.started":"2022-05-30T22:34:17.763787Z"},"trusted":true},"outputs":[],"source":["import os\n","from pathlib import Path\n","\n","iskaggle = os.environ.get('KAGGLE_KERNEL_RUN_TYPE', '')\n","if iskaggle: path = Path('../input/titanic')\n","else:\n","    path = Path('titanic')\n","    if not path.exists():\n","        import zipfile,kaggle\n","        kaggle.api.competition_download_cli(str(path))\n","        zipfile.ZipFile(f'{path}.zip').extractall(path)"]},{"cell_type":"markdown","metadata":{"hidden":true},"source":["Note that the data for Kaggle comps always lives in the `../input` folder. The easiest way to get the path is to click the \"K\" button in the top-right of the Kaggle notebook, click on the folder shown there, and click the copy button.\n","\n","We'll be using *numpy* and *pytorch* for array calculations in this notebook, and *pandas* for working with tabular data, so we'll import them and set them to display using a bit more space than they default to."]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:34:17.811857Z","iopub.status.busy":"2022-05-30T22:34:17.810967Z","iopub.status.idle":"2022-05-30T22:34:17.817725Z","shell.execute_reply":"2022-05-30T22:34:17.816849Z","shell.execute_reply.started":"2022-05-30T22:34:17.811797Z"},"trusted":true},"outputs":[],"source":["import torch, numpy as np, pandas as pd\n","np.set_printoptions(linewidth=140)\n","torch.set_printoptions(linewidth=140, sci_mode=False, edgeitems=7)\n","pd.set_option('display.width', 140)"]},{"cell_type":"markdown","metadata":{"heading_collapsed":true},"source":["## Cleaning the data"]},{"cell_type":"markdown","metadata":{"hidden":true},"source":["This is a *tabular data* competition -- the data is in the form of a table. It's provided as a Comma Separated Values (CSV) file. We can open it using the *pandas* library, which will create a `DataFrame`."]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:34:17.899238Z","iopub.status.busy":"2022-05-30T22:34:17.898249Z","iopub.status.idle":"2022-05-30T22:34:17.932714Z","shell.execute_reply":"2022-05-30T22:34:17.931738Z","shell.execute_reply.started":"2022-05-30T22:34:17.899131Z"},"hidden":true,"scrolled":true,"trusted":true},"outputs":[],"source":["df = pd.read_csv(path/'train.csv')\n","df"]},{"cell_type":"markdown","metadata":{"hidden":true},"source":["As we learned in the *How does a neural net really work* notebook, we going to want to multiply each column by some coefficients. But we can see in the `Cabin` column that there are `NaN` values, which is how Pandas refers to missing values. We can't multiply something by a missing value!\n","\n","Let's check which columns contain `NaN` values. Pandas' `isna()` function returns `True` (which is treated as `1` when used as a number) for `NaN` values, so we can just add them up for each column:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:34:17.955557Z","iopub.status.busy":"2022-05-30T22:34:17.95524Z","iopub.status.idle":"2022-05-30T22:34:17.966199Z","shell.execute_reply":"2022-05-30T22:34:17.96534Z","shell.execute_reply.started":"2022-05-30T22:34:17.955525Z"},"hidden":true,"trusted":true},"outputs":[],"source":["df.isna().sum()"]},{"cell_type":"markdown","metadata":{"hidden":true},"source":["Notice that by default Pandas sums over columns.\n","\n","We'll need to replace the missing values with something. It doesn't generally matter too much what we choose. We'll use the most common value (the \"*mode*\"). We can use the `mode` function for that. One wrinkle is that it returns more than one row in the case of ties, so we just grab the first row with `iloc[0]`:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:34:18.012387Z","iopub.status.busy":"2022-05-30T22:34:18.011876Z","iopub.status.idle":"2022-05-30T22:34:18.030165Z","shell.execute_reply":"2022-05-30T22:34:18.029316Z","shell.execute_reply.started":"2022-05-30T22:34:18.012338Z"},"hidden":true,"trusted":true},"outputs":[],"source":["modes = df.mode().iloc[0]\n","modes"]},{"cell_type":"markdown","metadata":{"hidden":true},"source":["BTW, it's never a good idea to use functions without understanding them. So be sure to google for anything you're not familiar with. E.g if you want to learn about `iloc` (which is a very important function indeed!) then Google will give you a link to a [great tutorial](https://www.shanelynn.ie/pandas-iloc-loc-select-rows-and-columns-dataframe/).\n","\n","Now that we've got the mode of each column, we can use `fillna` to replace the missing values with the mode of each column. We'll do it \"in place\" -- meaning that we'll change the dataframe itself, rather than returning a new one."]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:34:18.069624Z","iopub.status.busy":"2022-05-30T22:34:18.069123Z","iopub.status.idle":"2022-05-30T22:34:18.078805Z","shell.execute_reply":"2022-05-30T22:34:18.077978Z","shell.execute_reply.started":"2022-05-30T22:34:18.069573Z"},"hidden":true,"trusted":true},"outputs":[],"source":["df.fillna(modes, inplace=True)"]},{"cell_type":"markdown","metadata":{"hidden":true},"source":["We can now check there's no missing values left:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:34:18.139844Z","iopub.status.busy":"2022-05-30T22:34:18.13936Z","iopub.status.idle":"2022-05-30T22:34:18.149896Z","shell.execute_reply":"2022-05-30T22:34:18.148884Z","shell.execute_reply.started":"2022-05-30T22:34:18.139793Z"},"hidden":true,"trusted":true},"outputs":[],"source":["df.isna().sum()"]},{"cell_type":"markdown","metadata":{"hidden":true},"source":["Here's how we get a quick summary of all the numeric columns in the dataset:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:34:18.205953Z","iopub.status.busy":"2022-05-30T22:34:18.205483Z","iopub.status.idle":"2022-05-30T22:34:18.240196Z","shell.execute_reply":"2022-05-30T22:34:18.239106Z","shell.execute_reply.started":"2022-05-30T22:34:18.205897Z"},"hidden":true,"trusted":true},"outputs":[],"source":["import numpy as np\n","\n","df.describe(include=(np.number))"]},{"cell_type":"markdown","metadata":{"execution":{"iopub.execute_input":"2022-05-13T11:02:34.328433Z","iopub.status.busy":"2022-05-13T11:02:34.327999Z","iopub.status.idle":"2022-05-13T11:02:34.336993Z","shell.execute_reply":"2022-05-13T11:02:34.335466Z","shell.execute_reply.started":"2022-05-13T11:02:34.32838Z"},"hidden":true},"source":["We can see that `Fare` contains mainly values of around `0` to `30`, but there's a few really big ones. This is very common with fields contain monetary values, and it can cause problems for our model, because once that column is multiplied by a coefficient later, the few rows with really big values will dominate the result.\n","\n","You can see the issue most clearly visually by looking at a histogram, which shows a long tail to the right (and don't forget: if you're not entirely sure what a histogram is, Google \"[histogram tutorial](https://www.google.com/search?q=histogram+tutorial&oq=histogram+tutorial)\" and do a bit of reading before continuing on):"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:34:18.242123Z","iopub.status.busy":"2022-05-30T22:34:18.241874Z","iopub.status.idle":"2022-05-30T22:34:18.471439Z","shell.execute_reply":"2022-05-30T22:34:18.470441Z","shell.execute_reply.started":"2022-05-30T22:34:18.242094Z"},"hidden":true,"trusted":true},"outputs":[],"source":["df['Fare'].hist();"]},{"cell_type":"markdown","metadata":{"execution":{"iopub.execute_input":"2022-05-13T11:02:34.328433Z","iopub.status.busy":"2022-05-13T11:02:34.327999Z","iopub.status.idle":"2022-05-13T11:02:34.336993Z","shell.execute_reply":"2022-05-13T11:02:34.335466Z","shell.execute_reply.started":"2022-05-13T11:02:34.32838Z"},"hidden":true},"source":["To fix this, the most common approach is to take the logarithm, which squishes the big numbers and makes the distribution more reasonable. Note, however, that there are zeros in the `Fare` column, and `log(0)` is infinite -- to fix this, we'll simply add `1` to all values first:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:34:18.47382Z","iopub.status.busy":"2022-05-30T22:34:18.473109Z","iopub.status.idle":"2022-05-30T22:34:18.478776Z","shell.execute_reply":"2022-05-30T22:34:18.478149Z","shell.execute_reply.started":"2022-05-30T22:34:18.473778Z"},"hidden":true,"trusted":true},"outputs":[],"source":["df['LogFare'] = np.log(df['Fare']+1)"]},{"cell_type":"markdown","metadata":{"hidden":true},"source":["The histogram now shows a more even distribution of values without the long tail:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:34:18.480926Z","iopub.status.busy":"2022-05-30T22:34:18.480088Z","iopub.status.idle":"2022-05-30T22:34:18.855605Z","shell.execute_reply":"2022-05-30T22:34:18.854915Z","shell.execute_reply.started":"2022-05-30T22:34:18.480885Z"},"hidden":true,"trusted":true},"outputs":[],"source":["df['LogFare'].hist();"]},{"cell_type":"markdown","metadata":{"hidden":true},"source":["It looks from the `describe()` output like `Pclass` contains just 3 values, which we can confirm by looking at the [Data Dictionary](https://www.kaggle.com/competitions/titanic/data) (which you should always study carefully for any project!) -- "]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:34:18.857676Z","iopub.status.busy":"2022-05-30T22:34:18.857312Z","iopub.status.idle":"2022-05-30T22:34:18.863672Z","shell.execute_reply":"2022-05-30T22:34:18.862791Z","shell.execute_reply.started":"2022-05-30T22:34:18.857643Z"},"hidden":true,"trusted":true},"outputs":[],"source":["pclasses = sorted(df.Pclass.unique())\n","pclasses"]},{"cell_type":"markdown","metadata":{"hidden":true},"source":["Here's how we get a quick summary of all the non-numeric columns in the dataset:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:34:18.865154Z","iopub.status.busy":"2022-05-30T22:34:18.864893Z","iopub.status.idle":"2022-05-30T22:34:18.89758Z","shell.execute_reply":"2022-05-30T22:34:18.896706Z","shell.execute_reply.started":"2022-05-30T22:34:18.865115Z"},"hidden":true,"trusted":true},"outputs":[],"source":["df.describe(include=[object])"]},{"cell_type":"markdown","metadata":{"hidden":true},"source":["Clearly we can't multiply strings like `male` or `S` by coefficients, so we need to replace those with numbers.\n","\n","We do that by creating new columns containing *dummy variables*. A dummy variable is a column that contains a `1` where a particular column contains a particular value, or a `0` otherwise. For instance, we could create a dummy variable for `Sex='male'`, which would be a new column containing `1` for rows where `Sex` is `'male'`, and 0 for rows where it isn't.\n","\n","Pandas can create these automatically using `get_dummies`, which also remove the original columns. We'll create dummy variables for `Pclass`, even although it's numeric, since the numbers `1`, `2`, and `3` correspond to first, second, and third class cabins - not to counts or measures that make sense to multiply by. We'll also create dummies for `Sex` and `Embarked` since we'll want to use those as predictors in our model. On the other hand, `Cabin`, `Name`, and `Ticket` have too many unique values for it to make sense creating dummy variables for them."]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:34:18.899369Z","iopub.status.busy":"2022-05-30T22:34:18.899127Z","iopub.status.idle":"2022-05-30T22:34:18.914993Z","shell.execute_reply":"2022-05-30T22:34:18.914167Z","shell.execute_reply.started":"2022-05-30T22:34:18.899338Z"},"hidden":true,"trusted":true},"outputs":[],"source":["df = pd.get_dummies(df, columns=[\"Sex\",\"Pclass\",\"Embarked\"])\n","df.columns"]},{"cell_type":"markdown","metadata":{"hidden":true},"source":["We can see that 5 columns have been added to the end -- one for each of the possible values of each of the three columns we requested, and that those three requested columns have been removed.\n","\n","Here's what the first few rows of those newly added columns look like:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:34:18.916456Z","iopub.status.busy":"2022-05-30T22:34:18.916186Z","iopub.status.idle":"2022-05-30T22:34:18.933746Z","shell.execute_reply":"2022-05-30T22:34:18.933135Z","shell.execute_reply.started":"2022-05-30T22:34:18.916426Z"},"hidden":true,"trusted":true},"outputs":[],"source":["added_cols = ['Sex_male', 'Sex_female', 'Pclass_1', 'Pclass_2', 'Pclass_3', 'Embarked_C', 'Embarked_Q', 'Embarked_S']\n","df[added_cols].head()"]},{"cell_type":"markdown","metadata":{"hidden":true},"source":["Now we can create our independent (predictors) and dependent (target) variables. They both need to be PyTorch tensors. Our dependent variable is `Survived`:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:34:18.935422Z","iopub.status.busy":"2022-05-30T22:34:18.934648Z","iopub.status.idle":"2022-05-30T22:34:18.944596Z","shell.execute_reply":"2022-05-30T22:34:18.943444Z","shell.execute_reply.started":"2022-05-30T22:34:18.935384Z"},"hidden":true,"trusted":true},"outputs":[],"source":["from torch import tensor\n","\n","t_dep = tensor(df.Survived)"]},{"cell_type":"markdown","metadata":{"hidden":true},"source":["Our independent variables are all the continuous variables of interest plus all the dummy variables we just created:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:34:18.946772Z","iopub.status.busy":"2022-05-30T22:34:18.94652Z","iopub.status.idle":"2022-05-30T22:34:18.96487Z","shell.execute_reply":"2022-05-30T22:34:18.963667Z","shell.execute_reply.started":"2022-05-30T22:34:18.946741Z"},"hidden":true,"trusted":true},"outputs":[],"source":["indep_cols = ['Age', 'SibSp', 'Parch', 'LogFare'] + added_cols\n","\n","t_indep = tensor(df[indep_cols].values, dtype=torch.float)\n","t_indep"]},{"cell_type":"markdown","metadata":{"hidden":true},"source":["Here's the number of rows and columns we have for our independent variables:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:34:18.969136Z","iopub.status.busy":"2022-05-30T22:34:18.968005Z","iopub.status.idle":"2022-05-30T22:34:18.98114Z","shell.execute_reply":"2022-05-30T22:34:18.980184Z","shell.execute_reply.started":"2022-05-30T22:34:18.969092Z"},"hidden":true,"trusted":true},"outputs":[],"source":["t_indep.shape"]},{"cell_type":"markdown","metadata":{},"source":["## Setting up a linear model"]},{"cell_type":"markdown","metadata":{},"source":["Now that we've got a matrix of independent variables and a dependent variable vector, we can work on calculating our predictions and our loss. In this section, we're going to manually do a single step of calculating predictions and loss for every row of our data.\n","\n","Our first model will be a simple linear model. We'll need a coefficient for each column in `t_indep`. We'll pick random numbers in the range `(-0.5,0.5)`, and set our manual seed so that my explanations in the prose in this notebook will be consistent with what you see when you run it."]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:34:18.983237Z","iopub.status.busy":"2022-05-30T22:34:18.982492Z","iopub.status.idle":"2022-05-30T22:34:18.995038Z","shell.execute_reply":"2022-05-30T22:34:18.994437Z","shell.execute_reply.started":"2022-05-30T22:34:18.983187Z"},"trusted":true},"outputs":[],"source":["torch.manual_seed(442)\n","\n","n_coeff = t_indep.shape[1]\n","coeffs = torch.rand(n_coeff)-0.5\n","coeffs"]},{"cell_type":"markdown","metadata":{},"source":["Our predictions will be calculated by multiplying each row by the coefficients, and adding them up. One interesting point here is that we don't need a separate constant term (also known as a \"bias\" or \"intercept\" term), or a column of all `1`s to give the same effect has having a constant term. That's because our dummy variables already cover the entire dataset -- e.g. there's a column for \"male\" and a column for \"female\", and everyone in the dataset is in exactly one of these; therefore, we don't need a separate intercept term to cover rows that aren't otherwise part of a column.\n","\n","Here's what the multiplication looks like:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:34:19.000637Z","iopub.status.busy":"2022-05-30T22:34:18.999409Z","iopub.status.idle":"2022-05-30T22:34:19.009362Z","shell.execute_reply":"2022-05-30T22:34:19.00847Z","shell.execute_reply.started":"2022-05-30T22:34:19.000588Z"},"trusted":true},"outputs":[],"source":["t_indep*coeffs"]},{"cell_type":"markdown","metadata":{},"source":["We can see we've got a problem here. The sums of each row will be dominated by the first column, which is `Age`, since that's bigger on average than all the others.\n","\n","Let's make all the columns contain numbers from `0` to `1`, by dividing each column by its `max()`:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:34:19.011202Z","iopub.status.busy":"2022-05-30T22:34:19.010954Z","iopub.status.idle":"2022-05-30T22:34:19.02202Z","shell.execute_reply":"2022-05-30T22:34:19.02133Z","shell.execute_reply.started":"2022-05-30T22:34:19.011171Z"},"trusted":true},"outputs":[],"source":["vals,indices = t_indep.max(dim=0)\n","t_indep = t_indep / vals"]},{"cell_type":"markdown","metadata":{},"source":["As we see, that removes the problem of one column dominating all the others:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:34:19.04269Z","iopub.status.busy":"2022-05-30T22:34:19.042223Z","iopub.status.idle":"2022-05-30T22:34:19.050475Z","shell.execute_reply":"2022-05-30T22:34:19.049515Z","shell.execute_reply.started":"2022-05-30T22:34:19.042652Z"},"trusted":true},"outputs":[],"source":["t_indep*coeffs"]},{"cell_type":"markdown","metadata":{},"source":["One thing you hopefully noticed is how amazingly cool this line of code is:\n","\n","    t_indep = t_indep / vals\n","\n","That is dividing a matrix by a vector -- what on earth does that mean?!? The trick here is that we're taking advantage of a technique in numpy and PyTorch (and many other languages, going all the way back to APL) called [broadcasting](https://numpy.org/doc/stable/user/basics.broadcasting.html). In short, this acts as if there's a separate copy of the vector for every row of the matrix, so it divides each row of the matrix by the vector. In practice, it doesn't actually make any copies, and does the whole thing in a highly optimized way, taking full advantage of modern CPUs (or, indeed, GPUs, if we're using them). Broadcasting is one of the most important techniques for making your code concise, maintainable, and fast, so it's well worth studying and practicing.\n","\n","We can now create predictions from our linear model, by adding up the rows of the product:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:34:19.097604Z","iopub.status.busy":"2022-05-30T22:34:19.096685Z","iopub.status.idle":"2022-05-30T22:34:19.1028Z","shell.execute_reply":"2022-05-30T22:34:19.10218Z","shell.execute_reply.started":"2022-05-30T22:34:19.097545Z"},"trusted":true},"outputs":[],"source":["preds = (t_indep*coeffs).sum(axis=1)"]},{"cell_type":"markdown","metadata":{},"source":["Let's take a look at the first few:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:34:19.160794Z","iopub.status.busy":"2022-05-30T22:34:19.160318Z","iopub.status.idle":"2022-05-30T22:34:19.168117Z","shell.execute_reply":"2022-05-30T22:34:19.167355Z","shell.execute_reply.started":"2022-05-30T22:34:19.160761Z"},"trusted":true},"outputs":[],"source":["preds[:10]"]},{"cell_type":"markdown","metadata":{},"source":["Of course, these predictions aren't going to be any use, since our coefficients are random -- they're just a starting point for our gradient descent process.\n","\n","To do gradient descent, we need a loss function. Taking the average error of the rows (i.e. the absolute value of the difference between the prediction and the dependent) is generally a reasonable approach:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:34:19.235623Z","iopub.status.busy":"2022-05-30T22:34:19.235318Z","iopub.status.idle":"2022-05-30T22:34:19.24312Z","shell.execute_reply":"2022-05-30T22:34:19.242356Z","shell.execute_reply.started":"2022-05-30T22:34:19.235589Z"},"trusted":true},"outputs":[],"source":["loss = torch.abs(preds-t_dep).mean()\n","loss"]},{"cell_type":"markdown","metadata":{},"source":["Now that we've tested out a way of calculating predictions, and loss, let's pop them into functions to make life easier:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:34:19.296558Z","iopub.status.busy":"2022-05-30T22:34:19.295577Z","iopub.status.idle":"2022-05-30T22:34:19.301478Z","shell.execute_reply":"2022-05-30T22:34:19.300683Z","shell.execute_reply.started":"2022-05-30T22:34:19.296517Z"},"trusted":true},"outputs":[],"source":["def calc_preds(coeffs, indeps): return (indeps*coeffs).sum(axis=1)\n","def calc_loss(coeffs, indeps, deps): return torch.abs(calc_preds(coeffs, indeps)-deps).mean()"]},{"cell_type":"markdown","metadata":{},"source":["## Doing a gradient descent step"]},{"cell_type":"markdown","metadata":{},"source":["In this section, we're going to do a single \"epoch\" of gradient descent manually. The only thing we're going to automate is calculating gradients, because let's face it that's pretty tedious and entirely pointless to do by hand! To get PyTorch to calculate gradients, we'll need to call `requires_grad_()` on our `coeffs` (if you're not sure why, review the previous notebook, [How does a neural net really work?](https://www.kaggle.com/code/jhoward/how-does-a-neural-net-really-work), before continuing):"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:34:19.376212Z","iopub.status.busy":"2022-05-30T22:34:19.375387Z","iopub.status.idle":"2022-05-30T22:34:19.382205Z","shell.execute_reply":"2022-05-30T22:34:19.381536Z","shell.execute_reply.started":"2022-05-30T22:34:19.376163Z"},"trusted":true},"outputs":[],"source":["coeffs.requires_grad_()"]},{"cell_type":"markdown","metadata":{},"source":["Now when we calculate our loss, PyTorch will keep track of all the steps, so we'll be able to get the gradients afterwards:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:34:19.438026Z","iopub.status.busy":"2022-05-30T22:34:19.437208Z","iopub.status.idle":"2022-05-30T22:34:19.444641Z","shell.execute_reply":"2022-05-30T22:34:19.443791Z","shell.execute_reply.started":"2022-05-30T22:34:19.437985Z"},"trusted":true},"outputs":[],"source":["loss = calc_loss(coeffs, t_indep, t_dep)\n","loss"]},{"cell_type":"markdown","metadata":{},"source":["Use `backward()` to ask PyTorch to calculate gradients now:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:34:19.49239Z","iopub.status.busy":"2022-05-30T22:34:19.491601Z","iopub.status.idle":"2022-05-30T22:34:19.496556Z","shell.execute_reply":"2022-05-30T22:34:19.495831Z","shell.execute_reply.started":"2022-05-30T22:34:19.492353Z"},"trusted":true},"outputs":[],"source":["loss.backward()"]},{"cell_type":"markdown","metadata":{},"source":["Let's see what they look like:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:34:19.550501Z","iopub.status.busy":"2022-05-30T22:34:19.549651Z","iopub.status.idle":"2022-05-30T22:34:19.555905Z","shell.execute_reply":"2022-05-30T22:34:19.555257Z","shell.execute_reply.started":"2022-05-30T22:34:19.550456Z"},"trusted":true},"outputs":[],"source":["coeffs.grad"]},{"cell_type":"markdown","metadata":{},"source":["Note that each time we call `backward`, the gradients are actually *added* to whatever is in the `.grad` attribute. Let's try running the above steps again:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:34:19.603236Z","iopub.status.busy":"2022-05-30T22:34:19.60283Z","iopub.status.idle":"2022-05-30T22:34:19.610594Z","shell.execute_reply":"2022-05-30T22:34:19.609647Z","shell.execute_reply.started":"2022-05-30T22:34:19.603206Z"},"trusted":true},"outputs":[],"source":["loss = calc_loss(coeffs, t_indep, t_dep)\n","loss.backward()\n","coeffs.grad"]},{"cell_type":"markdown","metadata":{},"source":["As you see, our `.grad` values are have doubled. That's because it added the gradients a second time. For this reason, after we use the gradients to do a gradient descent step, we need to set them back to zero.\n","\n","We can now do one gradient descent step, and check that our loss decreases:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:34:19.665739Z","iopub.status.busy":"2022-05-30T22:34:19.665238Z","iopub.status.idle":"2022-05-30T22:34:19.674839Z","shell.execute_reply":"2022-05-30T22:34:19.673727Z","shell.execute_reply.started":"2022-05-30T22:34:19.665705Z"},"trusted":true},"outputs":[],"source":["loss = calc_loss(coeffs, t_indep, t_dep)\n","loss.backward()\n","with torch.no_grad():\n","    coeffs.sub_(coeffs.grad * 0.1)\n","    coeffs.grad.zero_()\n","    print(calc_loss(coeffs, t_indep, t_dep))"]},{"cell_type":"markdown","metadata":{},"source":["Note that `a.sub_(b)` subtracts `b` from `a` in-place. In PyTorch, any method that ends in `_` changes its object in-place. Similarly, `a.zero_()` sets all elements of a tensor to zero."]},{"cell_type":"markdown","metadata":{},"source":["## Training the linear model"]},{"cell_type":"markdown","metadata":{},"source":["Before we begin training our model, we'll need to ensure that we hold out a validation set for calculating our metrics (for details on this, see \"[Getting started with NLP for absolute beginners](https://www.kaggle.com/code/jhoward/getting-started-with-nlp-for-absolute-beginners#Test-and-validation-sets)\".\n","\n","There's lots of different ways we can do this. In the next notebook we'll be comparing our approach here to what the fastai library does, so we'll want to ensure we split the data in the same way. So let's use `RandomSplitter` to get indices that will split our data into training and validation sets:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:35:26.57635Z","iopub.status.busy":"2022-05-30T22:35:26.57605Z","iopub.status.idle":"2022-05-30T22:35:27.821258Z","shell.execute_reply":"2022-05-30T22:35:27.820329Z","shell.execute_reply.started":"2022-05-30T22:35:26.57632Z"},"trusted":true},"outputs":[],"source":["from fastai.data.transforms import RandomSplitter\n","trn_split,val_split=RandomSplitter(seed=42)(df)"]},{"cell_type":"markdown","metadata":{},"source":["Now we can apply those indicies to our independent and dependent variables:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:35:27.822887Z","iopub.status.busy":"2022-05-30T22:35:27.822671Z","iopub.status.idle":"2022-05-30T22:35:27.836937Z","shell.execute_reply":"2022-05-30T22:35:27.836081Z","shell.execute_reply.started":"2022-05-30T22:35:27.822859Z"},"trusted":true},"outputs":[],"source":["trn_indep,val_indep = t_indep[trn_split],t_indep[val_split]\n","trn_dep,val_dep = t_dep[trn_split],t_dep[val_split]\n","len(trn_indep),len(val_indep)"]},{"cell_type":"markdown","metadata":{},"source":["We'll create functions for the three things we did manually above: updating `coeffs`, doing one full gradient descent step, and initilising `coeffs` to random numbers:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:35:35.029177Z","iopub.status.busy":"2022-05-30T22:35:35.028649Z","iopub.status.idle":"2022-05-30T22:35:35.034346Z","shell.execute_reply":"2022-05-30T22:35:35.033261Z","shell.execute_reply.started":"2022-05-30T22:35:35.029143Z"},"trusted":true},"outputs":[],"source":["def update_coeffs(coeffs, lr):\n","    coeffs.sub_(coeffs.grad * lr)\n","    coeffs.grad.zero_()"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:35:35.633995Z","iopub.status.busy":"2022-05-30T22:35:35.633703Z","iopub.status.idle":"2022-05-30T22:35:35.639103Z","shell.execute_reply":"2022-05-30T22:35:35.63814Z","shell.execute_reply.started":"2022-05-30T22:35:35.633964Z"},"trusted":true},"outputs":[],"source":["def one_epoch(coeffs, lr):\n","    loss = calc_loss(coeffs, trn_indep, trn_dep)\n","    loss.backward()\n","    with torch.no_grad(): update_coeffs(coeffs, lr)\n","    print(f\"{loss:.3f}\", end=\"; \")"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:35:36.293837Z","iopub.status.busy":"2022-05-30T22:35:36.293565Z","iopub.status.idle":"2022-05-30T22:35:36.297457Z","shell.execute_reply":"2022-05-30T22:35:36.296816Z","shell.execute_reply.started":"2022-05-30T22:35:36.293808Z"},"trusted":true},"outputs":[],"source":["def init_coeffs(): return (torch.rand(n_coeff)-0.5).requires_grad_()"]},{"cell_type":"markdown","metadata":{},"source":["We can now use these functions to train our model:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:35:38.772314Z","iopub.status.busy":"2022-05-30T22:35:38.771827Z","iopub.status.idle":"2022-05-30T22:35:38.777313Z","shell.execute_reply":"2022-05-30T22:35:38.776439Z","shell.execute_reply.started":"2022-05-30T22:35:38.772247Z"},"trusted":true},"outputs":[],"source":["def train_model(epochs=30, lr=0.01):\n","    torch.manual_seed(442)\n","    coeffs = init_coeffs()\n","    for i in range(epochs): one_epoch(coeffs, lr=lr)\n","    return coeffs"]},{"cell_type":"markdown","metadata":{},"source":["Let's try it. Our loss will print at the end of every step, so we hope we'll see it going down:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:36:14.16788Z","iopub.status.busy":"2022-05-30T22:36:14.16706Z","iopub.status.idle":"2022-05-30T22:36:14.181652Z","shell.execute_reply":"2022-05-30T22:36:14.180496Z","shell.execute_reply.started":"2022-05-30T22:36:14.167812Z"},"trusted":true},"outputs":[],"source":["coeffs = train_model(18, lr=0.2)"]},{"cell_type":"markdown","metadata":{},"source":["It does!\n","\n","Let's take a look at the coefficients for each column:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:36:21.980389Z","iopub.status.busy":"2022-05-30T22:36:21.97957Z","iopub.status.idle":"2022-05-30T22:36:21.990088Z","shell.execute_reply":"2022-05-30T22:36:21.989021Z","shell.execute_reply.started":"2022-05-30T22:36:21.98035Z"},"trusted":true},"outputs":[],"source":["def show_coeffs(): return dict(zip(indep_cols, coeffs.requires_grad_(False)))\n","show_coeffs()"]},{"cell_type":"markdown","metadata":{},"source":["## Measuring accuracy"]},{"cell_type":"markdown","metadata":{},"source":["The Kaggle competition is not, however, scored by absolute error (which is our loss function). It's scored by *accuracy* -- the proportion of rows where we correctly predict survival. Let's see how accurate we were on the validation set. First, calculate the predictions:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:36:30.826372Z","iopub.status.busy":"2022-05-30T22:36:30.825856Z","iopub.status.idle":"2022-05-30T22:36:30.831613Z","shell.execute_reply":"2022-05-30T22:36:30.830756Z","shell.execute_reply.started":"2022-05-30T22:36:30.826322Z"},"trusted":true},"outputs":[],"source":["preds = calc_preds(coeffs, val_indep)"]},{"cell_type":"markdown","metadata":{},"source":["We'll assume that any passenger with a score of over `0.5` is predicted to survive. So that means we're correct for each row where `preds>0.5` is the same as the dependent variable:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:36:33.618899Z","iopub.status.busy":"2022-05-30T22:36:33.618455Z","iopub.status.idle":"2022-05-30T22:36:33.62703Z","shell.execute_reply":"2022-05-30T22:36:33.625949Z","shell.execute_reply.started":"2022-05-30T22:36:33.618867Z"},"trusted":true},"outputs":[],"source":["results = val_dep.bool()==(preds>0.5)\n","results[:16]"]},{"cell_type":"markdown","metadata":{},"source":["Let's see what our average accuracy is:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:36:35.725637Z","iopub.status.busy":"2022-05-30T22:36:35.725112Z","iopub.status.idle":"2022-05-30T22:36:35.732969Z","shell.execute_reply":"2022-05-30T22:36:35.732241Z","shell.execute_reply.started":"2022-05-30T22:36:35.725599Z"},"trusted":true},"outputs":[],"source":["results.float().mean()"]},{"cell_type":"markdown","metadata":{},"source":["That's not a bad start at all! We'll create a function so we can calcuate the accuracy easy for other models we train:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:36:40.356505Z","iopub.status.busy":"2022-05-30T22:36:40.356043Z","iopub.status.idle":"2022-05-30T22:36:40.365187Z","shell.execute_reply":"2022-05-30T22:36:40.364153Z","shell.execute_reply.started":"2022-05-30T22:36:40.356471Z"},"trusted":true},"outputs":[],"source":["def acc(coeffs): return (val_dep.bool()==(calc_preds(coeffs, val_indep)>0.5)).float().mean()\n","acc(coeffs)"]},{"cell_type":"markdown","metadata":{},"source":["## Using sigmoid"]},{"cell_type":"markdown","metadata":{},"source":["Looking at our predictions, there's one obvious problem -- some of our predictions of the probability of survival are `>1`, and some are `<0`:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:36:42.344823Z","iopub.status.busy":"2022-05-30T22:36:42.344533Z","iopub.status.idle":"2022-05-30T22:36:42.352948Z","shell.execute_reply":"2022-05-30T22:36:42.351968Z","shell.execute_reply.started":"2022-05-30T22:36:42.344794Z"},"trusted":true},"outputs":[],"source":["preds[:28]"]},{"cell_type":"markdown","metadata":{},"source":["To fix this, we should pass every prediction through the *sigmoid function*, which has a minimum at zero and maximum at one, and is defined as follows:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:36:44.914015Z","iopub.status.busy":"2022-05-30T22:36:44.913101Z","iopub.status.idle":"2022-05-30T22:36:46.311818Z","shell.execute_reply":"2022-05-30T22:36:46.311008Z","shell.execute_reply.started":"2022-05-30T22:36:44.913968Z"},"trusted":true},"outputs":[],"source":["import sympy\n","sympy.plot(\"1/(1+exp(-x))\", xlim=(-5,5));"]},{"cell_type":"markdown","metadata":{},"source":["PyTorch already defines that function for us, so we can modify `calc_preds` to use it:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:36:46.313644Z","iopub.status.busy":"2022-05-30T22:36:46.313435Z","iopub.status.idle":"2022-05-30T22:36:46.317749Z","shell.execute_reply":"2022-05-30T22:36:46.3169Z","shell.execute_reply.started":"2022-05-30T22:36:46.313618Z"},"trusted":true},"outputs":[],"source":["def calc_preds(coeffs, indeps): return torch.sigmoid((indeps*coeffs).sum(axis=1))"]},{"cell_type":"markdown","metadata":{},"source":["Let's train a new model now, using this updated function to calculate predictions:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:38:23.22576Z","iopub.status.busy":"2022-05-30T22:38:23.225051Z","iopub.status.idle":"2022-05-30T22:38:23.250206Z","shell.execute_reply":"2022-05-30T22:38:23.249321Z","shell.execute_reply.started":"2022-05-30T22:38:23.225722Z"},"trusted":true},"outputs":[],"source":["coeffs = train_model(lr=100)"]},{"cell_type":"markdown","metadata":{},"source":["The loss has improved by a lot. Let's check the accuracy:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:38:28.519132Z","iopub.status.busy":"2022-05-30T22:38:28.518642Z","iopub.status.idle":"2022-05-30T22:38:28.527145Z","shell.execute_reply":"2022-05-30T22:38:28.526248Z","shell.execute_reply.started":"2022-05-30T22:38:28.519078Z"},"trusted":true},"outputs":[],"source":["acc(coeffs)"]},{"cell_type":"markdown","metadata":{},"source":["That's improved too! Here's the coefficients of our trained model:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:38:32.01697Z","iopub.status.busy":"2022-05-30T22:38:32.015953Z","iopub.status.idle":"2022-05-30T22:38:32.02724Z","shell.execute_reply":"2022-05-30T22:38:32.026125Z","shell.execute_reply.started":"2022-05-30T22:38:32.016924Z"},"trusted":true},"outputs":[],"source":["show_coeffs()"]},{"cell_type":"markdown","metadata":{},"source":["These coefficients seem reasonable -- in general, older people and males were less likely to survive, and first class passengers were more likely to survive."]},{"cell_type":"markdown","metadata":{},"source":["## Submitting to Kaggle"]},{"cell_type":"markdown","metadata":{},"source":["Now that we've got a trained model, we can prepare a submission to Kaggle. To do that, first we need to read the test set:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:38:35.172909Z","iopub.status.busy":"2022-05-30T22:38:35.172343Z","iopub.status.idle":"2022-05-30T22:38:35.188597Z","shell.execute_reply":"2022-05-30T22:38:35.187826Z","shell.execute_reply.started":"2022-05-30T22:38:35.172873Z"},"trusted":true},"outputs":[],"source":["tst_df = pd.read_csv(path/'test.csv')"]},{"cell_type":"markdown","metadata":{},"source":["In this case, it turns out that the test set is missing `Fare` for one passenger. We'll just fill it with `0` to avoid problems:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:38:36.353392Z","iopub.status.busy":"2022-05-30T22:38:36.352687Z","iopub.status.idle":"2022-05-30T22:38:36.358237Z","shell.execute_reply":"2022-05-30T22:38:36.35739Z","shell.execute_reply.started":"2022-05-30T22:38:36.353355Z"},"trusted":true},"outputs":[],"source":["tst_df['Fare'] = tst_df.Fare.fillna(0)"]},{"cell_type":"markdown","metadata":{},"source":["Now we can just copy the same steps we did to our training set and do the same exact things on our test set to preprocess the data:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:38:38.198549Z","iopub.status.busy":"2022-05-30T22:38:38.198257Z","iopub.status.idle":"2022-05-30T22:38:38.220592Z","shell.execute_reply":"2022-05-30T22:38:38.219629Z","shell.execute_reply.started":"2022-05-30T22:38:38.198519Z"},"trusted":true},"outputs":[],"source":["tst_df.fillna(modes, inplace=True)\n","tst_df['LogFare'] = np.log(tst_df['Fare']+1)\n","tst_df = pd.get_dummies(tst_df, columns=[\"Sex\",\"Pclass\",\"Embarked\"])\n","\n","tst_indep = tensor(tst_df[indep_cols].values, dtype=torch.float)\n","tst_indep = tst_indep / vals"]},{"cell_type":"markdown","metadata":{},"source":["Let's calculate our predictions of which passengers survived in the test set:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:38:39.206547Z","iopub.status.busy":"2022-05-30T22:38:39.206216Z","iopub.status.idle":"2022-05-30T22:38:39.212386Z","shell.execute_reply":"2022-05-30T22:38:39.211631Z","shell.execute_reply.started":"2022-05-30T22:38:39.206512Z"},"trusted":true},"outputs":[],"source":["tst_df['Survived'] = (calc_preds(tst_indep, coeffs)>0.5).int()"]},{"cell_type":"markdown","metadata":{},"source":["The sample submission on the Kaggle competition site shows that we're expected to upload a CSV with just `PassengerId` and `Survived`, so let's create that and save it:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:38:40.161382Z","iopub.status.busy":"2022-05-30T22:38:40.161034Z","iopub.status.idle":"2022-05-30T22:38:40.173242Z","shell.execute_reply":"2022-05-30T22:38:40.17258Z","shell.execute_reply.started":"2022-05-30T22:38:40.161336Z"},"trusted":true},"outputs":[],"source":["sub_df = tst_df[['PassengerId','Survived']]\n","sub_df.to_csv('sub.csv', index=False)"]},{"cell_type":"markdown","metadata":{},"source":["We can check the first few rows of the file to make sure it looks reasonable:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:38:42.869402Z","iopub.status.busy":"2022-05-30T22:38:42.86855Z","iopub.status.idle":"2022-05-30T22:38:43.638832Z","shell.execute_reply":"2022-05-30T22:38:43.637559Z","shell.execute_reply.started":"2022-05-30T22:38:42.869362Z"},"trusted":true},"outputs":[],"source":["!head sub.csv"]},{"cell_type":"markdown","metadata":{},"source":["When you click \"save version\" in Kaggle, and wait for the notebook to run, you'll see that `sub.csv` appears in the \"Data\" tab. Clicking on that file will show a *Submit* button, which allows you to submit to the competition."]},{"cell_type":"markdown","metadata":{},"source":["## Using matrix product"]},{"cell_type":"markdown","metadata":{},"source":["We can make things quite a bit neater...\n","\n","Take a look at the inner-most calculation we're doing to get the predictions:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:38:48.248084Z","iopub.status.busy":"2022-05-30T22:38:48.247768Z","iopub.status.idle":"2022-05-30T22:38:48.258935Z","shell.execute_reply":"2022-05-30T22:38:48.258184Z","shell.execute_reply.started":"2022-05-30T22:38:48.248052Z"},"trusted":true},"outputs":[],"source":["(val_indep*coeffs).sum(axis=1)"]},{"cell_type":"markdown","metadata":{},"source":["Multiplying elements together and then adding across rows is identical to doing a matrix-vector product! Python uses the `@` operator to indicate matrix products, and is supported by PyTorch tensors. Therefore, we can replicate the above calculate more simply like so:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:38:51.959798Z","iopub.status.busy":"2022-05-30T22:38:51.959362Z","iopub.status.idle":"2022-05-30T22:38:51.97614Z","shell.execute_reply":"2022-05-30T22:38:51.975461Z","shell.execute_reply.started":"2022-05-30T22:38:51.959765Z"},"trusted":true},"outputs":[],"source":["val_indep@coeffs"]},{"cell_type":"markdown","metadata":{},"source":["It also turns out that this is much faster, because matrix products in PyTorch are very highly optimised.\n","\n","Let's use this to replace how `calc_preds` works:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:38:56.322255Z","iopub.status.busy":"2022-05-30T22:38:56.321807Z","iopub.status.idle":"2022-05-30T22:38:56.326812Z","shell.execute_reply":"2022-05-30T22:38:56.32606Z","shell.execute_reply.started":"2022-05-30T22:38:56.322213Z"},"trusted":true},"outputs":[],"source":["def calc_preds(coeffs, indeps): return torch.sigmoid(indeps@coeffs)"]},{"cell_type":"markdown","metadata":{},"source":["In order to do matrix-matrix products (which we'll need in the next section), we need to turn `coeffs` into a column vector (i.e. a matrix with a single column), which we can do by passing a second argument `1` to `torch.rand()`, indicating that we want our coefficients to have one column:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:38:58.895779Z","iopub.status.busy":"2022-05-30T22:38:58.895467Z","iopub.status.idle":"2022-05-30T22:38:58.900851Z","shell.execute_reply":"2022-05-30T22:38:58.899931Z","shell.execute_reply.started":"2022-05-30T22:38:58.895744Z"},"trusted":true},"outputs":[],"source":["def init_coeffs(): return (torch.rand(n_coeff, 1)*0.1).requires_grad_()"]},{"cell_type":"markdown","metadata":{},"source":["We'll also need to turn our dependent variable into a column vector, which we can do by indexing the column dimension with the special value `None`, which tells PyTorch to add a new dimension in this position:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:38:59.789678Z","iopub.status.busy":"2022-05-30T22:38:59.788799Z","iopub.status.idle":"2022-05-30T22:38:59.794227Z","shell.execute_reply":"2022-05-30T22:38:59.793326Z","shell.execute_reply.started":"2022-05-30T22:38:59.789625Z"},"trusted":true},"outputs":[],"source":["trn_dep = trn_dep[:,None]\n","val_dep = val_dep[:,None]"]},{"cell_type":"markdown","metadata":{},"source":["We can now train our model as before and confirm we get identical outputs...:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:39:38.071003Z","iopub.status.busy":"2022-05-30T22:39:38.070545Z","iopub.status.idle":"2022-05-30T22:39:38.094666Z","shell.execute_reply":"2022-05-30T22:39:38.093641Z","shell.execute_reply.started":"2022-05-30T22:39:38.070972Z"},"trusted":true},"outputs":[],"source":["coeffs = train_model(lr=100)"]},{"cell_type":"markdown","metadata":{},"source":["...and identical accuracy:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:39:40.463735Z","iopub.status.busy":"2022-05-30T22:39:40.463301Z","iopub.status.idle":"2022-05-30T22:39:40.469684Z","shell.execute_reply":"2022-05-30T22:39:40.468652Z","shell.execute_reply.started":"2022-05-30T22:39:40.463702Z"},"trusted":true},"outputs":[],"source":["acc(coeffs)"]},{"cell_type":"markdown","metadata":{},"source":["## A neural network"]},{"cell_type":"markdown","metadata":{},"source":["We've now got what we need to implement our neural network.\n","\n","First, we'll need to create coefficients for each of our layers. Our first set of coefficients will take our `n_coeff` inputs, and create `n_hidden` outputs. We can choose whatever `n_hidden` we like -- a higher number gives our network more flexibility, but makes it slower and harder to train. So we need a matrix of size `n_coeff` by `n_hidden`. We'll divide these coefficients by `n_hidden` so that when we sum them up in the next layer we'll end up with similar magnitude numbers to what we started with.\n","\n","Then our second layer will need to take the `n_hidden` inputs and create a single output, so that means we need a `n_hidden` by `1` matrix there. The second layer will also need a constant term added."]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:39:44.428599Z","iopub.status.busy":"2022-05-30T22:39:44.428254Z","iopub.status.idle":"2022-05-30T22:39:44.434009Z","shell.execute_reply":"2022-05-30T22:39:44.433164Z","shell.execute_reply.started":"2022-05-30T22:39:44.428563Z"},"trusted":true},"outputs":[],"source":["def init_coeffs(n_hidden=20):\n","    layer1 = (torch.rand(n_coeff, n_hidden)-0.5)/n_hidden\n","    layer2 = torch.rand(n_hidden, 1)-0.3\n","    const = torch.rand(1)[0]\n","    return layer1.requires_grad_(),layer2.requires_grad_(),const.requires_grad_()"]},{"cell_type":"markdown","metadata":{},"source":["Now we have our coefficients, we can create our neural net. The key steps are the two matrix products, `indeps@l1` and `res@l2` (where `res` is the output of the first layer). The first layer output is passed to `F.relu` (that's our non-linearity), and the second is passed to `torch.sigmoid` as before."]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:39:45.302903Z","iopub.status.busy":"2022-05-30T22:39:45.302573Z","iopub.status.idle":"2022-05-30T22:39:45.309472Z","shell.execute_reply":"2022-05-30T22:39:45.308498Z","shell.execute_reply.started":"2022-05-30T22:39:45.302864Z"},"trusted":true},"outputs":[],"source":["import torch.nn.functional as F\n","\n","def calc_preds(coeffs, indeps):\n","    l1,l2,const = coeffs\n","    res = F.relu(indeps@l1)\n","    res = res@l2 + const\n","    return torch.sigmoid(res)"]},{"cell_type":"markdown","metadata":{},"source":["Finally, now that we have more than one set of coefficients, we need to add a loop to update each one:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:39:55.366945Z","iopub.status.busy":"2022-05-30T22:39:55.3665Z","iopub.status.idle":"2022-05-30T22:39:55.371578Z","shell.execute_reply":"2022-05-30T22:39:55.370699Z","shell.execute_reply.started":"2022-05-30T22:39:55.366914Z"},"trusted":true},"outputs":[],"source":["def update_coeffs(coeffs, lr):\n","    for layer in coeffs:\n","        layer.sub_(layer.grad * lr)\n","        layer.grad.zero_()"]},{"cell_type":"markdown","metadata":{},"source":["That's it -- we're now ready to train our model!"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:39:58.189982Z","iopub.status.busy":"2022-05-30T22:39:58.189651Z","iopub.status.idle":"2022-05-30T22:39:58.227202Z","shell.execute_reply":"2022-05-30T22:39:58.226226Z","shell.execute_reply.started":"2022-05-30T22:39:58.189951Z"},"trusted":true},"outputs":[],"source":["coeffs = train_model(lr=1.4)"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:40:16.338016Z","iopub.status.busy":"2022-05-30T22:40:16.337512Z","iopub.status.idle":"2022-05-30T22:40:16.368327Z","shell.execute_reply":"2022-05-30T22:40:16.367439Z","shell.execute_reply.started":"2022-05-30T22:40:16.337959Z"},"trusted":true},"outputs":[],"source":["coeffs = train_model(lr=20)"]},{"cell_type":"markdown","metadata":{},"source":["It's looking good -- our loss is lower than before. Let's see if that translates to a better result on the validation set:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:40:18.644458Z","iopub.status.busy":"2022-05-30T22:40:18.644153Z","iopub.status.idle":"2022-05-30T22:40:18.651372Z","shell.execute_reply":"2022-05-30T22:40:18.650102Z","shell.execute_reply.started":"2022-05-30T22:40:18.644427Z"},"trusted":true},"outputs":[],"source":["acc(coeffs)"]},{"cell_type":"markdown","metadata":{},"source":["In this case our neural net isn't showing better results than the linear model. That's not surprising; this dataset is very small and very simple, and isn't the kind of thing we'd expect to see neural networks excel at. Furthermore, our validation set is too small to reliably see much accuracy difference. But the key thing is that we now know exactly what a real neural net looks like!"]},{"cell_type":"markdown","metadata":{},"source":["## Deep learning"]},{"cell_type":"markdown","metadata":{},"source":["The neural net in the previous section only uses one hidden layer, so it doesn't count as \"deep\" learning. But we can use the exact same technique to make our neural net deep, by adding more matrix multiplications.\n","\n","First, we'll need to create additional coefficients for each layer:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:40:55.255291Z","iopub.status.busy":"2022-05-30T22:40:55.25457Z","iopub.status.idle":"2022-05-30T22:40:55.261806Z","shell.execute_reply":"2022-05-30T22:40:55.261271Z","shell.execute_reply.started":"2022-05-30T22:40:55.255242Z"},"trusted":true},"outputs":[],"source":["def init_coeffs():\n","    hiddens = [10, 10]  # <-- set this to the size of each hidden layer you want\n","    sizes = [n_coeff] + hiddens + [1]\n","    n = len(sizes)\n","    layers = [(torch.rand(sizes[i], sizes[i+1])-0.3)/sizes[i+1]*4 for i in range(n-1)]\n","    consts = [(torch.rand(1)[0]-0.5)*0.1 for i in range(n-1)]\n","    for l in layers+consts: l.requires_grad_()\n","    return layers,consts"]},{"cell_type":"markdown","metadata":{},"source":["You'll notice here that there's a lot of messy constants to get the random numbers in just the right ranges. When you train the model in a moment, you'll see that the tiniest changes to these initialisations can cause our model to fail to train at all! This is a key reason that deep learning failed to make much progress in the early days -- it's very finicky to get a good starting point for our coefficients. Nowadays, we have ways to deal with that, which we'll learn about in other notebooks.\n","\n","Our deep learning `calc_preds` looks much the same as before, but now we loop through each layer, instead of listing them separately:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:40:57.610974Z","iopub.status.busy":"2022-05-30T22:40:57.610142Z","iopub.status.idle":"2022-05-30T22:40:57.618154Z","shell.execute_reply":"2022-05-30T22:40:57.617329Z","shell.execute_reply.started":"2022-05-30T22:40:57.610916Z"},"trusted":true},"outputs":[],"source":["import torch.nn.functional as F\n","\n","def calc_preds(coeffs, indeps):\n","    layers,consts = coeffs\n","    n = len(layers)\n","    res = indeps\n","    for i,l in enumerate(layers):\n","        res = res@l + consts[i]\n","        if i!=n-1: res = F.relu(res)\n","    return torch.sigmoid(res)"]},{"cell_type":"markdown","metadata":{},"source":["We also need a minor update to `update_coeffs` since we've got `layers` and `consts` separated now:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:41:08.513049Z","iopub.status.busy":"2022-05-30T22:41:08.512494Z","iopub.status.idle":"2022-05-30T22:41:08.519219Z","shell.execute_reply":"2022-05-30T22:41:08.518093Z","shell.execute_reply.started":"2022-05-30T22:41:08.512999Z"},"trusted":true},"outputs":[],"source":["def update_coeffs(coeffs, lr):\n","    layers,consts = coeffs\n","    for layer in layers+consts:\n","        layer.sub_(layer.grad * lr)\n","        layer.grad.zero_()"]},{"cell_type":"markdown","metadata":{},"source":["Let's train our model..."]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:41:23.633004Z","iopub.status.busy":"2022-05-30T22:41:23.632516Z","iopub.status.idle":"2022-05-30T22:41:23.666981Z","shell.execute_reply":"2022-05-30T22:41:23.666048Z","shell.execute_reply.started":"2022-05-30T22:41:23.632953Z"},"trusted":true},"outputs":[],"source":["coeffs = train_model(lr=4)"]},{"cell_type":"markdown","metadata":{},"source":["...and check its accuracy:"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-05-30T22:41:25.491182Z","iopub.status.busy":"2022-05-30T22:41:25.490656Z","iopub.status.idle":"2022-05-30T22:41:25.497888Z","shell.execute_reply":"2022-05-30T22:41:25.49695Z","shell.execute_reply.started":"2022-05-30T22:41:25.491146Z"},"trusted":true},"outputs":[],"source":["acc(coeffs)"]},{"cell_type":"markdown","metadata":{},"source":["## Final thoughts"]},{"cell_type":"markdown","metadata":{},"source":["It's actually pretty cool that we've managed to create a real deep learning model from scratch and trained it to get over 80% accuracy on this task, all in the course of a single notebook!\n","\n","The \"real\" deep learning models that are used in research and industry look very similar to this, and in fact if you look inside the source code of any deep learning model you'll recognise the basic steps are the same.\n","\n","The biggest differences in practical models to what we have above are:\n","\n","- How initialisation and normalisation is done to ensure the model trains correctly every time\n","- Regularization (to avoid over-fitting)\n","- Modifying the neural net itself to take advantage of knowledge of the problem domain\n","- Doing gradient descent steps on smaller batches, rather than the whole dataset.\n","\n","I'll be adding notebooks about all these later, and will add links here once they're ready.\n","\n","If you found this notebook useful, please remember to click the little up-arrow at the top to upvote it, since I like to know when people have found my work useful, and it helps others find it too. (BTW, be sure you're looking at my [original notebook here](https://www.kaggle.com/code/jhoward/linear-model-and-neural-net-from-scratch) when you do that, and are not on your own copy of it, otherwise your upvote won't get counted!) And if you have any questions or comments, please pop them below -- I read every comment I receive!"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.4"}},"nbformat":4,"nbformat_minor":4}