10-1_sklearn_traintestsplit.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn import linear_model, preprocessing, model_selection\n",
    "from sklearn.model_selection import train_test_split, cross_val_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![img](https://algotrading101.com/learn/wp-content/uploads/2020/06/training-validation-test-data-set.png)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Machine Learning, the primary way we select the best model is through the use of test sets and cross-validation.\n",
    "\n",
    "The idea of a test set works as follows:\n",
    "\n",
    "- Take all of the available data and split it into two parts - a training set and a test set.\n",
    "- Using only the training portion, you will estimate the parameters of several competing models.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# train_test_split\n",
    "\n",
    "This function takes a list of arrays and splits each array into two arrays (a training set and a test set) by randomly selecting rows or values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# x is our predictor matrix\n",
    "# y is a numeric output - for regression methods\n",
    "# z is a categorical output - for classification methods\n",
    "X = np.arange(20).reshape((2, -1)).T\n",
    "y = np.arange(10)\n",
    "z = np.array([0,0,0,0,0,1,1,1,1,1])\n",
    "print(X)\n",
    "print(y)\n",
    "print(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use train_test_split on each array individually.\n",
    "\n",
    "It returns a tuple that can be unpacked into train and test arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test = train_test_split(X, test_size = 1/4, random_state = 1)\n",
    "print(X_train)\n",
    "print(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train, y_test = train_test_split(y, test_size = 1/4, random_state = 1)\n",
    "print(y_train)\n",
    "print(y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also apply it to multiple arrays simultaneously."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 1/4, random_state = 1)\n",
    "print(X_train)\n",
    "print(X_test)\n",
    "print(y_train)\n",
    "print(y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## if you have a categorical variable, the stratify argument ensures \n",
    "# that you'll get an appropriate number of each category in the resulting split\n",
    "X_train, X_test, z_train, z_test = train_test_split(X, z, test_size = 0.4, random_state = 1, \n",
    "                                                    stratify = z)\n",
    "print(X_train)\n",
    "print(X_test)\n",
    "print(z_train)\n",
    "print(z_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Iron Slag example\n",
    "\n",
    "magnetic test is cheaper. chemical test is more accurate.\n",
    "\n",
    "Can we use the magnetic test to predict the chemical test result?\n",
    "\n",
    "X = magnetic test result\n",
    "y = chemical test\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iron = pd.read_csv('ironslag.csv')\n",
    "iron.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iron.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plot of the full dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.scatter(iron.magnetic, iron.chemical)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a hold-out set using train-test split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train, test = train_test_split(iron, test_size = 1/5, random_state = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plot of the training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.scatter(train.magnetic, train.chemical)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## We will use only the training data to try out possible models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# sklearn requires our predictor variables to be in a two dimensional array\n",
    "# reshape to have 1 column\n",
    "# the -1 in reshape means I don't want to figure out all the necessary dimensions\n",
    "# i want 1 column, and numpy, you figure out how many rows I need\n",
    "X = train.magnetic.values.reshape(-1,1)\n",
    "X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = train.chemical.values\n",
    "y.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.corrcoef(train.magnetic.values, train.chemical.values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# r-squared\n",
    "np.corrcoef(train.magnetic.values, train.chemical.values)[0,1] ** 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fit a linear model between x and y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "linear = linear_model.LinearRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "linear.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# linear.score is the R^2 value\n",
    "# how much error is reduced from no model (variance or MSE)\n",
    "# vs having the regression model\n",
    "linear.score(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_predict = np.arange(10, 40).reshape(-1,1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lin_y_hat = linear.predict(x_predict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.scatter(X, y)\n",
    "plt.plot(x_predict, lin_y_hat, c = 'red')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cross Validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# shuffle split says 'shuffle the data' and split it into 5 equal parts\n",
    "cv = model_selection.ShuffleSplit(n_splits = 5, test_size = 0.3, random_state=0)\n",
    "cv_linear = model_selection.cross_val_score(linear, X, y, cv = cv)\n",
    "print(cv_linear)\n",
    "print(np.mean(cv_linear))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Technically, the above is all you need to do. But I went ahead and wrote this loop which fits the model on the training data, and makes predictions for the validation data. \n",
    "\n",
    "In each plot, the light blue dots are the training data.\n",
    "\n",
    "The green dots are the validation data.\n",
    "\n",
    "The flat green line is the mean of the validation data. That would be the prediction if no model was fit.\n",
    "\n",
    "The red line is the linear model that was trained on the training data. We hope that the red line does a better job of predicting the green points than the green line. In some cases, it does not, and we actually get a negative cross-validation score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for train_index, test_index in cv.split(X):\n",
    "    # create a subset of the data using the training cases in cross validation\n",
    "    tX = X[train_index, : ]\n",
    "    ty = y[train_index]\n",
    "    \n",
    "    # initialize and fit a new linear regression model\n",
    "    clin = linear_model.LinearRegression()\n",
    "    \n",
    "    # fit only on the training cases\n",
    "    clin.fit(tX, ty)\n",
    "    \n",
    "    # create a subset for the test cases\n",
    "    testX = X[test_index, :]\n",
    "    testy = y[test_index]\n",
    "    \n",
    "    # plot the training data in blue and the fitted line in red\n",
    "    plt.scatter(tX, ty, c = 'blue', alpha = 0.5)\n",
    "    plt.plot(x_predict, clin.predict(x_predict), c = 'red')\n",
    "    plt.show()\n",
    "    \n",
    "    # plot the test cases in green against the fitted line\n",
    "    plt.scatter(testX, testy, c = 'green')\n",
    "    plt.plot(x_predict, clin.predict(x_predict), c = 'red')\n",
    "    plt.plot(x_predict, np.repeat(np.mean(testy), len(x_predict)), c = 'lightgreen')\n",
    "    \n",
    "    plt.show()\n",
    "    \n",
    "    # the MS of having no model = variance of the test data\n",
    "    mse = np.var(testy)\n",
    "    print(\"MS Error: \" + str(mse))\n",
    "    \n",
    "    # the MS regression\n",
    "    msr = sum((testy - clin.predict(testX))**2)/len(testy)\n",
    "    print(\"MS Regre: \" + str(msr))\n",
    "    \n",
    "    # the score is the proportion of reduction by having regression\n",
    "    red = (mse - msr)/mse\n",
    "    print(\"score: \" + str(red))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Polynomial fit - quadratic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# preprocessing polynomial features creates a polynomial based on X\n",
    "poly2 = preprocessing.PolynomialFeatures(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "polyX = poly2.fit_transform(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "poly2reg = linear_model.LinearRegression(fit_intercept = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "poly2reg.fit(polyX, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "poly2reg.score(polyX, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "poly2_X_new = poly2.fit_transform(x_predict)\n",
    "poly2_y_hat = poly2reg.predict(poly2_X_new)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.scatter(X, y)\n",
    "plt.plot(x_predict, poly2_y_hat, c = 'red')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cv_quad = model_selection.cross_val_score(poly2reg, polyX, y, cv=cv)\n",
    "print(cv_quad)\n",
    "print(np.mean(cv_quad))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def polycv(degree, X, y, train_index, test_index):\n",
    "    \n",
    "    # create a subset of the data using the training cases in cross validation\n",
    "    tX = X[train_index, : ]\n",
    "    ty = y[train_index]\n",
    "    \n",
    "    # create a subset for the test cases\n",
    "    testX = X[test_index, :]\n",
    "    testy = y[test_index]\n",
    "    \n",
    "    poly = preprocessing.PolynomialFeatures(degree)\n",
    "    polytX = poly.fit_transform(tX)\n",
    "    \n",
    "    # initialize and fite a new linear regression model\n",
    "    clin = linear_model.LinearRegression()\n",
    "    \n",
    "    # fit only on the training cases\n",
    "    clin.fit(polytX, ty)\n",
    "    \n",
    "    # plot the training data in blue and the prediction line in red\n",
    "    plt.scatter(tX,ty, c = 'blue', alpha = 0.5)\n",
    "    plt.plot(x_predict, clin.predict(poly.fit_transform(x_predict)), c = 'red')\n",
    "    plt.show()\n",
    "    \n",
    "    # plot the prediction line in red against the test cases in green\n",
    "    plt.scatter(testX, testy, c = 'green')\n",
    "    plt.plot(x_predict, clin.predict(poly.fit_transform(x_predict)), c = 'red')\n",
    "    \n",
    "    # plot the mean of the test cases to show what having no model looks like\n",
    "    plt.plot(x_predict, np.repeat(np.mean(testy), len(x_predict)), c = 'lightgreen')\n",
    "    \n",
    "    plt.show()\n",
    "    \n",
    "    # the MS of having no model = variance of the test data\n",
    "    mse = np.var(testy)\n",
    "    print(\"MS Error: \" + str(mse))\n",
    "    \n",
    "    # the MS regression\n",
    "    msr = sum((testy - clin.predict(poly.fit_transform(testX)))**2)/len(testy)\n",
    "    print(\"MS Regre: \" + str(msr))\n",
    "    \n",
    "    # the score is the proportion of reduction by having regression\n",
    "    red = (mse - msr)/mse\n",
    "    print(\"Score: \" + str(red))\n",
    "\n",
    "for train_index, test_index in cv.split(X):\n",
    "    polycv(2, X, y, train_index, test_index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# cubic fit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "poly3 = preprocessing.PolynomialFeatures(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "poly3X = poly3.fit_transform(X)\n",
    "poly3reg = linear_model.LinearRegression(fit_intercept = False)\n",
    "poly3reg.fit(poly3X, y)\n",
    "print(poly3reg.score(poly3X,y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The R^2 value of the cubic fit is better, but we will see with cross validation that it is not a better model. It is overfitting our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "poly3_X_new = poly3.fit_transform(x_predict)\n",
    "poly3_y_hat = poly3reg.predict(poly3_X_new)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.scatter(X,y)\n",
    "plt.plot(x_predict, poly3_y_hat, c = 'red')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cv_cube = model_selection.cross_val_score(poly3reg, poly3X, y, cv=cv)\n",
    "print(cv_cube)\n",
    "print(np.mean(cv_cube))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for train_index, test_index in cv.split(X):\n",
    "    polycv(3, X, y, train_index, test_index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## higher order polynomials overfit the data: degree 4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# degree4\n",
    "poly4 = preprocessing.PolynomialFeatures(4)\n",
    "poly4X = poly4.fit_transform(X)\n",
    "poly4reg = linear_model.LinearRegression(fit_intercept = False)\n",
    "\n",
    "cv_4th = model_selection.cross_val_score(poly4reg, poly4X, y, cv=cv)\n",
    "print(cv_4th)\n",
    "print(np.mean(cv_4th))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for train_index, test_index in cv.split(X):\n",
    "    polycv(4, X, y, train_index, test_index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## degree 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# degree 5\n",
    "poly5 = preprocessing.PolynomialFeatures(5)\n",
    "poly5X = poly5.fit_transform(X)\n",
    "poly5reg = linear_model.LinearRegression(fit_intercept = False)\n",
    "\n",
    "cv_5th = model_selection.cross_val_score(poly5reg, poly5X, y, cv=cv)\n",
    "print(cv_5th)\n",
    "print(np.mean(cv_5th))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for train_index, test_index in cv.split(X):\n",
    "    polycv(5, X, y, train_index, test_index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Assessing models without all the graphs:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "linear = linear_model.LinearRegression()\n",
    "linear.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# shuffle split says 'shuffle the data' and split it into 5 equal parts\n",
    "cv = model_selection.ShuffleSplit(n_splits = 5, test_size = 0.3, random_state=0)\n",
    "cv_linear = model_selection.cross_val_score(linear, X, y, cv = cv)\n",
    "print(cv_linear)\n",
    "print(np.mean(cv_linear))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "quad = preprocessing.PolynomialFeatures(2)\n",
    "quadX = quad.fit_transform(X)\n",
    "quad_model = linear_model.LinearRegression()\n",
    "quad_model.fit(quadX, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cv_quad = model_selection.cross_val_score(quad_model, quadX, y, cv = cv)\n",
    "print(cv_quad)\n",
    "print(np.mean(cv_quad))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cube = preprocessing.PolynomialFeatures(3)\n",
    "cubeX = cube.fit_transform(X)\n",
    "cube_model = linear_model.LinearRegression()\n",
    "cube_model.fit(cubeX, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cv_cube = model_selection.cross_val_score(cube_model, cubeX, y, cv = cv)\n",
    "print(cv_cube)\n",
    "print(np.mean(cv_cube))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Y ~ logX model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "log_transform = preprocessing.FunctionTransformer(np.log)\n",
    "logX = log_transform.fit_transform(X)\n",
    "logX_model = linear_model.LinearRegression()\n",
    "logX_model.fit(logX, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cv_logX = model_selection.cross_val_score(logX_model, logX, y, cv = cv)\n",
    "print(cv_logX)\n",
    "print(np.mean(cv_logX))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}