examples/notebooks/scala/mortgage-gpu.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to XGBoost Spark with GPU\n",
    "\n",
    "Mortgage is an example of xgboost classifier. In this notebook, we will show you how to load data, train the xgboost model and use this model to predict if someone is \"deliquency\". Comparing to original XGBoost Spark codes, there're only two API differences.\n",
    "\n",
    "\n",
    "## Load libraries\n",
    "First we load some common libraries that both GPU version and CPU version xgboost will use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ml.dmlc.xgboost4j.scala.spark.{XGBoostClassifier, XGBoostClassificationModel}\n",
    "import org.apache.spark.sql.SparkSession\n",
    "import org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator\n",
    "import org.apache.spark.sql.types.{DoubleType, IntegerType, StructField, StructType}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "what is new to xgboost-spark users is only `rapids.GpuDataReader`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ml.dmlc.xgboost4j.scala.spark.rapids.{GpuDataReader, GpuDataset}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some libraries needed for CPU version are not needed in GPU version any more. The extra libraries needed for CPU are like below:\n",
    "\n",
    "```scala\n",
    "import org.apache.spark.ml.feature.VectorAssembler\n",
    "import org.apache.spark.sql.DataFrame\n",
    "import org.apache.spark.sql.functions._\n",
    "import org.apache.spark.sql.types.FloatType\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set your dataset path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "trainPath = /data/mortgage/csv/train/\n",
       "trainWithEvalPath = /data/mortgage/csv/trainWithEval/\n",
       "evalPath = /data/mortgage/csv/eval/\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "/data/mortgage/csv/eval/"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "// Set the paths of datasets for training and prediction\n",
    "// You need to update them to your real paths!\n",
    "val trainPath = \"/data/mortgage/csv/train/\"\n",
    "val trainWithEvalPath = \"/data/mortgage/csv/trainWithEval/\"\n",
    "val evalPath  = \"/data/mortgage/csv/eval/\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set the schema of the dataset\n",
    "for mortgage example, the data has 27 columns: 26 features and 1 label. \"deinquency_12\" is set as label column. The schema will be used to help load data in the future. We also defined some key parameters used in xgboost training process. We also set some basic xgboost parameters here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "labelColName = delinquency_12\n",
       "schema = StructType(StructField(orig_channel,DoubleType,true), StructField(first_home_buyer,DoubleType,true), StructField(loan_purpose,DoubleType,true), StructField(property_type,DoubleType,true), StructField(occupancy_status,DoubleType,true), StructField(property_state,DoubleType,true), StructField(product_type,DoubleType,true), StructField(relocation_mortgage_indicator,DoubleType,true), StructField(seller_name,DoubleType,true), StructField(mod_flag,DoubleType,true), StructField(orig_interest_rate,DoubleType,true), StructField(orig_upb,IntegerType,true), StructField(orig_loan_term,IntegerType,true), StructField(orig_ltv,DoubleType,true), StructField(orig_cltv,DoubleType,true), StructField(num_borrowers,DoubleT...\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "StructType(StructField(orig_channel,DoubleType,true), StructField(first_home_buyer,DoubleType,true), StructField(loan_purpose,DoubleType,true), StructField(property_type,DoubleType,true), StructField(occupancy_status,DoubleType,true), StructField(property_state,DoubleType,true), StructField(product_type,DoubleType,true), StructField(relocation_mortgage_indicator,DoubleType,true), StructField(seller_name,DoubleType,true), StructField(mod_flag,DoubleType,true), StructField(orig_interest_rate,DoubleType,true), StructField(orig_upb,IntegerType,true), StructField(orig_loan_term,IntegerType,true), StructField(orig_ltv,DoubleType,true), StructField(orig_cltv,DoubleType,true), StructField(num_borrowers,DoubleT..."
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val labelColName = \"delinquency_12\"\n",
    "val schema = StructType(List(\n",
    "  StructField(\"orig_channel\", DoubleType),\n",
    "  StructField(\"first_home_buyer\", DoubleType),\n",
    "  StructField(\"loan_purpose\", DoubleType),\n",
    "  StructField(\"property_type\", DoubleType),\n",
    "  StructField(\"occupancy_status\", DoubleType),\n",
    "  StructField(\"property_state\", DoubleType),\n",
    "  StructField(\"product_type\", DoubleType),\n",
    "  StructField(\"relocation_mortgage_indicator\", DoubleType),\n",
    "  StructField(\"seller_name\", DoubleType),\n",
    "  StructField(\"mod_flag\", DoubleType),\n",
    "  StructField(\"orig_interest_rate\", DoubleType),\n",
    "  StructField(\"orig_upb\", IntegerType),\n",
    "  StructField(\"orig_loan_term\", IntegerType),\n",
    "  StructField(\"orig_ltv\", DoubleType),\n",
    "  StructField(\"orig_cltv\", DoubleType),\n",
    "  StructField(\"num_borrowers\", DoubleType),\n",
    "  StructField(\"dti\", DoubleType),\n",
    "  StructField(\"borrower_credit_score\", DoubleType),\n",
    "  StructField(\"num_units\", IntegerType),\n",
    "  StructField(\"zip\", IntegerType),\n",
    "  StructField(\"mortgage_insurance_percent\", DoubleType),\n",
    "  StructField(\"current_loan_delinquency_status\", IntegerType),\n",
    "  StructField(\"current_actual_upb\", DoubleType),\n",
    "  StructField(\"interest_rate\", DoubleType),\n",
    "  StructField(\"loan_age\", DoubleType),\n",
    "  StructField(\"msa\", DoubleType),\n",
    "  StructField(\"non_interest_bearing_upb\", DoubleType),\n",
    "  StructField(labelColName, IntegerType)))\n",
    "\n",
    "val commParamMap = Map(\n",
    "  \"eta\" -> 0.1,\n",
    "  \"gamma\" -> 0.1,\n",
    "  \"missing\" -> 0.0,\n",
    "  \"max_depth\" -> 10,\n",
    "  \"max_leaves\" -> 256,\n",
    "  \"grow_policy\" -> \"depthwise\",\n",
    "  \"min_child_weight\" -> 30,\n",
    "  \"lambda\" -> 1,\n",
    "  \"scale_pos_weight\" -> 2,\n",
    "  \"subsample\" -> 1,\n",
    "  \"nthread\" -> 1,\n",
    "  \"num_round\" -> 100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a new spark session and load data\n",
    "we must create a new spark session to continue all spark operations. It will also be used to initilize the `GpuDataReader` which is a data reader powered by GPU.\n",
    "\n",
    "Here's the first API difference, we now use GpuDataReader to load dataset. Similar to original Spark data loading API, GpuDataReader also uses chaining call of \"option\", \"schema\",\"csv\". For CPU verions data reader, the code is like below:\n",
    "\n",
    "```scala\n",
    "val dataReader = spark.read\n",
    "```\n",
    "\n",
    "NOTE: in this notebook, we have uploaded dependency jars when installing toree kernel. If we don't upload them at installation time, we can also upload in notebook by [%AddJar magic](https://toree.incubator.apache.org/docs/current/user/faq/). However, there's one restriction for `%AddJar`: the jar uploaded can only be available when `AddJar` is called after a new spark session is created. We must use it as below:\n",
    "\n",
    "```scala\n",
    "import org.apache.spark.sql.SparkSession\n",
    "val spark = SparkSession.builder().appName(\"mortgage-GPU\").getOrCreate\n",
    "%AddJar file:/data/libs/cudf-0.9.2-cuda10.jar\n",
    "%AddJar file:/data/libs/xgboost4j_2.x-1.0.0-Beta3.jar\n",
    "%AddJar file:/data/libs/xgboost4j-spark_2.x-1.0.0-Beta3.jar\n",
    "// ...\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "spark = org.apache.spark.sql.SparkSession@6ec3b17c\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "org.apache.spark.sql.SparkSession@6ec3b17c"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val spark = SparkSession.builder().appName(\"mortgage-gpu\").getOrCreate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's the first API difference, we now use `GpuDataReader` to load dataset. Similar to original Spark data loading API, `GpuDataReader` also uses chaining call of \"option\", \"schema\",\"csv\". For `CPU` verion data reader, the code is like below:\n",
    "\n",
    "```scala\n",
    "val dataReader = spark.read\n",
    "```\n",
    "\n",
    "`featureNames` is used to tell xgboost which columns are `feature` and while column is `label`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "reader = ml.dmlc.xgboost4j.scala.spark.rapids.GpuDataReader@46845449\n",
       "featureNames = List(orig_channel, first_home_buyer, loan_purpose, property_type, occupancy_status, property_state, product_type, relocation_mortgage_indicator, seller_name, mod_flag, orig_interest_rate, orig_upb, orig_loan_term, orig_ltv, orig_cltv, num_borrowers, dti, borrower_credit_score, num_units, zip, mortgage_insurance_percent, current_loan_delinquency_status, current_actual_upb, interest_rate, loan_age, msa, non_interest_bearing_upb)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "List(orig_channel, first_home_buyer, loan_purpose, property_type, occupancy_status, property_state, product_type, relocation_mortgage_indicator, seller_name, mod_flag, orig_interest_rate, orig_upb, orig_loan_term, orig_ltv, orig_cltv, num_borrowers, dti, borrower_credit_score, num_units, zip, mortgage_insurance_percent, current_loan_delinquency_status, current_actual_upb, interest_rate, loan_age, msa, non_interest_bearing_upb)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val reader = new GpuDataReader(spark).option(\"header\", true).schema(schema)\n",
    "val featureNames = schema.filter(_.name != labelColName).map(_.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can use `dataReader` to read data directly. However, in CPU version, we have to use `VectorAssembler` to assemble all feature columns into one column. The reason will be explained later. the CPU version code is as below:\n",
    "\n",
    "```scala\n",
    "object Vectorize {\n",
    "  def apply(df: DataFrame, featureNames: Seq[String], labelName: String): DataFrame = {\n",
    "    val toFloat = df.schema.map(f => col(f.name).cast(FloatType))\n",
    "    new VectorAssembler()\n",
    "      .setInputCols(featureNames.toArray)\n",
    "      .setOutputCol(\"features\")\n",
    "      .transform(df.select(toFloat:_*))\n",
    "      .select(col(\"features\"), col(labelName))\n",
    "  }\n",
    "}\n",
    "\n",
    "val trainSet = reader.csv(trainPath)\n",
    "val evalSet = reader.csv(evalPath)\n",
    "trainSet = Vectorize(trainSet, featureCols, labelName)\n",
    "evalSet = Vectorize(evalSet, featureCols, labelName)\n",
    "\n",
    "```\n",
    "\n",
    "While with GpuDataReader, `VectorAssembler` is not needed any more. We can simply read data by:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "trainSet = ml.dmlc.xgboost4j.scala.spark.rapids.GpuDataset@3ca0708c\n",
       "trainWithEvalSet = ml.dmlc.xgboost4j.scala.spark.rapids.GpuDataset@234184fd\n",
       "evalSet = ml.dmlc.xgboost4j.scala.spark.rapids.GpuDataset@2dbe1638\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "ml.dmlc.xgboost4j.scala.spark.rapids.GpuDataset@2dbe1638"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val trainSet = reader.csv(trainPath)\n",
    "val trainWithEvalSet = reader.csv(trainWithEvalPath)\n",
    "val evalSet = reader.csv(evalPath)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add XGBoost parameters for GPU version\n",
    "\n",
    "Another modification is `num_workers` should be set to the number of machines with GPU in Spark cluster, while it can be set to the number of your CPU cores in CPU version. CPU version parameters:\n",
    "\n",
    "```scala\n",
    "// difference in parameters\n",
    "\"tree_method\" -> \"hist\", \n",
    "\"num_workers\" -> 12\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "xgbParamFinal = Map(min_child_weight -> 30, grow_policy -> depthwise, scale_pos_weight -> 2, num_workers -> 1, subsample -> 1, lambda -> 1, max_depth -> 10, num_round -> 100, missing -> 0.0, tree_method -> gpu_hist, eta -> 0.1, max_leaves -> 256, gamma -> 0.1, nthread -> 1)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "Map(min_child_weight -> 30, grow_policy -> depthwise, scale_pos_weight -> 2, num_workers -> 1, subsample -> 1, lambda -> 1, max_depth -> 10, num_round -> 100, missing -> 0.0, tree_method -> gpu_hist, eta -> 0.1, max_leaves -> 256, gamma -> 0.1, nthread -> 1)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val xgbParamFinal = commParamMap ++ Map(\"tree_method\" -> \"gpu_hist\", \"num_workers\" -> 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "xgbClassifier = xgbc_12a415c93943\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "xgbc_12a415c93943"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val xgbClassifier = new XGBoostClassifier(xgbParamFinal)\n",
    "      .setLabelCol(labelColName)\n",
    "      // === diff ===\n",
    "      .setFeaturesCols(featureNames)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Benchmark and train\n",
    "The benchmark object is for calculating training time. We will use it to compare with xgboost in CPU version.\n",
    "\n",
    "We also support training with evaluation sets in 2 ways as same as CPU version behavior:\n",
    "\n",
    "* API `setEvalSets` after initializing an XGBoostClassifier\n",
    "\n",
    "```scala\n",
    "xgbClassifier.setEvalSets(Map(\"eval\" -> evalSet))\n",
    "\n",
    "```\n",
    "\n",
    "* parameter `eval_sets` when initializing an XGBoostClassifier\n",
    "\n",
    "```scala\n",
    "val paramMapWithEval = paramMap + (\"eval_sets\" -> Map(\"eval\" -> evalSet))\n",
    "val xgbClassifierWithEval = new XGBoostClassifier(paramMapWithEval)\n",
    "```\n",
    "\n",
    "in this notebook, we use API method to set evaluation sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "xgbc_12a415c93943"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgbClassifier.setEvalSets(Map(\"eval\" -> trainWithEvalSet))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "------ Training ------\n",
      "Tracker started, with env={DMLC_NUM_SERVER=0, DMLC_TRACKER_URI=10.19.183.124, DMLC_TRACKER_PORT=9092, DMLC_NUM_WORKER=1}\n",
      "Elapsed time [train]: 14.013s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "defined object Benchmark\n",
       "xgbClassificationModel = xgbc_12a415c93943\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "xgbc_12a415c93943"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "object Benchmark {\n",
    "  def time[R](phase: String)(block: => R): (R, Float) = {\n",
    "    val t0 = System.currentTimeMillis\n",
    "    val result = block // call-by-name\n",
    "    val t1 = System.currentTimeMillis\n",
    "    println(\"Elapsed time [\" + phase + \"]: \" + ((t1 - t0).toFloat / 1000) + \"s\")\n",
    "    (result, (t1 - t0).toFloat / 1000)\n",
    "  }\n",
    "}\n",
    "\n",
    "// Start training\n",
    "println(\"\\n------ Training ------\")\n",
    "val (xgbClassificationModel, _) = Benchmark.time(\"train\") {\n",
    "  xgbClassifier.fit(trainSet)\n",
    "}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transformation and evaluation\n",
    "We use `evalSet` to evaluate our model and use some key columns to show our predictions. Finally we use `MulticlassClassificationEvaluator` to calculate an overall accuracy of our predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "------ Transforming ------\n",
      "Elapsed time [transform]: 5.483s\n",
      "+------------+--------------+--------------------+--------------------+----------+\n",
      "|orig_channel|delinquency_12|       rawPrediction|         probability|prediction|\n",
      "+------------+--------------+--------------------+--------------------+----------+\n",
      "|         0.0|           0.0|[-1.3482570648193...|[0.99986517429351...|       0.0|\n",
      "|         0.0|           0.0|[3.86238098144531...|[1.00003862380981...|       0.0|\n",
      "|         0.0|           0.0|[9.65595245361328...|[1.00000965595245...|       0.0|\n",
      "|         0.0|           0.0|[-1.1876225471496...|[0.99988123774528...|       0.0|\n",
      "|         0.0|           0.0|[-7.2091817855834...|[0.99992790818214...|       0.0|\n",
      "|         0.0|           0.0|[4.38451766967773...|[1.00043845176696...|       0.0|\n",
      "|         0.0|           0.0|[7.64369964599609...|[1.00076436996459...|       0.0|\n",
      "|         0.0|           0.0|[4.24027442932128...|[1.00042402744293...|       0.0|\n",
      "|         0.0|           0.0|[-5.7399272918701...|[0.99994260072708...|       0.0|\n",
      "|         0.0|           0.0|[-3.9398670196533...|[0.99996060132980...|       0.0|\n",
      "+------------+--------------+--------------------+--------------------+----------+\n",
      "only showing top 10 rows\n",
      "\n",
      "\n",
      "------Accuracy of Evaluation------\n",
      "0.9990008308279212\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "results = [orig_channel: float, first_home_buyer: float ... 29 more fields]\n",
       "evaluator = mcEval_037d0924816e\n",
       "accuracy = 0.9990008308279212\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "0.9990008308279212"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "println(\"\\n------ Transforming ------\")\n",
    "val (results, _) = Benchmark.time(\"transform\") {\n",
    "  val ret = xgbClassificationModel.transform(evalSet).cache()\n",
    "  ret.foreachPartition(_ => ())\n",
    "  ret\n",
    "}\n",
    "results.select(\"orig_channel\", labelColName,\"rawPrediction\",\"probability\",\"prediction\").show(10)\n",
    "\n",
    "println(\"\\n------Accuracy of Evaluation------\")\n",
    "val evaluator = new MulticlassClassificationEvaluator().setLabelCol(labelColName)\n",
    "val accuracy = evaluator.evaluate(results)\n",
    "println(accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save the model to disk and load model\n",
    "We save the model to disk and then load it to memory. We can use the loaded model to do a new prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Elapsed time [transform2]: 0.04s\n",
      "+------------+----------------+------------+-------------+----------------+--------------+------------+-----------------------------+-----------+--------+------------------+--------+--------------+--------+---------+-------------+----+---------------------+---------+-----+--------------------------+-------------------------------+------------------+-------------+--------+-------+------------------------+--------------+--------------------+--------------------+----------+\n",
      "|orig_channel|first_home_buyer|loan_purpose|property_type|occupancy_status|property_state|product_type|relocation_mortgage_indicator|seller_name|mod_flag|orig_interest_rate|orig_upb|orig_loan_term|orig_ltv|orig_cltv|num_borrowers| dti|borrower_credit_score|num_units|  zip|mortgage_insurance_percent|current_loan_delinquency_status|current_actual_upb|interest_rate|loan_age|    msa|non_interest_bearing_upb|delinquency_12|       rawPrediction|         probability|prediction|\n",
      "+------------+----------------+------------+-------------+----------------+--------------+------------+-----------------------------+-----------+--------+------------------+--------+--------------+--------+---------+-------------+----+---------------------+---------+-----+--------------------------+-------------------------------+------------------+-------------+--------+-------+------------------------+--------------+--------------------+--------------------+----------+\n",
      "|         0.0|             0.0|         0.0|          0.0|             0.0|           0.0|         0.0|                          0.0|        0.0|     0.0|             2.375|142000.0|         180.0|    65.0|     65.0|          1.0|15.0|                817.0|      1.0|208.0|                       0.0|                            0.0|         122703.74|        2.375|     9.0|47900.0|                     0.0|           0.0|[-1.3482570648193...|[0.99986517429351...|       0.0|\n",
      "|         0.0|             0.0|         0.0|          0.0|             0.0|           0.0|         0.0|                          0.0|        0.0|     0.0|             2.375|228000.0|         180.0|    48.0|     48.0|          1.0|30.0|                780.0|      1.0|787.0|                       0.0|                            0.0|               0.0|        2.375|     4.0|12420.0|                     0.0|           0.0|[3.86238098144531...|[1.00003862380981...|       0.0|\n",
      "|         0.0|             0.0|         0.0|          0.0|             0.0|           0.0|         0.0|                          0.0|        0.0|     0.0|             2.375|228000.0|         180.0|    48.0|     48.0|          1.0|30.0|                780.0|      1.0|787.0|                       0.0|                            0.0|               0.0|        2.375|     5.0|12420.0|                     0.0|           0.0|[9.65595245361328...|[1.00000965595245...|       0.0|\n",
      "|         0.0|             0.0|         0.0|          0.0|             0.0|           0.0|         0.0|                          0.0|        0.0|     0.0|             2.375|228000.0|         180.0|    48.0|     48.0|          1.0|30.0|                780.0|      1.0|787.0|                       0.0|                            0.0|          213309.6|        2.375|    14.0|12420.0|                     0.0|           0.0|[-1.1876225471496...|[0.99988123774528...|       0.0|\n",
      "|         0.0|             0.0|         0.0|          0.0|             0.0|           0.0|         0.0|                          0.0|        0.0|     0.0|             2.375|228000.0|         180.0|    48.0|     48.0|          1.0|30.0|                780.0|      1.0|787.0|                       0.0|                            0.0|          219784.9|        2.375|     8.0|12420.0|                     0.0|           0.0|[-7.2091817855834...|[0.99992790818214...|       0.0|\n",
      "|         0.0|             0.0|         0.0|          0.0|             0.0|           0.0|         0.0|                          0.0|        0.0|     0.0|               2.5|110000.0|         120.0|    69.0|     69.0|          2.0|11.0|                710.0|      1.0|532.0|                       0.0|                            0.0|               0.0|          2.5|     0.0|33340.0|                     0.0|           0.0|[4.38451766967773...|[1.00043845176696...|       0.0|\n",
      "|         0.0|             0.0|         0.0|          0.0|             0.0|           0.0|         0.0|                          0.0|        0.0|     0.0|               2.5|110000.0|         120.0|    69.0|     69.0|          2.0|11.0|                710.0|      1.0|532.0|                       0.0|                            0.0|          86753.46|          2.5|    14.0|33340.0|                     0.0|           0.0|[7.64369964599609...|[1.00076436996459...|       0.0|\n",
      "|         0.0|             0.0|         0.0|          0.0|             0.0|           0.0|         0.0|                          0.0|        0.0|     0.0|               2.5|110000.0|         120.0|    69.0|     69.0|          2.0|11.0|                710.0|      1.0|532.0|                       0.0|                            0.0|          89782.52|          2.5|    12.0|33340.0|                     0.0|           0.0|[4.24027442932128...|[1.00042402744293...|       0.0|\n",
      "|         0.0|             0.0|         0.0|          0.0|             0.0|           0.0|         0.0|                          0.0|        0.0|     0.0|               2.5|116000.0|         180.0|    74.0|     74.0|          2.0|23.0|                753.0|      1.0|457.0|                       0.0|                            0.0|               0.0|          2.5|     2.0|    0.0|                     0.0|           0.0|[-5.7399272918701...|[0.99994260072708...|       0.0|\n",
      "|         0.0|             0.0|         0.0|          0.0|             0.0|           0.0|         0.0|                          0.0|        0.0|     0.0|               2.5|116000.0|         180.0|    74.0|     74.0|          2.0|23.0|                753.0|      1.0|457.0|                       0.0|                            0.0|         110441.06|          2.5|    10.0|    0.0|                     0.0|           0.0|[-3.9398670196533...|[0.99996060132980...|       0.0|\n",
      "+------------+----------------+------------+-------------+----------------+--------------+------------+-----------------------------+-----------+--------+------------------+--------+--------------+--------+---------+-------------+----+---------------------+---------+-----+--------------------------+-------------------------------+------------------+-------------+--------+-------+------------------------+--------------+--------------------+--------------------+----------+\n",
      "only showing top 10 rows\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "modelFromDisk = xgbc_12a415c93943\n",
       "results2 = [orig_channel: float, first_home_buyer: float ... 29 more fields]\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "[orig_channel: float, first_home_buyer: float ... 29 more fields]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgbClassificationModel.write.overwrite.save(\"/data/model/mortgage\")\n",
    "\n",
    "val modelFromDisk = XGBoostClassificationModel.load(\"/data/model/mortgage\")\n",
    "\n",
    "val (results2, _) = Benchmark.time(\"transform2\") {\n",
    "  modelFromDisk.transform(evalSet)\n",
    "}\n",
    "results2.show(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "spark.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "0904 - Scala",
   "language": "scala",
   "name": "0904_scala"
  },
  "language_info": {
   "codemirror_mode": "text/x-scala",
   "file_extension": ".scala",
   "mimetype": "text/x-scala",
   "name": "scala",
   "pygments_lexer": "scala",
   "version": "2.11.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}