rap_example_pipeline_python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# RAP Example Python Pipeline - Interactive Exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/nhsengland/HACA-2024-python-in-an-hour/blob/main/rap_example_pipeline_python.ipynb\">\n",
    "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
    "</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Intro\n",
    "\n",
    "This notebook will show you how straight-forward it is to do an analytical pipeline in Python.\n",
    "\n",
    "The core of any of piece of analytical work is to:\n",
    "- load some data\n",
    "- do something to do that, e.g. process it, do some analysis\n",
    "- create some output\n",
    "\n",
    "This notebook will go briefly through each of these showing *one* way of doing it in Python (there are many more!). \n",
    "\n",
    "Open this notebook in google colab and have a play - try changing bits and see what happens!\n",
    "\n",
    "**NOTE**: to make the workshop more straight forward, we haven't completely followed good practice. If you want to see a pipeline how it should be, well laid out and modularised, [see our Example Python pipeline](https://github.com/NHSDigital/RAP_example_pipeline_python)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will need to install a few things before we can get going.\n",
    "\n",
    "First, if this is running in Google Colab, we need to clone the repo and install the right python packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this forces google collab to install the dependencies\n",
    "if \"google.colab\" in str(get_ipython()):\n",
    "    print(\"Running on Colab\")\n",
    "    !git clone https://github.com/nhsengland/HACA-2024-python-in-an-hour.git -q\n",
    "    %cd HACA-2024-python-in-an-hour\n",
    "    !pip install -r requirements.txt -q"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we need to import the right libraries for this piece of work:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# These libraries will help us download the file\n",
    "import zipfile\n",
    "import io\n",
    "from pathlib import Path\n",
    "import requests\n",
    "\n",
    "\n",
    "import pandas as pd # this allows us to work with dataframes\n",
    "import matplotlib.pyplot as plt # this allows us to create some graphs\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Config"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Quite often, when we create an analytical process in code, there will be parts of the code that we need to update - dates, URLs, file paths, and so on.\n",
    " \n",
    "We usually put these in a separate file called a config file. That way, all our settings are in one place. Here we'll just put our config settings in this cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "zip_file_url = \"https://files.digital.nhs.uk/assets/Services/Artificial%20data/Artificial%20HES%20final/artificial_hes_ae_202302_v1_sample.zip\"\n",
    "path_to_downloaded_data = \"data_in/artificial_hes_ae_202302_v1_sample.zip/artificial_hes_ae_202302_v1_sample/artificial_hes_ae_2122.csv\"\n",
    "\n",
    "# The column(s) we are going to investigate\n",
    "col_to_group = \"AEARRIVALMODE\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we will load the data: we're going to use an artificial fake version of the NHS Hospital Episode Statistics Accident and Emergency (HES AE) data from 2003. \n",
    "\n",
    "\n",
    "This code:\n",
    "- gets the location of the data from the config file\n",
    "- downloads the CSV\n",
    "- saves it to our data_in folder\n",
    "\n",
    "This is just an example - in another setting we could make it load the data from a SQL server, or from a database, S3 bucket, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filename = Path(zip_file_url).name\n",
    "output_path = f\"data_in/{filename}\"\n",
    "\n",
    "response = requests.get(zip_file_url, stream=True,timeout=3600)\n",
    "downloaded_zip = zipfile.ZipFile(io.BytesIO(response.content))\n",
    "downloaded_zip.extractall(output_path)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we need to read in our data and store it so we can continue to use and manipulate it. We'll use the pandas method read_csv to turn the data within the csv file to a pandas dataframe (commonly referred to as df).\n",
    "\n",
    "A pandas DataFrame is a two-dimensional, labeled data structure in Python, similar to a table in a database or an Excel spreadsheet, that allows for the storage and manipulation of data across rows and columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_hes = pd.read_csv(path_to_downloaded_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see what this data looks like, and pull the first 5 rows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_hes.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the fun part - we get to do some interesting processing on the data.\n",
    "\n",
    "Let's group the dataframe by the specified columns in cols_to_group, counts the number of rows in each group, and create a new column \"Count\" to show this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_hes_grouped = (df_hes\n",
    "    .groupby(col_to_group) # we state the column we want to group by. We defined col_to_group above!\n",
    "    .size() # Here we tell python we want to count the number of each value in the column. It's like count() in SQL\n",
    "    .to_frame('Count') # This turns the counts into a dataframe, and gives a title to our counts column\n",
    ")\n",
    "\n",
    "display(df_hes_grouped)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's a little confusing what those numbers mean, so let's put them into plain English"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ae_arrival_mode_map = {\n",
    "    1: \"Ambulance\",\n",
    "    2: \"Other\",\n",
    "    9: \"Not known\"\n",
    "}\n",
    "df_hes_grouped = df_hes_grouped.rename(index=ae_arrival_mode_map)\n",
    "print(df_hes_grouped)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's better!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualising"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it's time to take our dataframe and turn it into something nice to look at and easy to interpret!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_hes_grouped.plot.bar(rot = 0)\n",
    "plt.title(f\"Count by {col_to_group}\") # Here we add a title. Try changing it! You can put anything you want, it just has to be in quotes\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Changing Outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Due to how the code is written, you can easily swap what column you are investigating. Under \"Config\", try swapping the column in col_to_group with one of the other columns listed here. Remember to update the name mapping too.\n",
    "\n",
    "AEATTENDDISP - Attendance Disposal\n",
    "\n",
    "AEINCLOCTYPE - Where incident occurred\n",
    "\n",
    "AEREFSOURCE - Source of referral\n",
    "\n",
    "ETHNOS - Ethnicity of patient\n",
    "\n",
    "RESGOR_ONS - Region of Residence\n",
    "\n",
    "SEX - Patient's Sex\n",
    "\n",
    "Check out the HES Technical Output Specification to understand what each column mean and the possible options.\n",
    "\n",
    "https://digital.nhs.uk/data-and-information/data-tools-and-services/data-services/hospital-episode-statistics/hospital-episode-statistics-data-dictionary\n",
    "\n",
    "Then just run the notebook again to see your results!\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advanced"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You could try grouping by two or more columns, to understand the data a little deeper.\n",
    "\n",
    "How about we make a grouped bar chart?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Config"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This time we'll need to list two columns. Pick two columns from the list above and put them into the square brackets below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cols_to_group = []"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Grouped bar chart"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copy the code below into the cell underneath. Most of the code is ready for you, but there are some bits missing.\n",
    "\n",
    "Can you fill in the ????s to complete the grouped bar chart?\n",
    "\n",
    "For bonus points, try adding a title to the chart!\n",
    "\n",
    "\n",
    "```\n",
    "df_hes_grouped_bar = df_hes[????]\n",
    "\n",
    "df_hes_grouped_bar = df_hes_grouped_bar.groupby(????).size().unstack()\n",
    "\n",
    "df_hes_grouped_bar.plot.bar(rot = 0)\n",
    "\n",
    "plt.show()\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}