Introduction_to_python_day_2_session_3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# An introduction to solving biological problems with Python\n",
    "\n",
    "## Session 2.3: Files\n",
    "\n",
    "- [Using files](#Using-files)\n",
    "- [Exercises 2.3.1](#Exercises-2.3.1)\n",
    "- [Data formats](#Data-formats)\n",
    "- [Importing modules and libraries](#Importing-modules-and-libraries)\n",
    "- [Using the `csv` module](#Using-the-csv-module)\n",
    "- [Python file library](#Python-file-library)\n",
    "- [Exercises 2.3.2](#Exercises-2.3.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data input and output (I/O)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far, all that data we have been working with has been written by us into our scripts, and the results of out computation has just been displayed in the terminal output. In the real world data will be supplied by the user of our programs (who may be you!) by some means, and we will often want to save the results of some analysis somewhere more permanent than just printing it to the screen. In this session we cover 2 widely used ways of reading data into our programs, via the command line and by reading files from dish, we also discuss writing out data to files. \n",
    "\n",
    "There are, of course, many other ways of accessing data, such as querying a database or retrieving data from a network such as the internet. We don't cover these here, but python has excellent support for interacting with databases and networks either in the standard library or using external modules."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Frequently the data we want to operate on or analyse will be stored in files, so in our programs we need to be able to open files, read through them (perhaps all at once, perhaps not), and then close them. \n",
    "\n",
    "We will also frequently want to be able to print output to files rather than always printing out results to the terminal.\n",
    "\n",
    "Python supports all of these modes of operations on files, and provides a number of useful functions and syntax to make dealing with files straightforward."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Opening files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To open a file, python provides the `open` function, which takes a filename as its first argument and returns a _file object_ which is python's internal representation of the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "path = \"data/datafile.txt\"\n",
    "fileObj = open( path )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`open` takes an optional second argument specifying the _mode_ in which the file is opened, either for reading, writing or appending.\n",
    "\n",
    "It defaults to `'r'` which means open for reading in text mode. Other common values are `'w'` for writing (truncating the file if it already exists) and `'a'` for appending."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "open( \"data/myfile.txt\", \"r\" ) # open for reading, default"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "open( \"data/myfile.txt\", \"w\" ) # open for writing (existing files will be overwritten)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "open( \"data/myfile.txt\", \"a\" ) # open for appending"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Closing files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To close a file once you finished with it, you can call the `.close` method on a file object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fileObj.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mode modifiers\n",
    "\n",
    "These mode strings can include some extra modifier characters to deal with issues with files across multiple platforms.\n",
    "\n",
    "`'b'`: binary mode, e.g. `'rb'`. No translation for end-of-line characters to platform specific setting value.\n",
    "\n",
    "|Character | Meaning |\n",
    "|----------|---------|\n",
    "|`'r'` |\topen for reading (default) |\n",
    "|`'w'` |\topen for writing, truncating the file first |\n",
    "|`'x'` |\topen for exclusive creation, failing if the file already exists |\n",
    "|`'a'` |\topen for writing, appending to the end of the file if it exists |\n",
    "|`'b'` |\tbinary mode |\n",
    "|`'t'` |\ttext mode (default) |\n",
    "|`'+'` |\topen a disk file for updating (reading and writing) |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reading from files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have opened a file for reading, file objects provide a number of methods for accessing the data in a file. The simplest of these is the `.read` method that reads the entire contents of the file into a string variable.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fileObj = open( \"data/datafile.txt\" )\n",
    "print(fileObj.read()) # everything\n",
    "fileObj.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that this means the entire file will be read into memory. If you are operating on a large file and don't actually need all the data at the same time this is rather inefficient.\n",
    "\n",
    "Frequently, we just need to operate on individual lines of the file, and you can use the `.readline` method to read a line from a file and return it as a python string.\n",
    "\n",
    "File objects internally keep track of your current location in a file, so to get following lines from the file you can call this method multiple times.\n",
    "\n",
    "It is important to note that the string representing each line will have a trailing newline `\"\\n\"` character, which you may want to remove with the `.rstrip` string method.\n",
    "\n",
    "Once the end of the file is reached, `.readline` will return an empty string `''`. This is different from an apparently empty line in a file, as even an empty line will contain a newline character. Recall that the empty string is considered as `False` in python, so you can readily check for this condition with an `if` statement etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# one line at a time\n",
    "fileObj = open( \"data/datafile.txt\" )\n",
    "print(\"1st line:\", fileObj.readline())\n",
    "print(\"2nd line:\", fileObj.readline())\n",
    "print(\"3rd line:\", fileObj.readline())\n",
    "print(\"4th line:\", fileObj.readline())\n",
    "fileObj.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To read in all lines from a file as a list of strings containing the data from each line, use the `.readlines` method (though note that this will again read all data into memory)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# all lines\n",
    "fileObj = open( \"data/datafile.txt\" )\n",
    "\n",
    "lines = fileObj.readlines()\n",
    "\n",
    "print(\"The file has\", len(lines), \"lines\")\n",
    "\n",
    "fileObj.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looping over the lines in a file is a very common operation and python lets you iterate over a file using a `for` loop just as if it were an array of strings. This does not read all data into memory at once, and so is much more efficient that reading the file with `.readlines` and then looping over the resulting list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# as an iterable\n",
    "fileObj = open( \"data/datafile.txt\" )\n",
    "\n",
    "for line in fileObj:\n",
    "    print(line.rstrip().upper())\n",
    "\n",
    "fileObj.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The with statement"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is important that files are closed when they are no longer required, but writing ``fileObj.close()`` is tedious (and more importantly, easy to forget). An alternative syntax is to open the files within a ``with`` statement, in which case the file will automatically be closed at the end of the `with` block."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# fileObj will be closed when leaving the block\n",
    "with open( \"data/datafile.txt\" ) as fileObj:\n",
    "    for ( i, line ) in enumerate( fileObj, start = 1 ):\n",
    "        print(\"%s: %r\" % ( i, line ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Writing to files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once a file has been opened for writing, you can use the `.write()` method on a file object to write data to the file.\n",
    "\n",
    "The argument to the `.write()` method must be a string, so if you want to write out numerical data to a file you will have to convert it to a string somehow beforehand."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert-warning\">**Remember** to include a newline character `\\n` to separate lines of your output, unlike the `print()` statement, `.write()` does not include this by default.</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "read_counts = {\n",
    "    'BRCA2': 43234,\n",
    "    'FOXP2': 3245,\n",
    "    'SORT1': 343792\n",
    "}\n",
    "\n",
    "with open( \"out.txt\", \"w\" ) as output:\n",
    "    output.write(\"GENE\\tREAD_COUNT\\n\")\n",
    "\n",
    "    for gene in read_counts:\n",
    "        line = \"\\t\".join( [ gene, str(read_counts[gene]) ] )\n",
    "        output.write(line + \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "To view the output file, open a terminal window, go to the directory where the file has been written, and print the content of the file using `cat` command or open it using your favourite editor:\n",
    "\n",
    "```bash\n",
    "cat out.txt\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Be cautious when opening a file for writing, as python will happily let you overwrite any existing data in the file. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercises 2.3.1\n",
    "\n",
    "- a. Create a script that writes the values of a list of numbers to a file, with each number on a seperate line.\n",
    "- b. Write a script that reads a file containing many lines of nucleotide sequence. For each line in the file, print out the line number, the length of the sequence and the sequence (There is an example file <a href=\"http://www.ebi.ac.uk/~grsr/perl/dna.txt\">here</a> or in `data/dna.txt` from the course materials )."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data formats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bioinformaticians love creating endless new file formats for their data, but there is one very common standard format\n",
    "that it is good to get used to parsing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Delimited file example:\n",
    "```\n",
    "X 169008682 1 111267453 1.0976\n",
    "2 8265484 5 69763543 4.9825\n",
    "MT 10924 MT 81934 7.2357\n",
    "3 127 8 10908776 1.2509\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reading delimited files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the various string manipulation techniques covered earlier to process delimited files in a fairly straightforward way. Here we loop through a file with columns delimited by spaces, reading the data for each row into a list, and storing each of these lists into a main results list."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "To view the an example of a delimited file, open a terminal window, go to the course directory, and print the content of the file using `cat` command or open it using your favourite editor:\n",
    "\n",
    "```bash\n",
    "cat data/mydata.txt\n",
    "```\n",
    "\n",
    "```\n",
    "Index Organism Score\n",
    "1 Human 1.076\n",
    "2 Mouse 1.202\n",
    "3 Frog 2.2362\n",
    "4 Fly 0.9853\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "results = []\n",
    "\n",
    "with open(\"data/mydata.txt\", \"r\") as data:\n",
    "    header = data.readline()\n",
    "    for line in data:\n",
    "        line = line.strip()\n",
    "        results.append(line.split(\" \"))\n",
    "        \n",
    "print(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we show a slightly more complicated example where we are reading the results into a more convenient data structure, a list of dictionaries with the dictionary keys corresponding to the column headers and the values to the values from each line. We also convert the columns to an appropriate type as we go."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "results = []\n",
    "\n",
    "with open(\"data/mydata.txt\", \"r\") as data:\n",
    "    header = data.readline()\n",
    "    for line in data:\n",
    "        idx, org, score = line.strip().split(\" \")\n",
    "        row = {'Index': int(idx), 'Organism': org, 'Score': float(score)}\n",
    "        results.append(row)\n",
    "        \n",
    "print(results)\n",
    "print('Score of first row:', results[0]['Score'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Writing delimited files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Writing out a delimited file is also straightforward using the `join` method. Here, as an example we will recreate our original file from above, but this time we will delimit the columns with a comma."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "mydata = [{'Organism': 'Human', 'Index': 1, 'Score': 1.076}, \n",
    "          {'Organism': 'Mouse', 'Index': 2, 'Score': 1.202}, \n",
    "          {'Organism': 'Frog', 'Index': 3, 'Score': 2.2362}, \n",
    "          {'Organism': 'Fly', 'Index': 4, 'Score': 0.9853}]\n",
    "\n",
    "with open('data/mydata.csv', 'w') as output:\n",
    "    # write a header\n",
    "    header = \",\".join(['Index', 'Organism', 'Score'])\n",
    "    output.write(header + \"\\n\")\n",
    "    for row in mydata:\n",
    "        line = \",\".join([str(row['Index']), row['Organism'], str(row['Score'])])\n",
    "        output.write(line + \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "To view the output file, open a terminal window, go to the course directory, and print the content of the file using `cat` command or open it using your favourite editor:\n",
    "\n",
    "```bash\n",
    "cat data/mydata.csv\n",
    "```\n",
    "\n",
    "```\n",
    "Index,Organism,Score\n",
    "1,Human,1.076\n",
    "2,Mouse,1.202\n",
    "3,Frog,2.2362\n",
    "4,Fly,0.9853\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert-warning\">**NOTE** that there is actually a module in the standard library called `csv` which can also be used to read and write delimited files. There is example code reading this same file using this library towards the end of this notebook.</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing modules and libraries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like other laguages, Python has the ability to import external modules (or libraries) into the current program. These modules may be part of the standard library that is automatically included with the Python installation, they may be extra libraries which you install separately or they may be other Python programs you have written yourself. Whatever the source of the module, they are imported into a program via an <tt>import</tt> command.\n",
    "\n",
    "For example, if we wish to access the mathematical constants pi and e we can use the import keyword to get [the module named `math`](https://docs.python.org/3/library/math.html) and access its contents with the dot notation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import math\n",
    "print(math.pi, math.e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also we can use the `as` keyword to give the module a different name in our code, which can be useful for brevity and avoiding name conflicts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import math as m\n",
    "print(m.pi, m.e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively we can import the separate components using the `from … import` keyword combination, like we've seen in the [previous session](Introduction_to_python_day_2_session_2.ipynb#Modules):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from math import pi, e\n",
    "print(pi, e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can import multiple components from a single module, either on one line like as seen above or on separate lines:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from math import pi\n",
    "from math import e"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Listing module contents\n",
    "\n",
    "Using the [method `dir()`](https://docs.python.org/3/library/functions.html?highlight=dir#dir) and passing the module name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import math\n",
    "dir(math)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "or directly using an instance, like with this String:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dir(\"mystring\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "or using the object type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dir(str)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting quick help on method\n",
    "\n",
    "After listing all contents, you may wish to display specific information on a method using [`help()`](https://docs.python.org/3/library/functions.html?highlight=help#help)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "help(str.title)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting help from the official Python documentation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The most useful information is online on https://www.python.org/ website and should  be used as a reference guide.\n",
    "\n",
    "- [Python 3.5.2 documentation](https://docs.python.org/3/) is the starting page with links to tutorials and libraries' documentation for Python 3\n",
    "    - [The Python Tutorial](https://docs.python.org/3/tutorial/index.html)\n",
    "        - [Modules](https://docs.python.org/3/tutorial/modules.html)\n",
    "        - [Brief Tour of the Standard Library: Mathematics](https://docs.python.org/3/tutorial/stdlib.html#mathematics)\n",
    "    - [The Python Standard Library Reference](https://docs.python.org/3/library/index.html) is the documentation of all libraries included within Python as well as built-in functions and data types like:\n",
    "        - [Text Sequence Type — `str`](https://docs.python.org/3/library/stdtypes.html#text-sequence-type-str)\n",
    "        - [`math` — Mathematical functions](https://docs.python.org/3/library/math.html)\n",
    "        - [`csv` — CSV File Reading and Writing](https://docs.python.org/3/library/csv.html)\n",
    "        - [`os` — Miscellaneous operating system interfaces](https://docs.python.org/3/library/os.html)\n",
    "        - [`os.path` — Common pathname manipulations](https://docs.python.org/3/library/os.path.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using the `csv` module\n",
    "\n",
    "The so-called CSV (Comma Separated Values) format is the most common import and export format for spreadsheets and databases. The csv module implements methods to read and write tabular data in CSV format.\n",
    "\n",
    "The csv module’s `reader()` and `writer()` methods read and write CSV files. You can also read and write data into dictionary form using the `DictReader()` and `DictWriter()` methods.\n",
    "\n",
    "For more information about this built-in Python library about [CSV File Reading and Writing documentation](https://docs.python.org/3/library/csv.html).\n",
    "\n",
    "Let's now read our `data/mydata.txt` file using the `csv` module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import csv\n",
    "with open(\"data/mydata.txt\") as f:\n",
    "    reader = csv.reader(f, delimiter = \" \") # default delimiter is \",\"\n",
    "    for row in reader:\n",
    "        print(row)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Change the `csv.reader()` by the `csv.DictReader()` and it builds up a dictionary automatically based on the column headers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "with open(\"data/mydata.txt\") as f:\n",
    "    reader = csv.DictReader(f, delimiter = \" \")\n",
    "    for row in reader:\n",
    "        print(row)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Write tab delimited files using the csv module\n",
    "import csv\n",
    "\n",
    "mydata = [\n",
    "    ['1', 'Human', '1.076'], \n",
    "    ['2', 'Mouse', '1.202'], \n",
    "    ['3', 'Frog', '2.2362'], \n",
    "    ['4', 'Fly', '0.9853']\n",
    "]\n",
    "\n",
    "with open(\"csvdata.tsv\", \"w\") as f:\n",
    "    writer = csv.writer(f, delimiter='\\t' )\n",
    "    writer.writerow( [ \"Index\", \"Organism\", \"Score\" ] ) # write header\n",
    "    for record in mydata:\n",
    "        writer.writerow( record )\n",
    "\n",
    "# Open the output file and print out its content\n",
    "with open(\"csvdata.tsv\") as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Write delimited files using the csv module from a list of dictionaries \n",
    "import csv\n",
    "\n",
    "mydata = [\n",
    "    {'Index': '1', 'Score': '1.076', 'Organism': 'Human'}, \n",
    "    {'Index': '2', 'Score': '1.202', 'Organism': 'Mouse'}, \n",
    "    {'Index': '3', 'Score': '2.2362', 'Organism': 'Frog'}, \n",
    "    {'Index': '4', 'Score': '0.9853', 'Organism': 'Fly'}\n",
    "]\n",
    "\n",
    "with open(\"csvdictdata.tsv\", \"w\") as f:\n",
    "    writer = csv.DictWriter(f, mydata[0].keys(), delimiter='\\t')\n",
    "    writer.writeheader() # write header\n",
    "\n",
    "    for record in mydata:\n",
    "        writer.writerow( record )\n",
    "\n",
    "# Open the output file and print out its content\n",
    "with open(\"csvdictdata.tsv\") as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python file library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [`os.path` — Common pathname manipulations](https://docs.python.org/3/library/os.path.html)\n",
    "\n",
    "- `exists(path)` : returns whether path exists\n",
    "- `isfile(path)` : returns whether path is a “regular” file (as opposed to a directory)\n",
    "- `isdir(path)` : returns whether path is a directory\n",
    "- `islink(path)` : returns whether path is a symbolic link\n",
    "- `join(*paths)` : joins the paths together into one long path\n",
    "- `dirname(path)` : returns directory containing the path\n",
    "- `basename(path)` : returns the path minus the dirname(path) in front\n",
    "- `split(path)` : returns (dirname(path), basename(path))\n",
    "\n",
    "### [`os` — Miscellaneous operating system interfaces](https://docs.python.org/3/library/os.html)\n",
    "\n",
    "- `chdir(path)` : change the current working directory to be path\n",
    "- `getcwd()` : return the current working directory\n",
    "- `listdir(path)` : returns a list of files/directories in the directory path\n",
    "- `mkdir(path)` : create the directory path\n",
    "- `rmdir(path)` : remove the directory path\n",
    "- `remove(path)` : remove the file path\n",
    "- `rename(src, dst)` : move the file/directory from src to dst"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Building the path to your file from a list of directory and filename makes your script able to run on any platforms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os.path\n",
    "os.path.join(\"data\", \"mydata.txt\")\n",
    "# data/mydata.txt - Unix\n",
    "# data\\mydata.txt - Windows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check if a file exists before opening it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os.path\n",
    "data_file = os.path.join(\"data\", \"mydata.txt\")\n",
    "if os.path.exists(data_file):\n",
    "    print(\"file\", data_file, \"exists\")\n",
    "    with open(data_file) as f:\n",
    "        print(f.read())\n",
    "else:\n",
    "    print(\"file\", data_file, \"not found!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercises 2.3.2\n",
    "\n",
    "- a. Write a script that reads a tab delimited file which has 4 columns: gene, chromosome, start and end coordinates. Check if the file exists, then compute the length of each gene and store its name and corresponding length into a dictionary. Write the results into a new tab separated file. You can find a data file <a href=\"http://www.ebi.ac.uk/~grsr/perl/genes.txt\">here</a>, or in ` data/genes.txt` directory of the course materials.\n",
    "\n",
    "- b. Now change your script to make use of the `csv` module."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next session\n",
    "\n",
    "Go to our next notebook: [Introduction_to_python_day_2_session_4](Introduction_to_python_day_2_session_4.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}