colibricore-python-tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Colibri Core Python Tutorial: Efficiently working with n-grams, skipgrams and flexgrams"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(*for Colibri Core v2.4* - revision 2016-06-10)\n",
    "\n",
    "\n",
    "https://proycon.github.io/colibri-core\n",
    "\n",
    "*by Maarten van Gompel, Radboud University Nijmegen*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial will show you how to work with Colibri Core's Python API, a tool for Natural Language Processing. It is assumed that you have already read the Colibri Core documentation, followed the installation instructions, and are familiar its purpose and concepts. The documentation also provides an API reference for all the Python classes and method. This tutorial is in the form of a Python Notebook, allowing you to interactively participate. Press ``shift+enter`` in code field to evaluate it.\n",
    "\n",
    "Colibri Core is written in C++ and the Python binding is writting in Cython. This offers the advantage of native-speed and memory efficiency, combined with the ease of a high-level pythonic interface. We will be using Python 3 conventions here, but Colibri Core and this tutorial can also work with Python 2.7."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We obviously start our adventure with an import of colibricore, so make sure you installed it properly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#These first imports are for Python 2.7 compatibility, we approximate Python 3 as much as possible\n",
    "from __future__ import print_function\n",
    "from sys import version\n",
    "PYTHON2 = version[0] == '2'\n",
    "if PYTHON2:\n",
    "    from io import open\n",
    "\n",
    "\n",
    "import colibricore\n",
    "\n",
    "TMPDIR = \"/tmp/\" #this is where we'll store intermediate files\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Class encoding/decoding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To give us something to work with, we will take an excerpt of Shakespeare's Hamlet as our corpus text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "corpustext = \"\"\"To be, or not to be, that is the question\n",
    "Whether 'tis Nobler in the mind to suffer\n",
    "The Slings and Arrows of outrageous Fortune,\n",
    "Or to take Arms against a Sea of troubles,\n",
    "And by opposing end them? To die, to sleep\n",
    "No more; and by a sleep, to say we end\n",
    "The Heart-ache, and the thousand Natural shocks\n",
    "That Flesh is heir to? 'Tis a consummation\n",
    "Devoutly to be wished. To die, to sleep,\n",
    "To sleep, perchance to Dream; Aye, there's the rub,\n",
    "For in that sleep of death, what dreams may come,\n",
    "When we have shuffled off this mortal coil,\n",
    "Must give us pause. There's the respect\n",
    "That makes Calamity of so long life:\n",
    "For who would bear the Whips and Scorns of time,\n",
    "Th' Oppressor's wrong, the proud man's Contumely,\n",
    "The pangs of despised Love, the Law’s delay,\n",
    "The insolence of Office, and the Spurns\n",
    "That patient merit of the unworthy takes,\n",
    "When he himself might his Quietus make\n",
    "With a bare Bodkin? Who would these Fardels bear,\n",
    "To grunt and sweat under a weary life,\n",
    "But that the dread of something after death,\n",
    "The undiscovered Country, from whose bourn\n",
    "No Traveler returns, Puzzles the will,\n",
    "And makes us rather bear those ills we have,\n",
    "Than fly to others that we know not of.\n",
    "Thus Conscience does make Cowards of us all,\n",
    "And thus the Native hue of Resolution\n",
    "Is sicklied o'er, with the pale cast of Thought,\n",
    "And enterprises of great pitch and moment,\n",
    "With this regard their Currents turn awry,\n",
    "And lose the name of Action. Soft you now,\n",
    "The fair Ophelia. Nymph, in all thy Orisons\n",
    "Be all my sins remembered\"\"\"\n",
    "\n",
    "#first we do some very rudimentary tokenisation\n",
    "# Yes, I realise this is a very stupid way ;)\n",
    "corpustext = corpustext.replace(',',' ,')\n",
    "corpustext = corpustext.replace('.',' .')\n",
    "corpustext = corpustext.replace(':',' :')\n",
    "\n",
    "if PYTHON2: corpustext = unicode(corpustext,'utf-8')\n",
    "\n",
    "corpusfile_plaintext = TMPDIR + \"hamlet.txt\"\n",
    "\n",
    "with open(corpusfile_plaintext,'w',encoding='utf-8') as f:\n",
    "    f.write(corpustext)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To work with this data with Colibri Core. We need to *class encode* it, assigning integer values to each word type. Using Python, a class encoder is built as follows:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Encoded  184  classes, well done!\n"
     ]
    }
   ],
   "source": [
    "classfile = TMPDIR + \"hamlet.colibri.cls\"\n",
    "\n",
    "#Instantiate class encoder\n",
    "classencoder = colibricore.ClassEncoder()\n",
    "\n",
    "#Build classes\n",
    "classencoder.build(corpusfile_plaintext)\n",
    "\n",
    "#Save class file\n",
    "classencoder.save(classfile)\n",
    "\n",
    "print(\"Encoded \", len(classencoder), \" classes, well done!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have a class encoder we can encode our corpus, producing a new encoded file (which tends to be about 50% compressed compared to the original):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "corpusfile = TMPDIR + \"hamlet.colibri.dat\" #this will be the encoded corpus file\n",
    "classencoder.encodefile(corpusfile_plaintext, corpusfile)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To check whether that worked as planned, we will construct a Class Decoder, load our class file, and decode the corpus:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "To be , or not to be , that is the question\n",
      "Whether 'tis Nobler in the mind to suffer\n",
      "The Slings and Arrows of outrageous Fortune ,\n",
      "Or to take Arms against a Sea of troubles ,\n",
      "And by opposing end them? To die , to sleep\n",
      "No more; and by a sleep , to say we end\n",
      "The Heart-ache , and the thousand Natural shocks\n",
      "That Flesh is heir to? 'Tis a consummation\n",
      "Devoutly to be wished . To die , to sleep ,\n",
      "To sleep , perchance to Dream; Aye , there's the rub ,\n",
      "For in that sleep of death , what dreams may come ,\n",
      "When we have shuffled off this mortal coil ,\n",
      "Must give us pause . There's the respect\n",
      "That makes Calamity of so long life :\n",
      "For who would bear the Whips and Scorns of time ,\n",
      "Th' Oppressor's wrong , the proud man's Contumely ,\n",
      "The pangs of despised Love , the Law’s delay ,\n",
      "The insolence of Office , and the Spurns\n",
      "That patient merit of the unworthy takes ,\n",
      "When he himself might his Quietus make\n",
      "With a bare Bodkin? Who would these Fardels bear ,\n",
      "To grunt and sweat under a weary life ,\n",
      "But that the dread of something after death ,\n",
      "The undiscovered Country , from whose bourn\n",
      "No Traveler returns , Puzzles the will ,\n",
      "And makes us rather bear those ills we have ,\n",
      "Than fly to others that we know not of .\n",
      "Thus Conscience does make Cowards of us all ,\n",
      "And thus the Native hue of Resolution\n",
      "Is sicklied o'er , with the pale cast of Thought ,\n",
      "And enterprises of great pitch and moment ,\n",
      "With this regard their Currents turn awry ,\n",
      "And lose the name of Action . Soft you now ,\n",
      "The fair Ophelia . Nymph , in all thy Orisons\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#Load class decoder from the classfile we just made\n",
    "classdecoder = colibricore.ClassDecoder(classfile)\n",
    "\n",
    "#Decode corpus data\n",
    "decoded = classdecoder.decodefile(corpusfile)\n",
    "\n",
    "#Show\n",
    "print(decoded)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Playing with patterns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have a class encoder and decoder, we can toy around with the most basic units in Colibri Core: **patterns**. These are using for n-grams, skipgrams, flexgrams and any kind of test. You would basically use an instance of ``Pattern`` where you'd normally use a string, as Patterns are much smaller in memory. Let's build a pattern from a string using the classencoder, note that we will only be able to use words that are known by the class encoder:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "To be or not to be\n",
      "6\n"
     ]
    }
   ],
   "source": [
    "#Build a pattern from a string, using the class encoder\n",
    "p = classencoder.buildpattern(\"To be or not to be\")\n",
    "\n",
    "#To print it we need the decoder\n",
    "print(p.tostring(classdecoder))\n",
    "print(len(p))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Iteration over a pattern will produce all the tokens that it is made up of. Note that the concepts of characters is gone from patterns! As a consequence, the ability to lowercase or uppercase text is also lost."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "To\n",
      "be\n",
      "or\n",
      "not\n",
      "to\n",
      "be\n"
     ]
    }
   ],
   "source": [
    "#Iterate over the token in a pattern, each token will be a Pattern instance\n",
    "\n",
    "for token in p:\n",
    "    print(token.tostring(classdecoder))\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "To\n",
      "be\n",
      "or not\n"
     ]
    }
   ],
   "source": [
    "#Extracting subpatterns by offset\n",
    "\n",
    "#Get first token\n",
    "print(p[0].tostring(classdecoder))\n",
    "\n",
    "#Get last token\n",
    "print(p[-1].tostring(classdecoder))\n",
    "\n",
    "#Get slice\n",
    "print(p[2:4].tostring(classdecoder))\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given a pattern, we can now very easily extract all n-grams in it, one of the most common NLP tasks:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "To be\n",
      "be or\n",
      "or not\n",
      "not to\n",
      "to be\n"
     ]
    }
   ],
   "source": [
    "#let's get all bigrams\n",
    "for ngram in p.ngrams(2):\n",
    "    print(ngram.tostring(classdecoder))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "To\n",
      "be\n",
      "or\n",
      "not\n",
      "to\n",
      "be\n",
      "To be\n",
      "be or\n",
      "or not\n",
      "not to\n",
      "to be\n",
      "To be or\n",
      "be or not\n",
      "or not to\n",
      "not to be\n",
      "To be or not\n",
      "be or not to\n",
      "or not to be\n",
      "To be or not to\n",
      "be or not to be\n"
     ]
    }
   ],
   "source": [
    "#or all n-grams:\n",
    "for ngram in p.ngrams():\n",
    "    print(ngram.tostring(classdecoder))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "To\n",
      "be\n",
      "or\n",
      "not\n",
      "to\n",
      "be\n",
      "To be\n",
      "be or\n",
      "or not\n",
      "not to\n",
      "to be\n",
      "To be or\n",
      "be or not\n",
      "or not to\n",
      "not to be\n"
     ]
    }
   ],
   "source": [
    "#or particular ngrams, such as unigrams up to trigrams:\n",
    "for ngram in p.ngrams(1,3):\n",
    "    print(ngram.tostring(classdecoder))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ``in`` operator can be used to check if a token **OR** ngram is part of a pattern"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "#token\n",
    "p2 = classencoder.buildpattern(\"be\")\n",
    "print(p2 in p)\n",
    "\n",
    "#ngram\n",
    "p3 = classencoder.buildpattern(\"or not\")\n",
    "print(p3 in p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The follow snippet is here just to prove that our Pattern representation is usually smaller than a string representation, and offers a sneak peek under the hood:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'\\x0e\\x16\\x81\\x01\\x1d\\t\\x16' 7\n",
      "b'To be or not to be' 18\n"
     ]
    }
   ],
   "source": [
    "from sys import version\n",
    "if version[0] == '3': #This works on Python 3 only\n",
    "    print(bytes(p), len(bytes(p)))\n",
    "    print(b\"To be or not to be\", len(b\"To be or not to be\"))\n",
    "    len(bytes(p)) < len(b\"To be or not to be\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading a corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to read an entire corpus, we can use the ``IndexedCorpus`` class. This we can use, for example, if we are merely interested in moving a sliding window over our data and extracting n-grams without counting or storing them:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "To be , | be , or | , or not | or not to | not to be | to be , | be , that | , that is | that is the | is the question | Whether 'tis Nobler | 'tis Nobler in | Nobler in the | in the mind | the mind to | mind to suffer | The Slings and | Slings and Arrows | and Arrows of | Arrows of outrageous | of outrageous Fortune | outrageous Fortune , | Or to take | to take Arms | take Arms against | Arms against a | against a Sea | a Sea of | Sea of troubles | of troubles , | And by opposing | by opposing end | opposing end them? | end them? To | them? To die | To die , | die , to | , to sleep | No more; and | more; and by | and by a | by a sleep | a sleep , | sleep , to | , to say | to say we | say we end | The Heart-ache , | Heart-ache , and | , and the | and the thousand | the thousand Natural | thousand Natural shocks | That Flesh is | Flesh is heir | is heir to? | heir to? 'Tis | to? 'Tis a | 'Tis a consummation | Devoutly to be | to be wished | be wished . | wished . To | . To die | To die , | die , to | , to sleep | to sleep , | To sleep , | sleep , perchance | , perchance to | perchance to Dream; | to Dream; Aye | Dream; Aye , | Aye , there's | , there's the | there's the rub | the rub , | For in that | in that sleep | that sleep of | sleep of death | of death , | death , what | , what dreams | what dreams may | dreams may come | may come , | When we have | we have shuffled | have shuffled off | shuffled off this | off this mortal | this mortal coil | mortal coil , | Must give us | give us pause | us pause . | pause . There's | . There's the | There's the respect | That makes Calamity | makes Calamity of | Calamity of so | of so long | so long life | long life : | For who would | who would bear | would bear the | bear the Whips | the Whips and | Whips and Scorns | and Scorns of | Scorns of time | of time , | Th' Oppressor's wrong | Oppressor's wrong , | wrong , the | , the proud | the proud man's | proud man's Contumely | man's Contumely , | The pangs of | pangs of despised | of despised Love | despised Love , | Love , the | , the Law’s | the Law’s delay | Law’s delay , | The insolence of | insolence of Office | of Office , | Office , and | , and the | and the Spurns | That patient merit | patient merit of | merit of the | of the unworthy | the unworthy takes | unworthy takes , | When he himself | he himself might | himself might his | might his Quietus | his Quietus make | With a bare | a bare Bodkin? | bare Bodkin? Who | Bodkin? Who would | Who would these | would these Fardels | these Fardels bear | Fardels bear , | To grunt and | grunt and sweat | and sweat under | sweat under a | under a weary | a weary life | weary life , | But that the | that the dread | the dread of | dread of something | of something after | something after death | after death , | The undiscovered Country | undiscovered Country , | Country , from | , from whose | from whose bourn | No Traveler returns | Traveler returns , | returns , Puzzles | , Puzzles the | Puzzles the will | the will , | And makes us | makes us rather | us rather bear | rather bear those | bear those ills | those ills we | ills we have | we have , | Than fly to | fly to others | to others that | others that we | that we know | we know not | know not of | not of . | Thus Conscience does | Conscience does make | does make Cowards | make Cowards of | Cowards of us | of us all | us all , | And thus the | thus the Native | the Native hue | Native hue of | hue of Resolution | Is sicklied o'er | sicklied o'er , | o'er , with | , with the | with the pale | the pale cast | pale cast of | cast of Thought | of Thought , | And enterprises of | enterprises of great | of great pitch | great pitch and | pitch and moment | and moment , | With this regard | this regard their | regard their Currents | their Currents turn | Currents turn awry | turn awry , | And lose the | lose the name | the name of | name of Action | of Action . | Action . Soft | . Soft you | Soft you now | you now , | The fair Ophelia | fair Ophelia . | Ophelia . Nymph | . Nymph , | Nymph , in | , in all | in all thy | all thy Orisons | "
     ]
    }
   ],
   "source": [
    "corpusdata = colibricore.IndexedCorpus(corpusfile) #encoded data, will be loaded into memory entirely\n",
    "\n",
    "for sentence in corpusdata.sentences(): #will return a Pattern per sentence (generator)\n",
    "    for trigram in sentence.ngrams(3):\n",
    "        print(trigram.tostring(classdecoder), end= \" | \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you may be very tempted to start storing and counting n-grams this way, but **don't**. This method is only suitable for iterating and quickly discarding the ngrams. Colibri core has facilities to deal with storing and counting far more efficiently, these are *pattern models* which we will discuss in the next section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First some more about ``IndexedCorpus``. We can also obtain any pattern using its index, a ``(sentence,token)`` tuple:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "in\n"
     ]
    }
   ],
   "source": [
    "unigram = corpusdata[(2,3)]\n",
    "print(unigram.tostring(classdecoder))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A slice syntax is also supported, but may never cross line/sentence boundaries. As is customary in Python, the last index is non-inclusive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "in the mind to suffer\n"
     ]
    }
   ],
   "source": [
    "ngram = corpusdata[(2,3):(2,8)]\n",
    "print(ngram.tostring(classdecoder))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The number of sentences and the length of each sentence can be extracted as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Length of sentence 1: 12\n",
      "Length of sentence 2: 8\n",
      "Length of sentence 3: 8\n",
      "Length of sentence 4: 10\n",
      "Length of sentence 5: 10\n",
      "Length of sentence 6: 11\n",
      "Length of sentence 7: 8\n",
      "Length of sentence 8: 8\n",
      "Length of sentence 9: 11\n",
      "Length of sentence 10: 12\n",
      "Length of sentence 11: 12\n",
      "Length of sentence 12: 9\n",
      "Length of sentence 13: 8\n",
      "Length of sentence 14: 8\n",
      "Length of sentence 15: 11\n",
      "Length of sentence 16: 9\n",
      "Length of sentence 17: 10\n",
      "Length of sentence 18: 8\n",
      "Length of sentence 19: 8\n",
      "Length of sentence 20: 7\n",
      "Length of sentence 21: 10\n",
      "Length of sentence 22: 9\n",
      "Length of sentence 23: 9\n",
      "Length of sentence 24: 7\n",
      "Length of sentence 25: 8\n",
      "Length of sentence 26: 10\n",
      "Length of sentence 27: 10\n",
      "Length of sentence 28: 9\n",
      "Length of sentence 29: 7\n",
      "Length of sentence 30: 11\n",
      "Length of sentence 31: 8\n",
      "Length of sentence 32: 8\n",
      "Length of sentence 33: 11\n",
      "Length of sentence 34: 10\n"
     ]
    }
   ],
   "source": [
    "sentencecount = corpusdata.sentencecount()\n",
    "for i in range(1, sentencecount+1): #note the 1..+1 range, sentences are 1-indexed (whereas tokens are 0-indexed)\n",
    "    print(\"Length of sentence \" + str(i) + \":\", corpusdata.sentencelength(i))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also find specific patterns in ``IndexedCorpus`` data. However, it is usually more efficient to use a Pattern Model, as discussed in the next section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pattern found at: 1:3\n"
     ]
    }
   ],
   "source": [
    "searchpattern = classencoder.buildpattern(\"or not\")\n",
    "for (sentence,token), pattern in corpusdata.findpattern(searchpattern):\n",
    "    print(\"Pattern found at: \" + str(sentence) + \":\" + str(token))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can pass an extra parameter with a sentence index to ``findpattern()`` to limit your search to one particular sentence rather than the entire corpus."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pattern Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it's time to build our first pattern model on the Hamlet excerpt. We will extract all patterns occurring at least twice and with maximum length 8."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found  54  patterns:\n",
      "To die , to sleep | die , to sleep | To die , to | , and the | , to sleep | die , to | To die , | , the | we have | and the | , and | sleep , | to sleep | die , | To die | to be | be , | all | With | make | we | That | not | No | die | And | a | , | be | is | that | would | in | to | and | end | For | To | The | have | death | us | the | When | this | makes | death , | , to | by | life | sleep | bear | of | . | "
     ]
    }
   ],
   "source": [
    "#Set the options\n",
    "options = colibricore.PatternModelOptions(mintokens=2,maxlength=8)\n",
    "\n",
    "#Instantiate an empty unindexed model \n",
    "model = colibricore.UnindexedPatternModel()\n",
    "\n",
    "#Train it on our corpus file (class-encoded data, not plain text)\n",
    "model.train(corpusfile, options)\n",
    "\n",
    "print(\"Found \" , len(model), \" patterns:\")\n",
    "\n",
    "#Let's see what patterns are in our model (the order will be 'random')\n",
    "for pattern in model:\n",
    "    print(pattern.tostring(classdecoder), end=\" | \")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rather than just output the patterns, we of course now have the counts as well, let's output it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "To die , to sleep 2\n",
      "die , to sleep 2\n",
      "To die , to 2\n",
      ", and the 2\n",
      ", to sleep 2\n",
      "die , to 2\n",
      "To die , 2\n",
      ", the 2\n",
      "we have 2\n",
      "and the 2\n",
      ", and 2\n",
      "sleep , 3\n",
      "to sleep 2\n",
      "die , 2\n",
      "To die 2\n",
      "to be 2\n",
      "be , 2\n",
      "all 2\n",
      "With 2\n",
      "make 2\n",
      "we 4\n",
      "That 3\n",
      "not 2\n",
      "No 2\n",
      "die 2\n",
      "And 5\n",
      "a 5\n",
      ", 36\n",
      "be 3\n",
      "is 2\n",
      "that 4\n",
      "would 2\n",
      "in 3\n",
      "to 9\n",
      "and 7\n",
      "end 2\n",
      "For 2\n",
      "To 5\n",
      "The 6\n",
      "have 2\n",
      "death 2\n",
      "us 3\n",
      "the 15\n",
      "When 2\n",
      "this 2\n",
      "makes 2\n",
      "death , 2\n",
      ", to 3\n",
      "by 2\n",
      "life 2\n",
      "sleep 5\n",
      "bear 3\n",
      "of 15\n",
      ". 5\n"
     ]
    }
   ],
   "source": [
    "#Models behave much alike to Python dictionaries:\n",
    "for pattern, count in model.items():\n",
    "    print(pattern.tostring(classdecoder), count)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also query specific patterns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "How much sleep?\n",
      "5\n"
     ]
    }
   ],
   "source": [
    "\n",
    "querypattern = classencoder.buildpattern(\"sleep\")\n",
    "\n",
    "print(\"How much sleep?\")\n",
    "print(model[querypattern])\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "How much insolence?\n",
      "Nope, KeyError, no such pattern in model..\n"
     ]
    }
   ],
   "source": [
    "#Like dictionaries, unknown patterns will trigger a KeyError\n",
    "querypattern = classencoder.buildpattern(\"insolence\")\n",
    "\n",
    "print(\"How much insolence?\")\n",
    "try:\n",
    "    print(model[querypattern])\n",
    "except KeyError:\n",
    "    print(\"Nope, KeyError, no such pattern in model..\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can check whether a pattern is in a model in the usual pythonic fashion:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No insolence in model!\n"
     ]
    }
   ],
   "source": [
    "if querypattern in model:\n",
    "    print(\"Insolence in model!\")\n",
    "else:\n",
    "    print(\"No insolence in model!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rather than the absolute counts, we can get the frequency of a pattern *within its type and class*. For example the frequency of a bigram amongst all bigrams:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.07692307692307693\n"
     ]
    }
   ],
   "source": [
    "querypattern = classencoder.buildpattern(\"and the\")\n",
    "\n",
    "print(model.frequency(querypattern))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To analyse the distribution of occurrences, we can extract a histogram from our model as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2  occurrences by  34 patterns\n",
      "3  occurrences by  7 patterns\n",
      "4  occurrences by  2 patterns\n",
      "5  occurrences by  5 patterns\n",
      "6  occurrences by  1 patterns\n",
      "7  occurrences by  1 patterns\n",
      "9  occurrences by  1 patterns\n",
      "15  occurrences by  2 patterns\n",
      "36  occurrences by  1 patterns\n"
     ]
    }
   ],
   "source": [
    "for occurrencecount, frequency in model.histogram():\n",
    "    print(occurrencecount , \" occurrences by \", frequency , \"patterns\")\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have a model, we can save it to file, to reload later, loading is much faster than training:\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "patternmodelfile = TMPDIR + \"hamlet.colibri.patternmodel\"\n",
    "\n",
    "model.write(patternmodelfile)\n",
    "\n",
    "#and reload just to show we can:\n",
    "model = colibricore.UnindexedPatternModel(patternmodelfile, options)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unindexed models are much smaller in memory than indexed models, but their functionality is also limited. Let's take a look at *indexed models*. Indexed models keep a *forward index* to all locations in the original corpus where patterns occur. The references are 2-tuples in the form ``(sentence,token)``, where ``sentence`` is 1-indexed and ``token`` is 0-indexed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found  54  patterns:\n",
      "To die , to sleep (5, 5) (9, 5) \n",
      "die , to sleep (5, 6) (9, 6) \n",
      "To die , to (5, 5) (9, 5) \n",
      ", and the (7, 2) (18, 4) \n",
      ", to sleep (5, 7) (9, 7) \n",
      "die , to (5, 6) (9, 6) \n",
      "To die , (5, 5) (9, 5) \n",
      ", the (16, 3) (17, 5) \n",
      "we have (12, 1) (26, 7) \n",
      "and the (7, 3) (18, 5) \n",
      ", and (7, 2) (18, 4) \n",
      "sleep , (6, 5) (9, 9) (10, 1) \n",
      "to sleep (5, 8) (9, 8) \n",
      "die , (5, 6) (9, 6) \n",
      "To die (5, 5) (9, 5) \n",
      "to be (1, 5) (9, 1) \n",
      "be , (1, 1) (1, 6) \n",
      "all (28, 7) (34, 7) \n",
      "With (21, 0) (32, 0) \n",
      "make (20, 6) (28, 3) \n",
      "we (6, 9) (12, 1) (26, 7) (27, 5) \n",
      "That (8, 0) (14, 0) (19, 0) \n",
      "not (1, 4) (27, 7) \n",
      "No (6, 0) (25, 0) \n",
      "die (5, 6) (9, 6) \n",
      "And (5, 0) (26, 0) (29, 0) (31, 0) (33, 0) \n",
      "a (4, 5) (6, 4) (8, 6) (21, 1) (22, 5) \n",
      ", (1, 2) (1, 7) (3, 7) (4, 9) (5, 7) (6, 6) (7, 2) (9, 7) (9, 10) (10, 2) (10, 7) (10, 11) (11, 6) (11, 11) (12, 8) (15, 10) (16, 3) (16, 8) (17, 5) (17, 9) (18, 4) (19, 7) (21, 9) (22, 8) (23, 8) (24, 3) (25, 3) (25, 7) (26, 9) (28, 8) (30, 3) (30, 10) (31, 7) (32, 7) (33, 10) (34, 5) \n",
      "be (1, 1) (1, 6) (9, 2) \n",
      "is (1, 9) (8, 2) \n",
      "that (1, 8) (11, 2) (23, 1) (27, 4) \n",
      "would (15, 2) (21, 5) \n",
      "in (2, 3) (11, 1) (34, 6) \n",
      "to (1, 5) (2, 6) (4, 1) (5, 8) (6, 7) (9, 1) (9, 8) (10, 4) (27, 2) \n",
      "and (3, 2) (6, 2) (7, 3) (15, 6) (18, 5) (22, 2) (31, 5) \n",
      "end (5, 3) (6, 10) \n",
      "For (11, 0) (15, 0) \n",
      "To (1, 0) (5, 5) (9, 5) (10, 0) (22, 0) \n",
      "The (3, 0) (7, 0) (17, 0) (18, 0) (24, 0) (34, 0) \n",
      "have (12, 2) (26, 8) \n",
      "death (11, 5) (23, 7) \n",
      "us (13, 2) (26, 2) (28, 6) \n",
      "the (1, 10) (2, 4) (7, 4) (10, 9) (13, 6) (15, 4) (16, 4) (17, 6) (18, 6) (19, 4) (23, 2) (25, 5) (29, 2) (30, 5) (33, 2) \n",
      "When (12, 0) (20, 0) \n",
      "this (12, 5) (32, 1) \n",
      "makes (14, 1) (26, 1) \n",
      "death , (11, 5) (23, 7) \n",
      ", to (5, 7) (6, 6) (9, 7) \n",
      "by (5, 1) (6, 3) \n",
      "life (14, 6) (22, 7) \n",
      "sleep (5, 9) (6, 5) (9, 9) (10, 1) (11, 3) \n",
      "bear (15, 3) (21, 8) (26, 4) \n",
      "of (3, 4) (4, 7) (11, 4) (14, 3) (15, 8) (17, 2) (18, 2) (19, 3) (23, 4) (27, 8) (28, 5) (29, 5) (30, 8) (31, 2) (33, 4) \n",
      ". (9, 4) (13, 4) (27, 9) (33, 6) (34, 3) \n"
     ]
    }
   ],
   "source": [
    "#Set the options\n",
    "options = colibricore.PatternModelOptions(mintokens=2,maxlength=8)\n",
    "\n",
    "#Instantiate an empty indexed model \n",
    "model = colibricore.IndexedPatternModel()\n",
    "\n",
    "#Train it on our corpus file (class-encoded data, not plain text)\n",
    "model.train(corpusfile, options)\n",
    "\n",
    "print(\"Found \" , len(model), \" patterns:\")\n",
    "\n",
    "#Let's see what patterns are in our model (the order will be 'random')\n",
    "for pattern, indices in model.items():\n",
    "    print(pattern.tostring(classdecoder),end=\" \")\n",
    "    for index in indices:\n",
    "        print(index,end=\" \") #(sentence,token) tuple, sentences start with 1, tokens with 0\n",
    "    print()\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One interesting feature we can get from indexed models, is coverage information. This shows how many of the tokens in the original corpus data are covered by a particular pattern."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.012698412698412698\n"
     ]
    }
   ],
   "source": [
    "querypattern = classencoder.buildpattern(\"and the\")\n",
    "\n",
    "print(model.coverage(querypattern))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some numbers on the original corpus data can be obtained from the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total amount of tokens in the corpus data: 315\n",
      "Total amount of word types in the corpus data: 180\n"
     ]
    }
   ],
   "source": [
    "print(\"Total amount of tokens in the corpus data:\" , model.tokens() )\n",
    "print(\"Total amount of word types in the corpus data:\" , model.types() )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Whilst we have a *forward index*, we can also include a *reverse index* in our pattern model, which then allows us to look what patterns *begin* at a particular location. To build a reverse model, we explicitly need to instantiate an ``IndexedCorpus`` and pass it to the ``IndexedPatternModel`` constructor:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found  54  patterns:\n"
     ]
    }
   ],
   "source": [
    "#Set the options\n",
    "options = colibricore.PatternModelOptions(mintokens=2,maxlength=8)\n",
    "\n",
    "#Load the corpus\n",
    "corpus = colibricore.IndexedCorpus(corpusfile)\n",
    "\n",
    "#Instantiate an empty unindexed model \n",
    "model = colibricore.IndexedPatternModel(reverseindex=corpus)\n",
    "\n",
    "#Train it on our corpus file (class-encoded data, not plain text)\n",
    "model.train(corpusfile, options)\n",
    "\n",
    "print(\"Found \" , len(model), \" patterns:\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have a model with a reverse index, we can compute what patterns from our model *begin* at a certain index, expressed as a ``(sentence,token)`` tuple:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Patterns at (1,5): \n",
      "to be\n",
      "to\n"
     ]
    }
   ],
   "source": [
    "print(\"Patterns at (1,5): \")\n",
    "for pattern in model.getreverseindex( (1,5) ):\n",
    "    print(pattern.tostring(classdecoder))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also use this to easily get all patterns in a sentence that are in the model:\n",
    "    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Patterns in first sentence\n",
      "1 0  --  To\n",
      "1 1  --  be ,\n",
      "1 1  --  be\n",
      "1 2  --  ,\n",
      "1 4  --  not\n",
      "1 5  --  to be\n",
      "1 5  --  to\n",
      "1 6  --  be ,\n",
      "1 6  --  be\n",
      "1 7  --  ,\n",
      "1 8  --  that\n",
      "1 9  --  is\n",
      "1 10  --  the\n"
     ]
    }
   ],
   "source": [
    "print(\"Patterns in first sentence\")\n",
    "for (sentence, token), pattern in model.getreverseindex_bysentence(1):\n",
    "    print(sentence,token, \" -- \", pattern.tostring(classdecoder))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is easy to iterate over all indices in the reverse index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 0) (1, 1) (1, 2) (1, 3) (1, 4) (1, 5) (1, 6) (1, 7) (1, 8) (1, 9) (1, 10) (1, 11) (2, 0) (2, 1) (2, 2) (2, 3) (2, 4) (2, 5) (2, 6) (2, 7) (3, 0) (3, 1) (3, 2) (3, 3) (3, 4) (3, 5) (3, 6) (3, 7) (4, 0) (4, 1) (4, 2) (4, 3) (4, 4) (4, 5) (4, 6) (4, 7) (4, 8) (4, 9) (5, 0) (5, 1) (5, 2) (5, 3) (5, 4) (5, 5) (5, 6) (5, 7) (5, 8) (5, 9) (6, 0) (6, 1) (6, 2) (6, 3) (6, 4) (6, 5) (6, 6) (6, 7) (6, 8) (6, 9) (6, 10) (7, 0) (7, 1) (7, 2) (7, 3) (7, 4) (7, 5) (7, 6) (7, 7) (8, 0) (8, 1) (8, 2) (8, 3) (8, 4) (8, 5) (8, 6) (8, 7) (9, 0) (9, 1) (9, 2) (9, 3) (9, 4) (9, 5) (9, 6) (9, 7) (9, 8) (9, 9) (9, 10) (10, 0) (10, 1) (10, 2) (10, 3) (10, 4) (10, 5) (10, 6) (10, 7) (10, 8) (10, 9) (10, 10) (10, 11) (11, 0) (11, 1) (11, 2) (11, 3) (11, 4) (11, 5) (11, 6) (11, 7) (11, 8) (11, 9) (11, 10) (11, 11) (12, 0) (12, 1) (12, 2) (12, 3) (12, 4) (12, 5) (12, 6) (12, 7) (12, 8) (13, 0) (13, 1) (13, 2) (13, 3) (13, 4) (13, 5) (13, 6) (13, 7) (14, 0) (14, 1) (14, 2) (14, 3) (14, 4) (14, 5) (14, 6) (14, 7) (15, 0) (15, 1) (15, 2) (15, 3) (15, 4) (15, 5) (15, 6) (15, 7) (15, 8) (15, 9) (15, 10) (16, 0) (16, 1) (16, 2) (16, 3) (16, 4) (16, 5) (16, 6) (16, 7) (16, 8) (17, 0) (17, 1) (17, 2) (17, 3) (17, 4) (17, 5) (17, 6) (17, 7) (17, 8) (17, 9) (18, 0) (18, 1) (18, 2) (18, 3) (18, 4) (18, 5) (18, 6) (18, 7) (19, 0) (19, 1) (19, 2) (19, 3) (19, 4) (19, 5) (19, 6) (19, 7) (20, 0) (20, 1) (20, 2) (20, 3) (20, 4) (20, 5) (20, 6) (21, 0) (21, 1) (21, 2) (21, 3) (21, 4) (21, 5) (21, 6) (21, 7) (21, 8) (21, 9) (22, 0) (22, 1) (22, 2) (22, 3) (22, 4) (22, 5) (22, 6) (22, 7) (22, 8) (23, 0) (23, 1) (23, 2) (23, 3) (23, 4) (23, 5) (23, 6) (23, 7) (23, 8) (24, 0) (24, 1) (24, 2) (24, 3) (24, 4) (24, 5) (24, 6) (25, 0) (25, 1) (25, 2) (25, 3) (25, 4) (25, 5) (25, 6) (25, 7) (26, 0) (26, 1) (26, 2) (26, 3) (26, 4) (26, 5) (26, 6) (26, 7) (26, 8) (26, 9) (27, 0) (27, 1) (27, 2) (27, 3) (27, 4) (27, 5) (27, 6) (27, 7) (27, 8) (27, 9) (28, 0) (28, 1) (28, 2) (28, 3) (28, 4) (28, 5) (28, 6) (28, 7) (28, 8) (29, 0) (29, 1) (29, 2) (29, 3) (29, 4) (29, 5) (29, 6) (30, 0) (30, 1) (30, 2) (30, 3) (30, 4) (30, 5) (30, 6) (30, 7) (30, 8) (30, 9) (30, 10) (31, 0) (31, 1) (31, 2) (31, 3) (31, 4) (31, 5) (31, 6) (31, 7) (32, 0) (32, 1) (32, 2) (32, 3) (32, 4) (32, 5) (32, 6) (32, 7) (33, 0) (33, 1) (33, 2) (33, 3) (33, 4) (33, 5) (33, 6) (33, 7) (33, 8) (33, 9) (33, 10) (34, 0) (34, 1) (34, 2) (34, 3) (34, 4) (34, 5) (34, 6) (34, 7) (34, 8) (34, 9) "
     ]
    }
   ],
   "source": [
    "for ref in model.reverseindex():\n",
    "    print(ref, end=\" \") #ref is a (sentence,token) tuple"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, use ``model.items()`` to get the pattern as well, it will return ``(ref, pattern)`` tuples.\n",
    "\n",
    "The reverse index, as returned by the ``reverseindex()`` method, is just the same instance of ``IndexedCorpus``, which we passed to the constructor earlier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Skipgrams and flexgrams and relations between patterns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Skipgrams are n-grams with one or more *gaps* of a particular size. Flexgrams have a gap of dynamic size. Colibri Core can deal with both. Let's start with a new, and somewhat bigger, corpus. As the data in our previous example was too sparse to find any skipgrams. To that end, we will download Plato's *Republic*, this version is already tokenised and has one sentence per line, just as Colibri Core likes it: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloaded to /tmp/republic.txt\n"
     ]
    }
   ],
   "source": [
    "corpusfile_plato_plaintext = TMPDIR + \"republic.txt\"\n",
    "\n",
    "if PYTHON2:\n",
    "    from urllib import urlopen\n",
    "else:\n",
    "    from urllib.request import urlopen\n",
    "f = urlopen('http://lst.science.ru.nl/~proycon/republic.txt')\n",
    "with open(corpusfile_plato_plaintext,'wb') as of:\n",
    "    of.write(f.read())\n",
    "print(\"Downloaded to \" + corpusfile_plato_plaintext)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we create a class file and class encode the corpus, but because we may later on want to compare Shakespeare's Hamlet with Plato's Republic, we ensure that we use the same vocabulary. Note that it would have been better (more optimal classes, better compression) if we had built the original class encoder on both files right away, but you don't always have the luxury of foresight.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done\n"
     ]
    }
   ],
   "source": [
    "classfile_plato = TMPDIR + \"republic.colibri.cls\"\n",
    "corpusfile_plato  = TMPDIR + \"republic.colibri.dat\"\n",
    "\n",
    "#Build classes, re-using our classencoder from Hamlet! Let's reload it just for completion's sake\n",
    "classencoder = colibricore.ClassEncoder(TMPDIR + \"hamlet.colibri.cls\")\n",
    "\n",
    "#Now we will extend it by buildiing classes on Plato's data. If we had done this earlier, \n",
    "# we could have passed a list of filenames, ensuring more optimal encoding.\n",
    "classencoder.build(corpusfile_plato_plaintext)\n",
    "\n",
    "#Save new class file, this will be a superset of the original one.\n",
    "classencoder.save(classfile_plato)\n",
    "\n",
    "#Encode the corpus\n",
    "classencoder.encodefile(corpusfile_plato_plaintext, corpusfile_plato)\n",
    "\n",
    "#Load decoder because the old one will only handle Hamlet\n",
    "classdecoder = colibricore.ClassDecoder(classfile_plato)\n",
    "\n",
    "print(\"Done\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have a proper class file and encoded corpus, we can build an indexed pattern model with skipgrams. Skipgrams can only be build most efficiently using indexed models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training\n",
      "Found  84957  patterns:\n"
     ]
    }
   ],
   "source": [
    "#Set the options, doskipgrams=True is the key to enabling skipgrams\n",
    "options = colibricore.PatternModelOptions(mintokens=2,maxlength=8, doskipgrams=True)\n",
    "\n",
    "#Instantiate an empty indexed model \n",
    "corpus_plato = colibricore.IndexedCorpus(corpusfile_plato)\n",
    "model = colibricore.IndexedPatternModel(reverseindex=corpus_plato)\n",
    "\n",
    "#Train it on our corpus file (class-encoded data, not plain text)\n",
    "print(\"Training\")\n",
    "model.train(corpusfile_plato, options)\n",
    "\n",
    "print(\"Found \" , len(model), \" patterns:\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now how many of those patterns are skipgrams? We can find out ourselves by iterating over the patterns and checking their *category*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 8568  skipgrams\n"
     ]
    }
   ],
   "source": [
    "skipgrams = 0\n",
    "for pattern in model:\n",
    "    if pattern.category() == colibricore.Category.SKIPGRAM:\n",
    "        skipgrams += 1\n",
    "print(\"Found\",skipgrams,\" skipgrams\")\n",
    "        \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, it is much faster to do this using the built-in ``filter()`` method, which can also be used to filter patterns above a certain occurrence threshold, we can constrain it to a specific type such as skipgrams, and to a specific length (third argument, not used here):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 8568  skipgrams\n"
     ]
    }
   ],
   "source": [
    "skipgrams = 0\n",
    "for pattern, occurrencecount in model.filter(0,colibricore.Category.SKIPGRAM): #the first parameter is the occurrence threshold\n",
    "    skipgrams += 1\n",
    "print(\"Found\",skipgrams,\" skipgrams\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similar to ``filter()`` is the ``top()`` method, which we can use to extract the top patterns, let's get the top 20 of skipgrams. We will still need to relay it through a sorting function to get it in descending order:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the {*} of  --  3392\n",
      ", {*} the  --  1058\n",
      "of {*} ,  --  947\n",
      "the {*} ,  --  918\n",
      ", {*} said  --  857\n",
      "the {*} {*} the  --  805\n",
      ", {*} {*} ,  --  777\n",
      "the {*} of the  --  672\n",
      ", {*} ,  --  562\n",
      ", {*} is  --  536\n",
      ", {*} {*} the  --  496\n",
      ", {*} said ,  --  492\n",
      "the {*} .  --  454\n",
      "I {*} ,  --  445\n",
      "of {*} .  --  395\n",
      "the {*} and  --  391\n",
      "he {*} .  --  389\n",
      "of {*} and  --  388\n",
      ", {*} he  --  386\n",
      "of {*} {*} ,  --  379\n",
      ", {*} {*} .  --  372\n"
     ]
    }
   ],
   "source": [
    "for pattern, occurrencecount in sorted( model.top(20,colibricore.Category.SKIPGRAM), key=lambda x:x[1]*-1 ):\n",
    "    print(pattern.tostring(classdecoder), \" -- \", occurrencecount)\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each occurrence of `{*}` expresses a gap of exactly one word/token. We can create skipgrams from scratch using the same syntax with the classencoder, you can also use `{*2*}` for a gap covering two words etc..:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "skipgram = classencoder.buildpattern(\"To {*} or not to {*} is the question\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The consecutive non-gap parts of a skipgram can be obtained using the ``parts()`` method. The skipgram above consists of three parts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "To\n",
      "or not to\n",
      "is the question\n"
     ]
    }
   ],
   "source": [
    "for part in skipgram.parts():\n",
    "    print(part.tostring(classdecoder))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because an indexed model stores all the locations at which a pattern occurs, and a reverse index allows us to fill missing gaps, we can easily obtain all n-grams of which the skipgram is an abstraction:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "to the words of  -- occurring  2  times\n",
      "to the Republic of  -- occurring  2  times\n",
      "to the rest of  -- occurring  4  times\n",
      "to the level of  -- occurring  2  times\n",
      "to the plain of  -- occurring  2  times\n",
      "to the sight of  -- occurring  3  times\n",
      "to the devastation of  -- occurring  2  times\n",
      "to the terms of  -- occurring  2  times\n",
      "to the happiness of  -- occurring  2  times\n",
      "to the authority of  -- occurring  3  times\n",
      "to the tale of  -- occurring  2  times\n",
      "to the eye of  -- occurring  2  times\n",
      "to the voice of  -- occurring  2  times\n",
      "to the importance of  -- occurring  2  times\n",
      "to the question of  -- occurring  2  times\n",
      "to the end of  -- occurring  3  times\n",
      "to the mind of  -- occurring  2  times\n",
      "to the idea of  -- occurring  8  times\n",
      "to the law of  -- occurring  2  times\n",
      "to the conditions of  -- occurring  3  times\n",
      "to the class of  -- occurring  4  times\n",
      "to the contemplation of  -- occurring  5  times\n",
      "to the injury of  -- occurring  3  times\n",
      "to the wants of  -- occurring  2  times\n",
      "to the relation of  -- occurring  3  times\n",
      "to the good of  -- occurring  5  times\n",
      "to the vision of  -- occurring  2  times\n",
      "to the sum of  -- occurring  4  times\n",
      "to the knowledge of  -- occurring  3  times\n",
      "to the number of  -- occurring  2  times\n",
      "to the nature of  -- occurring  2  times\n"
     ]
    }
   ],
   "source": [
    "#let's pick a common skipgram from the data:\n",
    "skipgram = classencoder.buildpattern(\"to the {*} of\")\n",
    "\n",
    "for ngram, occurrences in model.getinstances(skipgram):\n",
    "    print(ngram.tostring(classdecoder), \" -- occurring \", occurrences, \" times\" )\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The reverse is also possible, given an ngram we can find what skipgrams are abstractions, or *templates* of it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "to {*} {*} of  -- occurring  2  times\n",
      "to the {*} of  -- occurring  2  times\n"
     ]
    }
   ],
   "source": [
    "#let's pick something that should be covered by a skipgram from the data:\n",
    "ngram = classencoder.buildpattern(\"to the question of\")\n",
    "\n",
    "for skipgram, occurrences in model.gettemplates(ngram):\n",
    "    print(skipgram.tostring(classdecoder), \" -- occurring \", occurrences, \" times\" )    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another trait of indexed pattern models is the ability to extract co-occurrence information using the ``getcooc()`` method. Let's see with what patterns the ngram \"the law of\" co-occurs more than five times (the second argument specifies this threshold, using it is always more efficient than doing a check on the variable ``occurrences`` that is returned):\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ",  -- occurring  99  times\n",
      "the  -- occurring  97  times\n",
      "and  -- occurring  75  times\n",
      "of  -- occurring  72  times\n",
      "to  -- occurring  45  times\n",
      "in  -- occurring  34  times\n",
      "is  -- occurring  34  times\n",
      ".  -- occurring  32  times\n",
      "a  -- occurring  30  times\n",
      "the {*} of  -- occurring  28  times\n",
      ";  -- occurring  24  times\n",
      ", and  -- occurring  24  times\n",
      "which  -- occurring  21  times\n",
      "or  -- occurring  19  times\n",
      "they  -- occurring  19  times\n",
      "their  -- occurring  18  times\n",
      "not  -- occurring  18  times\n",
      "by  -- occurring  16  times\n",
      "that  -- occurring  15  times\n",
      "be  -- occurring  15  times\n",
      "the {*} {*} the  -- occurring  14  times\n",
      "of the  -- occurring  14  times\n",
      "have  -- occurring  14  times\n",
      "them  -- occurring  13  times\n",
      "he  -- occurring  12  times\n",
      "are  -- occurring  12  times\n",
      "the {*} ,  -- occurring  12  times\n",
      "this  -- occurring  11  times\n",
      ", {*} {*} {*} ,  -- occurring  10  times\n",
      "the {*} {*} {*} {*} {*} {*} the  -- occurring  10  times\n",
      ", {*} the  -- occurring  9  times\n",
      "we  -- occurring  9  times\n",
      "; and  -- occurring  9  times\n",
      "as  -- occurring  9  times\n",
      ", the  -- occurring  9  times\n",
      "man  -- occurring  8  times\n",
      "in the  -- occurring  8  times\n",
      "and {*} ,  -- occurring  8  times\n",
      "and {*} {*} {*} the  -- occurring  8  times\n",
      "the {*} of the  -- occurring  8  times\n",
      "the {*} {*} {*} {*} ,  -- occurring  8  times\n",
      "but  -- occurring  8  times\n",
      ", {*} ,  -- occurring  8  times\n",
      "his  -- occurring  8  times\n",
      "of {*} {*} {*} {*} {*} the  -- occurring  8  times\n",
      "when  -- occurring  8  times\n",
      "these  -- occurring  8  times\n",
      "for  -- occurring  7  times\n",
      "is {*} {*} {*} the  -- occurring  7  times\n",
      "to be  -- occurring  7  times\n",
      ", {*} {*} {*} of  -- occurring  7  times\n",
      ", {*} {*} {*} the  -- occurring  7  times\n",
      "of a  -- occurring  7  times\n",
      ", {*} {*} {*} {*} and  -- occurring  7  times\n",
      "is not  -- occurring  7  times\n",
      "And  -- occurring  7  times\n",
      ", {*} {*} {*} to  -- occurring  7  times\n",
      "I  -- occurring  7  times\n",
      "the {*} {*} {*} and  -- occurring  6  times\n",
      "the {*} {*} {*} {*} and  -- occurring  6  times\n",
      "him  -- occurring  6  times\n",
      "to the  -- occurring  6  times\n",
      "--  -- occurring  6  times\n",
      ", {*} {*} {*} {*} {*} the  -- occurring  6  times\n",
      "with  -- occurring  6  times\n",
      "of {*} {*} {*} {*} ,  -- occurring  6  times\n",
      "of {*} and  -- occurring  6  times\n",
      "the {*} {*} {*} {*} {*} {*} ,  -- occurring  6  times\n",
      ", or  -- occurring  6  times\n",
      "and {*} {*} {*} and  -- occurring  6  times\n",
      "there  -- occurring  6  times\n",
      "of {*} {*} the  -- occurring  6  times\n",
      "of {*} ,  -- occurring  6  times\n",
      "in {*} {*} ,  -- occurring  6  times\n",
      "of these  -- occurring  6  times\n",
      "of {*} {*} ,  -- occurring  6  times\n",
      "by the  -- occurring  6  times\n",
      "her  -- occurring  5  times\n",
      ", {*} {*} to  -- occurring  5  times\n",
      "father  -- occurring  5  times\n",
      "who  -- occurring  5  times\n",
      "in which  -- occurring  5  times\n",
      "own  -- occurring  5  times\n",
      "the {*} {*} of  -- occurring  5  times\n",
      "what  -- occurring  5  times\n",
      "all  -- occurring  5  times\n",
      "they have  -- occurring  5  times\n",
      "the {*} {*} {*} {*} {*} the  -- occurring  5  times\n",
      ", {*} {*} {*} {*} {*} ,  -- occurring  5  times\n",
      ", {*} {*} of  -- occurring  5  times\n",
      ", {*} {*} and  -- occurring  5  times\n",
      "no  -- occurring  5  times\n",
      "of {*} {*} {*} ,  -- occurring  5  times\n",
      "state  -- occurring  5  times\n",
      "has  -- occurring  5  times\n",
      "the {*} {*} {*} ,  -- occurring  5  times\n",
      "one  -- occurring  5  times\n",
      ", {*} {*} the  -- occurring  5  times\n",
      "State  -- occurring  5  times\n",
      "'  -- occurring  5  times\n",
      ", {*} {*} {*} and  -- occurring  5  times\n",
      "of {*} {*} {*} {*} and  -- occurring  5  times\n",
      "is the  -- occurring  5  times\n",
      "the {*} {*} {*} {*} {*} ,  -- occurring  5  times\n"
     ]
    }
   ],
   "source": [
    "ngram = classencoder.buildpattern(\"the law\")\n",
    "\n",
    "for coocngram, occurrences in sorted( model.getcooc(ngram,5), key=lambda x: x[1] *-1): #let's sort the output too\n",
    "    print(coocngram.tostring(classdecoder), \" -- occurring \", occurrences, \" times\")\n",
    "        \n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are also specific methods for extracting co-occurrences left or right of the pattern: ``getleftcooc()`` and ``getrightcooc()``. Other relationships can be extracted in an identical fashion:\n",
    "\n",
    " * **``getleftneighbours``**``(pattern,threshold=0,category=0,size=0)`` -- returns the neighbours to the immediate left of a pattern (threshold, category and size are constraints which are set to 0 by default)\n",
    " * **``getrightneighbours``**``(pattern,threshold=0,category=0,size=0)``-- returns the neighbours to the immediate right of a pattern\n",
    " * **``getsubchildren``**``(pattern,threshold=0,category=0,size=0)``-- returns patterns that are a subpart (subsumed by) the specified\n",
    " * **``getsubparents``**``(pattern,threshold=0,category=0,size=0)``-- the reverse of the above, returns patterns which subsume the specified patterns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to skipgrams, Colibri Core core also supports *flexgrams*. Whereas the gaps in skipgrams are of a predefined size, in flexgrams they are by definition variable. A gap in a flexgram is represented as `{**}`. All of the existing functions that work on skipgrams, including the methods to extract relationsships, should also work on flexgrams.\n",
    "\n",
    "\n",
    "Flexgrams can be computed in two ways, but only on indexed pattern models:\n",
    "* by abstracting from the skipgrams in the model using the ``computeflexgrams_fromskipgrams()`` method.\n",
    "* by co-occurrence based on *normalised mutual pointwise information*. In this case a flexgram will have only one gap. Use the ``computeflexgrams_fromcooc(threshold)`` method for this.\n",
    "\n",
    "You have to explicitly choose one of these methods. An example of the first strategy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found  6880  flexgrams\n"
     ]
    }
   ],
   "source": [
    "#Set the options, doskipgrams=True is the key to enabling skipgrams\n",
    "options = colibricore.PatternModelOptions(mintokens=2,maxlength=8, doskipgrams=True)\n",
    "\n",
    "#Instantiate an empty indexed model \n",
    "corpus_plato = colibricore.IndexedCorpus(corpusfile_plato)\n",
    "flexmodel = colibricore.IndexedPatternModel(reverseindex=corpus_plato)\n",
    "\n",
    "#Train it on our corpus file (class-encoded data, not plain text)\n",
    "flexmodel.train(corpusfile_plato, options)\n",
    "\n",
    "#compute the flexgrams\n",
    "found = flexmodel.computeflexgrams_fromskipgrams()\n",
    "\n",
    "print(\"Found \" , str(found), \" flexgrams\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparing pattern models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pattern Models can be used in a train/test paradigm. You can create a Pattern Model on the training corpus and then generated a Pattern Model on the test corpus **constrained** by the training model. This allows you to test what patterns from the training corpus also occur in the test corpus, and how often. Statistics on these two differing counts can provide insight into how much corpora differ.\n",
    "\n",
    "We already saw the *coverage* metric previously, when applied to a train/test scenario it measures the number or ratio of tokens in the test corpus covered by patterns found during training. Let's perform such a comparison.\n",
    "\n",
    "We made a Pattern Model on Plato's Republic and we have a small excerpt from Hamlet. Let's use the former as training and the letter as test."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When doing any kind of comparison, it is absolutely crucial that you make sure the training and test data are class encoded with the same classes. The best method for this is to build the class files for all data in advance. In the previous class encoding example we saw ``classencoder.build()`` which does nothing more than provide us with a shortcut to call ``classencoder.processcorpus()`` followed by ``classencoder.buildclasses()``. To process multiple corpora, we do this ourselves:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Encoded  11540  classes, well done!\n"
     ]
    }
   ],
   "source": [
    "classfile2 = TMPDIR + \"platoandhamlet.colibri.cls\"\n",
    "\n",
    "#Instantiate class encoder\n",
    "classencoder2 = colibricore.ClassEncoder()\n",
    "\n",
    "#Build classes\n",
    "classencoder2.processcorpus(corpusfile_plato_plaintext)\n",
    "classencoder2.processcorpus(corpusfile_plaintext)\n",
    "classencoder2.buildclasses()\n",
    "\n",
    "#Save class file\n",
    "classencoder2.save(classfile2)\n",
    "\n",
    "print(\"Encoded \", len(classencoder2), \" classes, well done!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is important to realise that the Class Encoder we just built  (``classencoder2``) is now not compatible with the earlier class encoder used for previous examples! \n",
    "\n",
    "Often, however, you do not have all data available in advance. You may add a different test set later on, long after training. The way to make sure you have a proper class encoding is to extend your original class encoding. Rather than using the class encoder we just build, let us opt for that method, as this will keep all the classes we already had for the training data (Plato's Republic). This we do by calling the ``encodefile()`` method with two extra arguments set to True, indicating respectively that unknown words are allowed, and that unknown words are automatically added the the class encoding. If the second boolean is set to False, all unknown words would be encoded by one single class reserved for unknown words.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Class encoder has  11544  classes prior to extension\n",
      "Class encoder has  11544  classes after extension\n"
     ]
    }
   ],
   "source": [
    "print(\"Class encoder has \", len(classencoder), \" classes prior to extension\")\n",
    "\n",
    "testcorpusfile = TMPDIR + \"hamlet_test.colibri.dat\" #this will be the encoded test corpus file\n",
    "classencoder.encodefile(corpusfile_plaintext, testcorpusfile, True, True)\n",
    "\n",
    "classfile_test = TMPDIR + \"platoplushamlet.colibri.cls\"\n",
    "classencoder.save(classfile_test)\n",
    "\n",
    "print(\"Class encoder has \", len(classencoder), \" classes after extension\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do note that this method of encoding is not optimal, only encoding everything in one go ensures the smallest possible memory footprint."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We already created a pattern model on the training data in one of our earlier steps (called ``model``), to create our test model we *train* a constrained model on the test set, this model is constrained by the training model we made earlier. This will result in a new pattern model. The nomenclature may be a bit confusing at first. We simply do all this by instantiating a new model and calling the ``train()`` method and passing the contraining model as the last argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#Set the options\n",
    "options = colibricore.PatternModelOptions(mintokens=2,maxlength=8)\n",
    "\n",
    "#Instantiate an empty indexed model \n",
    "testmodel = colibricore.IndexedPatternModel()\n",
    "\n",
    "#Train it on our test corpus file (class-encoded data, not plain text)\n",
    "testmodel.train(testcorpusfile, options, model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have a test model (effectively the intersection an unconstrained model of the test corpus and the training model). We can see what patterns from the training corpus occur in the test corpus:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "When\n",
      "death ,\n",
      ", to\n",
      "by\n",
      "die\n",
      "No\n",
      "death\n",
      "and the\n",
      "of\n",
      "To\n",
      "end\n",
      "sleep ,\n",
      "And\n",
      "The\n",
      "For\n",
      "be\n",
      "in\n",
      ",\n",
      "a\n",
      "have\n",
      "sleep\n",
      "the\n",
      "bear\n",
      "not\n",
      "That\n",
      "is\n",
      ", and the\n",
      "to sleep\n",
      ", the\n",
      ".\n",
      "and\n",
      "to be\n",
      "die ,\n",
      "we have\n",
      "us\n",
      "makes\n",
      "be ,\n",
      "life\n",
      "this\n",
      "would\n",
      ", and\n",
      "to\n",
      "all\n",
      "we\n",
      "With\n",
      "that\n",
      "make\n"
     ]
    }
   ],
   "source": [
    "for pattern in testmodel:\n",
    "    print(pattern.tostring(classdecoder))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can inspect the differences between the counts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "When  ---  in training:  78 , in test:  2\n",
      "death ,  ---  in training:  14 , in test:  2\n",
      ", to  ---  in training:  131 , in test:  3\n",
      "by  ---  in training:  1324 , in test:  2\n",
      "die  ---  in training:  13 , in test:  2\n",
      "No  ---  in training:  85 , in test:  2\n",
      "death  ---  in training:  60 , in test:  2\n",
      "and the  ---  in training:  736 , in test:  2\n",
      "of  ---  in training:  10374 , in test:  15\n",
      "To  ---  in training:  88 , in test:  5\n",
      "end  ---  in training:  103 , in test:  2\n",
      "sleep ,  ---  in training:  3 , in test:  3\n",
      "And  ---  in training:  1071 , in test:  5\n",
      "The  ---  in training:  797 , in test:  6\n",
      "For  ---  in training:  169 , in test:  2\n",
      "be  ---  in training:  2930 , in test:  3\n",
      "in  ---  in training:  4319 , in test:  3\n",
      ",  ---  in training:  15352 , in test:  36\n",
      "a  ---  in training:  3924 , in test:  5\n",
      "have  ---  in training:  1485 , in test:  2\n",
      "sleep  ---  in training:  13 , in test:  5\n",
      "the  ---  in training:  14783 , in test:  15\n",
      "bear  ---  in training:  30 , in test:  3\n",
      "not  ---  in training:  2267 , in test:  2\n",
      "That  ---  in training:  197 , in test:  3\n",
      "is  ---  in training:  4619 , in test:  2\n",
      ", and the  ---  in training:  310 , in test:  2\n",
      "to sleep  ---  in training:  4 , in test:  2\n",
      ", the  ---  in training:  551 , in test:  2\n",
      ".  ---  in training:  7014 , in test:  5\n",
      "and  ---  in training:  8517 , in test:  7\n",
      "to be  ---  in training:  940 , in test:  2\n",
      "die ,  ---  in training:  3 , in test:  2\n",
      "we have  ---  in training:  138 , in test:  2\n",
      "us  ---  in training:  436 , in test:  3\n",
      "makes  ---  in training:  76 , in test:  2\n",
      "be ,  ---  in training:  23 , in test:  2\n",
      "life  ---  in training:  395 , in test:  2\n",
      "this  ---  in training:  936 , in test:  2\n",
      "would  ---  in training:  590 , in test:  2\n",
      ", and  ---  in training:  2899 , in test:  2\n",
      "to  ---  in training:  5917 , in test:  9\n",
      "all  ---  in training:  867 , in test:  2\n",
      "we  ---  in training:  1314 , in test:  4\n",
      "With  ---  in training:  9 , in test:  2\n",
      "that  ---  in training:  2830 , in test:  4\n",
      "make  ---  in training:  224 , in test:  2\n"
     ]
    }
   ],
   "source": [
    "for pattern in testmodel:\n",
    "    print(pattern.tostring(classdecoder), \" ---  in training: \", model.occurrencecount(pattern), \", in test: \", testmodel.occurrencecount(pattern)   )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This isn't so informative unless we apply some normalisation, so let's get the coverage instead:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "When  ---  in training:  0.0003101058733257265 , in test:  7.94148712287863e-06\n",
      "death ,  ---  in training:  0.00011132005709128642 , in test:  1.588297424575726e-05\n",
      ", to  ---  in training:  0.0010416376770684657 , in test:  2.382446136863589e-05\n",
      "by  ---  in training:  0.005263848413887972 , in test:  7.94148712287863e-06\n",
      "die  ---  in training:  5.168431222095441e-05 , in test:  7.94148712287863e-06\n",
      "No  ---  in training:  0.0003379358875985481 , in test:  7.94148712287863e-06\n",
      "death  ---  in training:  0.00023854297948132806 , in test:  7.94148712287863e-06\n",
      "and the  ---  in training:  0.005852254429941915 , in test:  1.588297424575726e-05\n",
      "of  ---  in training:  0.04124408115232162 , in test:  5.9561153421589724e-05\n",
      "To  ---  in training:  0.00034986303657261447 , in test:  1.9853717807196577e-05\n",
      "end  ---  in training:  0.0004094987814429465 , in test:  7.94148712287863e-06\n",
      "sleep ,  ---  in training:  2.3854297948132806e-05 , in test:  2.382446136863589e-05\n",
      "And  ---  in training:  0.004257992183741705 , in test:  1.9853717807196577e-05\n",
      "The  ---  in training:  0.003168645910776974 , in test:  2.382446136863589e-05\n",
      "For  ---  in training:  0.0006718960588724073 , in test:  7.94148712287863e-06\n",
      "be  ---  in training:  0.011648848831338186 , in test:  1.1912230684317945e-05\n",
      "in  ---  in training:  0.01717111880633093 , in test:  1.1912230684317945e-05\n",
      ",  ---  in training:  0.06103519701662247 , in test:  0.00014294676821181534\n",
      "a  ---  in training:  0.015600710858078855 , in test:  1.9853717807196577e-05\n",
      "have  ---  in training:  0.005903938742162869 , in test:  7.94148712287863e-06\n",
      "sleep  ---  in training:  5.168431222095441e-05 , in test:  1.9853717807196577e-05\n",
      "the  ---  in training:  0.05877301442787454 , in test:  5.9561153421589724e-05\n",
      "bear  ---  in training:  0.00011927148974066403 , in test:  1.1912230684317945e-05\n",
      "not  ---  in training:  0.009012948908069512 , in test:  7.94148712287863e-06\n",
      "That  ---  in training:  0.0007832161159636937 , in test:  1.1912230684317945e-05\n",
      "is  ---  in training:  0.01836383370373757 , in test:  7.94148712287863e-06\n",
      ", and the  ---  in training:  0.003697416181960585 , in test:  2.382446136863589e-05\n",
      "to sleep  ---  in training:  3.1805730597510405e-05 , in test:  1.588297424575726e-05\n",
      ", the  ---  in training:  0.0043812393898070585 , in test:  1.588297424575726e-05\n",
      ".  ---  in training:  0.02788567430136725 , in test:  1.9853717807196577e-05\n",
      "and  ---  in training:  0.03386117593737452 , in test:  2.7795204930075206e-05\n",
      "to be  ---  in training:  0.0074743466904149455 , in test:  1.588297424575726e-05\n",
      "die ,  ---  in training:  2.3854297948132806e-05 , in test:  1.588297424575726e-05\n",
      "we have  ---  in training:  0.001097297705614109 , in test:  1.588297424575726e-05\n",
      "us  ---  in training:  0.0017334123175643172 , in test:  1.1912230684317945e-05\n",
      "makes  ---  in training:  0.0003021544406763489 , in test:  7.94148712287863e-06\n",
      "be ,  ---  in training:  0.00018288295093568484 , in test:  1.588297424575726e-05\n",
      "life  ---  in training:  0.0015704079482520763 , in test:  7.94148712287863e-06\n",
      "this  ---  in training:  0.0037212704799087174 , in test:  7.94148712287863e-06\n",
      "would  ---  in training:  0.0023456726315663925 , in test:  7.94148712287863e-06\n",
      ", and  ---  in training:  0.023051203250545667 , in test:  1.588297424575726e-05\n",
      "to  ---  in training:  0.023524313493183634 , in test:  3.5736692052953834e-05\n",
      "all  ---  in training:  0.0034469460535051905 , in test:  7.94148712287863e-06\n",
      "we  ---  in training:  0.005224091250641084 , in test:  1.588297424575726e-05\n",
      "With  ---  in training:  3.578144692219921e-05 , in test:  7.94148712287863e-06\n",
      "that  ---  in training:  0.011251277198869307 , in test:  1.588297424575726e-05\n",
      "make  ---  in training:  0.0008905604567302914 , in test:  7.94148712287863e-06\n"
     ]
    }
   ],
   "source": [
    "for pattern in testmodel:\n",
    "    print(pattern.tostring(classdecoder), \" ---  in training: \", model.coverage(pattern), \", in test: \", testmodel.coverage(pattern)   )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Particularly the total coverage may be an interesting metric for similarity accross of corpora, which we can compute as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.0006750264054446836\n"
     ]
    }
   ],
   "source": [
    "coverage = testmodel.totaltokensingroup() / testmodel.tokens()\n",
    "\n",
    "print(coverage)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get a more traditional frequency metric for a pattern, you have to be aware that the total that is used in normalisation is impacted by the fact that the model is constrained! It will not include any unseen n-grams, for that you'd need an unconstrained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Frequency in training: 2.1719535636328094e-05\n",
      "Frequency in test (constrained): 0.08333333333333333\n",
      "Coverage in test (constrained): 1.588297424575726e-05\n",
      "Frequency in test (unconstrained): 0.07692307692307693\n",
      "Coverage in test (unconstrained): 0.012698412698412698\n"
     ]
    }
   ],
   "source": [
    "sleep = classencoder.buildpattern(\"to sleep\")\n",
    "\n",
    "print(\"Frequency in training:\", model.frequency(sleep))\n",
    "\n",
    "print(\"Frequency in test (constrained):\", testmodel.frequency(sleep) )\n",
    "print(\"Coverage in test (constrained):\", testmodel.coverage(sleep) )\n",
    "\n",
    "fullmodel = colibricore.IndexedPatternModel()\n",
    "fullmodel.train(testcorpusfile, options)\n",
    "print(\"Frequency in test (unconstrained):\", fullmodel.frequency(sleep) )\n",
    "print(\"Coverage in test (unconstrained):\", fullmodel.coverage(sleep) )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### Efficiently finding specific patterns in corpus data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Constrained models can be used if you want to search for a limited set of specific patterns in corpus data without the need to compute a full pattern model on the data. You create a pattern model from a pattern list, a plain text file with one pattern per line. Then you can use this model as a constraint model on your actual corpus data (the test data) and extract only occurrences of the patterns you are interested in, and in so doing conserving a lot of memory.\n",
    "\n",
    "\n",
    "To do this most efficiently we are going to use in-place rebuilding, where we simply load the contraint model, reset any count information, and then we recompute the patterns anew on the test data instead, \n",
    "telling it to *constrain on itself*.\n",
    "\n",
    "First, however, we construct a patternlist file, containing the patterns we want to extract from Plato's Republic. The pattern list can include skipgrams and flexgrams as well:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "with open(TMPDIR + '/patternlist.txt','w', encoding='utf-8') as f:\n",
    "    f.write(u'irony\\n') #(the u'' syntax is so Python 2 works as well)\n",
    "    f.write(u'and all\\n')\n",
    "    f.write(u'one part of\\n')\n",
    "    f.write(u'to {*} the\\n') #skipgram\n",
    "    f.write(u'both {**} and\\n') #flexgram\n",
    "    \n",
    "\n",
    "#Load our existing class encoding that already cover's Plato's Republic (and Hamlet, though we don't need it)\n",
    "classencoder = colibricore.ClassEncoder(TMPDIR + \"/platoplushamlet.colibri.cls\")\n",
    "\n",
    "#Encode the patternlist, adding any new classes unseen classes to the encoder (none in this case though)\n",
    "classencoder.encodefile(TMPDIR+\"/patternlist.txt\",TMPDIR+\"/patternlist.colibri.dat\",True,True)\n",
    "\n",
    "#Save it\n",
    "classencoder.save(TMPDIR+\"/withpatternlist.colibri.cls\")\n",
    "#Load a class decoder\n",
    "classdecoder = colibricore.ClassDecoder(TMPDIR+\"/withpatternlist.colibri.cls\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we train an unindexed pattern model from our pattern list, by setting the ``dopatternperline`` option, which means our resulting model will only include exactly those patterns we specified in the list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of patterns in the model:  5\n"
     ]
    }
   ],
   "source": [
    "#Set the option to say we are dealing with a pattern list here\n",
    "options = colibricore.PatternModelOptions(dopatternperline=True)\n",
    "\n",
    "#Create an unindexed model for our patternlist, this will be the constraint model\n",
    "patternlistmodel = colibricore.UnindexedPatternModel()\n",
    "patternlistmodel.train(TMPDIR+'/patternlist.colibri.dat',options)\n",
    "\n",
    "patternlistmodel.write(TMPDIR+\"/patternlist.colibri.patternmodel\")\n",
    "\n",
    "print(\"Number of patterns in the model: \", len(patternlistmodel))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final step is to load our constraint model, and train it on the test data with ``doreset=True`` and using the model as its own contraint model. We are going to build an indexed model. It is possible to load unindexed models as indexed, but since you have no indices you automatically lose all counts. As we would explicitly force this with ``doreset=True``, this is of no concern. A reverse index is always required for this, so we load the corpus data into an ``IndexedCorpus``:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 5 patterns\n"
     ]
    }
   ],
   "source": [
    "testcorpus = colibricore.IndexedCorpus(corpusfile_plato)\n",
    "testmodel = colibricore.IndexedPatternModel(TMPDIR+\"/patternlist.colibri.patternmodel\", reverseindex=testcorpus)\n",
    "\n",
    "options = colibricore.PatternModelOptions(doreset=True,doskipgrams=True, mintokens=1)\n",
    "\n",
    "\n",
    "testmodel.train(\"\",options,testmodel)\n",
    "#Notes:\n",
    "#1 - No need to pass a filename as first parameter since we already have a testcorpus loaded     \n",
    "#2 - The 3rd parameter is our contraint model, which is our own model. We thus constrain on our own pre-loaded model\n",
    "\n",
    "\n",
    "print(\"Found \" + str(len(testmodel)) + \" patterns\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Iterate over all patterns found and instantiate any skipgrams/flexgrams using ``getinstance()``:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "both {**} and: (730, 10)[both , and] (2683, 31)[both from and] \n",
      "one part of: (101, 24) (1143, 56) (7644, 31) (7662, 56) (8987, 36) \n",
      "irony: (33, 7) (183, 6) (359, 4) (836, 13) (1061, 58) (1722, 97) (2811, 31) (4303, 25) \n",
      "to {*} the: (50, 5)[to represent the] (94, 3)[to determine the] (121, 17)[to express the] (135, 26)[to bear the] (159, 3)[to raise the] (177, 23)[to escape the] (187, 4)[to continue the] (271, 34)[to be the] (303, 13)[to be the] (312, 16)[to speak the] (334, 4)[to accept the] (342, 23)[to know the] (365, 23)[to admit the] (366, 15)[to understand the] (402, 14)[to all the] (404, 9)[to extend the] (432, 18)[to consider the] (456, 23)[to them the] (456, 31)[to be the] (482, 19)[to strengthen the] (483, 27)[to us the] (514, 19)[to which the] (517, 2)[to feel the] (525, 2)[to establish the] (555, 14)[to read the] (556, 3)[to construct the] (566, 21)[to attract the] (629, 14)[to trace the] (630, 30)[to save the] (655, 21)[to identify the] (725, 4)[to play the] (744, 3)[to affirm the] (749, 14)[to corrupt the] (751, 24)[to it the] (811, 6)[to confine the] (1014, 5)[to Apollo the] (1021, 6)[to us the] (1051, 28)[to be the] (1068, 14)[to make the] (1113, 2)[to discover the] (1120, 2)[to be the] (1134, 4)[to identify the] (1193, 11)[to explain the] (1214, 23)[to acknowledge the] (1225, 7)[to Plato the] (1231, 5)[to enumerate the] (1255, 4)[to have the] (1279, 17)[to be the] (1387, 5)[to describe the] (1418, 6)[to describe the] (1431, 34)[to distinguish the] (1464, 10)[to view the] (1485, 11)[to lose the] (1550, 2)[to make the] (1557, 18)[to take the] (1561, 1)[to others the] (1630, 15)[to be the] (1682, 13)[to another the] (1771, 48)[to be the] (1839, 16)[to draw the] (1926, 3)[to combine the] (1995, 15)[to analyse the] (2083, 17)[to lead the] (2092, 36)[to pay the] (2097, 25)[to all the] (2109, 15)[to restrain the] (2161, 4)[to depress the] (2172, 14)[to lighten the] (2184, 15)[to admit the] (2184, 34)[to be the] (2217, 15)[to repel the] (2295, 4)[to be the] (2446, 12)[to strengthen the] (2451, 13)[to degrade the] (2490, 5)[to estimate the] (2514, 18)[to be the] (2532, 5)[to be the] (2619, 6)[to say the] (2675, 33)[to refuse the] (2677, 37)[to avoid the] (2733, 30)[to afford the] (2770, 15)[to enlist the] (2771, 21)[to express the] (2848, 9)[to express the] (2849, 3)[to represent the] (2856, 25)[to connect the] (2865, 5)[to refuse the] (2865, 10)[to choose the] (2927, 20)[to prepare the] (2933, 9)[to unite the] (2949, 8)[to become the] (2953, 3)[to address the] (2960, 4)[to consider the] (2976, 7)[to oppose the] (3009, 2)[to use the] (3043, 17)[to be the] (3065, 6)[to be the] (3117, 20)[to prove the] (3132, 23)[to which the] (3146, 7)[to be the] (3227, 5)[to probe the] (3232, 22)[to make the] (3261, 16)[to be the] (3282, 8)[to ravish the] (3288, 12)[to view the] (3303, 11)[to govern the] (3310, 3)[to be the] (3314, 6)[to paraphrase the] (3331, 2)[to be the] (3359, 11)[to comprehend the] (3362, 17)[to modify the] (3363, 4)[to admit the] (3381, 8)[to him the] (3389, 17)[to them the] (3395, 16)[to be the] (3444, 2)[to find the] (3445, 4)[to recognize the] (3468, 22)[to distinguish the] (3497, 17)[to impress the] (3499, 6)[to train the] (3566, 7)[to be the] (3640, 10)[to be the] (3684, 12)[to be the] (3686, 2)[to be the] (3691, 5)[to maintain the] (3706, 8)[to be the] (3742, 8)[to include the] (3842, 24)[to be the] (3961, 4)[to be the] (3996, 4)[to doubt the] (4164, 1)[to them the] (4202, 6)[to be the] (4332, 24)[to pieces the] (4557, 3)[to determine the] (4650, 10)[to be the] (4755, 10)[to take the] (4775, 4)[to exceed the] (4851, 3)[to view the] (5031, 22)[to enjoy the] (5084, 3)[to be the] (5198, 11)[to disprove the] (5223, 10)[to let the] (5348, 1)[to mention the] (5438, 6)[to support the] (5500, 15)[to overtake the] (5987, 50)[to be the] (5987, 59)[to do the] (5996, 12)[to introduce the] (6042, 7)[to have the] (6089, 5)[to make the] (6291, 5)[to imitate the] (6343, 4)[to admit the] (6493, 9)[to discern the] (6649, 6)[to require the] (6764, 14)[to practise the] (6768, 10)[to stimulate the] (6839, 2)[to be the] (6903, 11)[to possess the] (6904, 12)[to possess the] (7232, 13)[to them the] (7256, 16)[to alter the] (7299, 6)[to make the] (7411, 34)[to take the] (7490, 7)[to be the] (7571, 9)[to be the] (7598, 4)[to be the] (7729, 9)[to be the] (8031, 3)[to tell the] (8064, 14)[to require the] (8109, 5)[to have the] (8173, 7)[to reach the] (8180, 11)[to have the] (8272, 6)[to have the] (8366, 18)[to consider the] (8498, 8)[to which the] (8503, 14)[to be the] (8586, 6)[to be the] (8588, 18)[to utter the] (8607, 7)[to be the] (8653, 4)[to mention the] (8747, 8)[to be the] (8803, 9)[to order the] (8900, 16)[to devastate the] (8934, 11)[to prove the] (8976, 13)[to show the] (9069, 18)[to distinguish the] \n",
      "and all: (43, 30) (136, 40) (445, 55) (540, 12) (970, 51) (1131, 30) (1437, 38) (1519, 9) (1573, 50) (1592, 25) (1735, 8) (1739, 15) (2102, 38) (2128, 10) (2538, 15) (2555, 12) (2942, 33) (2980, 7) (2988, 49) (3018, 9) (3622, 53) (3955, 94) (5422, 86) (5676, 29) (6299, 26) (6319, 44) (6331, 8) (6644, 101) (6674, 24) (7010, 10) (7021, 31) (7027, 107) (7048, 14) (7992, 31) (8653, 37) (9056, 24) (9374, 16) (9435, 87) (9892, 69) (10073, 7) (10861, 46) (10895, 7) (10994, 42) (11165, 54) (11332, 28) (11361, 46) (11361, 95) (11597, 7) (11730, 111) (12064, 17) (12090, 26) (12157, 13) (12623, 23) (12623, 42) (12939, 22) (12950, 58) (13058, 18) (13062, 25) (13356, 10) (13447, 24) (13488, 18) (13501, 69) (13593, 37) (13664, 7) (13779, 25) \n"
     ]
    }
   ],
   "source": [
    "\n",
    "for pattern, indices in testmodel.items():\n",
    "    print(pattern.tostring(classdecoder), end=\": \")\n",
    "    for index in indices:\n",
    "        if pattern.category() == colibricore.Category.NGRAM:\n",
    "            print(index, end=\" \")\n",
    "        else:\n",
    "            #let's find out the precise instance of the skipgram/flexgram\n",
    "            instance = testmodel.getinstance(index, pattern)\n",
    "            print(str(index) + \"[\" + instance.tostring(classdecoder) + \"]\", end=\" \")\n",
    "    print()\n",
    "    "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}