examples/qa/qa_deep_dive.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "99f717ef-4cba-4300-b258-0b1c248cb873",
   "metadata": {},
   "source": [
    "# QA Deep Dive\n",
    "\n",
    "In this QA Deep Dive notebook, we take a closer look at semantic ingestion, RAG, and retrieval, presenting the various customization options, and providing hints to help you make the most out of your QA application.\n",
    "\n",
    "For getting started with basic QA usage, check out [QA Quick Start](./qa_quick_start.ipynb).\n",
    "\n",
    "### Access required\n",
    "\n",
    "The content of this notebook requires access to Deep Search capabilities which are not\n",
    "available on the public access system.\n",
    "\n",
    "[Contact us](https://ds4sd.github.io) if you are interested in exploring\n",
    "these Deep Search capabilities.\n",
    "\n",
    "\n",
    "### GenAI Integration required\n",
    "\n",
    "When interacting with the virtual assistant, Deep Search requires a connection to a Generative AI API. Currently, we support connections to [watsonx.ai](https://www.ibm.com/products/watsonx-ai) or the IBM-internal GenAI platform BAM.\n",
    "\n",
    "Deep Search allows custom GenAI configurations for each project.\n",
    "In the following example you will require to work in a project which has such GenAI capabilities activated."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "256aef50-71a1-4278-9b22-17cb99a6566e",
   "metadata": {},
   "source": [
    "### Set notebook parameters\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "5b244bdd-1b52-41ff-b63e-9a203570d210",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from dotenv import load_dotenv\n",
    "from pydantic import TypeAdapter\n",
    "\n",
    "load_dotenv()\n",
    "\n",
    "PROFILE_NAME = os.environ.get(\"DS_NB_PROFILE\")  # profile to use; defaults to active one\n",
    "PROJ_KEY = os.environ[\"DS_NB_PROJ_KEY\"]  # project to use\n",
    "INDEX_KEY = os.environ[\"DS_NB_QA_IDX_KEY\"]\n",
    "DOC_HASH = os.environ.get(\"DS_NB_QA_DOC_HASH\")  # set only when targeting a specific doc\n",
    "QUESTION = os.environ[\"DS_NB_QUESTION\"]\n",
    "\n",
    "# whether to ingest incrementally:\n",
    "SKIP_INGESTED_DOCS = TypeAdapter(bool).validate_python(\n",
    "    os.environ.get(\"DS_NB_SKIP_INGESTED_DOCS\", True)\n",
    ")\n",
    "RETR_K = os.environ.get(\"DS_NB_RETR_K\", 3)  # number of search results to retrieve\n",
    "GEN_TIMEOUT = os.environ.get(\"DS_NB_GEN_TIMEOUT\", 10)  # generation timeout in seconds\n",
    "GEN_CTX_WIN_SIZE = os.environ.get(\"DS_NB_GEN_CTX_WIN_SIZE\")  # max gen ctx window size"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5269060-bb5f-4fe3-9b64-547202db6714",
   "metadata": {},
   "source": [
    "### Import example dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "5d236ea0-db1c-4171-8e11-cdd0bad69d66",
   "metadata": {},
   "outputs": [],
   "source": [
    "import rich\n",
    "from typing import Union\n",
    "from IPython.display import display, Markdown\n",
    "from deepsearch.cps.client.api import CpsApi\n",
    "from deepsearch.cps.client.components.documents import create_private_data_source\n",
    "from deepsearch.cps.client.components.elastic import (\n",
    "    ElasticDataCollectionSource,\n",
    "    ElasticProjectDataCollectionSource,\n",
    ")\n",
    "from deepsearch.cps.queries import RAGQuery, SemanticQuery\n",
    "from deepsearch.cps.queries.results import RAGResult, SearchResult, SearchResultItem"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77c2dac3",
   "metadata": {},
   "source": [
    "### Connect to Deep Search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "9c108432-a285-4c7b-a996-008ac3ff3d7a",
   "metadata": {},
   "outputs": [],
   "source": [
    "api = CpsApi.from_env(profile_name=PROFILE_NAME)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ef952e7",
   "metadata": {},
   "source": [
    "### Notebook utils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "aaf88811",
   "metadata": {},
   "outputs": [],
   "source": [
    "def render_provenance_url(\n",
    "    api: CpsApi,\n",
    "    coords: Union[ElasticDataCollectionSource, ElasticProjectDataCollectionSource],\n",
    "    retr_item: SearchResultItem,\n",
    "):\n",
    "    ## compute URL to the document in the Deep Search UI\n",
    "    item_index = int(retr_item.main_path[retr_item.main_path.rfind(\".\") + 1 :])\n",
    "    doc_url = api.documents.generate_url(\n",
    "        document_hash=retr_item.doc_hash,\n",
    "        data_source=coords,\n",
    "        item_index=item_index,\n",
    "    )\n",
    "    display(\n",
    "        Markdown(\n",
    "            f\"The provenance of the answer can be inspected on the [source document]({doc_url}).\"\n",
    "        )\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd011dd3",
   "metadata": {},
   "source": [
    "### Prepare data source\n",
    "\n",
    "All semantic operations, i.e. ingestion, RAG, and retrieval, require a *data source* for defining the docs to operate on.\n",
    "\n",
    "The cell below shows how to configure a *private* data source, i.e. for operating on a whole private collection or a given doc within one.\n",
    "- set `document_hash` only when targeting a specific doc; when targeting the whole private collection instead, omit it or set it to `None`\n",
    "- to use a *public* data source instead, switch to the commented code on the bottom of the cell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "152766b0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# configuring a private data source:\n",
    "data_source = create_private_data_source(\n",
    "    proj_key=PROJ_KEY,\n",
    "    index_key=INDEX_KEY,\n",
    "    document_hash=DOC_HASH,\n",
    ")\n",
    "\n",
    "# # configuring a public data source:\n",
    "# from deepsearch.cps.client.components.documents import create_public_data_source\n",
    "# data_source = create_public_data_source(\n",
    "#     elastic_id=\"default\",\n",
    "#     index_key=INDEX_KEY,\n",
    "#     document_hash=DOC_HASH,\n",
    "# )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35bbc30e",
   "metadata": {},
   "source": [
    "### Ingestion\n",
    "\n",
    "If your data source has not yet been semantically indexed, you can ingest it into the vector DB as shown below. Otherwise you can skip this step.\n",
    "\n",
    "Ingestion of already indexed docs is controlled via param `skip_ingested_docs`. \n",
    "\n",
    "Particularly when indexing whole collections, note that the larger the data source, the longer the ingestion duration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "a4dcad49",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/pva/work/github.com/DS4SD/deepsearch-examples/.venv/lib/python3.10/site-packages/pydantic/main.py:347: UserWarning: Pydantic serializer warnings:\n",
      "  Expected `list[str]` but got `_LiteralGenericAlias` - serialized value may not be as expected\n",
      "  return self.__pydantic_serializer__.to_python(\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'ing_out': {}}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "task = api.documents.semantic_ingest(\n",
    "    project=PROJ_KEY,\n",
    "    data_source=data_source,\n",
    "    skip_ingested_docs=SKIP_INGESTED_DOCS,\n",
    ")\n",
    "\n",
    "# wait for the ingestion task to finish\n",
    "api.tasks.wait_for(PROJ_KEY, task.task_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e796c83c",
   "metadata": {},
   "source": [
    "### RAG\n",
    "\n",
    "Besides the standard RAG usage shown in [QA Quick Start](./qa_quick_start.ipynb), `RAGQuery` has numerous additional parameters for customizing aspects of retrieval, generation, and overall RAG pipeline:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "5da78ce5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function RAGQuery in module deepsearch.cps.queries:\n",
      "\n",
      "RAGQuery(question: str, *, project: Union[str, deepsearch.cps.client.components.projects.Project], data_source: Union[deepsearch.cps.client.components.documents.PrivateDataDocumentSource, deepsearch.cps.client.components.documents.PrivateDataCollectionSource, deepsearch.cps.client.components.documents.PublicDataDocumentSource], retr_k: int = 10, rerank: bool = False, text_weight: typing.Annotated[float, FieldInfo(default=PydanticUndefined, ge=0.0, le=1.0, multiple_of=0.1, extra={'strict': True})] = 0.1, model_id: Optional[str] = None, prompt_template: Optional[str] = None, gen_params: Optional[Dict[str, Any]] = None, gen_ctx_extr_method: Literal['window', 'page'] = 'window', gen_ctx_window_size: int = 5000, gen_ctx_window_lead_weight: float = 0.5, return_prompt: bool = False, chunk_refs: Optional[List[deepsearch.cps.queries.results.ChunkRef]] = None, gen_timeout: Optional[float] = None) -> deepsearch.cps.client.queries.query.Query\n",
      "    Create a RAG query\n",
      "    \n",
      "    Args:\n",
      "        question (str): the natural-language query\n",
      "        project (Union[str, Project]): project to use\n",
      "        data_source (DataSource): the data source to query\n",
      "        retr_k (int, optional): num of items to retrieve; defaults to 10\n",
      "        rerank (bool, optional): whether to rerank retrieval results; defaults to False\n",
      "        text_weight (ConstrainedWeight, optional): lexical weight for hybrid search; allowed values: {0.0, 0.1, 0.2, ..., 1.0}; defaults to 0.1\n",
      "        model_id (str, optional): the LLM to use for generation; defaults to None, i.e. determined by system\n",
      "        prompt_template (str, optional): the prompt template to use; defaults to None, i.e. determined by system\n",
      "        gen_params (dict, optional): the generation params to send to the Gen AI platforms; defaults to None, i.e. determined by system\n",
      "        gen_ctx_extr_method (Literal[\"window\", \"page\"], optional): method for gen context extraction from document; defaults to \"window\"\n",
      "        gen_ctx_window_size (int, optional): (relevant only if gen_ctx_extr_method==\"window\") max chars to use for extracted gen context (actual extraction quantized on doc item level); defaults to 5000\n",
      "        gen_ctx_window_lead_weight (float, optional): (relevant only if gen_ctx_extr_method==\"window\") weight of leading text for distributing remaining window size after extracting the `main_path`; defaults to 0.5 (centered around `main_path`)\n",
      "        return_prompt (bool, optional): whether to return the instantiated prompt; defaults to False\n",
      "        chunk_refs (Optional[List[ChunkRef]], optional): list of explicit chunk references to use instead of performing retrieval; defaults to None (i.e. retrieval-mode)\n",
      "        gen_timeout (float, optional): timeout for LLM generation; defaults to None, i.e. determined by system\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(RAGQuery)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b835f6e6",
   "metadata": {},
   "source": [
    "For instance, below we set the `return_prompt` parameter to get the actual instantiated prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "f4db6844",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Where was the first European IBM lab located?\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Where was the first European IBM lab located?\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">RAGResult</span><span style=\"font-weight: bold\">(</span>\n",
       "    <span style=\"color: #808000; text-decoration-color: #808000\">answers</span>=<span style=\"font-weight: bold\">[</span>\n",
       "        <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">RAGAnswerItem</span><span style=\"font-weight: bold\">(</span>\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">answer</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'The first European IBM laboratory was located in Adliswil, Switzerland, near Zurich.'</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">grounding</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">RAGGroundingInfo</span><span style=\"font-weight: bold\">(</span>\n",
       "                <span style=\"color: #808000; text-decoration-color: #808000\">retr_items</span>=<span style=\"font-weight: bold\">[</span>\n",
       "                    <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">SearchResultItem</span><span style=\"font-weight: bold\">(</span>\n",
       "                        <span style=\"color: #808000; text-decoration-color: #808000\">doc_hash</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'b30bc667a324ae111d025526563b674a8d3fd869bc07c8fd204aa95b05d41f0c'</span>,\n",
       "                        <span style=\"color: #808000; text-decoration-color: #808000\">main_path</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'main-text.71'</span>,\n",
       "                        <span style=\"color: #808000; text-decoration-color: #808000\">path_group</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008000; text-decoration-color: #008000\">'main-text.69'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'main-text.71'</span><span style=\"font-weight: bold\">]</span>,\n",
       "                        <span style=\"color: #808000; text-decoration-color: #808000\">chunk</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'Switzerland\\nIn 1956, IBM opened their first European research laboratory in </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">Adliswil, Switzerland, near Zurich. The lab moved to its own campus in neighboring Rüschlikon in 1962. The Zurich </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">lab is staffed by a multicultural and interdisciplinary team of a few hundred permanent research staff members, </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">graduate students and post-doctoral fellows, representing about 45 nationalities. Collocated with the lab is a </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">Client Center (formerly the Industry Solutions Lab), an executive briefing facility demonstrating technology </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">prototypes and solutions.'</span>,\n",
       "                        <span style=\"color: #808000; text-decoration-color: #808000\">source_is_text</span>=<span style=\"color: #00ff00; text-decoration-color: #00ff00; font-style: italic\">True</span>\n",
       "                    <span style=\"font-weight: bold\">)</span>\n",
       "                <span style=\"font-weight: bold\">]</span>,\n",
       "                <span style=\"color: #808000; text-decoration-color: #808000\">gen_ctx_paths</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008000; text-decoration-color: #008000\">'main-text.71'</span><span style=\"font-weight: bold\">]</span>\n",
       "            <span style=\"font-weight: bold\">)</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">prompt</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'Context information is below.\\n---------------------\\nIn 1956, IBM opened their first European </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">research laboratory in Adliswil, Switzerland, near Zurich. The lab moved to its own campus in neighboring </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">Rüschlikon in 1962. The Zurich lab is staffed by a multicultural and interdisciplinary team of a few hundred </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">permanent research staff members, graduate students and post-doctoral fellows, representing about 45 nationalities.</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">Collocated with the lab is a Client Center (formerly the Industry Solutions Lab), an executive briefing facility </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">demonstrating technology prototypes and solutions.\\n---------------------\\nGiven the context information and not </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">prior knowledge, answer the query.\\nQuery: Where was the first European IBM lab located?\\nAnswer:\\n'</span>\n",
       "        <span style=\"font-weight: bold\">)</span>\n",
       "    <span style=\"font-weight: bold\">]</span>,\n",
       "    <span style=\"color: #808000; text-decoration-color: #808000\">search_result_items</span>=<span style=\"font-weight: bold\">[</span>\n",
       "        <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">SearchResultItem</span><span style=\"font-weight: bold\">(</span>\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">doc_hash</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'b30bc667a324ae111d025526563b674a8d3fd869bc07c8fd204aa95b05d41f0c'</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">main_path</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'main-text.71'</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">path_group</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008000; text-decoration-color: #008000\">'main-text.69'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'main-text.71'</span><span style=\"font-weight: bold\">]</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">chunk</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'Switzerland\\nIn 1956, IBM opened their first European research laboratory in Adliswil, </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">Switzerland, near Zurich. The lab moved to its own campus in neighboring Rüschlikon in 1962. The Zurich lab is </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">staffed by a multicultural and interdisciplinary team of a few hundred permanent research staff members, graduate </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">students and post-doctoral fellows, representing about 45 nationalities. Collocated with the lab is a Client Center</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">(formerly the Industry Solutions Lab), an executive briefing facility demonstrating technology prototypes and </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">solutions.'</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">source_is_text</span>=<span style=\"color: #00ff00; text-decoration-color: #00ff00; font-style: italic\">True</span>\n",
       "        <span style=\"font-weight: bold\">)</span>,\n",
       "        <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">SearchResultItem</span><span style=\"font-weight: bold\">(</span>\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">doc_hash</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'b30bc667a324ae111d025526563b674a8d3fd869bc07c8fd204aa95b05d41f0c'</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">main_path</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'main-text.70'</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">path_group</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008000; text-decoration-color: #008000\">'main-text.69'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'main-text.70'</span><span style=\"font-weight: bold\">]</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">chunk</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'Switzerland\\nIBM Research-Zurich (previously called IBM Zurich Research Laboratory, ZRL) is the </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">European branch of IBM Research. It was opened in 1956 and is located in Rüschlikon, near Zurich, Switzerland.'</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">source_is_text</span>=<span style=\"color: #00ff00; text-decoration-color: #00ff00; font-style: italic\">True</span>\n",
       "        <span style=\"font-weight: bold\">)</span>,\n",
       "        <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">SearchResultItem</span><span style=\"font-weight: bold\">(</span>\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">doc_hash</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'b30bc667a324ae111d025526563b674a8d3fd869bc07c8fd204aa95b05d41f0c'</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">main_path</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'main-text.8'</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">path_group</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008000; text-decoration-color: #008000\">'main-text.7'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'main-text.8'</span><span style=\"font-weight: bold\">]</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">chunk</span>=<span style=\"color: #008000; text-decoration-color: #008000\">\"History\\nThe roots of today's IBM Research began with the 1945 opening of the Watson Scientific </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">Computing Laboratory at Columbia University. $^{[4]}$ This was the first IBM laboratory devoted to pure science and</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">later expanded into additional IBM Research locations in Westchester County, New York, starting in the 1950s, </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">$^{[5][6]}$ including the Thomas J. Watson Research Center in 1961. [5][6]\"</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">source_is_text</span>=<span style=\"color: #00ff00; text-decoration-color: #00ff00; font-style: italic\">True</span>\n",
       "        <span style=\"font-weight: bold\">)</span>,\n",
       "        <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">SearchResultItem</span><span style=\"font-weight: bold\">(</span>\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">doc_hash</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'234bc5cf2c860d49574b0ff7191c354b7bbc11472a09972c8b6108c62375ffc3'</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">main_path</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'main-text.86'</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">path_group</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008000; text-decoration-color: #008000\">'main-text.85'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'main-text.86'</span><span style=\"font-weight: bold\">]</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">chunk</span>=<span style=\"color: #008000; text-decoration-color: #008000\">\"Research\\nResearch has been part of IBM since its founding, and its organized efforts trace </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">their roots back to 1945, when the Watson Scientific Computing Laboratory was founded at Columbia University in New</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">York City, converting a renovated fraternity house on Manhattan's West Side into IBM's first laboratory. Now, IBM </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">Research constitutes the largest industrial research organization in the world, with 12 labs on 6 continents. [161]</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">IBM Research is headquartered at the Thomas J. Watson Research Center in New York, and facilities include the </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">Almaden lab in California, Austin lab in Texas, Australia lab in Melbourne, Brazil lab in São Paulo and Rio de </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">Janeiro, China lab in Beijing and Shanghai, Ireland lab in Dublin, Haifa lab in Israel, India lab in Delhi and </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">Bangalore, Tokyo lab, Zurichlab and Africa lab in Nairobi.\"</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">source_is_text</span>=<span style=\"color: #00ff00; text-decoration-color: #00ff00; font-style: italic\">True</span>\n",
       "        <span style=\"font-weight: bold\">)</span>,\n",
       "        <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">SearchResultItem</span><span style=\"font-weight: bold\">(</span>\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">doc_hash</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'b30bc667a324ae111d025526563b674a8d3fd869bc07c8fd204aa95b05d41f0c'</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">main_path</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'main-text.62'</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">path_group</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008000; text-decoration-color: #008000\">'main-text.61'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'main-text.62'</span><span style=\"font-weight: bold\">]</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">chunk</span>=<span style=\"color: #008000; text-decoration-color: #008000\">\"Israel\\nIBM Research-Haifa, previously known as the Haifa Research Lab (HRL) was founded as a </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">small scientific center in 1972. $^{[55]}$ Since then, it has grown into a major lab that leads the development of </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">innovative technologies and solutions for the IBM corporation. The lab's offices are situated in three locations </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">across Israel: Haifa, Tel Aviv, and Beer Sheva.\"</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">source_is_text</span>=<span style=\"color: #00ff00; text-decoration-color: #00ff00; font-style: italic\">True</span>\n",
       "        <span style=\"font-weight: bold\">)</span>,\n",
       "        <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">SearchResultItem</span><span style=\"font-weight: bold\">(</span>\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">doc_hash</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'b30bc667a324ae111d025526563b674a8d3fd869bc07c8fd204aa95b05d41f0c'</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">main_path</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'main-text.30'</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">path_group</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008000; text-decoration-color: #008000\">'main-text.16'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'main-text.30'</span><span style=\"font-weight: bold\">]</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">chunk</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'Laboratories\\nHistoric research centers for IBM also include IBM La Gaude (Nice), the Cambridge </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">Scientific Center, the IBM New York Scientific Center, 330 North Wabash (Chicago), IBM Austin Research Laboratory, </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">and IBM Laboratory Vienna. [33]'</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">source_is_text</span>=<span style=\"color: #00ff00; text-decoration-color: #00ff00; font-style: italic\">True</span>\n",
       "        <span style=\"font-weight: bold\">)</span>\n",
       "    <span style=\"font-weight: bold\">]</span>\n",
       "<span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;35mRAGResult\u001b[0m\u001b[1m(\u001b[0m\n",
       "    \u001b[33manswers\u001b[0m=\u001b[1m[\u001b[0m\n",
       "        \u001b[1;35mRAGAnswerItem\u001b[0m\u001b[1m(\u001b[0m\n",
       "            \u001b[33manswer\u001b[0m=\u001b[32m'The first European IBM laboratory was located in Adliswil, Switzerland, near Zurich.'\u001b[0m,\n",
       "            \u001b[33mgrounding\u001b[0m=\u001b[1;35mRAGGroundingInfo\u001b[0m\u001b[1m(\u001b[0m\n",
       "                \u001b[33mretr_items\u001b[0m=\u001b[1m[\u001b[0m\n",
       "                    \u001b[1;35mSearchResultItem\u001b[0m\u001b[1m(\u001b[0m\n",
       "                        \u001b[33mdoc_hash\u001b[0m=\u001b[32m'b30bc667a324ae111d025526563b674a8d3fd869bc07c8fd204aa95b05d41f0c'\u001b[0m,\n",
       "                        \u001b[33mmain_path\u001b[0m=\u001b[32m'main-text.71'\u001b[0m,\n",
       "                        \u001b[33mpath_group\u001b[0m=\u001b[1m[\u001b[0m\u001b[32m'main-text.69'\u001b[0m, \u001b[32m'main-text.71'\u001b[0m\u001b[1m]\u001b[0m,\n",
       "                        \u001b[33mchunk\u001b[0m=\u001b[32m'Switzerland\\nIn 1956, IBM opened their first European research laboratory in \u001b[0m\n",
       "\u001b[32mAdliswil, Switzerland, near Zurich. The lab moved to its own campus in neighboring Rüschlikon in 1962. The Zurich \u001b[0m\n",
       "\u001b[32mlab is staffed by a multicultural and interdisciplinary team of a few hundred permanent research staff members, \u001b[0m\n",
       "\u001b[32mgraduate students and post-doctoral fellows, representing about 45 nationalities. Collocated with the lab is a \u001b[0m\n",
       "\u001b[32mClient Center \u001b[0m\u001b[32m(\u001b[0m\u001b[32mformerly the Industry Solutions Lab\u001b[0m\u001b[32m)\u001b[0m\u001b[32m, an executive briefing facility demonstrating technology \u001b[0m\n",
       "\u001b[32mprototypes and solutions.'\u001b[0m,\n",
       "                        \u001b[33msource_is_text\u001b[0m=\u001b[3;92mTrue\u001b[0m\n",
       "                    \u001b[1m)\u001b[0m\n",
       "                \u001b[1m]\u001b[0m,\n",
       "                \u001b[33mgen_ctx_paths\u001b[0m=\u001b[1m[\u001b[0m\u001b[32m'main-text.71'\u001b[0m\u001b[1m]\u001b[0m\n",
       "            \u001b[1m)\u001b[0m,\n",
       "            \u001b[33mprompt\u001b[0m=\u001b[32m'Context information is below.\\n---------------------\\nIn 1956, IBM opened their first European \u001b[0m\n",
       "\u001b[32mresearch laboratory in Adliswil, Switzerland, near Zurich. The lab moved to its own campus in neighboring \u001b[0m\n",
       "\u001b[32mRüschlikon in 1962. The Zurich lab is staffed by a multicultural and interdisciplinary team of a few hundred \u001b[0m\n",
       "\u001b[32mpermanent research staff members, graduate students and post-doctoral fellows, representing about 45 nationalities.\u001b[0m\n",
       "\u001b[32mCollocated with the lab is a Client Center \u001b[0m\u001b[32m(\u001b[0m\u001b[32mformerly the Industry Solutions Lab\u001b[0m\u001b[32m)\u001b[0m\u001b[32m, an executive briefing facility \u001b[0m\n",
       "\u001b[32mdemonstrating technology prototypes and solutions.\\n---------------------\\nGiven the context information and not \u001b[0m\n",
       "\u001b[32mprior knowledge, answer the query.\\nQuery: Where was the first European IBM lab located?\\nAnswer:\\n'\u001b[0m\n",
       "        \u001b[1m)\u001b[0m\n",
       "    \u001b[1m]\u001b[0m,\n",
       "    \u001b[33msearch_result_items\u001b[0m=\u001b[1m[\u001b[0m\n",
       "        \u001b[1;35mSearchResultItem\u001b[0m\u001b[1m(\u001b[0m\n",
       "            \u001b[33mdoc_hash\u001b[0m=\u001b[32m'b30bc667a324ae111d025526563b674a8d3fd869bc07c8fd204aa95b05d41f0c'\u001b[0m,\n",
       "            \u001b[33mmain_path\u001b[0m=\u001b[32m'main-text.71'\u001b[0m,\n",
       "            \u001b[33mpath_group\u001b[0m=\u001b[1m[\u001b[0m\u001b[32m'main-text.69'\u001b[0m, \u001b[32m'main-text.71'\u001b[0m\u001b[1m]\u001b[0m,\n",
       "            \u001b[33mchunk\u001b[0m=\u001b[32m'Switzerland\\nIn 1956, IBM opened their first European research laboratory in Adliswil, \u001b[0m\n",
       "\u001b[32mSwitzerland, near Zurich. The lab moved to its own campus in neighboring Rüschlikon in 1962. The Zurich lab is \u001b[0m\n",
       "\u001b[32mstaffed by a multicultural and interdisciplinary team of a few hundred permanent research staff members, graduate \u001b[0m\n",
       "\u001b[32mstudents and post-doctoral fellows, representing about 45 nationalities. Collocated with the lab is a Client Center\u001b[0m\n",
       "\u001b[32m(\u001b[0m\u001b[32mformerly the Industry Solutions Lab\u001b[0m\u001b[32m)\u001b[0m\u001b[32m, an executive briefing facility demonstrating technology prototypes and \u001b[0m\n",
       "\u001b[32msolutions.'\u001b[0m,\n",
       "            \u001b[33msource_is_text\u001b[0m=\u001b[3;92mTrue\u001b[0m\n",
       "        \u001b[1m)\u001b[0m,\n",
       "        \u001b[1;35mSearchResultItem\u001b[0m\u001b[1m(\u001b[0m\n",
       "            \u001b[33mdoc_hash\u001b[0m=\u001b[32m'b30bc667a324ae111d025526563b674a8d3fd869bc07c8fd204aa95b05d41f0c'\u001b[0m,\n",
       "            \u001b[33mmain_path\u001b[0m=\u001b[32m'main-text.70'\u001b[0m,\n",
       "            \u001b[33mpath_group\u001b[0m=\u001b[1m[\u001b[0m\u001b[32m'main-text.69'\u001b[0m, \u001b[32m'main-text.70'\u001b[0m\u001b[1m]\u001b[0m,\n",
       "            \u001b[33mchunk\u001b[0m=\u001b[32m'Switzerland\\nIBM Research-Zurich \u001b[0m\u001b[32m(\u001b[0m\u001b[32mpreviously called IBM Zurich Research Laboratory, ZRL\u001b[0m\u001b[32m)\u001b[0m\u001b[32m is the \u001b[0m\n",
       "\u001b[32mEuropean branch of IBM Research. It was opened in 1956 and is located in Rüschlikon, near Zurich, Switzerland.'\u001b[0m,\n",
       "            \u001b[33msource_is_text\u001b[0m=\u001b[3;92mTrue\u001b[0m\n",
       "        \u001b[1m)\u001b[0m,\n",
       "        \u001b[1;35mSearchResultItem\u001b[0m\u001b[1m(\u001b[0m\n",
       "            \u001b[33mdoc_hash\u001b[0m=\u001b[32m'b30bc667a324ae111d025526563b674a8d3fd869bc07c8fd204aa95b05d41f0c'\u001b[0m,\n",
       "            \u001b[33mmain_path\u001b[0m=\u001b[32m'main-text.8'\u001b[0m,\n",
       "            \u001b[33mpath_group\u001b[0m=\u001b[1m[\u001b[0m\u001b[32m'main-text.7'\u001b[0m, \u001b[32m'main-text.8'\u001b[0m\u001b[1m]\u001b[0m,\n",
       "            \u001b[33mchunk\u001b[0m=\u001b[32m\"History\u001b[0m\u001b[32m\\nThe roots of today's IBM Research began with the 1945 opening of the Watson Scientific \u001b[0m\n",
       "\u001b[32mComputing Laboratory at Columbia University. $^\u001b[0m\u001b[32m{\u001b[0m\u001b[32m[\u001b[0m\u001b[32m4\u001b[0m\u001b[32m]\u001b[0m\u001b[32m}\u001b[0m\u001b[32m$ This was the first IBM laboratory devoted to pure science and\u001b[0m\n",
       "\u001b[32mlater expanded into additional IBM Research locations in Westchester County, New York, starting in the 1950s, \u001b[0m\n",
       "\u001b[32m$^\u001b[0m\u001b[32m{\u001b[0m\u001b[32m[\u001b[0m\u001b[32m5\u001b[0m\u001b[32m]\u001b[0m\u001b[32m[\u001b[0m\u001b[32m6\u001b[0m\u001b[32m]\u001b[0m\u001b[32m}\u001b[0m\u001b[32m$ including the Thomas J. Watson Research Center in 1961. \u001b[0m\u001b[32m[\u001b[0m\u001b[32m5\u001b[0m\u001b[32m]\u001b[0m\u001b[32m[\u001b[0m\u001b[32m6\u001b[0m\u001b[32m]\u001b[0m\u001b[32m\"\u001b[0m,\n",
       "            \u001b[33msource_is_text\u001b[0m=\u001b[3;92mTrue\u001b[0m\n",
       "        \u001b[1m)\u001b[0m,\n",
       "        \u001b[1;35mSearchResultItem\u001b[0m\u001b[1m(\u001b[0m\n",
       "            \u001b[33mdoc_hash\u001b[0m=\u001b[32m'234bc5cf2c860d49574b0ff7191c354b7bbc11472a09972c8b6108c62375ffc3'\u001b[0m,\n",
       "            \u001b[33mmain_path\u001b[0m=\u001b[32m'main-text.86'\u001b[0m,\n",
       "            \u001b[33mpath_group\u001b[0m=\u001b[1m[\u001b[0m\u001b[32m'main-text.85'\u001b[0m, \u001b[32m'main-text.86'\u001b[0m\u001b[1m]\u001b[0m,\n",
       "            \u001b[33mchunk\u001b[0m=\u001b[32m\"Research\u001b[0m\u001b[32m\\nResearch has been part of IBM since its founding, and its organized efforts trace \u001b[0m\n",
       "\u001b[32mtheir roots back to 1945, when the Watson Scientific Computing Laboratory was founded at Columbia University in New\u001b[0m\n",
       "\u001b[32mYork City, converting a renovated fraternity house on Manhattan's West Side into IBM's first laboratory. Now, IBM \u001b[0m\n",
       "\u001b[32mResearch constitutes the largest industrial research organization in the world, with 12 labs on 6 continents. \u001b[0m\u001b[32m[\u001b[0m\u001b[32m161\u001b[0m\u001b[32m]\u001b[0m\n",
       "\u001b[32mIBM Research is headquartered at the Thomas J. Watson Research Center in New York, and facilities include the \u001b[0m\n",
       "\u001b[32mAlmaden lab in California, Austin lab in Texas, Australia lab in Melbourne, Brazil lab in São Paulo and Rio de \u001b[0m\n",
       "\u001b[32mJaneiro, China lab in Beijing and Shanghai, Ireland lab in Dublin, Haifa lab in Israel, India lab in Delhi and \u001b[0m\n",
       "\u001b[32mBangalore, Tokyo lab, Zurichlab and Africa lab in Nairobi.\"\u001b[0m,\n",
       "            \u001b[33msource_is_text\u001b[0m=\u001b[3;92mTrue\u001b[0m\n",
       "        \u001b[1m)\u001b[0m,\n",
       "        \u001b[1;35mSearchResultItem\u001b[0m\u001b[1m(\u001b[0m\n",
       "            \u001b[33mdoc_hash\u001b[0m=\u001b[32m'b30bc667a324ae111d025526563b674a8d3fd869bc07c8fd204aa95b05d41f0c'\u001b[0m,\n",
       "            \u001b[33mmain_path\u001b[0m=\u001b[32m'main-text.62'\u001b[0m,\n",
       "            \u001b[33mpath_group\u001b[0m=\u001b[1m[\u001b[0m\u001b[32m'main-text.61'\u001b[0m, \u001b[32m'main-text.62'\u001b[0m\u001b[1m]\u001b[0m,\n",
       "            \u001b[33mchunk\u001b[0m=\u001b[32m\"Israel\u001b[0m\u001b[32m\\nIBM Research-Haifa, previously known as the Haifa Research Lab \u001b[0m\u001b[32m(\u001b[0m\u001b[32mHRL\u001b[0m\u001b[32m)\u001b[0m\u001b[32m was founded as a \u001b[0m\n",
       "\u001b[32msmall scientific center in 1972. $^\u001b[0m\u001b[32m{\u001b[0m\u001b[32m[\u001b[0m\u001b[32m55\u001b[0m\u001b[32m]\u001b[0m\u001b[32m}\u001b[0m\u001b[32m$ Since then, it has grown into a major lab that leads the development of \u001b[0m\n",
       "\u001b[32minnovative technologies and solutions for the IBM corporation. The lab's offices are situated in three locations \u001b[0m\n",
       "\u001b[32macross Israel: Haifa, Tel Aviv, and Beer Sheva.\"\u001b[0m,\n",
       "            \u001b[33msource_is_text\u001b[0m=\u001b[3;92mTrue\u001b[0m\n",
       "        \u001b[1m)\u001b[0m,\n",
       "        \u001b[1;35mSearchResultItem\u001b[0m\u001b[1m(\u001b[0m\n",
       "            \u001b[33mdoc_hash\u001b[0m=\u001b[32m'b30bc667a324ae111d025526563b674a8d3fd869bc07c8fd204aa95b05d41f0c'\u001b[0m,\n",
       "            \u001b[33mmain_path\u001b[0m=\u001b[32m'main-text.30'\u001b[0m,\n",
       "            \u001b[33mpath_group\u001b[0m=\u001b[1m[\u001b[0m\u001b[32m'main-text.16'\u001b[0m, \u001b[32m'main-text.30'\u001b[0m\u001b[1m]\u001b[0m,\n",
       "            \u001b[33mchunk\u001b[0m=\u001b[32m'Laboratories\\nHistoric research centers for IBM also include IBM La Gaude \u001b[0m\u001b[32m(\u001b[0m\u001b[32mNice\u001b[0m\u001b[32m)\u001b[0m\u001b[32m, the Cambridge \u001b[0m\n",
       "\u001b[32mScientific Center, the IBM New York Scientific Center, 330 North Wabash \u001b[0m\u001b[32m(\u001b[0m\u001b[32mChicago\u001b[0m\u001b[32m)\u001b[0m\u001b[32m, IBM Austin Research Laboratory, \u001b[0m\n",
       "\u001b[32mand IBM Laboratory Vienna. \u001b[0m\u001b[32m[\u001b[0m\u001b[32m33\u001b[0m\u001b[32m]\u001b[0m\u001b[32m'\u001b[0m,\n",
       "            \u001b[33msource_is_text\u001b[0m=\u001b[3;92mTrue\u001b[0m\n",
       "        \u001b[1m)\u001b[0m\n",
       "    \u001b[1m]\u001b[0m\n",
       "\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "question_query = RAGQuery(\n",
    "    question=QUESTION,\n",
    "    project=PROJ_KEY,\n",
    "    data_source=data_source,\n",
    "    retr_k=RETR_K,\n",
    "    return_prompt=True,\n",
    "    gen_timeout=GEN_TIMEOUT,\n",
    "    gen_ctx_window_size=GEN_CTX_WIN_SIZE,\n",
    ")\n",
    "api_output = api.queries.run(question_query)\n",
    "rag_result = RAGResult.from_api_output(api_output)\n",
    "\n",
    "rich.print(QUESTION)\n",
    "rich.print(rag_result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "066b7945",
   "metadata": {},
   "source": [
    "Additionally, we can generate a provenance URL to the document in the Deep Search UI:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "18a7ede6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "The provenance of the answer can be inspected on the [source document](https://sds.app.accelerate.science/projects/b09ae7561a01dc7c4b0fd21a43bfd93d140766d1/library/private/6b70072911ad2794a3844dd44d1705a5ba37ca0b?search=JTdCJTIycHJpdmF0ZUNvbGxlY3Rpb24lMjIlM0ElMjI2YjcwMDcyOTExYWQyNzk0YTM4NDRkZDQ0ZDE3MDVhNWJhMzdjYTBiJTIyJTJDJTIydHlwZSUyMiUzQSUyMkRvY3VtZW50JTIyJTJDJTIyZXhwcmVzc2lvbiUyMiUzQSUyMmZpbGUtaW5mby5kb2N1bWVudC1oYXNoJTNBJTIwJTVDJTIyYjMwYmM2NjdhMzI0YWUxMTFkMDI1NTI2NTYzYjY3NGE4ZDNmZDg2OWJjMDdjOGZkMjA0YWE5NWIwNWQ0MWYwYyU1QyUyMiUyMiUyQyUyMmZpbHRlcnMlMjIlM0ElNUIlNUQlMkMlMjJzZWxlY3QlMjIlM0ElNUIlMjJfbmFtZSUyMiUyQyUyMmRlc2NyaXB0aW9uLmNvbGxlY3Rpb24lMjIlMkMlMjJwcm92JTIyJTJDJTIyZGVzY3JpcHRpb24udGl0bGUlMjIlMkMlMjJkZXNjcmlwdGlvbi5wdWJsaWNhdGlvbl9kYXRlJTIyJTJDJTIyZGVzY3JpcHRpb24udXJsX3JlZnMlMjIlNUQlMkMlMjJpdGVtSW5kZXglMjIlM0EwJTJDJTIycGFnZVNpemUlMjIlM0ExMCUyQyUyMnNlYXJjaEFmdGVySGlzdG9yeSUyMiUzQSU1QiU1RCUyQyUyMnZpZXdUeXBlJTIyJTNBJTIyc25pcHBldHMlMjIlMkMlMjJyZWNvcmRTZWxlY3Rpb24lMjIlM0ElN0IlMjJyZWNvcmQlMjIlM0ElN0IlMjJpZCUyMiUzQSUyMmIzMGJjNjY3YTMyNGFlMTExZDAyNTUyNjU2M2I2NzRhOGQzZmQ4NjliYzA3YzhmZDIwNGFhOTViMDVkNDFmMGMlMjIlN0QlMkMlMjJpdGVtSW5kZXglMjIlM0E3MSU3RCU3RA%3D%3D)."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "render_provenance_url(\n",
    "    api=api,\n",
    "    coords=data_source.source,\n",
    "    retr_item=rag_result.answers[0].grounding.retr_items[0],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0f98608",
   "metadata": {},
   "source": [
    "As shown below, we can also inspect the timing information of the query execution, including the time spent on each step of the pipeline:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "99fa6498",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">QueryTimings</span><span style=\"font-weight: bold\">(</span>\n",
       "    <span style=\"color: #808000; text-decoration-color: #808000\">overall</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.6126572471112013</span>,\n",
       "    <span style=\"color: #808000; text-decoration-color: #808000\">tasks</span>=<span style=\"font-weight: bold\">{</span>\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'QA'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TaskTimings</span><span style=\"font-weight: bold\">(</span>\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">overall</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.6123144757002592</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">details</span>=<span style=\"font-weight: bold\">{</span>\n",
       "                <span style=\"color: #008000; text-decoration-color: #008000\">'pipeline'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.5076748670544475</span>,\n",
       "                <span style=\"color: #008000; text-decoration-color: #008000\">'encode'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.02719842502847314</span>,\n",
       "                <span style=\"color: #008000; text-decoration-color: #008000\">'search'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.025227335980162024</span>,\n",
       "                <span style=\"color: #008000; text-decoration-color: #008000\">'rerank'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.0</span>,\n",
       "                <span style=\"color: #008000; text-decoration-color: #008000\">'extr_gen_ctx'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.03297079307958484</span>,\n",
       "                <span style=\"color: #008000; text-decoration-color: #008000\">'generate'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.42107926099561155</span>\n",
       "            <span style=\"font-weight: bold\">}</span>\n",
       "        <span style=\"font-weight: bold\">)</span>\n",
       "    <span style=\"font-weight: bold\">}</span>\n",
       "<span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;35mQueryTimings\u001b[0m\u001b[1m(\u001b[0m\n",
       "    \u001b[33moverall\u001b[0m=\u001b[1;36m0\u001b[0m\u001b[1;36m.6126572471112013\u001b[0m,\n",
       "    \u001b[33mtasks\u001b[0m=\u001b[1m{\u001b[0m\n",
       "        \u001b[32m'QA'\u001b[0m: \u001b[1;35mTaskTimings\u001b[0m\u001b[1m(\u001b[0m\n",
       "            \u001b[33moverall\u001b[0m=\u001b[1;36m0\u001b[0m\u001b[1;36m.6123144757002592\u001b[0m,\n",
       "            \u001b[33mdetails\u001b[0m=\u001b[1m{\u001b[0m\n",
       "                \u001b[32m'pipeline'\u001b[0m: \u001b[1;36m0.5076748670544475\u001b[0m,\n",
       "                \u001b[32m'encode'\u001b[0m: \u001b[1;36m0.02719842502847314\u001b[0m,\n",
       "                \u001b[32m'search'\u001b[0m: \u001b[1;36m0.025227335980162024\u001b[0m,\n",
       "                \u001b[32m'rerank'\u001b[0m: \u001b[1;36m0.0\u001b[0m,\n",
       "                \u001b[32m'extr_gen_ctx'\u001b[0m: \u001b[1;36m0.03297079307958484\u001b[0m,\n",
       "                \u001b[32m'generate'\u001b[0m: \u001b[1;36m0.42107926099561155\u001b[0m\n",
       "            \u001b[1m}\u001b[0m\n",
       "        \u001b[1m)\u001b[0m\n",
       "    \u001b[1m}\u001b[0m\n",
       "\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "rich.print(api_output.timings)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b86295f6",
   "metadata": {},
   "source": [
    "### Generating next-best answers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a275567",
   "metadata": {},
   "source": [
    "To generate next-best answers, we can make follow-up queries with respective `chunk_refs` based on a reference RAG result.\n",
    "\n",
    "For example, below we use the previous RAG result as reference and generate answers based on its second-best and third-best items."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "117eec49",
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import List\n",
    "\n",
    "from deepsearch.cps.queries.results import RAGAnswerItem\n",
    "\n",
    "reference_rag_result = rag_result\n",
    "\n",
    "answer_items: List[RAGAnswerItem] = []\n",
    "for i in range(1, 3):\n",
    "\n",
    "    chunk_refs = [reference_rag_result.search_result_items[i]]\n",
    "\n",
    "    question_query = RAGQuery(\n",
    "        question=QUESTION,\n",
    "        project=PROJ_KEY,\n",
    "        data_source=data_source,\n",
    "        retr_k=RETR_K,\n",
    "        return_prompt=True,\n",
    "        chunk_refs=chunk_refs,  # <==\n",
    "        gen_ctx_window_size=GEN_CTX_WIN_SIZE,\n",
    "        gen_timeout=GEN_TIMEOUT,\n",
    "    )\n",
    "    api_output = api.queries.run(question_query)\n",
    "    new_rag_result = RAGResult.from_api_output(api_output)\n",
    "    answer_items.append(new_rag_result.answers[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc337982",
   "metadata": {},
   "source": [
    "Here we print the resulting next-best answers in a table:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "ae92b1de",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                                         Next-best answers                                          </span>\n",
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\"> Prompt                                          </span>┃<span style=\"font-weight: bold\"> Answer                                         </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│<span style=\"color: #808000; text-decoration-color: #808000\"> Context information is below.                   </span>│<span style=\"color: #008000; text-decoration-color: #008000\"> The first European IBM lab was located in      </span>│\n",
       "│<span style=\"color: #808000; text-decoration-color: #808000\"> ---------------------                           </span>│<span style=\"color: #008000; text-decoration-color: #008000\"> Zurich, Switzerland, and opened in 1956. It is </span>│\n",
       "│<span style=\"color: #808000; text-decoration-color: #808000\"> IBM Research-Zurich (previously called IBM      </span>│<span style=\"color: #008000; text-decoration-color: #008000\"> currently known as IBM Research-Zurich.        </span>│\n",
       "│<span style=\"color: #808000; text-decoration-color: #808000\"> Zurich Research Laboratory, ZRL) is the         </span>│<span style=\"color: #008000; text-decoration-color: #008000\">                                                </span>│\n",
       "│<span style=\"color: #808000; text-decoration-color: #808000\"> European branch of IBM Research. It was opened  </span>│<span style=\"color: #008000; text-decoration-color: #008000\">                                                </span>│\n",
       "│<span style=\"color: #808000; text-decoration-color: #808000\"> in 1956 and is located in Rüschlikon, near      </span>│<span style=\"color: #008000; text-decoration-color: #008000\">                                                </span>│\n",
       "│<span style=\"color: #808000; text-decoration-color: #808000\"> Zurich, Switzerland.                            </span>│<span style=\"color: #008000; text-decoration-color: #008000\">                                                </span>│\n",
       "│<span style=\"color: #808000; text-decoration-color: #808000\"> ---------------------                           </span>│<span style=\"color: #008000; text-decoration-color: #008000\">                                                </span>│\n",
       "│<span style=\"color: #808000; text-decoration-color: #808000\"> Given the context information and not prior     </span>│<span style=\"color: #008000; text-decoration-color: #008000\">                                                </span>│\n",
       "│<span style=\"color: #808000; text-decoration-color: #808000\"> knowledge, answer the query.                    </span>│<span style=\"color: #008000; text-decoration-color: #008000\">                                                </span>│\n",
       "│<span style=\"color: #808000; text-decoration-color: #808000\"> Query: Where was the first European IBM lab     </span>│<span style=\"color: #008000; text-decoration-color: #008000\">                                                </span>│\n",
       "│<span style=\"color: #808000; text-decoration-color: #808000\"> located?                                        </span>│<span style=\"color: #008000; text-decoration-color: #008000\">                                                </span>│\n",
       "│<span style=\"color: #808000; text-decoration-color: #808000\"> Answer:                                         </span>│<span style=\"color: #008000; text-decoration-color: #008000\">                                                </span>│\n",
       "│<span style=\"color: #808000; text-decoration-color: #808000\">                                                 </span>│<span style=\"color: #008000; text-decoration-color: #008000\">                                                </span>│\n",
       "├─────────────────────────────────────────────────┼────────────────────────────────────────────────┤\n",
       "│<span style=\"color: #808000; text-decoration-color: #808000\"> Context information is below.                   </span>│<span style=\"color: #008000; text-decoration-color: #008000\"> The information does not directly state where  </span>│\n",
       "│<span style=\"color: #808000; text-decoration-color: #808000\"> ---------------------                           </span>│<span style=\"color: #008000; text-decoration-color: #008000\"> the first European IBM lab was located; it     </span>│\n",
       "│<span style=\"color: #808000; text-decoration-color: #808000\"> The roots of today's IBM Research began with    </span>│<span style=\"color: #008000; text-decoration-color: #008000\"> only mentions the Watson Scientific Computing  </span>│\n",
       "│<span style=\"color: #808000; text-decoration-color: #808000\"> the 1945 opening of the Watson Scientific       </span>│<span style=\"color: #008000; text-decoration-color: #008000\"> Laboratory at Columbia University as the first </span>│\n",
       "│<span style=\"color: #808000; text-decoration-color: #808000\"> Computing Laboratory at Columbia University.    </span>│<span style=\"color: #008000; text-decoration-color: #008000\"> IBM laboratory devoted to pure science.        </span>│\n",
       "│<span style=\"color: #808000; text-decoration-color: #808000\"> $^{[4]}$ This was the first IBM laboratory      </span>│<span style=\"color: #008000; text-decoration-color: #008000\"> However, it does mention the expansion of IBM  </span>│\n",
       "│<span style=\"color: #808000; text-decoration-color: #808000\"> devoted to pure science and later expanded into </span>│<span style=\"color: #008000; text-decoration-color: #008000\"> Research into locations in Westchester County, </span>│\n",
       "│<span style=\"color: #808000; text-decoration-color: #808000\"> additional IBM Research locations in            </span>│<span style=\"color: #008000; text-decoration-color: #008000\"> New York, in the 1950s. To find out where the  </span>│\n",
       "│<span style=\"color: #808000; text-decoration-color: #808000\"> Westchester County, New York, starting in the   </span>│<span style=\"color: #008000; text-decoration-color: #008000\"> first European IBM lab was located, one would  </span>│\n",
       "│<span style=\"color: #808000; text-decoration-color: #808000\"> 1950s, $^{[5][6]}$ including the Thomas J.      </span>│<span style=\"color: #008000; text-decoration-color: #008000\"> need additional information.                   </span>│\n",
       "│<span style=\"color: #808000; text-decoration-color: #808000\"> Watson Research Center in 1961. [5][6]          </span>│<span style=\"color: #008000; text-decoration-color: #008000\">                                                </span>│\n",
       "│<span style=\"color: #808000; text-decoration-color: #808000\"> ---------------------                           </span>│<span style=\"color: #008000; text-decoration-color: #008000\">                                                </span>│\n",
       "│<span style=\"color: #808000; text-decoration-color: #808000\"> Given the context information and not prior     </span>│<span style=\"color: #008000; text-decoration-color: #008000\">                                                </span>│\n",
       "│<span style=\"color: #808000; text-decoration-color: #808000\"> knowledge, answer the query.                    </span>│<span style=\"color: #008000; text-decoration-color: #008000\">                                                </span>│\n",
       "│<span style=\"color: #808000; text-decoration-color: #808000\"> Query: Where was the first European IBM lab     </span>│<span style=\"color: #008000; text-decoration-color: #008000\">                                                </span>│\n",
       "│<span style=\"color: #808000; text-decoration-color: #808000\"> located?                                        </span>│<span style=\"color: #008000; text-decoration-color: #008000\">                                                </span>│\n",
       "│<span style=\"color: #808000; text-decoration-color: #808000\"> Answer:                                         </span>│<span style=\"color: #008000; text-decoration-color: #008000\">                                                </span>│\n",
       "│<span style=\"color: #808000; text-decoration-color: #808000\">                                                 </span>│<span style=\"color: #008000; text-decoration-color: #008000\">                                                </span>│\n",
       "└─────────────────────────────────────────────────┴────────────────────────────────────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                                         Next-best answers                                          \u001b[0m\n",
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1mPrompt                                         \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mAnswer                                        \u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[33m \u001b[0m\u001b[33mContext information is below.                  \u001b[0m\u001b[33m \u001b[0m│\u001b[32m \u001b[0m\u001b[32mThe first European IBM lab was located in     \u001b[0m\u001b[32m \u001b[0m│\n",
       "│\u001b[33m \u001b[0m\u001b[33m---------------------                          \u001b[0m\u001b[33m \u001b[0m│\u001b[32m \u001b[0m\u001b[32mZurich, Switzerland, and opened in 1956. It is\u001b[0m\u001b[32m \u001b[0m│\n",
       "│\u001b[33m \u001b[0m\u001b[33mIBM Research-Zurich (previously called IBM     \u001b[0m\u001b[33m \u001b[0m│\u001b[32m \u001b[0m\u001b[32mcurrently known as IBM Research-Zurich.       \u001b[0m\u001b[32m \u001b[0m│\n",
       "│\u001b[33m \u001b[0m\u001b[33mZurich Research Laboratory, ZRL) is the        \u001b[0m\u001b[33m \u001b[0m│\u001b[32m                                                \u001b[0m│\n",
       "│\u001b[33m \u001b[0m\u001b[33mEuropean branch of IBM Research. It was opened \u001b[0m\u001b[33m \u001b[0m│\u001b[32m                                                \u001b[0m│\n",
       "│\u001b[33m \u001b[0m\u001b[33min 1956 and is located in Rüschlikon, near     \u001b[0m\u001b[33m \u001b[0m│\u001b[32m                                                \u001b[0m│\n",
       "│\u001b[33m \u001b[0m\u001b[33mZurich, Switzerland.                           \u001b[0m\u001b[33m \u001b[0m│\u001b[32m                                                \u001b[0m│\n",
       "│\u001b[33m \u001b[0m\u001b[33m---------------------                          \u001b[0m\u001b[33m \u001b[0m│\u001b[32m                                                \u001b[0m│\n",
       "│\u001b[33m \u001b[0m\u001b[33mGiven the context information and not prior    \u001b[0m\u001b[33m \u001b[0m│\u001b[32m                                                \u001b[0m│\n",
       "│\u001b[33m \u001b[0m\u001b[33mknowledge, answer the query.                   \u001b[0m\u001b[33m \u001b[0m│\u001b[32m                                                \u001b[0m│\n",
       "│\u001b[33m \u001b[0m\u001b[33mQuery: Where was the first European IBM lab    \u001b[0m\u001b[33m \u001b[0m│\u001b[32m                                                \u001b[0m│\n",
       "│\u001b[33m \u001b[0m\u001b[33mlocated?                                       \u001b[0m\u001b[33m \u001b[0m│\u001b[32m                                                \u001b[0m│\n",
       "│\u001b[33m \u001b[0m\u001b[33mAnswer:                                        \u001b[0m\u001b[33m \u001b[0m│\u001b[32m                                                \u001b[0m│\n",
       "│\u001b[33m \u001b[0m\u001b[33m                                               \u001b[0m\u001b[33m \u001b[0m│\u001b[32m                                                \u001b[0m│\n",
       "├─────────────────────────────────────────────────┼────────────────────────────────────────────────┤\n",
       "│\u001b[33m \u001b[0m\u001b[33mContext information is below.                  \u001b[0m\u001b[33m \u001b[0m│\u001b[32m \u001b[0m\u001b[32mThe information does not directly state where \u001b[0m\u001b[32m \u001b[0m│\n",
       "│\u001b[33m \u001b[0m\u001b[33m---------------------                          \u001b[0m\u001b[33m \u001b[0m│\u001b[32m \u001b[0m\u001b[32mthe first European IBM lab was located; it    \u001b[0m\u001b[32m \u001b[0m│\n",
       "│\u001b[33m \u001b[0m\u001b[33mThe roots of today's IBM Research began with   \u001b[0m\u001b[33m \u001b[0m│\u001b[32m \u001b[0m\u001b[32monly mentions the Watson Scientific Computing \u001b[0m\u001b[32m \u001b[0m│\n",
       "│\u001b[33m \u001b[0m\u001b[33mthe 1945 opening of the Watson Scientific      \u001b[0m\u001b[33m \u001b[0m│\u001b[32m \u001b[0m\u001b[32mLaboratory at Columbia University as the first\u001b[0m\u001b[32m \u001b[0m│\n",
       "│\u001b[33m \u001b[0m\u001b[33mComputing Laboratory at Columbia University.   \u001b[0m\u001b[33m \u001b[0m│\u001b[32m \u001b[0m\u001b[32mIBM laboratory devoted to pure science.       \u001b[0m\u001b[32m \u001b[0m│\n",
       "│\u001b[33m \u001b[0m\u001b[33m$^{[4]}$ This was the first IBM laboratory     \u001b[0m\u001b[33m \u001b[0m│\u001b[32m \u001b[0m\u001b[32mHowever, it does mention the expansion of IBM \u001b[0m\u001b[32m \u001b[0m│\n",
       "│\u001b[33m \u001b[0m\u001b[33mdevoted to pure science and later expanded into\u001b[0m\u001b[33m \u001b[0m│\u001b[32m \u001b[0m\u001b[32mResearch into locations in Westchester County,\u001b[0m\u001b[32m \u001b[0m│\n",
       "│\u001b[33m \u001b[0m\u001b[33madditional IBM Research locations in           \u001b[0m\u001b[33m \u001b[0m│\u001b[32m \u001b[0m\u001b[32mNew York, in the 1950s. To find out where the \u001b[0m\u001b[32m \u001b[0m│\n",
       "│\u001b[33m \u001b[0m\u001b[33mWestchester County, New York, starting in the  \u001b[0m\u001b[33m \u001b[0m│\u001b[32m \u001b[0m\u001b[32mfirst European IBM lab was located, one would \u001b[0m\u001b[32m \u001b[0m│\n",
       "│\u001b[33m \u001b[0m\u001b[33m1950s, $^{[5][6]}$ including the Thomas J.     \u001b[0m\u001b[33m \u001b[0m│\u001b[32m \u001b[0m\u001b[32mneed additional information.                  \u001b[0m\u001b[32m \u001b[0m│\n",
       "│\u001b[33m \u001b[0m\u001b[33mWatson Research Center in 1961. [5][6]         \u001b[0m\u001b[33m \u001b[0m│\u001b[32m                                                \u001b[0m│\n",
       "│\u001b[33m \u001b[0m\u001b[33m---------------------                          \u001b[0m\u001b[33m \u001b[0m│\u001b[32m                                                \u001b[0m│\n",
       "│\u001b[33m \u001b[0m\u001b[33mGiven the context information and not prior    \u001b[0m\u001b[33m \u001b[0m│\u001b[32m                                                \u001b[0m│\n",
       "│\u001b[33m \u001b[0m\u001b[33mknowledge, answer the query.                   \u001b[0m\u001b[33m \u001b[0m│\u001b[32m                                                \u001b[0m│\n",
       "│\u001b[33m \u001b[0m\u001b[33mQuery: Where was the first European IBM lab    \u001b[0m\u001b[33m \u001b[0m│\u001b[32m                                                \u001b[0m│\n",
       "│\u001b[33m \u001b[0m\u001b[33mlocated?                                       \u001b[0m\u001b[33m \u001b[0m│\u001b[32m                                                \u001b[0m│\n",
       "│\u001b[33m \u001b[0m\u001b[33mAnswer:                                        \u001b[0m\u001b[33m \u001b[0m│\u001b[32m                                                \u001b[0m│\n",
       "│\u001b[33m \u001b[0m\u001b[33m                                               \u001b[0m\u001b[33m \u001b[0m│\u001b[32m                                                \u001b[0m│\n",
       "└─────────────────────────────────────────────────┴────────────────────────────────────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from rich.console import Console\n",
    "from rich.table import Table\n",
    "\n",
    "table = Table(title=\"Next-best answers\", show_lines=True, width=100)\n",
    "\n",
    "table.add_column(\"Prompt\", style=\"yellow\")\n",
    "table.add_column(\"Answer\", style=\"green\")\n",
    "\n",
    "for answer_item in answer_items:\n",
    "    table.add_row(answer_item.prompt, answer_item.answer)\n",
    "\n",
    "console = Console()\n",
    "console.print(table)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61e1cca5",
   "metadata": {},
   "source": [
    "### Semantic retrieval\n",
    "\n",
    "Besides the standard semantic retrieval usage shown in [QA Quick Start](./qa_quick_start.ipynb), `SemanticQuery` has numerous additional parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "1598dbe8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function SemanticQuery in module deepsearch.cps.queries:\n",
      "\n",
      "SemanticQuery(question: str, *, project: Union[str, deepsearch.cps.client.components.projects.Project], data_source: Union[deepsearch.cps.client.components.documents.PrivateDataDocumentSource, deepsearch.cps.client.components.documents.PrivateDataCollectionSource, deepsearch.cps.client.components.documents.PublicDataDocumentSource], retr_k: int = 10, rerank: bool = False, text_weight: typing.Annotated[float, FieldInfo(default=PydanticUndefined, ge=0.0, le=1.0, multiple_of=0.1, extra={'strict': True})] = 0.1) -> deepsearch.cps.client.queries.query.Query\n",
      "    Create a semantic retrieval query\n",
      "    \n",
      "    Args:\n",
      "        question (str): the natural-language query\n",
      "        document_hash (str): hash of target document\n",
      "        project (Union[str, Project]): project to use\n",
      "        data_source (DataSource): the data source to query\n",
      "        retr_k (int, optional): num of items to retrieve; defaults to 10\n",
      "        rerank (bool, optional): whether to rerank retrieval results; defaults to False\n",
      "        text_weight (ConstrainedWeight, optional): lexical weight for hybrid search; allowed values: {0.0, 0.1, 0.2, ..., 1.0}; defaults to 0.1\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(SemanticQuery)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8c2e781",
   "metadata": {},
   "source": [
    "For instance, below we set:\n",
    "- `rerank` to `True` to rerank the retrieved chunks, and\n",
    "- `text_weight` to `0.8` to favor the lexical component of hybrid search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "22953ad3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Where was the first European IBM lab located?\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Where was the first European IBM lab located?\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">SearchResult</span><span style=\"font-weight: bold\">(</span>\n",
       "    <span style=\"color: #808000; text-decoration-color: #808000\">search_result_items</span>=<span style=\"font-weight: bold\">[</span>\n",
       "        <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">SearchResultItem</span><span style=\"font-weight: bold\">(</span>\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">doc_hash</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'b30bc667a324ae111d025526563b674a8d3fd869bc07c8fd204aa95b05d41f0c'</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">main_path</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'main-text.71'</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">path_group</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008000; text-decoration-color: #008000\">'main-text.69'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'main-text.71'</span><span style=\"font-weight: bold\">]</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">chunk</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'Switzerland\\nIn 1956, IBM opened their first European research laboratory in Adliswil, </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">Switzerland, near Zurich. The lab moved to its own campus in neighboring Rüschlikon in 1962. The Zurich lab is </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">staffed by a multicultural and interdisciplinary team of a few hundred permanent research staff members, graduate </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">students and post-doctoral fellows, representing about 45 nationalities. Collocated with the lab is a Client Center</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">(formerly the Industry Solutions Lab), an executive briefing facility demonstrating technology prototypes and </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">solutions.'</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">source_is_text</span>=<span style=\"color: #00ff00; text-decoration-color: #00ff00; font-style: italic\">True</span>\n",
       "        <span style=\"font-weight: bold\">)</span>,\n",
       "        <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">SearchResultItem</span><span style=\"font-weight: bold\">(</span>\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">doc_hash</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'234bc5cf2c860d49574b0ff7191c354b7bbc11472a09972c8b6108c62375ffc3'</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">main_path</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'main-text.86'</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">path_group</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008000; text-decoration-color: #008000\">'main-text.85'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'main-text.86'</span><span style=\"font-weight: bold\">]</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">chunk</span>=<span style=\"color: #008000; text-decoration-color: #008000\">\"Research\\nResearch has been part of IBM since its founding, and its organized efforts trace </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">their roots back to 1945, when the Watson Scientific Computing Laboratory was founded at Columbia University in New</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">York City, converting a renovated fraternity house on Manhattan's West Side into IBM's first laboratory. Now, IBM </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">Research constitutes the largest industrial research organization in the world, with 12 labs on 6 continents. [161]</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">IBM Research is headquartered at the Thomas J. Watson Research Center in New York, and facilities include the </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">Almaden lab in California, Austin lab in Texas, Australia lab in Melbourne, Brazil lab in São Paulo and Rio de </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">Janeiro, China lab in Beijing and Shanghai, Ireland lab in Dublin, Haifa lab in Israel, India lab in Delhi and </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">Bangalore, Tokyo lab, Zurichlab and Africa lab in Nairobi.\"</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">source_is_text</span>=<span style=\"color: #00ff00; text-decoration-color: #00ff00; font-style: italic\">True</span>\n",
       "        <span style=\"font-weight: bold\">)</span>,\n",
       "        <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">SearchResultItem</span><span style=\"font-weight: bold\">(</span>\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">doc_hash</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'b30bc667a324ae111d025526563b674a8d3fd869bc07c8fd204aa95b05d41f0c'</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">main_path</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'main-text.70'</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">path_group</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008000; text-decoration-color: #008000\">'main-text.69'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'main-text.70'</span><span style=\"font-weight: bold\">]</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">chunk</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'Switzerland\\nIBM Research-Zurich (previously called IBM Zurich Research Laboratory, ZRL) is the </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">European branch of IBM Research. It was opened in 1956 and is located in Rüschlikon, near Zurich, Switzerland.'</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">source_is_text</span>=<span style=\"color: #00ff00; text-decoration-color: #00ff00; font-style: italic\">True</span>\n",
       "        <span style=\"font-weight: bold\">)</span>,\n",
       "        <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">SearchResultItem</span><span style=\"font-weight: bold\">(</span>\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">doc_hash</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'b30bc667a324ae111d025526563b674a8d3fd869bc07c8fd204aa95b05d41f0c'</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">main_path</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'main-text.62'</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">path_group</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008000; text-decoration-color: #008000\">'main-text.61'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'main-text.62'</span><span style=\"font-weight: bold\">]</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">chunk</span>=<span style=\"color: #008000; text-decoration-color: #008000\">\"Israel\\nIBM Research-Haifa, previously known as the Haifa Research Lab (HRL) was founded as a </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">small scientific center in 1972. $^{[55]}$ Since then, it has grown into a major lab that leads the development of </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">innovative technologies and solutions for the IBM corporation. The lab's offices are situated in three locations </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">across Israel: Haifa, Tel Aviv, and Beer Sheva.\"</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">source_is_text</span>=<span style=\"color: #00ff00; text-decoration-color: #00ff00; font-style: italic\">True</span>\n",
       "        <span style=\"font-weight: bold\">)</span>,\n",
       "        <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">SearchResultItem</span><span style=\"font-weight: bold\">(</span>\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">doc_hash</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'b30bc667a324ae111d025526563b674a8d3fd869bc07c8fd204aa95b05d41f0c'</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">main_path</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'main-text.44'</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">path_group</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008000; text-decoration-color: #008000\">'main-text.43'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'main-text.44'</span><span style=\"font-weight: bold\">]</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">chunk</span>=<span style=\"color: #008000; text-decoration-color: #008000\">\"Australia\\nIBM Research-Australia is a research and development laboratory established by IBM </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">Research in 2009 in Melbourne. [39] It is involved in social media, interactive content, healthcare analytics and </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">services research, multimedia analytics, and genomics. The lab is headed by Vice President and Lab Director Joanna </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">Batstone. [40] It was to be the company's first laboratory combining research and development in a single </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">organisation. [41]\"</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">source_is_text</span>=<span style=\"color: #00ff00; text-decoration-color: #00ff00; font-style: italic\">True</span>\n",
       "        <span style=\"font-weight: bold\">)</span>,\n",
       "        <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">SearchResultItem</span><span style=\"font-weight: bold\">(</span>\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">doc_hash</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'fa7ce2f66a7a5e061813d36348425f81d9e7ebc23454d8e63359fb5e28ab8dd0'</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">main_path</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'main-text.1'</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">path_group</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008000; text-decoration-color: #008000\">'main-text.0'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'main-text.1'</span><span style=\"font-weight: bold\">]</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">chunk</span>=<span style=\"color: #008000; text-decoration-color: #008000\">\"Deep Blue (chess computer)\\nDeep Blue was a chess-playing expert system run on a unique </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">purpose-built IBM supercomputer. It was the first computer to win a game, and the first to win a match, against a </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">reigning world champion under regular time controls. Development began in 1985 at Carnegie Mellon University under </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">the name ChipTest. It then moved to IBM, where it was first renamed Deep Thought, then again in 1989 to Deep Blue. </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">It first played world champion Garry Kasparov in a six-game match in 1996, where it lost four games to two. It was </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">upgraded in 1997 and in a six-game rematch, it defeated Kasparov by winning two games and drawing three. Deep </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">Blue's victory is considered a milestone in the history of artificial intelligence and has been the subject of </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">several books and films.\"</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">source_is_text</span>=<span style=\"color: #00ff00; text-decoration-color: #00ff00; font-style: italic\">True</span>\n",
       "        <span style=\"font-weight: bold\">)</span>\n",
       "    <span style=\"font-weight: bold\">]</span>\n",
       "<span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;35mSearchResult\u001b[0m\u001b[1m(\u001b[0m\n",
       "    \u001b[33msearch_result_items\u001b[0m=\u001b[1m[\u001b[0m\n",
       "        \u001b[1;35mSearchResultItem\u001b[0m\u001b[1m(\u001b[0m\n",
       "            \u001b[33mdoc_hash\u001b[0m=\u001b[32m'b30bc667a324ae111d025526563b674a8d3fd869bc07c8fd204aa95b05d41f0c'\u001b[0m,\n",
       "            \u001b[33mmain_path\u001b[0m=\u001b[32m'main-text.71'\u001b[0m,\n",
       "            \u001b[33mpath_group\u001b[0m=\u001b[1m[\u001b[0m\u001b[32m'main-text.69'\u001b[0m, \u001b[32m'main-text.71'\u001b[0m\u001b[1m]\u001b[0m,\n",
       "            \u001b[33mchunk\u001b[0m=\u001b[32m'Switzerland\\nIn 1956, IBM opened their first European research laboratory in Adliswil, \u001b[0m\n",
       "\u001b[32mSwitzerland, near Zurich. The lab moved to its own campus in neighboring Rüschlikon in 1962. The Zurich lab is \u001b[0m\n",
       "\u001b[32mstaffed by a multicultural and interdisciplinary team of a few hundred permanent research staff members, graduate \u001b[0m\n",
       "\u001b[32mstudents and post-doctoral fellows, representing about 45 nationalities. Collocated with the lab is a Client Center\u001b[0m\n",
       "\u001b[32m(\u001b[0m\u001b[32mformerly the Industry Solutions Lab\u001b[0m\u001b[32m)\u001b[0m\u001b[32m, an executive briefing facility demonstrating technology prototypes and \u001b[0m\n",
       "\u001b[32msolutions.'\u001b[0m,\n",
       "            \u001b[33msource_is_text\u001b[0m=\u001b[3;92mTrue\u001b[0m\n",
       "        \u001b[1m)\u001b[0m,\n",
       "        \u001b[1;35mSearchResultItem\u001b[0m\u001b[1m(\u001b[0m\n",
       "            \u001b[33mdoc_hash\u001b[0m=\u001b[32m'234bc5cf2c860d49574b0ff7191c354b7bbc11472a09972c8b6108c62375ffc3'\u001b[0m,\n",
       "            \u001b[33mmain_path\u001b[0m=\u001b[32m'main-text.86'\u001b[0m,\n",
       "            \u001b[33mpath_group\u001b[0m=\u001b[1m[\u001b[0m\u001b[32m'main-text.85'\u001b[0m, \u001b[32m'main-text.86'\u001b[0m\u001b[1m]\u001b[0m,\n",
       "            \u001b[33mchunk\u001b[0m=\u001b[32m\"Research\u001b[0m\u001b[32m\\nResearch has been part of IBM since its founding, and its organized efforts trace \u001b[0m\n",
       "\u001b[32mtheir roots back to 1945, when the Watson Scientific Computing Laboratory was founded at Columbia University in New\u001b[0m\n",
       "\u001b[32mYork City, converting a renovated fraternity house on Manhattan's West Side into IBM's first laboratory. Now, IBM \u001b[0m\n",
       "\u001b[32mResearch constitutes the largest industrial research organization in the world, with 12 labs on 6 continents. \u001b[0m\u001b[32m[\u001b[0m\u001b[32m161\u001b[0m\u001b[32m]\u001b[0m\n",
       "\u001b[32mIBM Research is headquartered at the Thomas J. Watson Research Center in New York, and facilities include the \u001b[0m\n",
       "\u001b[32mAlmaden lab in California, Austin lab in Texas, Australia lab in Melbourne, Brazil lab in São Paulo and Rio de \u001b[0m\n",
       "\u001b[32mJaneiro, China lab in Beijing and Shanghai, Ireland lab in Dublin, Haifa lab in Israel, India lab in Delhi and \u001b[0m\n",
       "\u001b[32mBangalore, Tokyo lab, Zurichlab and Africa lab in Nairobi.\"\u001b[0m,\n",
       "            \u001b[33msource_is_text\u001b[0m=\u001b[3;92mTrue\u001b[0m\n",
       "        \u001b[1m)\u001b[0m,\n",
       "        \u001b[1;35mSearchResultItem\u001b[0m\u001b[1m(\u001b[0m\n",
       "            \u001b[33mdoc_hash\u001b[0m=\u001b[32m'b30bc667a324ae111d025526563b674a8d3fd869bc07c8fd204aa95b05d41f0c'\u001b[0m,\n",
       "            \u001b[33mmain_path\u001b[0m=\u001b[32m'main-text.70'\u001b[0m,\n",
       "            \u001b[33mpath_group\u001b[0m=\u001b[1m[\u001b[0m\u001b[32m'main-text.69'\u001b[0m, \u001b[32m'main-text.70'\u001b[0m\u001b[1m]\u001b[0m,\n",
       "            \u001b[33mchunk\u001b[0m=\u001b[32m'Switzerland\\nIBM Research-Zurich \u001b[0m\u001b[32m(\u001b[0m\u001b[32mpreviously called IBM Zurich Research Laboratory, ZRL\u001b[0m\u001b[32m)\u001b[0m\u001b[32m is the \u001b[0m\n",
       "\u001b[32mEuropean branch of IBM Research. It was opened in 1956 and is located in Rüschlikon, near Zurich, Switzerland.'\u001b[0m,\n",
       "            \u001b[33msource_is_text\u001b[0m=\u001b[3;92mTrue\u001b[0m\n",
       "        \u001b[1m)\u001b[0m,\n",
       "        \u001b[1;35mSearchResultItem\u001b[0m\u001b[1m(\u001b[0m\n",
       "            \u001b[33mdoc_hash\u001b[0m=\u001b[32m'b30bc667a324ae111d025526563b674a8d3fd869bc07c8fd204aa95b05d41f0c'\u001b[0m,\n",
       "            \u001b[33mmain_path\u001b[0m=\u001b[32m'main-text.62'\u001b[0m,\n",
       "            \u001b[33mpath_group\u001b[0m=\u001b[1m[\u001b[0m\u001b[32m'main-text.61'\u001b[0m, \u001b[32m'main-text.62'\u001b[0m\u001b[1m]\u001b[0m,\n",
       "            \u001b[33mchunk\u001b[0m=\u001b[32m\"Israel\u001b[0m\u001b[32m\\nIBM Research-Haifa, previously known as the Haifa Research Lab \u001b[0m\u001b[32m(\u001b[0m\u001b[32mHRL\u001b[0m\u001b[32m)\u001b[0m\u001b[32m was founded as a \u001b[0m\n",
       "\u001b[32msmall scientific center in 1972. $^\u001b[0m\u001b[32m{\u001b[0m\u001b[32m[\u001b[0m\u001b[32m55\u001b[0m\u001b[32m]\u001b[0m\u001b[32m}\u001b[0m\u001b[32m$ Since then, it has grown into a major lab that leads the development of \u001b[0m\n",
       "\u001b[32minnovative technologies and solutions for the IBM corporation. The lab's offices are situated in three locations \u001b[0m\n",
       "\u001b[32macross Israel: Haifa, Tel Aviv, and Beer Sheva.\"\u001b[0m,\n",
       "            \u001b[33msource_is_text\u001b[0m=\u001b[3;92mTrue\u001b[0m\n",
       "        \u001b[1m)\u001b[0m,\n",
       "        \u001b[1;35mSearchResultItem\u001b[0m\u001b[1m(\u001b[0m\n",
       "            \u001b[33mdoc_hash\u001b[0m=\u001b[32m'b30bc667a324ae111d025526563b674a8d3fd869bc07c8fd204aa95b05d41f0c'\u001b[0m,\n",
       "            \u001b[33mmain_path\u001b[0m=\u001b[32m'main-text.44'\u001b[0m,\n",
       "            \u001b[33mpath_group\u001b[0m=\u001b[1m[\u001b[0m\u001b[32m'main-text.43'\u001b[0m, \u001b[32m'main-text.44'\u001b[0m\u001b[1m]\u001b[0m,\n",
       "            \u001b[33mchunk\u001b[0m=\u001b[32m\"Australia\u001b[0m\u001b[32m\\nIBM Research-Australia is a research and development laboratory established by IBM \u001b[0m\n",
       "\u001b[32mResearch in 2009 in Melbourne. \u001b[0m\u001b[32m[\u001b[0m\u001b[32m39\u001b[0m\u001b[32m]\u001b[0m\u001b[32m It is involved in social media, interactive content, healthcare analytics and \u001b[0m\n",
       "\u001b[32mservices research, multimedia analytics, and genomics. The lab is headed by Vice President and Lab Director Joanna \u001b[0m\n",
       "\u001b[32mBatstone. \u001b[0m\u001b[32m[\u001b[0m\u001b[32m40\u001b[0m\u001b[32m]\u001b[0m\u001b[32m It was to be the company's first laboratory combining research and development in a single \u001b[0m\n",
       "\u001b[32morganisation. \u001b[0m\u001b[32m[\u001b[0m\u001b[32m41\u001b[0m\u001b[32m]\u001b[0m\u001b[32m\"\u001b[0m,\n",
       "            \u001b[33msource_is_text\u001b[0m=\u001b[3;92mTrue\u001b[0m\n",
       "        \u001b[1m)\u001b[0m,\n",
       "        \u001b[1;35mSearchResultItem\u001b[0m\u001b[1m(\u001b[0m\n",
       "            \u001b[33mdoc_hash\u001b[0m=\u001b[32m'fa7ce2f66a7a5e061813d36348425f81d9e7ebc23454d8e63359fb5e28ab8dd0'\u001b[0m,\n",
       "            \u001b[33mmain_path\u001b[0m=\u001b[32m'main-text.1'\u001b[0m,\n",
       "            \u001b[33mpath_group\u001b[0m=\u001b[1m[\u001b[0m\u001b[32m'main-text.0'\u001b[0m, \u001b[32m'main-text.1'\u001b[0m\u001b[1m]\u001b[0m,\n",
       "            \u001b[33mchunk\u001b[0m=\u001b[32m\"Deep\u001b[0m\u001b[32m Blue \u001b[0m\u001b[32m(\u001b[0m\u001b[32mchess computer\u001b[0m\u001b[32m)\u001b[0m\u001b[32m\\nDeep Blue was a chess-playing expert system run on a unique \u001b[0m\n",
       "\u001b[32mpurpose-built IBM supercomputer. It was the first computer to win a game, and the first to win a match, against a \u001b[0m\n",
       "\u001b[32mreigning world champion under regular time controls. Development began in 1985 at Carnegie Mellon University under \u001b[0m\n",
       "\u001b[32mthe name ChipTest. It then moved to IBM, where it was first renamed Deep Thought, then again in 1989 to Deep Blue. \u001b[0m\n",
       "\u001b[32mIt first played world champion Garry Kasparov in a six-game match in 1996, where it lost four games to two. It was \u001b[0m\n",
       "\u001b[32mupgraded in 1997 and in a six-game rematch, it defeated Kasparov by winning two games and drawing three. Deep \u001b[0m\n",
       "\u001b[32mBlue's victory is considered a milestone in the history of artificial intelligence and has been the subject of \u001b[0m\n",
       "\u001b[32mseveral books and films.\"\u001b[0m,\n",
       "            \u001b[33msource_is_text\u001b[0m=\u001b[3;92mTrue\u001b[0m\n",
       "        \u001b[1m)\u001b[0m\n",
       "    \u001b[1m]\u001b[0m\n",
       "\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# submit natural-language query on document\n",
    "question_query = SemanticQuery(\n",
    "    question=QUESTION,\n",
    "    project=PROJ_KEY,\n",
    "    data_source=data_source,\n",
    "    retr_k=RETR_K,\n",
    "    rerank=True,\n",
    "    text_weight=0.8,\n",
    ")\n",
    "api_output = api.queries.run(question_query)\n",
    "rag_result = SearchResult.from_api_output(api_output)\n",
    "\n",
    "rich.print(QUESTION)\n",
    "rich.print(rag_result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c383a49",
   "metadata": {},
   "source": [
    "Again here, we can see the time spent on each step of the pipeline:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "4b6ce608",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">QueryTimings</span><span style=\"font-weight: bold\">(</span>\n",
       "    <span style=\"color: #808000; text-decoration-color: #808000\">overall</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.1080491673201323</span>,\n",
       "    <span style=\"color: #808000; text-decoration-color: #808000\">tasks</span>=<span style=\"font-weight: bold\">{</span>\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'QA'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TaskTimings</span><span style=\"font-weight: bold\">(</span>\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">overall</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.1076935632154346</span>,\n",
       "            <span style=\"color: #808000; text-decoration-color: #808000\">details</span>=<span style=\"font-weight: bold\">{</span>\n",
       "                <span style=\"color: #008000; text-decoration-color: #008000\">'pipeline'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.79485984146595</span>,\n",
       "                <span style=\"color: #008000; text-decoration-color: #008000\">'encode'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.028432318940758705</span>,\n",
       "                <span style=\"color: #008000; text-decoration-color: #008000\">'search'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.023948708549141884</span>,\n",
       "                <span style=\"color: #008000; text-decoration-color: #008000\">'rerank'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.7413393948227167</span>\n",
       "            <span style=\"font-weight: bold\">}</span>\n",
       "        <span style=\"font-weight: bold\">)</span>\n",
       "    <span style=\"font-weight: bold\">}</span>\n",
       "<span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;35mQueryTimings\u001b[0m\u001b[1m(\u001b[0m\n",
       "    \u001b[33moverall\u001b[0m=\u001b[1;36m2\u001b[0m\u001b[1;36m.1080491673201323\u001b[0m,\n",
       "    \u001b[33mtasks\u001b[0m=\u001b[1m{\u001b[0m\n",
       "        \u001b[32m'QA'\u001b[0m: \u001b[1;35mTaskTimings\u001b[0m\u001b[1m(\u001b[0m\n",
       "            \u001b[33moverall\u001b[0m=\u001b[1;36m2\u001b[0m\u001b[1;36m.1076935632154346\u001b[0m,\n",
       "            \u001b[33mdetails\u001b[0m=\u001b[1m{\u001b[0m\n",
       "                \u001b[32m'pipeline'\u001b[0m: \u001b[1;36m1.79485984146595\u001b[0m,\n",
       "                \u001b[32m'encode'\u001b[0m: \u001b[1;36m0.028432318940758705\u001b[0m,\n",
       "                \u001b[32m'search'\u001b[0m: \u001b[1;36m0.023948708549141884\u001b[0m,\n",
       "                \u001b[32m'rerank'\u001b[0m: \u001b[1;36m1.7413393948227167\u001b[0m\n",
       "            \u001b[1m}\u001b[0m\n",
       "        \u001b[1m)\u001b[0m\n",
       "    \u001b[1m}\u001b[0m\n",
       "\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "rich.print(api_output.timings)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}