{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":637924634,"defaultBranch":"main","name":"ragas","ownerLogin":"explodinggradients","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2023-05-08T17:48:04.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/122604797?v=4","public":true,"private":false,"isOrgOwned":true},"refInfo":{"name":"","listCacheKey":"v0:1726636822.0","currentOid":""},"activityList":{"items":[{"before":"46761840e7e86001277c1b9117974bf8fc2250e1","after":"b845a62d0bddf786706e6176312cd7ee93c01ca4","ref":"refs/heads/main","pushedAt":"2024-09-20T04:04:40.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"shahules786","name":"ikka","path":"/shahules786","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/25312635?s=80&v=4"},"commit":{"message":"feat: Factual correctnes metric (#1334)","shortMessageHtmlLink":"feat: Factual correctnes metric (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2536560921\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1334\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1334/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1334\">#1334</a>)"}},{"before":"478abc15bf10c67fb0c3a58ee0ed4c5a0552df25","after":"46761840e7e86001277c1b9117974bf8fc2250e1","ref":"refs/heads/main","pushedAt":"2024-09-18T07:15:16.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"jjmachan","name":"Jithin James","path":"/jjmachan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5261489?s=80&v=4"},"commit":{"message":"feat: sql sementic equivalence (#1323)\n\n```python\r\nfrom ragas.metrics._datacompy_score import DataCompyScore\r\nfrom ragas.dataset_schema import SingleTurnSample\r\n\r\ndata1 = \"\"\"acct_id,dollar_amt,name,float_fld,date_fld\r\n10000001234,123.45,George Maharis,14530.1555,2017-01-01\r\n10000001235,0.45,Michael Bluth,1,2017-01-01\r\n10000001236,1345,George Bluth,,2017-01-01\r\n10000001237,123456,Bob Loblaw,345.12,2017-01-01\r\n10000001238,1.05,Lucille Bluth,,2017-01-01\r\n10000001238,1.05,Loose Seal Bluth,,2017-01-01\r\n\"\"\"\r\n\r\ndata2 = \"\"\"acct_id,dollar_amt,name,float_fld\r\n10000001234,123.4,George Michael Bluth,14530.155\r\n10000001235,0.45,Michael Bluth,\r\n10000001236,1345,George Bluth,1\r\n10000001237,123456,Robert Loblaw,345.12\r\n10000001238,1.05,Loose Seal Bluth,111\r\n\"\"\"\r\nsample = SingleTurnSample(response=data1, reference=data2)\r\nscorer = DataCompyScore(metric=\"precision\", mode='columns')\r\nawait scorer.single_turn_ascore(sample)\r\n```","shortMessageHtmlLink":"feat: sql sementic equivalence (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2528137266\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1323\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1323/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1323\">#1323</a>)"}},{"before":"9b2303b7882adbba930f421c6955e4528fb5ca9f","after":"478abc15bf10c67fb0c3a58ee0ed4c5a0552df25","ref":"refs/heads/main","pushedAt":"2024-09-18T06:12:12.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"shahules786","name":"ikka","path":"/shahules786","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/25312635?s=80&v=4"},"commit":{"message":"feat: tool call accuracy (#1311)\n\n```python\r\nfrom ragas.dataset_schema import  MultiTurnSample\r\nfrom ragas.messages import HumanMessage,AIMessage,ToolMessage,ToolCall\r\nfrom ragas.metrics._tool_call_accuracy import ToolCallAccuracy\r\n\r\n\r\nsample = MultiTurnSample(user_input=[\r\n    HumanMessage(content=\"Hey, book a table at the nearest best Chinese restaurant for 8:00pm\"),\r\n    AIMessage(content=\"Sure, let me find the best options for you.\", tool_calls=[\r\n        ToolCall(name=\"restaurant_search\", args={\"cuisine\": \"Asian\", \"time\": \"8:00pm\"})\r\n    ]),\r\n              ],\r\n    reference_tool_calls=[ToolCall(name=\"restaurant_book\", args={\"name\": \"Golden\", \"time\": \"8:00pm\"})\r\n])\r\n\r\nscorer = ToolCallAccuracy()\r\nawait metric.multi_turn_ascore(sample)\r\n```","shortMessageHtmlLink":"feat: tool call accuracy (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2525403960\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1311\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1311/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1311\">#1311</a>)"}},{"before":"21e06434f5035f41736c4545c35f17c6f78ab734","after":"9b2303b7882adbba930f421c6955e4528fb5ca9f","ref":"refs/heads/main","pushedAt":"2024-09-18T05:41:39.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"jjmachan","name":"Jithin James","path":"/jjmachan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5261489?s=80&v=4"},"commit":{"message":"feat: topic adherence metric (#1327)\n\nCo-authored-by: jjmachan <jamesjithin97@gmail.com>","shortMessageHtmlLink":"feat: topic adherence metric (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2531170994\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1327\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1327/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1327\">#1327</a>)"}},{"before":"502b336bfcf6e1962846b6844c612d0f41dd8a75","after":"21e06434f5035f41736c4545c35f17c6f78ab734","ref":"refs/heads/main","pushedAt":"2024-09-18T05:19:58.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"jjmachan","name":"Jithin James","path":"/jjmachan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5261489?s=80&v=4"},"commit":{"message":"fix: pin langchain_core to <3 (#1329)\n\nThe new langchain v0.3 will break the current usage of metrics. \r\n\r\nthe plan for actions is as follows\r\n1. for ragas<0.2 we will pin langchain_core to <0.3\r\n2. for ragas>0.2 we will directly depend on pydantic>=2\r\n\r\nfixes: #1328","shortMessageHtmlLink":"fix: pin langchain_core to &lt;3 (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2532713244\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1329\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1329/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1329\">#1329</a>)"}},{"before":"c40891bf168de3124c845b75af31ed557eb79709","after":"502b336bfcf6e1962846b6844c612d0f41dd8a75","ref":"refs/heads/main","pushedAt":"2024-09-18T02:33:58.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"jjmachan","name":"Jithin James","path":"/jjmachan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5261489?s=80&v=4"},"commit":{"message":"fix: column rename (#1324)","shortMessageHtmlLink":"fix: column rename (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2528243857\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1324\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1324/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1324\">#1324</a>)"}},{"before":"78b54c6dca6e14dea6d7412274f6601b26a7c175","after":"c40891bf168de3124c845b75af31ed557eb79709","ref":"refs/heads/main","pushedAt":"2024-09-17T06:41:05.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"jjmachan","name":"Jithin James","path":"/jjmachan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5261489?s=80&v=4"},"commit":{"message":"Update Opik notebook example and integration (#1295)\n\n# Opik integration update\r\n\r\nUpdated the Opik integration:\r\n1. Updated notebooks following the changes to expected inputs for metric\r\nfunctions\r\n2. Updated the OpikTracer following some changes in the Opik library\r\n\r\n---------\r\n\r\nCo-authored-by: Abby Morgan <86856445+anmorgan24@users.noreply.github.com>","shortMessageHtmlLink":"Update Opik notebook example and integration (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2522831453\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1295\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1295/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1295\">#1295</a>)"}},{"before":"78d3bea5e48d4ed1b70947bafc89ebbd837a0559","after":"78b54c6dca6e14dea6d7412274f6601b26a7c175","ref":"refs/heads/main","pushedAt":"2024-09-15T04:18:49.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"shahules786","name":"ikka","path":"/shahules786","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/25312635?s=80&v=4"},"commit":{"message":"docs: agent metrics (#1314)","shortMessageHtmlLink":"docs: agent metrics (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2526038936\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1314\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1314/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1314\">#1314</a>)"}},{"before":"0ef19e651366e7a0cd3257565b1ef3e10c36e6bc","after":"78d3bea5e48d4ed1b70947bafc89ebbd837a0559","ref":"refs/heads/main","pushedAt":"2024-09-14T21:00:36.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"jjmachan","name":"Jithin James","path":"/jjmachan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5261489?s=80&v=4"},"commit":{"message":"feat: agent goal accuracy metric (#1303)","shortMessageHtmlLink":"feat: agent goal accuracy metric (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2523897886\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1303\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1303/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1303\">#1303</a>)"}},{"before":"e9fb7103a93bc20438e50e4e2d73e9097bb0ff9d","after":"0ef19e651366e7a0cd3257565b1ef3e10c36e6bc","ref":"refs/heads/main","pushedAt":"2024-09-13T05:49:23.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"shahules786","name":"ikka","path":"/shahules786","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/25312635?s=80&v=4"},"commit":{"message":"feat: Instance specifc rubrics metrics (#1304)\n\nCo-authored-by: Jithin James <jamesjithin97@gmail.com>\r\nCo-authored-by: Hao Ruan <ruanhao1116@gmail.com>","shortMessageHtmlLink":"feat: Instance specifc rubrics metrics (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2523902052\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1304\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1304/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1304\">#1304</a>)"}},{"before":"d219159cb33bd00f7b3ad2005a4926e9982daeff","after":"e9fb7103a93bc20438e50e4e2d73e9097bb0ff9d","ref":"refs/heads/main","pushedAt":"2024-09-13T04:32:22.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"shahules786","name":"ikka","path":"/shahules786","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/25312635?s=80&v=4"},"commit":{"message":"chore: move rubrics based metric into single file (#1287)\n\n```python\r\nfrom ragas.dataset_schema import SingleTurnSample\r\nfrom ragas.metrics._domain_specific_rubrics import RubricsScoreWithReference\r\nsample = SingleTurnSample(\r\n    user_input=\"Where is the Eiffel Tower located?\",\r\n    response=\"The Eiffel Tower is located in Paris.\",\r\n    reference=\"The Eiffel Tower is located in Paris.\",\r\n)\r\nrubrics = {\r\n    \"score1_description\": \"The response is incorrect, irrelevant, or does not align with the ground truth.\",\r\n    \"score2_description\": \"The response partially matches the ground truth but includes significant errors, omissions, or irrelevant information.\",\r\n    \"score3_description\": \"The response generally aligns with the ground truth but may lack detail, clarity, or have minor inaccuracies.\",\r\n    \"score4_description\": \"The response is mostly accurate and aligns well with the ground truth, with only minor issues or missing details.\",\r\n    \"score5_description\": \"The response is fully accurate, aligns completely with the ground truth, and is clear and detailed.\",\r\n}\r\nscorer =  RubricsScoreWithReference(rubrics=rubrics)\r\nscorer.llm = openai_model\r\nawait scorer.single_turn_ascore(sample)\r\n```\r\n\r\n---------\r\n\r\nCo-authored-by: Jithin James <jamesjithin97@gmail.com>","shortMessageHtmlLink":"chore: move rubrics based metric into single file (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2522417530\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1287\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1287/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1287\">#1287</a>)"}},{"before":"66285d151207eded4bee1879fd2f291584d8a04f","after":"d219159cb33bd00f7b3ad2005a4926e9982daeff","ref":"refs/heads/main","pushedAt":"2024-09-13T02:21:54.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"shahules786","name":"ikka","path":"/shahules786","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/25312635?s=80&v=4"},"commit":{"message":"Fix wrong input key name (#1300)\n\nValueError: The metric [context_utilization] that that is used requires\r\nthe following additional columns ['retreived_contexts'] to be present in\r\nthe dataset.","shortMessageHtmlLink":"Fix wrong input key name (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2523731890\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1300\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1300/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1300\">#1300</a>)"}},{"before":"e16b6f543ad91915393428f31e32a4c0111cb116","after":"66285d151207eded4bee1879fd2f291584d8a04f","ref":"refs/heads/main","pushedAt":"2024-09-13T01:12:39.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"jjmachan","name":"Jithin James","path":"/jjmachan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5261489?s=80&v=4"},"commit":{"message":"docs: add docs for new metrics (#1292)\n\nIgnore the title formatting, etc for now. This is just the content, the\r\nformatting can be re-redone once the new doc structure is ready\r\n\r\nDoc is not added to the index.","shortMessageHtmlLink":"docs: add docs for new metrics (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2522524303\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1292\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1292/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1292\">#1292</a>)"}},{"before":"6687cb52206bd1ea8ef69e4b0f3a10c1bec8e45f","after":"e16b6f543ad91915393428f31e32a4c0111cb116","ref":"refs/heads/main","pushedAt":"2024-09-13T01:11:36.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"jjmachan","name":"Jithin James","path":"/jjmachan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5261489?s=80&v=4"},"commit":{"message":"feat: add simple scorer metric (#1291)\n\n```python\r\nfrom ragas.dataset_schema import SingleTurnSample\r\nfrom ragas.metrics._simple_criteria import SimpleCriteriaScoreWithoutReference\r\n\r\n\r\nsample = SingleTurnSample(\r\n    user_input=\"Where is the Eiffel Tower located?\",\r\n    response=\"The Eiffel Tower is located in Paris.\",\r\n)\r\n\r\nscorer =  SimpleCriteriaScoreWithoutReference(name=\"course_grained_score\", definition=\"Score 0 to 5 for correctness\")\r\nscorer.llm = openai_model\r\nawait scorer.single_turn_ascore(sample)\r\n```","shortMessageHtmlLink":"feat: add simple scorer metric (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2522511113\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1291\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1291/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1291\">#1291</a>)"}},{"before":"d452bbceddbb1f10f66299df46930ca8a02430e0","after":"6687cb52206bd1ea8ef69e4b0f3a10c1bec8e45f","ref":"refs/heads/main","pushedAt":"2024-09-13T01:00:33.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"jjmachan","name":"Jithin James","path":"/jjmachan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5261489?s=80&v=4"},"commit":{"message":"feat: new aspect critic metrics (#1286)\n\n```python\r\nfrom ragas.dataset_schema import SingleTurnSample\r\nfrom ragas.metrics._aspect_critic import AspectCritic\r\n\r\n\r\nsample = SingleTurnSample(\r\n    user_input=\"Where is the Eiffel Tower located?\",\r\n    response=\"The Eiffel Tower is located in Paris.\",\r\n    reference=\"The Eiffel Tower is located in Egypt\"\r\n)\r\n\r\nscorer =  AspectCritic(name=\"harmlessness\",definition=\"is the submission harmful\")\r\nscorer.llm = openai_model\r\nawait scorer.single_turn_ascore(sample)\r\n```\r\n\r\n---------\r\n\r\nCo-authored-by: Jithin James <jamesjithin97@gmail.com>\r\nCo-authored-by: Freddie Vargus <freddie@quotientai.co>","shortMessageHtmlLink":"feat: new aspect critic metrics (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2522394987\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1286\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1286/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1286\">#1286</a>)"}},{"before":"5ac6f534550065d1d4f603d7b44bd511e2bd061b","after":"d452bbceddbb1f10f66299df46930ca8a02430e0","ref":"refs/heads/main","pushedAt":"2024-09-12T08:45:50.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"jjmachan","name":"Jithin James","path":"/jjmachan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5261489?s=80&v=4"},"commit":{"message":"Add option to disable progress bar in `evaluate` (#1279)\n\nHello there :wave: I noticed that there was no way to disable the\r\nprogress bar and thought it'd nice to have the option to silence the\r\n`tqdm` call in the executor, as it doesn’t play nicely with logs and\r\nadds noise.\r\n\r\nI've added the following changes:\r\n\r\nAdd `show_progress` flag to `Executor` class and update `evaluate`\r\nfunction to use it.\r\n\r\n* **Executor Class Changes:**\r\n- Add `show_progress` flag to the `Executor` class in\r\n`src/ragas/executor.py`.\r\n- Use the `show_progress` flag to control the display of the progress\r\nbar.\r\n- If `show_progress` is set to False, disable `tqdm` usage for\r\ndisplaying progress bars.\r\n\r\n* **Evaluate Function Changes:**\r\n- Add `show_progress` parameter to the `evaluate` function in\r\n`src/ragas/evaluation.py`.\r\n- Pass the `show_progress` flag to the `Executor` class during\r\ninitialization in the `evaluate` function.","shortMessageHtmlLink":"Add option to disable progress bar in <code>evaluate</code> (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2520638811\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1279\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1279/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1279\">#1279</a>)"}},{"before":"196127e1efa2e2cce819048d350ff7e859b8cf37","after":"5ac6f534550065d1d4f603d7b44bd511e2bd061b","ref":"refs/heads/main","pushedAt":"2024-09-12T06:34:34.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"shahules786","name":"ikka","path":"/shahules786","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/25312635?s=80&v=4"},"commit":{"message":"feat: added n and temperature passthrough to prompt (#1281)","shortMessageHtmlLink":"feat: added n and temperature passthrough to prompt (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2521376826\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1281\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1281/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1281\">#1281</a>)"}},{"before":"f123d3d855a61921230dec0ddcd537abe835df8c","after":"196127e1efa2e2cce819048d350ff7e859b8cf37","ref":"refs/heads/main","pushedAt":"2024-09-12T03:58:05.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"shahules786","name":"ikka","path":"/shahules786","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/25312635?s=80&v=4"},"commit":{"message":"feat: Non LLM based context precision (#1264)\n\n1. Add Non LLM based context precision\r\n2. Rename and introduce naming convention to metrics -\r\n<llm_or_non_llm><metric_name><with_or_without_reference>\r\n\r\n```python\r\n\r\nfrom ragas import SingleTurnSample\r\nfrom ragas.metrics._context_precision import NonLLMContextPrecisionWithReference\r\n\r\ncontext_precision = NonLLMContextPrecisionWithReference()\r\n\r\nsample = SingleTurnSample(\r\n    retrieved_contexts=[\"The Eiffel Tower is located in Paris.\"], \r\n    reference_contexts=[\"Paris is the capital of France.\", \"The Eiffel Tower is one of the most famous landmarks in Paris.\"]\r\n)\r\n\r\nawait context_precision.single_turn_ascore(sample)\r\n```\r\n\r\n---------\r\n\r\nCo-authored-by: Jithin James <jamesjithin97@gmail.com>","shortMessageHtmlLink":"feat: Non LLM based context precision (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2515644398\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1264\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1264/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1264\">#1264</a>)"}},{"before":"ae50d459904759e08ad926a7b598d0d15efaf647","after":"f123d3d855a61921230dec0ddcd537abe835df8c","ref":"refs/heads/main","pushedAt":"2024-09-12T03:57:15.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"shahules786","name":"ikka","path":"/shahules786","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/25312635?s=80&v=4"},"commit":{"message":"feat: add non llm based context recall (#1266)","shortMessageHtmlLink":"feat: add non llm based context recall (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2516691047\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1266\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1266/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1266\">#1266</a>)"}},{"before":"a4a3e5655665fc716b286c6590e1718a22edfd26","after":"ae50d459904759e08ad926a7b598d0d15efaf647","ref":"refs/heads/main","pushedAt":"2024-09-11T12:58:47.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"jjmachan","name":"Jithin James","path":"/jjmachan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5261489?s=80&v=4"},"commit":{"message":"fix: make score nested if loop_is_running (#1276)","shortMessageHtmlLink":"fix: make score nested if loop_is_running (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2519638954\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1276\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1276/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1276\">#1276</a>)"}},{"before":"3076f509dc5fe2b0418b1de60ece7dc991da0e84","after":"a4a3e5655665fc716b286c6590e1718a22edfd26","ref":"refs/heads/main","pushedAt":"2024-09-11T10:45:06.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"shahules786","name":"ikka","path":"/shahules786","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/25312635?s=80&v=4"},"commit":{"message":"fix: v1 to v2 dataset (#1275)\n\nfixes: #1271","shortMessageHtmlLink":"fix: v1 to v2 dataset (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2519368972\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1275\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1275/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1275\">#1275</a>)"}},{"before":"c615a9f89258b9f3370bc9b9eb9c3a447e9c2792","after":"3076f509dc5fe2b0418b1de60ece7dc991da0e84","ref":"refs/heads/main","pushedAt":"2024-09-11T04:28:21.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"shahules786","name":"ikka","path":"/shahules786","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/25312635?s=80&v=4"},"commit":{"message":"fix: non llm based metrics  (#1268)\n\n1) rename metrics\r\n2) delay import of optional dependencies","shortMessageHtmlLink":"fix: non llm based metrics (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2517352529\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1268\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1268/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1268\">#1268</a>)"}},{"before":"4e6a96a0ff3ac613f64e2d01199e42665cea643f","after":"c615a9f89258b9f3370bc9b9eb9c3a447e9c2792","ref":"refs/heads/main","pushedAt":"2024-09-10T17:29:47.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"shahules786","name":"ikka","path":"/shahules786","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/25312635?s=80&v=4"},"commit":{"message":"fix: inverse distance (#1267)","shortMessageHtmlLink":"fix: inverse distance (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2517035727\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1267\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1267/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1267\">#1267</a>)"}},{"before":"054c0e90ce98ddbee68010941f3385ae5e9b45ed","after":"4e6a96a0ff3ac613f64e2d01199e42665cea643f","ref":"refs/heads/main","pushedAt":"2024-09-10T13:27:10.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"jjmachan","name":"Jithin James","path":"/jjmachan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5261489?s=80&v=4"},"commit":{"message":"feat: set and get prompts for metrics (#1259)\n\n```python\r\nfrom ragas.experimental.metrics._faithfulness import FaithfulnessExperimental, LongFormAnswerPrompt\r\n\r\nfaithfulness = FaithfulnessExperimental() \r\nfaithfulness.get_prompts()\r\n\r\n#{'long_form_answer_prompt': <ragas.experimental.metrics._faithfulness.LongFormAnswerPrompt at 0x7fd7baa8efb0>,\r\n#'nli_statement_prompt': <ragas.experimental.metrics._faithfulness.NLIStatementPrompt at 0x7fd7baa8f010>}\r\n\r\nlong_form_prompt = LongFormAnswerPrompt()\r\nlong_form_prompt.instruction = \"my new instruction\"\r\n\r\nprompts = {\"long_form_answer_prompt\":long_form_prompt}\r\nfaithfulness.set_prompts(**prompts)\r\n```\r\n\r\n---------\r\n\r\nCo-authored-by: Jithin James <jamesjithin97@gmail.com>","shortMessageHtmlLink":"feat: set and get prompts for metrics (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2513095505\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1259\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1259/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1259\">#1259</a>)"}},{"before":"9709139f942770f2cc229bb48111042ec1262e1c","after":"054c0e90ce98ddbee68010941f3385ae5e9b45ed","ref":"refs/heads/main","pushedAt":"2024-09-10T12:27:33.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"jjmachan","name":"Jithin James","path":"/jjmachan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5261489?s=80&v=4"},"commit":{"message":"Non LLM based metrics (#1260)\n\nAdded support for \r\n1. BLEU SCORE\r\n2. ROGUE SCORE\r\n3. STRING MATCH\r\n4. EDIT DISTANCE","shortMessageHtmlLink":"Non LLM based metrics (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2513630390\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1260\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1260/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1260\">#1260</a>)"}},{"before":"8cd081e96d3c0cc27a72785df1fd2f2b9f0fcaf6","after":"9709139f942770f2cc229bb48111042ec1262e1c","ref":"refs/heads/main","pushedAt":"2024-09-10T04:48:28.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"shahules786","name":"ikka","path":"/shahules786","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/25312635?s=80&v=4"},"commit":{"message":"fix: handle v0.1 dataset (#1262)\n\nfixes: #1257","shortMessageHtmlLink":"fix: handle v0.1 dataset (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2515383034\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1262\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1262/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1262\">#1262</a>)"}},{"before":"2e6ac350c14e4886e121b9136789e4c8776d7742","after":"8cd081e96d3c0cc27a72785df1fd2f2b9f0fcaf6","ref":"refs/heads/main","pushedAt":"2024-09-09T14:43:49.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"shahules786","name":"ikka","path":"/shahules786","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/25312635?s=80&v=4"},"commit":{"message":"fix: add single_turn_ascore (#1261)","shortMessageHtmlLink":"fix: add single_turn_ascore (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2514148812\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1261\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1261/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1261\">#1261</a>)"}},{"before":"193947a688250015ba18f43d404612213e551b7b","after":"2e6ac350c14e4886e121b9136789e4c8776d7742","ref":"refs/heads/main","pushedAt":"2024-09-09T13:00:05.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"jjmachan","name":"Jithin James","path":"/jjmachan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5261489?s=80&v=4"},"commit":{"message":"Add support to MultiTurn Metrics (#1249)\n\n```python\r\nfrom ragas import evaluate\r\nfrom ragas.metrics import rubrics_score_without_reference\r\nfrom ragas.dataset_schema import EvaluationDataset, MultiTurnSample\r\nfrom ragas.messages import HumanMessage\r\n\r\nsample1 = MultiTurnSample(user_input=[HumanMessage(content=\"What is X\")])\r\nsample2 = MultiTurnSample(user_input=[HumanMessage(content=\"What is X\")])\r\nds = EvaluationDataset(samples=[sample1, sample2])\r\nevaluate(ds,metrics=[rubrics_score_without_reference])\r\n```","shortMessageHtmlLink":"Add support to MultiTurn Metrics (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2507554544\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1249\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1249/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1249\">#1249</a>)"}},{"before":"d5fe288405f35212e731ad2c4cd332c4595e6005","after":null,"ref":"refs/heads/ref/pull/1256/head","pushedAt":"2024-09-07T10:55:34.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"jjmachan","name":"Jithin James","path":"/jjmachan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5261489?s=80&v=4"}},{"before":"d7df217a78276bace60de887b74d5f0d758ea112","after":"193947a688250015ba18f43d404612213e551b7b","ref":"refs/heads/main","pushedAt":"2024-09-07T10:55:13.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"jjmachan","name":"Jithin James","path":"/jjmachan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5261489?s=80&v=4"},"commit":{"message":"Feat: Opik Integration (#1256)\n\nThis PR adds an integration to the [Opik\r\nplatform](https://github.com/comet-ml/opik), in includes both a new\r\ntracer and some documentation updates.\r\n\r\n---------\r\n\r\nCo-authored-by: Sid Mehta <siddharthm2350@gmail.com>\r\nCo-authored-by: Abby Morgan <86856445+anmorgan24@users.noreply.github.com>","shortMessageHtmlLink":"Feat: Opik Integration (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2510188208\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1256\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1256/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1256\">#1256</a>)"}}],"hasNextPage":true,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"Y3Vyc29yOnYyOpK7MjAyNC0wOS0yMFQwNDowNDo0MC4wMDAwMDBazwAAAAS7iWxN","startCursor":"Y3Vyc29yOnYyOpK7MjAyNC0wOS0yMFQwNDowNDo0MC4wMDAwMDBazwAAAAS7iWxN","endCursor":"Y3Vyc29yOnYyOpK7MjAyNC0wOS0wN1QxMDo1NToxMy4wMDAwMDBazwAAAASvmbC-"}},"title":"Activity · explodinggradients/ragas"}