Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(llmobs): poc ragas evaluation integration #10431

Draft
wants to merge 29 commits into
base: main
Choose a base branch
from
Draft

feat(llmobs): poc ragas evaluation integration #10431

wants to merge 29 commits into from
+734 −58

Commits on Aug 27, 2024

  1. ragas callback skeleton

    lievan committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    de70bd3 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' of github.com:DataDog/dd-trace-py into evan.li/ra… 

    …gas-integration
    lievan committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    0e749db View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' of github.com:DataDog/dd-trace-py into evan.li/ra… 

    …gas-integration
    lievan committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    1f96705 View commit details
    Browse the repository at this point in the history

Commits on Aug 28, 2024

  1. rename to eval callbacks

    lievan committed Aug 28, 2024
    Configuration menu
    Copy the full SHA
    2f96eab View commit details
    Browse the repository at this point in the history

  2. bug fixes'

    lievan committed Aug 28, 2024
    Configuration menu
    Copy the full SHA
    a0d7718 View commit details
    Browse the repository at this point in the history

Commits on Sep 1, 2024

  1. span by span evals

    lievan committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    e33875b View commit details
    Browse the repository at this point in the history

Commits on Sep 5, 2024

  1. ragas poc

    lievan committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    189ca4f View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2024

  1. trace tasks

    lievan committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    6240950 View commit details
    Browse the repository at this point in the history

  2. Update ddtrace/llmobs/utils.py 

    Co-authored-by: datadog-datadog-prod-us1[bot] <88084959+datadog-datadog-prod-us1[bot]@users.noreply.github.com>
    @lievan @datadog-datadog-prod-us1
    lievan and datadog-datadog-prod-us1[bot] committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    c34639b View commit details
    Browse the repository at this point in the history

  3. Update ddtrace/llmobs/utils.py 

    Co-authored-by: datadog-datadog-prod-us1[bot] <88084959+datadog-datadog-prod-us1[bot]@users.noreply.github.com>
    @lievan @datadog-datadog-prod-us1
    lievan and datadog-datadog-prod-us1[bot] committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    d8fae5c View commit details
    Browse the repository at this point in the history

  4. rm role constants

    lievan committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    62059ec View commit details
    Browse the repository at this point in the history

  5. Merge branch 'evan.li/ragas-integration' of github.com:DataDog/dd-tra… 

    …ce-py into evan.li/ragas-integration
    lievan committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    0260cc7 View commit details
    Browse the repository at this point in the history

  6. refactor

    lievan committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    b2567fc View commit details
    Browse the repository at this point in the history

  7. just focus on ragas faithfulness thing

    lievan committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    19f8a00 View commit details
    Browse the repository at this point in the history

  8. remove the prompt stuff

    lievan committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    cd26c15 View commit details
    Browse the repository at this point in the history

  9. remove dupe name

    lievan committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    9f3ce40 View commit details
    Browse the repository at this point in the history

Commits on Sep 9, 2024

  1. use futures, remember to cancel futures on exit

    lievan committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    82ad564 View commit details
    Browse the repository at this point in the history

Commits on Sep 10, 2024

  1. save

    lievan committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    7e539dd View commit details
    Browse the repository at this point in the history

  2. cleanup

    lievan committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    0443e1a View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2024

  1. annotate token usage

    lievan committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    795c84d View commit details
    Browse the repository at this point in the history

  2. support annotating prompt template data

    lievan committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    641fe72 View commit details
    Browse the repository at this point in the history

  3. rquirements

    lievan committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    07e09ef View commit details
    Browse the repository at this point in the history

  4. rel note

    lievan committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    55ea8fb View commit details
    Browse the repository at this point in the history

  5. llm span requirements

    lievan committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    e93f38f View commit details
    Browse the repository at this point in the history

  6. fix type annotaiton, pop prompt

    lievan committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    54826db View commit details
    Browse the repository at this point in the history

  7. prompt vars validation

    lievan committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    0bbdffe View commit details
    Browse the repository at this point in the history

  8. test for tradce processor

    lievan committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    6200025 View commit details
    Browse the repository at this point in the history

Commits on Sep 13, 2024

  1. merge in prompt templating stuff

    lievan committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    5e7910a View commit details
    Browse the repository at this point in the history

  2. improve the ragas traces

    lievan committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    d0a0304 View commit details
    Browse the repository at this point in the history