Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

In case an error is thrown during the structured data flow, capture the text value that failed #43

Closed
theoremonemehdi opened this issue Jun 20, 2024 · 3 comments
Assignees
Labels
enhancement New feature or request

Comments

@theoremonemehdi
Copy link
Collaborator

In some cases, Kurt fails during a structured data flow for the following reasons:

  1. The LLM response breaks a Zod constraint (i.e. a number between 0 and 100, or a missing key…)
  2. The LLM response is not a JSON structure (basically JSON.parse fails)

In order to build a follow-up prompt for the LLM, we need to capture the returned text value that failed.
Maybe expose the text value inside a KurtStructuredDataError

@theoremonemehdi
Copy link
Collaborator Author

theoremonemehdi commented Jun 21, 2024

@jemc noticed this: { index: 0, delta: {}, logprobs: null, finish_reason: 'length' }
Basically, the LLM response exceeded the remaining available tokens that's why JSON.parse is failing.
This is the successful finish_reason { index: 0, delta: {}, logprobs: null, finish_reason: 'stop' }

@jemc
Copy link
Collaborator

jemc commented Jun 26, 2024

@theoremonemehdi - thinking through the different use cases here... How does this sound for developer experience?

  • Any response with finish_reason: "length" throws a KurtResultLimitError, containing the text-so-far in a property called text

  • Any response which fails to parse JSON throws a KurtResultParseError, containing the text that failed to parse in a property called text

  • Any response which fails Zod constraints throws a KurtResultValidateError, containing the text in a property called text, and the not-valid data object in a property called data (or additionalData in the case of additional parallel tool calls). Note that all these property names match the corresponding property names in the KurtResult interface type, for consistency.

  • All of the above error classes would be subclasses of a common KurtResultError type, so that they could be handled together by applications that don't care to differentiate these cases.

  • KurtResultError itself would be a subclass of KurtError, which would be the superclass of any errors thrown by Kurt, including other categories like KurtFeatureIncompatibleError (see discussion in ticket Add parameter to set PRNG seed #47).

@jemc jemc added the enhancement New feature or request label Jun 26, 2024
@jemc jemc self-assigned this Jun 26, 2024
@jemc
Copy link
Collaborator

jemc commented Aug 1, 2024

This has been resolved in tickets #48 and #49.

@jemc jemc closed this as completed Aug 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants