Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Docs] Prompt management Post-app structure #2086

Draft
wants to merge 5 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,3 +52,4 @@ By inserting to these formatting conventions, you'll maintain the integrity and
## Notes

- Do not update any libraries or packages as this could disrupt the template structure and cause it to break.
- Please use kebab-case (this-way) instead of snake_case for naming files and folders
115 changes: 115 additions & 0 deletions docs/docs/prompt-management/01-overview.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
---
title: "Overview"
---

```mdx-code-block
import DocCard from '@theme/DocCard';
import clsx from 'clsx';

```

## What is prompt management?

Building LLM-powered applications is an iterative process. In each iteration, you aim to improve the application's performance by refining prompts, adjusting configurations, and evaluating outputs.

<img
style={{ width: "75%", display: "block", margin: "0 auto" }}
src="/images/prompt_management/llm_lifecycle.png"
/>

A prompt management system provides you the tools to do this process systematically by:

- **Versioning Prompts**: Keeping track of different prompts you've tested.
- **Linking Prompt Variants to Experiments**: Connecting each prompt variant to its evaluation metrics to understand the effect of changes and determine the best variant.
- **Publishing Prompts**: Providing a way to publish the best prompt variants to production and maintain a history of changes in production systems.
- **Associating Prompts with Traces**: Monitoring how changes in prompts affect production metrics.

## Capabilities in agenta

Agenta enables you to create **prompts** both from the **web UI** and **from code**. It allows you to publish these **prompts** to **endpoints** with specific environment names and version them. Additionally, it allows you to **run evaluations** from the web UI and code on these prompts and connect their results and **observability** to the respective version.

## Why do I need a prompt management system?

A prompt management system enables everyone on the team—from **product owners** to **subject matter experts**—to collaborate in creating prompts. Additionally it helps you answer the following questions:

- Which prompts have we tried?
- What were the outputs of these prompts?
- How do the evaluation results of these prompts compare?
- Which prompt was used for a specific generation in production?
- What was the effect of publishing the new version of this prompt in production?
- Who on the team made changes to a particular prompt in production?

## Configuration management in agenta

Agenta goes beyond prompt management to encompass the entire configuration of your LLM applications. If your LLM workflow is more complex than a single prompt (e.g., Retrieval-Augmented Generation (RAG) or a chain of prompts), you can version the whole configuration together.

In contrast to a **prompt**, a **configuration** of an LLM application can include additional parameters beyond prompt templates and models (with their parameters). For instance:

- An LLM application using a **chain of two prompts** would have a configuration that includes the two prompts and their respective model parameters.
- An application that includes a **RAG pipeline** would have a configuration that includes parameters such as `top_k` and `embedding`.

```json title="Example RAG configuration"
{
"top_k": 3,
"embedding": "text-embedding-3-large",
"prompt-query": "We have provided context information below. {context_str}. Given this information, please answer the question: {query_str}\n",
"model-query": "openai/gpt-o1",
"temperature-query": "1.0"
}
```

Agenta enables you to version the entire **configuration** of the LLM app as a unit. This is essential because there is a dependency between the parts of the configuration. For instance, in a chain of two prompts, changes to the first prompt depend on changes to the second prompt. Therefore, you need to version them together to ensure consistency and traceability.

### Get started

<section className='row'>
<article key='1' className="col col--6 margin-bottom--lg">

<DocCard
item={{
type: "link",
href: "/prompt-management/quick-start",
label: "Quick start",
description: "Create a prompt from the web UI, publish it to an endpoint, and integrate it into your code",
}}
/>
</article>

<article key='2' className="col col--6 margin-bottom--lg">
<DocCard
item={{
type: "link",
href: "/prompt-management/prompt-engineering-in-the-playground",
label: "Using the playground",
description: "Perform prompt engineering in the playground",
}}
/>
</article>
</section>

### Configuration lifecycle management

<section className='row'>

<article key="1" className="col col--6 margin-bottom--lg">
<DocCard
item={{
type: "link",
href: "/prompt-management/how-to-publish-a-prompt",
label: "Publishing a prompt",
description: "How to publish a prompt to an endpoint from the web UI.",
}}
/>
</article>

<article key='2' className="col col--6 margin-bottom--lg">
<DocCard
item={{
type: "link",
href: "/prompt-management/how-to-use-a-prompt",
label: "How to use a prompt",
description: "How to use a published a prompt in your code",
}}
/>
</article>
</section>
113 changes: 113 additions & 0 deletions docs/docs/prompt-management/02-quick-start.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
---
title: "Quick Start"
---

## Introduction

In this tutorial, we will **create a prompt** in the web UI, **publish** it to a deployment, **integrate** it with our code base.

:::note
If you want to do this whole process programatically, jump to [this guide](/prompt-management/prompt-management-from-sdk)
:::

## 1. Create a prompt

We will create a prompt from the web UI. This can be done simply by going to the app overview, clicking on create a prompt. You have here the choice between using a chat prompt or a text prompt:

- A text prompt is useful for single turn LLM applications such as question answering, text generation, entity extraction, and classification, etc.
- A chat application is designed for multi-turn applications like chatbots.

<img
style={{ width: "80%", display: "block", margin: "0 auto" }}
src="/images/prompt_management/create-prompt-modal.png"
/>
<br />

## 2. Publish a variant

You can create in each LLM application multiple variants. You can think of variants as git branches. Each variant is versioned with each variant version having it's own commit number and being immutable.

When you are satified with a commit (after evaluating it for instance) you can publish it to a deployment. The deployment is tagged with an environment (`production`, `development` or `staging`) and provides you access to endpoint for both the published configuration and to a proxy for the calls.

To publish a variant, go to overview, click on the three dots on the **variant** that you want to publish and select Deploy (see screenshto)

<img
style={{ width: "75%", display: "block", margin: "0 auto" }}
src="/images/prompt_management/deploy-action.png"
/>
<br />
You can select now which deployments you want to publish the variant to

<img
style={{ width: "75%", display: "block", margin: "0 auto" }}
src="/images/prompt_management/deployment-modal.png"
/>

<br />
:::caution

New change
to the **variant** will not be automaticly published to the **deployment** unless
we explicitly **publish it**.

The reason is that we have published the last **version**/**commit** of the **variant** to
that deployment and not the **variant** itself!

:::

## 3. Integrate with your code

To see the code snippet needed to use the prompt, click on the name of the deployment and you will see a drawer with the integration code. For example for python

```python
import agenta as ag
from agenta import DeploymentManager, PromptManager

# os.environment["AGENTA_API_KEY"] = "ag-xxx"
# os.environment["AGENTA_PROJECT_ID] = "ag-xxx"
ag.init()

# fetch the deployment (id)
deployment = DeploymentManager.get_deployment_by_name(deployment_name="production",
app_slug="my-app")
# fetch latest configuration deployed here
prompt_obj = PromptManager.get_from_registry(deployment_id=deployment.id)
```

The prompt object will have both the information about the configuration version and the data (under the field data)

```python
{
variant_slug: "my-variant",
app_slug: "my-app""
version: 3,
commit-id: "afae3232",
data: {
'temperature': 1.0,
'model': 'gpt-3.5-turbo',
'max_tokens': -1,
'prompt_system': 'You are an expert in geography.',
'prompt_user': 'What is the capital of {country}?',
'top_p': 1.0,
'frequence_penalty': 0.0,
'presence_penalty': 0.0,
'force_json': 0
}
```

## 4. Revert to previous deployment (optional)

:::note
This feature is only available in cloud and enterprise version
:::

Optionally you would like to revert to a previously published commit. For this click on the deployment in the overview view, then click on **history**, you will see all the previous published version. You can revert to a previous version by clicking on revert.

<img
style={{ width: "75%", display: "block", margin: "0 auto" }}
src="/images/prompt_management/revert-deployment.png"
/>

## Next steps

Now that you've created and published your first prompt, you can learn how to do [prompt engineering in the playground](/prompt_management/prompt_engineering) or dive deeper into [the capabilities of the prompt management SDK](/prompt-management/the-sdk)
Original file line number Diff line number Diff line change
Expand Up @@ -2,50 +2,6 @@
title: "Core Concepts"
---

# Core Concepts

This page describes the main concepts used in prompt management in **agenta**.

## Prompt and Configuration Management

### What Is Prompt Management?

Building LLM-powered applications is an iterative process. In each iteration, you aim to improve the application's performance by refining prompts, adjusting configurations, and evaluating outputs.

<img
style={{ width: "75%", display: "block", margin: "0 auto" }}
src="/images/prompt_management/llm_lifecycle.png"
/>

A prompt management system provides you the tools to do this process systematically by:

- **Versioning Prompts**: Keeping track of different prompts you've tested.
- **Linking Prompt Variants to Experiments**: Connecting each prompt variant to its evaluation metrics to understand the effect of changes and determine the best variant.
- **Publishing Prompts**: Providing a way to publish the best prompt variants to production and maintain a history of changes in production systems.
- **Associating Prompts with Traces**: Monitoring how changes in prompts affect production metrics.

### Why Do I Need a Prompt Management System?

A prompt management system enables everyone on the team—from product owners to subject matter experts—to collaborate in creating prompts. Additionally it helps you answer the following questions:

- Which prompts have we tried?
- What were the outputs of these prompts?
- How do the evaluation results of these prompts compare?
- Which prompt was used for a specific generation in production?
- What was the effect of publishing the new version of this prompt in production?
- Who on the team made changes to a particular prompt in production?

### What Is the Difference Between Prompt and Configuration Management?

Agenta goes beyond prompt management to encompass the entire configuration of your LLM applications.

**Prompts** are a special case of a **configuration**. A **prompt** includes the prompt template, the model, and the model parameters. However, a **configuration** of an LLM application can include additional parameters.
For instance, an LLM application using a chain of two prompts would have a configuration that includes the two prompts and their respective model parameters. Similarly, an application that includes a RAG pipeline would have a configuration that includes parameters such as `top_k` and `embedding`.

Agenta enables you to version the entire `configuration` of the LLM app as a unit. This makes sense since there is a dependency between the parts of the configuration. For instance in a chain of two prompts, the changes of the first depend on the changes of the second. Therefore you need to version them together to ensure consistency and traceability.

## Taxonomy of Terms and Concepts in agenta

Below are the description to the main terms and concepts used in agenta.

<img
Expand Down
69 changes: 0 additions & 69 deletions docs/docs/prompt_management/02-creating-a-prompt.mdx

This file was deleted.

Loading