Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chore: Split AI Gateway into its own section #7274

Merged
merged 2 commits into from
May 1, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 38 additions & 2 deletions app/_data/docs_nav_gateway_3.6.x.yml
Original file line number Diff line number Diff line change
Expand Up @@ -87,8 +87,6 @@ items:
url: /get-started/key-authentication
- text: Load-Balancing
url: /get-started/load-balancing
- text: AI Gateway
url: /get-started/ai-gateway

- title: Install Kong
icon: /assets/images/icons/documentation/icn-deployment-color.svg
Expand Down Expand Up @@ -351,6 +349,44 @@ items:
- text: Verify Build Provenance for Signed Kong Images
url: /kong-enterprise/provenance-verification

- title: Kong AI Gateway
icon: /assets/images/icons/documentation/icn-ai.svg
items:
- text: Overview
url: /ai-gateway/
- text: Get started with AI Gateway
url: /get-started/ai-gateway/
- text: LLM Provider Integration Guides
items:
- text: OpenAI
url: /hub/kong-inc/ai-proxy/how-to/openai/
generate: false
absolute_url: true
- text: Cohere
url: /hub/kong-inc/ai-proxy/how-to/cohere/
generate: false
absolute_url: true
- text: Azure
url: /hub/kong-inc/ai-proxy/how-to/azure/
generate: false
absolute_url: true
- text: Anthropic
url: /hub/kong-inc/ai-proxy/how-to/anthropic/
generate: false
absolute_url: true
- text: Mistral
url: /hub/kong-inc/ai-proxy/how-to/mistral/
generate: false
absolute_url: true
- text: Llama2
url: /hub/kong-inc/ai-proxy/how-to/llama2/
generate: false
absolute_url: true
- text: AI Gateway plugins
url: /hub/?category=ai
generate: false
absolute_url: true

- title: Kong Manager
icon: /assets/images/icons/documentation/icn-manager-color.svg
items:
Expand Down
40 changes: 38 additions & 2 deletions app/_data/docs_nav_gateway_3.7.x.yml
Original file line number Diff line number Diff line change
Expand Up @@ -87,8 +87,6 @@ items:
url: /get-started/key-authentication
- text: Load-Balancing
url: /get-started/load-balancing
- text: AI Gateway
url: /get-started/ai-gateway
- title: Kong in Production
icon: /assets/images/icons/documentation/icn-deployment-color.svg
items:
Expand Down Expand Up @@ -347,6 +345,44 @@ items:
- text: Verify Build Provenance for Signed Kong Images
url: /kong-enterprise/provenance-verification

- title: Kong AI Gateway
icon: /assets/images/icons/documentation/icn-ai.svg
items:
- text: Overview
url: /ai-gateway/
- text: Get started with AI Gateway
url: /get-started/ai-gateway/
- text: LLM Provider Integration Guides
items:
- text: OpenAI
url: /hub/kong-inc/ai-proxy/how-to/openai/
generate: false
absolute_url: true
- text: Cohere
url: /hub/kong-inc/ai-proxy/how-to/cohere/
generate: false
absolute_url: true
- text: Azure
url: /hub/kong-inc/ai-proxy/how-to/azure/
generate: false
absolute_url: true
- text: Anthropic
url: /hub/kong-inc/ai-proxy/how-to/anthropic/
generate: false
absolute_url: true
- text: Mistral
url: /hub/kong-inc/ai-proxy/how-to/mistral/
generate: false
absolute_url: true
- text: Llama2
url: /hub/kong-inc/ai-proxy/how-to/llama2/
generate: false
absolute_url: true
- text: AI Gateway plugins
url: /hub/?category=ai
generate: false
absolute_url: true

- title: Kong Manager
icon: /assets/images/icons/documentation/icn-manager-color.svg
items:
Expand Down
150 changes: 150 additions & 0 deletions app/_src/gateway/ai-gateway/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,150 @@
---
title: Kong AI Gateway
content-type: explanation
---

Kong AI Gateway is a powerful set of features built on top of [{{site.base_gateway}}](/gateway/latest/),
designed to help developers and organizations effectively adopt AI capabilities quickly and securely.

## How to get started

<div class="docs-grid-install docs-grid-install__bottom max-3">
<a href="/gateway/{{page.release}}/get-started/ai-gateway/" class="docs-grid-install-block docs-grid-install-block__bottom">
<img class="install-icon no-image-expand small" src="/assets/images/icons/documentation/icn-flag.svg" alt="">
<div class="install-block-column">
<div class="install-text">Get started</div>
<div class="install-description">Get started in one minute with our guide</div>
</div>
</a>

<a href="https://konghq.com/products/kong-ai-gateway#videos" class="docs-grid-install-block docs-grid-install-block__bottom no-link-icon">
<img class="install-icon no-image-expand small" src="/assets/images/icons/documentation/icn-learning.svg" alt="">
<div class="install-block-column">
<div class="install-text">Videos</div>
<div class="install-description">Watch video tutorials</div>
</div>
</a>

<a href="/hub/?category=ai" class="docs-grid-install-block docs-grid-install-block__bottom">
<img class="install-icon no-image-expand small" src="/assets/images/icons/documentation/icn-api-plugins-color.svg" alt="">
<div class="install-block-column">
<div class="install-text">AI Plugins</div>
<div class="install-description">Check out the AI plugins on the Kong Plugin Hub</div>
</div>
</a>
</div>

## What is the Kong AI Gateway?

With the rapid emergence of multiple AI LLM providers (including open source and self-hosted models),
the AI technology landscape is fragmented and lacking in standards and controls. This
significantly complicates how developers and organizations use and govern AI services. {{site.base_gateway}}'s
broad API management capabilities and plugin extensibility model make it well suited to
provide AI-specific API management and governance services.

While AI providers don't conform to a standard API specification, the AI Gateway provides a
normalized API layer allowing clients to consume multiple AI services from the same client code base.
The AI Gateway provides additional capabilities for credential management, AI usage observability,
governance, and tuning through prompt engineering. Developers can use no-code AI Plugins to enrich
existing API traffic, easily enhancing their existing application functionality.

You can enable the AI Gateway features through a set of modern and specialized plugins,
using the same model you use for any other {{site.base_gateway}} plugin.
When deployed alongside existing [{{site.base_gateway}} plugins](/hub/?category=ai),
{{site.base_gateway}} users can quickly assemble a sophisticated AI management platform
without custom code or deploying new and unfamiliar tools.

![AI Gateway](/assets/images/products/gateway/getting-started-guide/ai-gateway.png)

## AI Gateway capabilities

The following describes the broad capabilities of the AI Gateway. More details can be found
in the AI Gateway plugins found in the Kong [Plugin Hub](/hub/?category=ai).

### AI Provider Proxy

The core of the AI Gateway is the ability to route AI requests to various providers exposed via
a provider-agnostic API. This normalized API layer affords developers and organizations multiple benefits:

* Client applications are shielded from AI provider API specifics, promoting code reusability
* Centralized AI provider credential management
* The AI Gateway gives the developers and organizations a central point of governance and observability over AI data and usage
* Request routing can be dynamic, allowing AI usage to be optimized based on various metrics: cost, usage, response accuracy, and so on.
* AI services can be used by other {{site.base_gateway}} plugins to augment non-AI API traffic

This core AI Gateway feature is enabled with the [AI Proxy](/hub/kong-inc/ai-proxy/) plugin, which is
deployed by default in the getting started script referenced above.

The AI Proxy supports two types of LLM requests:

* _Completion_: A type of request where the AI system is asked to generate a textual output based on a single prompt.
Completions are configured using the configuration key `route_type` and a value of `llm/v1/completions`.
* _Chat_: A type of request that is part of a conversational AI interface. In a `chat` request, the AI is expected to return
a dialog response to user input and the AI system bases its response on the conversational history. Chats are
configured using the configuration key `route_type` and a value of `llm/v1/chat`.

The core proxy behavior supports the following hosted AI providers:

* [OpenAI](https://openai.com/product)
* [Cohere](https://docs.cohere.com/reference/about)
* [Azure](https://learn.microsoft.com/en-us/azure/ai-services/openai/reference)
* [Anthropic](https://www.anthropic.com/)

In addition to the hosted AI providers, self hosted models are supported as well. An example
tool that allows the running of local models is [Ollama](https://ollama.ai/). The following local
models are supported:

* [Mistral](https://mistral.ai/)
* [Llama2](https://llama.meta.com/)

See the [AI Proxy plugin configuration](/hub/kong-inc/ai-proxy/configuration/) for details on modifying the proxy behavior.

### AI usage governance

With the growing adoption of AI technologies, developers and their organizations are exposed to a set of new risk vectors.
In particular, the risk of having sensitive data leaked to AI Providers, exposing organizations and their customers to
data breaches and other security risks.

Kong's AI Gateway provides additional plugins to aid the developers in controlling AI data and usage. These
plugins are used in combination with the AI Proxy plugin, allowing you to build secure and specialized AI
experiences for your users.

#### Data governance

AI Gateway provides the ability to govern outgoing AI prompts via the
[AI Prompt Guard](/hub/kong-inc/ai-prompt-guard). This plugin allows the configuration of regular expressions
following an allow/deny list configuration. Denied prompts result in `4xx` HTTP code responses to clients preventing
the egress of offending requests.

#### Prompt engineering

AI systems are built around prompts, and manipulating those prompts is important for successful adoption of the technologies.
Prompt engineering is the methodology of manipulating the linguistic inputs that guide the AI system. The AI Gateway
supports a set of plugins that allow you to create a simplified and enhanced experience by setting default prompts or manipulating
prompts from clients as they pass through the gateway.

* The [AI Prompt Template](/hub/kong-inc/ai-prompt-template) plugin enables administrators to provide pre-configured AI prompts to users. These prompts contain variable
placeholders in the format `{% raw %}{{variable}}{% endraw %}` which users fill to adapt the template to their specific needs. This functionality
prohibits arbitrary prompt injection by sanitizing string inputs to ensure that JSON control characters are escaped.

* The [AI Prompt Decorator](/hub/kong-inc/ai-prompt-decorator) plugin injects an array of `llm/v1/chat` messages at the
start or end of a caller’s chat history. This capability allows the caller to create more complex prompts and have more control
over how a Large Language Model (LLM) is used when called via {{site.base_gateway}}.

#### Request transformations

Kong's AI Gateway also allows you to use AI technology to augment other API traffic. One example may be routing API responses
through an AI language translation prompt before returning it to the client. Kong's AI Gateway provides two plugins that can be
used in conjunction with other upstream API services to weave AI capabilities into API request processing. These plugins
can be configured independently of the AI Proxy plugin.

* The [AI Request Transformer](/hub/kong-inc/ai-request-transformer) plugin uses a configured LLM service to transform and introspect the
consumer's request body before proxying the request upstream. It extends the function of the AI Proxy plugin and runs after all of the
AI Prompt plugins, allowing it to introspect LLM requests against a different LLM. The transformed request is then sent to the backend service.
Once the LLM service returns a response, this is set as the upstream's request body.

* The [AI Response Transformer](/hub/kong-inc/ai-response-transformer) plugin uses a configured LLM service to introspect and transform the
HTTP(S) response from upstream before sending it back to the client. This plugin complements the AI Proxy plugin, facilitating
introspection of LLM responses against a different LLM. Importantly, it adjusts response headers, response status codes, and the body of the
response based on instructions from the LLM. The adjusted response is then sent back to the client.

Loading
Loading