Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ai-proxy 收口 SSE 消息分段机制 #1164

Open
CH3CHO opened this issue Jul 30, 2024 · 0 comments · May be fixed by #1184
Open

ai-proxy 收口 SSE 消息分段机制 #1164

CH3CHO opened this issue Jul 30, 2024 · 0 comments · May be fixed by #1184

Comments

@CH3CHO
Copy link
Collaborator

CH3CHO commented Jul 30, 2024

Why do you need it?

在处理 SSE 数据推送时,由于 envoy 推送到 Wasm 插件的 response data chunk 可能并不是一个完整的 SSE 消息,所以插件里需要对数据进行缓存,凑够一个完整的 SSE 消息后再进行处理。

目前 ai-proxy 的这部分工作是由各个 provider 单独实现的。考虑到这部分功能很多 provider 都需要对接,可以改成由 ai-proxy 插件主体集中处理。这样 provider 就只需要关注消息本身,不再需要关注分段这些公共逻辑了。

How could it be?

将目前分散在各个 provider 实现中的 SSE 消息分段逻辑集中到 ai-proxy 本体。

Other related information

相关代码:https://github.com/alibaba/higress/blob/main/plugins/wasm-go/extensions/ai-proxy/provider/qwen.go#L253

@CH3CHO CH3CHO removed the help wanted Extra attention is needed label Jul 30, 2024
@cr7258 cr7258 linked a pull request Aug 8, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Todo
Development

Successfully merging a pull request may close this issue.

2 participants