Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LangchainProccessor does not handle function calls by returning tool message #511

Open
agilebean opened this issue Sep 26, 2024 · 0 comments

Comments

@agilebean
Copy link

After a successful function call, LangchainProcessor correctly returns a function call message.
However, the LLM (e.g. OpenAI or Anthropic) requires to receive a so-called tool message response as acknowledgment that the function call is performed.

As the context contains the tools (as initialized with context = OpenAILLMContext(messages, tools),
I think (but not sure) that the acknowledgment is handled for OpenAI in pipecat/src/pipecat/services
/openai.py in this section for handling the context because ):

    async def process_frame(self, frame: Frame, direction: FrameDirection):
        await super().process_frame(frame, direction)

        context = None
        if isinstance(frame, OpenAILLMContextFrame):
            context: OpenAILLMContext = frame.context
...
        if context:
            await self.push_frame(LLMFullResponseStartFrame())
            await self.start_processing_metrics()
            await self._process_context(context)
            await self.stop_processing_metrics()
            await self.push_frame(LLMFullResponseEndFrame())

There is no equivalent of this code section in LangchainProcessor but crucial, as most people use langchain for agents that use tools, or their unified interface to function calling (bind_tools with pydantic class).

Workaround (which doesn't work)

From the function call response, I successfully assembled the required syntax for a tool message.

            tool_message_response = {
                "role": "tool",
                "tool_call_id": tool_call_id,
                "content": tool_output,
                "name": tool_call['name'],
            }

Failed sending tool message

However, I couldn't send this successfully to the OpenAI API.
Tried the following methods:

### Method 1: plain invoke
await self._chain.invoke(
    tool_message_response,
    {'configurable': {'session_id': self._participant_id}}
)

### Method 2: push TextFrame
await self.push_frame(TextFrame(tool_message_response))

### Method 3: push LLMMessagesFrame
await self.push_frame(LLMMessagesFrame(tool_message_response))

### Method 4: send context
context = OpenAILLMContext()
logger.info(f"context: {context}")

context.add_message(tool_message_response)
logger.info(f"context.add_message(tool_message_response): {context}")

frame = OpenAILLMContextFrame(context)
logger.info(f"frame: {frame}")

await self.push_frame(frame)
logger.info(f"Pushed frame: {frame}")

await self.push_frame(LLMFullResponseEndFrame())
logger.info(f"Pushed frame: LLMFullResponseEndFrame()")

There is no documentation whatsoever on how to correctly handle this with pipecat. Or is there?
Let's say that I can now understand other people's growing frustration with pipecat :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant