Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Net HTTP persistent adapter compatibility #509

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

madwork
Copy link

@madwork madwork commented Aug 7, 2024

Explanation

The idea is to allow connection reuse with net http persistent adapter. It should also improve memory footprint by lowering the number of Faraday::Connection objects and quoting net http persistent project description:

Using persistent HTTP connections can dramatically increase the speed of HTTP. Creating a new HTTP connection for every request involves an extra TCP round-trip and causes TCP congestion avoidance negotiation to start over.

Benchmark

The graph below display usage of the gem with the latest version (before) versus this branch (after) for the exact same job. It illustrates that it drastically reduces the number of open connections and port usage.

openai

Usage example

gem "faraday", "~> 2.7", ">= 2.7.11"
gem "faraday-net_http_persistent", "~> 2.1"
openai_client ||= OpenAI::Client.new do |faraday|
  faraday.adapter :net_http_persistent, pool_size: 5 do |http|
    http.idle_timeout = 30
  end
end
thread = openai_client.threads.create
openai_client.messages.create(
  thread_id: thread["id"],
  parameters: {
    role: "user",
    content: "content"
  }
)
run = openai_client.runs.create(
  thread_id: thread["id"],
  parameters: {
    assistant_id: "assistant_id"
  }
)
loop do
  response = openai_client.runs.retrieve(
    id: run["id"],
    thread_id: thread["id"]
  )

  # Begin polling
  case response["status"]
  when "queued", "in_progress"
    sleep 1 # Wait 1s and poll again
  when "completed"
    # TODO
  else
    raise
  end
end

All Submissions:

  • Have you followed the guidelines in our Contributing document?
  • Have you checked to ensure there aren't other open Pull Requests for the same update/change?
  • Have you added an explanation of what your changes do and why you'd like us to include them?

@madwork madwork force-pushed the improvements/connection-reuse branch from a32cee1 to 9f065c2 Compare August 7, 2024 16:45
The current implementation is breaking any attempt of connection pool usage.
@madwork madwork force-pushed the improvements/connection-reuse branch from 9f065c2 to 475b25e Compare August 7, 2024 16:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant