Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SSH Connection Timeouts During Full Deployment #857

Closed
jfanals opened this issue Jun 23, 2024 · 0 comments · May be fixed by #859
Closed

SSH Connection Timeouts During Full Deployment #857

jfanals opened this issue Jun 23, 2024 · 0 comments · May be fixed by #859

Comments

@jfanals
Copy link

jfanals commented Jun 23, 2024

I'm experiencing intermittent SSH connection timeouts during the full kamal deploy process, specifically in the pull method. Interestingly, running kamal build pull separately works fine, but the same operation fails during the full deployment.

Environment

  • Kamal version: 1.7.1

Steps to Reproduce

  1. Run kamal app remove && kamal deploy --verbose
  2. Observe that the process fails during the pull method with a timeout error
  3. Run kamal build pull separately
  4. Observe that it works without issues

Error Message

 DEBUG [4d2890b0] Running /usr/bin/env echo [2024-06-23T14:33:29Z] [myuser] Pulled image with version 3fbef4c6366422100f25a27d0376790eebd81461 >> .kamal/frontend_v2-audit.log on 5.161.193.74
 DEBUG [4d2890b0] Command: /usr/bin/env echo [2024-06-23T14:33:29Z] [myuser] Pulled image with version 3fbef4c6366422100f25a27d0376790eebd81461 >> .kamal/frontend_v2-audit.log
  Finished all in 231.3 seconds
  ERROR (IO::TimeoutError): Exception while executing on host 5.161.193.74: Blocking operation timed out!
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/net-ssh-7.2.3/lib/net/ssh/buffered_io.rb:64:in `recv'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/net-ssh-7.2.3/lib/net/ssh/buffered_io.rb:64:in `fill'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/net-ssh-7.2.3/lib/net/ssh/connection/session.rb:275:in `block in ev_do_handle_events'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/net-ssh-7.2.3/lib/net/ssh/connection/session.rb:271:in `each'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/net-ssh-7.2.3/lib/net/ssh/connection/session.rb:271:in `ev_do_handle_events'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/net-ssh-7.2.3/lib/net/ssh/connection/event_loop.rb:117:in `ev_select_and_postprocess'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/net-ssh-7.2.3/lib/net/ssh/connection/event_loop.rb:30:in `process'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/net-ssh-7.2.3/lib/net/ssh/connection/session.rb:226:in `process'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/net-ssh-7.2.3/lib/net/ssh/connection/session.rb:179:in `block in loop'
<internal:kernel>:187:in `loop'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/net-ssh-7.2.3/lib/net/ssh/connection/session.rb:179:in `loop'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/sshkit-1.22.2/lib/sshkit/backends/netssh.rb:182:in `block in execute_command'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/sshkit-1.22.2/lib/sshkit/backends/connection_pool.rb:65:in `with'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/kamal-1.7.1/lib/kamal/sshkit_with_ext.rb:84:in `with_ssh'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/sshkit-1.22.2/lib/sshkit/backends/netssh.rb:146:in `execute_command'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/sshkit-1.22.2/lib/sshkit/backends/abstract.rb:148:in `block in create_command_and_execute'
<internal:kernel>:90:in `tap'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/sshkit-1.22.2/lib/sshkit/backends/abstract.rb:148:in `create_command_and_execute'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/sshkit-1.22.2/lib/sshkit/backends/abstract.rb:80:in `execute'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/kamal-1.7.1/lib/kamal/cli/build.rb:61:in `block in pull'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/sshkit-1.22.2/lib/sshkit/backends/abstract.rb:31:in `instance_exec'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/sshkit-1.22.2/lib/sshkit/backends/abstract.rb:31:in `run'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/kamal-1.7.1/lib/kamal/sshkit_with_ext.rb:117:in `block (2 levels) in execute'

Additional Information

  • The issue occurs consistently during full deployment but not when running kamal build pull separately
  • SSH agent is not running, but SSH_AUTH_SOCK is set
  • SSH configuration shows forward_agent: true

Questions

  1. Why might the SSH connection work for kamal build pull but fail during kamal deploy?
  2. Are there any known issues with SSH connections timing out during longer processes?
  3. Could there be a problem with how the SSH connection is maintained or reused during the full deployment process?

Attempted Solutions

  • Increased SSH timeout to 300 seconds
  • Added more verbose logging and error handling
  • Verified SSH key presence and configuration

Any assistance in resolving this issue or suggestions for further debugging would be greatly appreciated.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: Ensure SSH connections are closed after each command execution jfanals/kamal
1 participant
@jfanals