docker-machine create using digitalocean driver needs droplet reboot to start listening on 2376

[bitlush]

I noticed that last week my deployment scripts stopped working. The main problems is:

docker-machine create "--driver" "digitalocean" [other driver settings] "droplet-name"

This errors with:

docker-machine : Error creating machine: Error running provisioning: Unable to verify the Docker daemon is 
listening: Maximum number of retries (10) exceeded

This happens consistently. It seems to be because the docker host fails to listen on port 2376. I presume because docker does not start.

Rebooting the droplet after issuing docker-machine create forces docker to start listening on port 2376 and everything then works as normal, so I have added this workaround to my deployment scripts.

[runforrestrun]

How do you restart your droplet whilst it is just in creation mode by the "docker-machine create" command.

Is it possible to share the snippet of this workaround? Thanks in advance

[bitlush]

How do you restart your droplet whilst it is just in creation mode by the "docker-machine create" command.

Is it possible to share the snippet of this workaround? Thanks in advance

The full PowerShell script workaround:

docker-machine create $docker.driver $vms.web.name

#workaround to get docker listening (was working fine for the last 3 years and only needed recently):

$droplets = doctl compute droplet list $digitalOcean.auth | ConvertFrom-Json

$droplet = $droplets | Where-Object {$_.Name -eq $vms.web.name}

doctl compute droplet-action reboot $droplet.id $digitalOcean.auth

$ip = docker-machine ip $vms.web.name

do
{
    sleep 1
}
until(Test-NetConnection $ip -Port 2376 | ? { $_.TcpTestSucceeded })

So, the docker-machine create almost works, it's just that docker fails to start listening on 2376 (presumably the last step inside docker-machine). So it's not necessary to inject a command into the docker-machine create but just reboot the droplet afterwards and ignore the final error from docker-machine. This magically wakes up docker to start listening on port 2376. Everything below the first line of the script was unnecessary two weeks ago. It had been working fine for 3 years before this workaround was recently needed.

Note: waiting on port 2376 without rebooting does not work.

[runforrestrun]

Thanks. With your insights I decided to it manually.

1 - reboot the droplet.
2 - run docker-machine regenerate-certs machine-name

About 2 months I need to execute the script again, by that time, I hope it is resolved :)

[bitlush]

About 2 months I need to execute the script again, by that time, I hope it is resolved :)

Make sure to +1 the issue then!

[juanluisrp]

For me I needed to restart the Docker service (in an Ubuntu host).
Before the service wasn't listening in the network interface:

systemctl status docker.service
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/docker.service.d
           └─10-machine.conf
   Active: active (running) since Mon 2021-01-11 16:38:43 UTC; 2min 24s ago
     Docs: https://docs.docker.com
 Main PID: 20707 (dockerd)
   CGroup: /system.slice/docker.service
           └─20707 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

Jan 11 16:38:42 true dockerd[20707]: time="2021-01-11T16:38:42.849273393Z" level=warning msg="Your kernel does not support swap memory limit"
Jan 11 16:38:42 true dockerd[20707]: time="2021-01-11T16:38:42.849585773Z" level=warning msg="Your kernel does not support CPU realtime scheduler"
Jan 11 16:38:42 true dockerd[20707]: time="2021-01-11T16:38:42.850137298Z" level=info msg="Loading containers: start."
Jan 11 16:38:42 true dockerd[20707]: time="2021-01-11T16:38:42.976737533Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a prefe
Jan 11 16:38:43 true dockerd[20707]: time="2021-01-11T16:38:43.021144594Z" level=info msg="Loading containers: done."
Jan 11 16:38:43 true dockerd[20707]: time="2021-01-11T16:38:43.065147486Z" level=info msg="Docker daemon" commit=8891c58 graphdriver(s)=overlay2 version=20.10.2
Jan 11 16:38:43 true dockerd[20707]: time="2021-01-11T16:38:43.065334208Z" level=info msg="Daemon has completed initialization"
Jan 11 16:38:43 true systemd[1]: Started Docker Application Container Engine.
Jan 11 16:38:43 true dockerd[20707]: time="2021-01-11T16:38:43.097588922Z" level=info msg="API listen on /var/run/docker.sock"
Jan 11 16:38:46 true systemd[1]: Started Docker Application Container Engine.

After restarting docker.service it listens to 0.0.0.0:2376 and the daemon can be contacted by docker-machine:

systemctl restart docker.service

systemctl status docker.service
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/docker.service.d
           └─10-machine.conf
   Active: active (running) since Mon 2021-01-11 16:41:18 UTC; 2s ago
     Docs: https://docs.docker.com
 Main PID: 21666 (dockerd)
    Tasks: 12
   Memory: 41.1M
      CPU: 438ms
   CGroup: /system.slice/docker.service
           └─21666 /usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --storage-driver overlay2 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/doc

Jan 11 16:41:18 true dockerd[21666]: time="2021-01-11T16:41:18.302975014Z" level=warning msg="Your kernel does not support swap memory limit"
Jan 11 16:41:18 true dockerd[21666]: time="2021-01-11T16:41:18.303558253Z" level=warning msg="Your kernel does not support CPU realtime scheduler"
Jan 11 16:41:18 true dockerd[21666]: time="2021-01-11T16:41:18.304190321Z" level=info msg="Loading containers: start."
Jan 11 16:41:18 true dockerd[21666]: time="2021-01-11T16:41:18.483956848Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a prefe
Jan 11 16:41:18 true dockerd[21666]: time="2021-01-11T16:41:18.528351736Z" level=info msg="Loading containers: done."
Jan 11 16:41:18 true dockerd[21666]: time="2021-01-11T16:41:18.564827725Z" level=info msg="Docker daemon" commit=8891c58 graphdriver(s)=overlay2 version=20.10.2
Jan 11 16:41:18 true dockerd[21666]: time="2021-01-11T16:41:18.564977429Z" level=info msg="Daemon has completed initialization"
Jan 11 16:41:18 true systemd[1]: Started Docker Application Container Engine.
Jan 11 16:41:18 true dockerd[21666]: time="2021-01-11T16:41:18.607117310Z" level=info msg="API listen on [::]:2376"
Jan 11 16:41:18 true dockerd[21666]: time="2021-01-11T16:41:18.617467835Z" level=info msg="API listen on /var/run/docker.sock"

[juanluisrp]

This seems to be the same problem than #4858.