Unable to connect to Docker TCP service #1374

crawford · 2014-06-17T08:50:55Z

MacOSX 10.9.3
OpenSSH_6.2p2
Virtualbox 4.3.12
Vagrant 1.6.3
Coreos v343.0.0 and v349.0.0

Setting $expose_docker_tcp=2375 in config.rb and invoking export DOCKER_HOST='tcp://127.0.0.1:2375' does not give the expected results.

local shell:

$ docker images
2014/06/17 10:37:10 Get http://127.0.0.1:2375/v1.12/images/json: EOF

VM:

$ cat /etc/systemd/system/docker-tcp.socket
[Unit]
Description=Docker Socket for the API

[Socket]
ListenStream=127.0.0.1:2375
Service=docker.service
BindIPv6Only=both

[Install]
WantedBy=sockets.target

Set via project user-data:

$ cat /var/lib/coreos-vagrant/vagrantfile-user-data
#cloud-config

coreos:
 etcd:
 #generate a new token for each unique cluster from https://discovery.etcd.io/new
 #discovery: https://discovery.etcd.io/<token>
 addr: $public_ipv4:4001
 peer-addr: $public_ipv4:7001
 units:
 - name: etcd.service
 command: start
 - name: fleet.service
 command: start
 runtime: no
 content: |
 [Unit]
 Description=fleet

 [Service]
 Environment=FLEET_PUBLIC_IP=$public_ipv4
 ExecStart=/usr/bin/fleet
 - name: docker-tcp.socket
 command: start
 enable: true
 content: |
 [Unit]
 Description=Docker Socket for the API

 [Socket]
 ListenStream=127.0.0.1:2375
 Service=docker.service
 BindIPv6Only=both

 [Install]
 WantedBy=sockets.target
 - name: enable-docker-tcp.service
 command: start
 content: |
 [Unit]
 Description=Enable the Docker Socket for the API

 [Service]
 Type=oneshot
 ExecStart=/usr/bin/systemctl enable docker-tcp.socket

On the VM docker commands are working. But the docker commands in the local shell fail.
Probably unrelated but I'm also seeing a ssh connection error in vagrant up. See: #131

The text was updated successfully, but these errors were encountered:

crawford · 2014-06-19T19:39:50Z

Comment by philips

@MarcDiethelm when you login to the machine and see if docker-tcp.socket is running?

crawford · 2014-06-19T19:40:01Z

Comment by philips

systemctl status docker-tcp.socket

crawford · 2014-06-19T19:40:22Z

Comment by philips

If it isn't running do:

systemctl stop docker.service
systemctl start docker-tcp.socket docker.socket

Then try again.

crawford · 2014-06-19T20:47:55Z

Comment by MarcDiethelm

Yeah, as mentioned in the VM the Docker commands work. I did manually stop and start the service previously but no joy.

crawford · 2014-06-19T20:49:44Z

Comment by MarcDiethelm

I'm on the road currently but will try the status check when I can.

crawford · 2014-06-20T08:04:00Z

Comment by MarcDiethelm

Yup.

$ systemctl status docker-tcp.socket
● docker-tcp.socket - Docker Socket for the API
 Loaded: loaded (/etc/systemd/system/docker-tcp.socket; enabled)
 Active: active (listening) since Fri 2014-06-20 07:58:53 UTC; 1min 29s ago
 Listen: 127.0.0.1:2375 (Stream)

Jun 20 07:58:53 core-01 systemd[1]: Starting Docker Socket for the API.
Jun 20 07:58:53 core-01 systemd[1]: Listening on Docker Socket for the API.
Jun 20 07:58:54 core-01 systemd[1]: Listening on Docker Socket for the API.

But no response in the local shell.

$ docker images
2014/06/20 09:59:25 Get http://127.0.0.1:2375/v1.12/images/json: EOF

crawford · 2014-09-17T02:59:29Z

Comment by razic

vagrant version

~/coreos-vagrant master vagrant --version
Vagrant 1.6.5

`user-data`:

#cloud-config
coreos:
 etcd:
 addr: $public_ipv4:4001
 peer-addr: $public_ipv4:7001
 discovery: https://discovery.etcd.io/e792d9ee556fddc746ca93aa6202cadf
 fleet:
 public-ip: $public_ipv4
 units:
 - name: etcd.service
 command: start
 - name: fleet.service
 command: start
 - name: docker-tcp.socket
 command: start
 enable: true
 content: |
 [Unit]
 Description=Docker Socket for the API

 [Socket]
 ListenStream=2375
 Service=docker.service
 BindIPv6Only=both

 [Install]
 WantedBy=sockets.target
 [Socket]
 ListenStream=2375
 Service=docker.service
 BindIPv6Only=both

 [Install]
 WantedBy=sockets.target

`config.rb`:

# To automatically replace the discovery token on 'vagrant up', uncomment
# the lines below:
if File.exists?('user-data') && ARGV[0].eql?('up')
 require 'open-uri'
 require 'yaml'

 token = open('https://discovery.etcd.io/new').read

 data = YAML.load(IO.readlines('user-data')[1..-1].join)
 data['coreos']['etcd']['discovery'] = token

 lines = YAML.dump(data).split("\n")
 lines[0] = '#cloud-config'

 open('user-data', 'r+') do |f|
 f.puts(lines.join("\n"))
 end
end

# coreos-vagrant is configured through a series of configuration
# options (global ruby variables) which are detailed below. To modify
# these options, first copy this file to "config.rb". Then simply
# uncomment the necessary lines, leaving the $, and replace everything
# after the equals sign..

# Size of the CoreOS cluster created by Vagrant
$num_instances=3

# Official CoreOS channel from which updates should be downloaded
#$update_channel='alpha'

# Log the serial consoles of CoreOS VMs to log/
# Enable by setting value to true, disable with false
# WARNING: Serial logging is known to result in extremely high CPU usage with
# VirtualBox, so should only be used in debugging situations
#$enable_serial_logging=false

# Enable port forwarding of Docker TCP socket
# Set to the TCP port you want exposed on the *host* machine, default is 2375
# If 2375 is used, Vagrant will auto-increment (e.g. in the case of $num_instances > 1)
# You can then use the docker tool locally by setting the following env var:
# export DOCKER_HOST='tcp://127.0.0.1:2375'
$expose_docker_tcp=2375

# Setting for VirtualBox VMs
#$vb_gui = false
#$vb_memory = 1024
#$vb_cpus = 1

Building the Cluster

~/coreos-vagrant master vagrant up
Bringing machine 'core-01' up with 'virtualbox' provider...
Bringing machine 'core-02' up with 'virtualbox' provider...
Bringing machine 'core-03' up with 'virtualbox' provider...
==> core-01: Importing base box 'coreos-alpha'...
==> core-01: Matching MAC address for NAT networking...
==> core-01: Checking if box 'coreos-alpha' is up to date...
==> core-01: Setting the name of the VM: coreos-vagrant_core-01_1410922149559_88128
==> core-01: Clearing any previously set network interfaces...
==> core-01: Preparing network interfaces based on configuration...
 core-01: Adapter 1: nat
 core-01: Adapter 2: hostonly
==> core-01: Forwarding ports...
 core-01: 2375 => 2375 (adapter 1)
 core-01: 22 => 2222 (adapter 1)
==> core-01: Running 'pre-boot' VM customizations...
==> core-01: Booting VM...
==> core-01: Waiting for machine to boot. This may take a few minutes...
 core-01: SSH address: 127.0.0.1:2222
 core-01: SSH username: core
 core-01: SSH auth method: private key
 core-01: Warning: Connection timeout. Retrying...
==> core-01: Machine booted and ready!
==> core-01: Setting hostname...
==> core-01: Configuring and enabling network interfaces...
==> core-01: Running provisioner: file...
==> core-01: Running provisioner: shell...
 core-01: Running: inline script
==> core-02: Importing base box 'coreos-alpha'...
==> core-02: Matching MAC address for NAT networking...
==> core-02: Checking if box 'coreos-alpha' is up to date...
==> core-02: Setting the name of the VM: coreos-vagrant_core-02_1410922168730_19927
==> core-02: Fixed port collision for 22 => 2222. Now on port 2200.
==> core-02: Clearing any previously set network interfaces...
==> core-02: Preparing network interfaces based on configuration...
 core-02: Adapter 1: nat
 core-02: Adapter 2: hostonly
==> core-02: Forwarding ports...
 core-02: 2375 => 2376 (adapter 1)
 core-02: 22 => 2200 (adapter 1)
==> core-02: Running 'pre-boot' VM customizations...
==> core-02: Booting VM...
==> core-02: Waiting for machine to boot. This may take a few minutes...
 core-02: SSH address: 127.0.0.1:2200
 core-02: SSH username: core
 core-02: SSH auth method: private key
 core-02: Warning: Connection timeout. Retrying...
==> core-02: Machine booted and ready!
==> core-02: Setting hostname...
==> core-02: Configuring and enabling network interfaces...
==> core-02: Running provisioner: file...
==> core-02: Running provisioner: shell...
 core-02: Running: inline script
==> core-03: Importing base box 'coreos-alpha'...
==> core-03: Matching MAC address for NAT networking...
==> core-03: Checking if box 'coreos-alpha' is up to date...
==> core-03: Setting the name of the VM: coreos-vagrant_core-03_1410922188019_86776
==> core-03: Fixed port collision for 22 => 2222. Now on port 2201.
==> core-03: Clearing any previously set network interfaces...
==> core-03: Preparing network interfaces based on configuration...
 core-03: Adapter 1: nat
 core-03: Adapter 2: hostonly
==> core-03: Forwarding ports...
 core-03: 2375 => 2377 (adapter 1)
 core-03: 22 => 2201 (adapter 1)
==> core-03: Running 'pre-boot' VM customizations...
==> core-03: Booting VM...
==> core-03: Waiting for machine to boot. This may take a few minutes...
 core-03: SSH address: 127.0.0.1:2201
 core-03: SSH username: core
 core-03: SSH auth method: private key
 core-03: Warning: Connection timeout. Retrying...
==> core-03: Machine booted and ready!
==> core-03: Setting hostname...
==> core-03: Configuring and enabling network interfaces...
==> core-03: Running provisioner: file...
==> core-03: Running provisioner: shell...
 core-03: Running: inline script

Trying to use Docker Over the TCP Port

~/coreos-vagrant master export DOCKER_HOST=tcp://localhost:2375
~/coreos-vagrant master docker ps
2014/09/16 19:52:04 Get http://localhost:2375/v1.14/containers/json: EOF

Checking the Docker Services with `systemctl` on `core-01`

~/coreos-vagrant master vagrant ssh core-01
CoreOS (alpha)
core@core-01 ~ $

core@core-01 ~ $ systemctl status docker
● docker.service - Docker Application Container Engine
 Loaded: loaded (/usr/lib64/systemd/system/docker.service; disabled)
 Active: inactive (dead)
 Docs: http://docs.docker.io

core@core-01 ~ $ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

wtf?

docker.service:

core@core-01 ~ $ systemctl status docker
● docker.service - Docker Application Container Engine
 Loaded: loaded (/usr/lib64/systemd/system/docker.service; disabled)
 Active: active (running) since Wed 2014-09-17 02:53:54 UTC; 2s ago
 Docs: http://docs.docker.io
 Process: 914 ExecStartPre=/bin/mount --make-rprivate / (code=exited, status=0/SUCCESS)
 Main PID: 916 (docker)
 CGroup: /system.slice/docker.service
 └─916 /usr/bin/docker --daemon --storage-driver=btrfs --host=fd://

Sep 17 02:53:54 core-01 docker[916]: [info] Listening for HTTP on fd ()
Sep 17 02:53:54 core-01 docker[916]: [473afc16.init_networkdriver()] getting iface addr
Sep 17 02:53:54 core-01 docker[916]: [473afc16] -job init_networkdriver() = OK (0)
Sep 17 02:53:55 core-01 docker[916]: [info] Loading containers:
Sep 17 02:53:55 core-01 docker[916]: [info] : done.
Sep 17 02:53:55 core-01 docker[916]: [473afc16] +job acceptconnections()
Sep 17 02:53:55 core-01 docker[916]: [473afc16] -job acceptconnections() = OK (0)
Sep 17 02:53:55 core-01 docker[916]: [info] GET /v1.14/containers/json
Sep 17 02:53:55 core-01 docker[916]: [473afc16] +job containers()
Sep 17 02:53:55 core-01 docker[916]: [473afc16] -job containers() = OK (0)

docker.socket:

core@core-01 ~ $ systemctl status docker.socket
● docker.socket - Docker Socket for the API
 Loaded: loaded (/usr/lib64/systemd/system/docker.socket; disabled)
 Active: active (running) since Wed 2014-09-17 02:49:19 UTC; 6min ago
 Listen: /var/run/docker.sock (Stream)

Sep 17 02:49:19 localhost systemd[1]: Starting Docker Socket for the API.
Sep 17 02:49:19 localhost systemd[1]: Listening on Docker Socket for the API.

docker-tcp.port:

core@core-01 ~ $ systemctl status docker-tcp.port
● docker-tcp.port.service
 Loaded: not-found (Reason: No such file or directory)
 Active: inactive (dead)
core@core-01 ~ $

core@core-01 ~ $ systemctl stop docker.service
Failed to stop docker.service: Access denied
Warning: Stopping docker.service, but it can still be activated by:
 docker.socket
core@core-01 ~ $ sudo !!
sudo systemctl stop docker.service
Warning: Stopping docker.service, but it can still be activated by:
 docker.socket
core@core-01 ~ $ systemctl start docker-tcp.socket docker.socket
Failed to start docker-tcp.socket: Access denied
Failed to start docker.socket: Access denied
core@core-01 ~ $ sudo !!
sudo systemctl start docker-tcp.socket docker.socket
Job for docker-tcp.socket failed. See 'systemctl status docker-tcp.socket' and 'journalctl -xn' for details.

why do i have to sudo? and why won't it start damn it....

Looking at `journalctl`

core@core-01 ~ $ journalctl -xn
-- Logs begin at Wed 2014-09-17 02:49:17 UTC, end at Wed 2014-09-17 02:58:58 UTC. --
Sep 17 02:57:59 core-01 update_engine[476]: [0917/025759:INFO:omaha_request_action.cc(433)] No update.
Sep 17 02:57:59 core-01 update_engine[476]: [0917/025759:INFO:action_processor.cc(82)] ActionProcessor::ActionComplete: finished OmahaR
Sep 17 02:57:59 core-01 update_engine[476]: [0917/025759:INFO:omaha_response_handler_action.cc(36)] There are no updates. Aborting.
Sep 17 02:57:59 core-01 update_engine[476]: [0917/025759:INFO:action_processor.cc(68)] ActionProcessor::ActionComplete: OmahaResponseHa
Sep 17 02:57:59 core-01 update_engine[476]: [0917/025759:INFO:action_processor.cc(73)] ActionProcessor::ActionComplete: finished last a
Sep 17 02:57:59 core-01 update_engine[476]: [0917/025759:INFO:update_attempter.cc(531)] Processing Done.
Sep 17 02:57:59 core-01 update_engine[476]: [0917/025759:INFO:update_attempter.cc(582)] No update.
Sep 17 02:57:59 core-01 update_engine[476]: [0917/025759:INFO:update_check_scheduler.cc(84)] Next update check in 42m10s
Sep 17 02:57:59 core-01 locksmithd[494]: LastCheckedTime=0 Progress=0 CurrentOperation="UPDATE_STATUS_IDLE" NewVersion=0.0.0.0 NewSize=
Sep 17 02:58:58 core-01 sudo[998]: core : TTY=pts/0 ; PWD=/home/core ; USER=root ; COMMAND=/bin/journalctl -xn

crawford · 2014-09-17T03:16:49Z

Comment by razic

Referenced from #169 this is the cause of this bug!

crawford · 2014-10-08T12:13:04Z

Comment by BugRoger

For what it's worth I solved this by using the following enable-docker-tcp.service. I'm pretty sure you can do this properly using systemd primitives. I monkeyed around with it quite a while but having the docker service already started when cloud-init runs and with something keep on restarting it through the socket activation I caved in and just manually orchestrated the socket start order.

 - name: enable-docker-tcp.service
 command: start
 content: |
 [Unit]
 Description=Enable the Docker Socket for the API

 [Service]
 ExecStartPre=/usr/bin/systemctl stop docker.socket
 ExecStartPre=/usr/bin/systemctl stop docker-tcp.socket
 ExecStartPre=/usr/bin/systemctl stop docker
 ExecStart=/usr/bin/systemctl enable docker-tcp.socket
 ExecStartPost=/usr/bin/systemctl start docker.socket
 ExecStartPost=/usr/bin/systemctl start docker-tcp.socket
 Type=oneshot
 RemainAfterExit=true

crawford · 2015-09-30T21:21:12Z

Comment by jeremyd

Thanks @BugRoger , +1 to your solution as the official copy/paste of cloud-config.yml does not appear to enable Dockersocket by itself. The service you posted is required to actually restart docker with the API support on boot.

ender74 · 2016-08-10T08:12:40Z

I did create a pull request for the solution from BugRoger

coreos/coreos-vagrant#296

crawford added the component/vagrant label Jun 3, 2016

ender74 mentioned this issue Aug 10, 2016

added enable-docker-tcp.service (see comments for issue 1374) coreos/coreos-vagrant#296

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to connect to Docker TCP service #1374

Unable to connect to Docker TCP service #1374

crawford commented Jun 17, 2014

crawford commented Jun 19, 2014

crawford commented Jun 19, 2014

crawford commented Jun 19, 2014

crawford commented Jun 19, 2014

crawford commented Jun 19, 2014

crawford commented Jun 20, 2014

crawford commented Sep 17, 2014

crawford commented Sep 17, 2014

crawford commented Oct 8, 2014

crawford commented Sep 30, 2015

ender74 commented Aug 10, 2016

Unable to connect to Docker TCP service #1374

Unable to connect to Docker TCP service #1374

Comments

crawford commented Jun 17, 2014

crawford commented Jun 19, 2014

crawford commented Jun 19, 2014

crawford commented Jun 19, 2014

crawford commented Jun 19, 2014

crawford commented Jun 19, 2014

crawford commented Jun 20, 2014

crawford commented Sep 17, 2014

vagrant version

user-data:

config.rb:

Building the Cluster

Trying to use Docker Over the TCP Port

Checking the Docker Services with systemctl on core-01

Looking at journalctl

crawford commented Sep 17, 2014

crawford commented Oct 8, 2014

crawford commented Sep 30, 2015

ender74 commented Aug 10, 2016

`user-data`:

`config.rb`:

Checking the Docker Services with `systemctl` on `core-01`

Looking at `journalctl`