allow docker host access to enclave ports under linux ?

[garyng2000]

What's your CLI version?

0.90.1

Description & steps to reproduce

I am using a kurtosis package at github.com/ethpandaops/ethereum-package

which maps host ports to docker container ones(created via kurtosis) but I cannot access it from the host. how should the kurtosis package to be configured in order to allow this ?

Desired behavior

should be able to access mapped port from host

What is the severity of this bug?

Critical; I am blocked and Kurtosis is unusable for me because of this bug.

What area of the product does this pertain to?

Other: anything not covered by the above

[tedim52]

Hey @garyng2000 ! How are you attempting to access the ports? eg. localhost:61234with 61234 being the port on the host mapped to the port of the service inside the enclave ?

[garyng2000]

curl -s http://localhost: as shown in 'kurtosis enclave inspect '

========================================== User Services ==========================================
UUID Name Ports Status
9115b4d644b6 cl-1-lighthouse-geth http: 4000/tcp -> http://127.0.0.1:32877 RUNNING
metrics: 5054/tcp -> http://127.0.0.1:32876
tcp-discovery: 9000/tcp -> 127.0.0.1:32875
udp-discovery: 9000/udp -> 127.0.0.1:32789
683d6f13726e cl-2-lighthouse-geth http: 4000/tcp -> http://127.0.0.1:32880 RUNNING
metrics: 5054/tcp -> http://127.0.0.1:32879
tcp-discovery: 9000/tcp -> 127.0.0.1:32878
udp-discovery: 9000/udp -> 127.0.0.1:32790
f5803bd182e3 cl-3-teku-geth http: 4000/tcp -> http://127.0.0.1:32883 RUNNING
metrics: 8008/tcp -> 127.0.0.1:32882
tcp-discovery: 9000/tcp -> 127.0.0.1:32881
udp-discovery: 9000/udp -> 127.0.0.1:32791
7691562c9fb5 dora http: 8080/tcp -> http://127.0.0.1:32886 RUNNING
4a24f863b367 el-1-geth-lighthouse engine-rpc: 8551/tcp -> 127.0.0.1:32862 RUNNING
metrics: 9001/tcp -> 127.0.0.1:32861
rpc: 8545/tcp -> http://127.0.0.1:32864
tcp-discovery: 30303/tcp -> 127.0.0.1:32860
udp-discovery: 30303/udp -> 127.0.0.1:32786
ws: 8546/tcp -> 127.0.0.1:32863
e90e9bb1ee16 el-2-geth-lighthouse engine-rpc: 8551/tcp -> 127.0.0.1:32867 RUNNING
metrics: 9001/tcp -> 127.0.0.1:32866
rpc: 8545/tcp -> http://127.0.0.1:32869
tcp-discovery: 30303/tcp -> 127.0.0.1:32865
udp-discovery: 30303/udp -> 127.0.0.1:32787
ws: 8546/tcp -> 127.0.0.1:32868
6c2021fe91ad el-3-geth-teku engine-rpc: 8551/tcp -> 127.0.0.1:32872 RUNNING
metrics: 9001/tcp -> 127.0.0.1:32871
rpc: 8545/tcp -> http://127.0.0.1:32874
tcp-discovery: 30303/tcp -> 127.0.0.1:32870
udp-discovery: 30303/udp -> 127.0.0.1:32788
ws: 8546/tcp -> 127.0.0.1:32873
68d7758bdc9c validator-key-generation-cl-validator-keystore RUNNING
4be8bbb05868 vc-1-geth-lighthouse metrics: 8080/tcp -> http://127.0.0.1:32884 RUNNING
22175e303736 vc-2-geth-lighthouse metrics: 8080/tcp -> http://127.0.0.1:32885 RUNNING

[garyng2000]

so in this case curl -s http://localhost:32864. I get reset by peer error
I can get into the the container shell and access it without issue

wondering if it has anything to do with this

host.docker.internal:host-gateway

which is needed for my other docker-compose setup

below is the exact input/output
root@hqprdubada:/mnt/data3/chatgpt# curl -X POST -H "Content-Type: application/json" --data '{"method":"eth_blockNumber","params":[],"id":1,"jsonrpc":"2.0"}' http://127.0.0.1:32864
curl: (56) Recv failure: Connection reset by peer

[tedim52]

what are the network params you're running with? or are you using defaults? Also how are you seeing the host.docker.internal config/does it work if you unset it?

my hypothesis is that this setting host.docker.internal:host-gateway might make services in the enclave communicate on the host network which means when the el-1-geth-lighthouse attempts to connect with other peers on the IP address assigned to them on the enclave subnetwork, that IP address is invalid on the host network?

[garyng2000]

I am not using additional param, just straight setup from this(basically default)

https://geth.ethereum.org/docs/fundamentals/kurtosis

it seems that the 'dora' service can communicate with the geth container so it must be some docker level setup issue but that being wrapped inside kurtosis, I don't know where to start to trace

[tedim52]

Any chance you can join our discord? I'm happy to continue assisting you there! https://discord.com/invite/HUapYX9RvV

[garyng2000]

thanks for the help, I would try to use that channel later.

I may have find the reason as inspecting the geth container itself shows me this about network

"Networks": {
"kt-private3151908net": {
"IPAMConfig": {
"IPv4Address": "172.16.0.13"
},
"Links": null,
"Aliases": [
"el-3-geth-teku"
],
"MacAddress": "02:42:ac:10:00:0d",
"NetworkID": "150bb87a432c106db2b39f86325828a0e210a8d264e137787858bc42d8f6664d",
"EndpointID": "2951b8e68dbfbce91a7a8454bdda0bd8a3af13c46837b4696ce0b20beae6fa7b",
"Gateway": "172.16.0.4",
"IPAddress": "172.16.0.13",
"IPPrefixLen": 22,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"DriverOpts": null,
"DNSNames": [
"el-3-geth-teku--6c2021fe91ad46d5b8e12db3a8fc7c82",
"el-3-geth-teku",
"81f987de1343"
]
}
}

which may be conflicting with my hosting machine which is also using 172.16.0.x, so the question may then be anyway to configure the network setup in kurtosis for the 'private docker network' ? though just a WAG at the moment

[tedim52]

That makes sense. Background - Kurtosis creates a Docker bridge subnetwork for each enclave so that services within each enclave can only communicate with other services in the enclave. Any custom config that might alter how Docker networking works might affect this.

[garyng2000]

something doesn't make sense, in the created brdige(which seems to be what kurtosis is using)

br-b4e4c03a202a: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1440
inet 172.16.0.2 netmask 255.255.252.0 broadcast 172.16.3.255
inet6 fe80::42:53ff:fe3f:35cc prefixlen 64 scopeid 0x20
ether 02:42:53:3f:35:cc txqueuelen 0 (Ethernet)
RX packets 215 bytes 26166 (26.1 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 51 bytes 6221 (6.2 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

so this is using a bridging network thus the host mapping probably is not working as my limited understanding of docker is it needs to be in 'host' network mode for the port mapping to work

[garyng2000]

ok, it seems that it is the package itself https://github.com/ethpandaops/ethereum-package/tree/main is having issue, the default quickstart package runs find as the proper network configurations are used for the various containers

need to bug the package maintainer :-)