Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Detect failures related to TCP simultaneous open and retry #2925

Open
elecbug opened this issue Aug 17, 2024 · 17 comments · May be fixed by #2950
Open

Detect failures related to TCP simultaneous open and retry #2925

elecbug opened this issue Aug 17, 2024 · 17 comments · May be fixed by #2950
Assignees

Comments

@elecbug
Copy link

elecbug commented Aug 17, 2024

While working on my personal project, I noticed that when using the noise protocol, both peers are intermittently getting error messages when they try to connect to each other at the same time.

Link's project is a reproduction of that part. and I solved the problem by storing each other in the peer store and connecting streams without trying to connect directly at the moment.

The file below is a log.txt file included in the link, which shows that noise errors occur intermittently when a docker is used to create an isolated environment and try to connect by repeatedly creating/removing another peer with one peer.

root@.../go-noise-problem# docker run -it test
2024/08/17 06:07:48 Self: [/ip4/127.0.0.1/tcp/39859 /ip4/172.17.0.2/tcp/39859] 12D3KooWEy9zuUv1qc6JfswHGgvZxCVw2RWQwmzMMgwQC6xQz2RT
2024/08/17 06:08:02 Found: {12D3KooWAn1oVgmzmYF64xQpPL52rVVWatCQw5s8WDbToaXTxvYw: [/ip4/127.0.0.1/tcp/41965 /ip4/172.17.0.3/tcp/41965]}
2024/08/17 06:08:09 Found: {12D3KooW9ySZUUY7b7xhEUkPny5NMGaMeLvuyQmmF7U6ieznJo7Y: [/ip4/127.0.0.1/tcp/33779 /ip4/172.17.0.3/tcp/33779]}
2024/08/17 06:08:24 Found: {12D3KooWNkiPAMLLorVuqjAPWjo6w6WE5CFT67b1dWVWyg5Ruh4g: [/ip4/127.0.0.1/tcp/37107 /ip4/172.17.0.3/tcp/37107]}
2024/08/17 06:08:27 Found: {12D3KooWHSbowE11HW22qdD8EGWMdBwD8cNXX334bWxYdNWk2n5g: [/ip4/127.0.0.1/tcp/39443 /ip4/172.17.0.3/tcp/39443]}
2024/08/17 06:08:29 Found: {12D3KooWAT1YDAZm5ux9ZTrjDCrGM8dGXLxYZcfZszsXGBx9xRRG: [/ip4/127.0.0.1/tcp/39481 /ip4/172.17.0.3/tcp/39481]}
2024/08/17 06:08:31 Found: {12D3KooWML2GZt9nP4JPprdno4H2EgcWcxmMh9gc1MvrfNdmF6AA: [/ip4/127.0.0.1/tcp/35611 /ip4/172.17.0.3/tcp/35611]}
2024/08/17 06:08:32 Found: {12D3KooWJZr4WhUoyNLRzaCaJSrbZ79GTEqGPNKhcwdcG2SqPpRd: [/ip4/127.0.0.1/tcp/37463 /ip4/172.17.0.3/tcp/37463]}
2024/08/17 06:08:34 failed to dial: failed to dial 12D3KooWGdPNKAWySFjmWCJXPNP4UBcdtE9uT9eGJd3HyvPChAgD: all dials failed
  * [/ip4/127.0.0.1/tcp/45231] dial tcp4 127.0.0.1:45231: connect: connection refused
  * [/ip4/172.17.0.3/tcp/45231] failed to negotiate security protocol: error reading handshake message: noise: message is too short

If you want to reproduce the situation, you can reproduce it if you follow the sequence of test.sh .

thanks for reading and efforting.

Version Information
noise-test
cloud.google.com/go v0.37.0
dmitri.shuralyov.com/app/changes v0.0.0-20180602232624-0a106ad413e3
dmitri.shuralyov.com/html/belt v0.0.0-20180602232347-f7d459c86be0
dmitri.shuralyov.com/service/change v0.0.0-20181023043359-a85b471d5412
dmitri.shuralyov.com/state v0.0.0-20180228185332-28bcc343414c
git.apache.org/thrift.git v0.0.0-20180902110319-2566ecd5d999
git.sr.ht/~sbinet/gg v0.5.0
github.com/AllenDang/gform v0.0.0-20130830051017-6bb9f0d6b77e
github.com/AllenDang/w32 v0.0.0-20180428130237-ad0a36d80adc
github.com/AndreasBriese/bbloom v0.0.0-20190825152654-46b345b51c96
github.com/BurntSushi/toml v0.3.1
github.com/Jorropo/jsync v1.0.1
github.com/aead/siphash v1.0.1
github.com/ajstarks/svgo v0.0.0-20211024235047-1546f124cd8b
github.com/alecthomas/kingpin/v2 v2.4.0
github.com/alecthomas/units v0.0.0-20231202071711-9a357b53e9c9
github.com/anmitsu/go-shlex v0.0.0-20161002113705-648efa622239
github.com/benbjohnson/clock v1.3.5
github.com/beorn7/perks v1.0.1
github.com/bradfitz/go-smtpd v0.0.0-20170404230938-deb6d6237625
github.com/btcsuite/btcd v0.22.1
github.com/btcsuite/btcd/btcec/v2 v2.1.3
github.com/btcsuite/btcd/chaincfg/chainhash v1.0.1
github.com/btcsuite/btclog v0.0.0-20170628155309-84c8d2346e9f
github.com/btcsuite/btcutil v0.0.0-20190207003914-4c204d697803
github.com/btcsuite/go-socks v0.0.0-20170105172521-4720035b7bfd
github.com/btcsuite/goleveldb v0.0.0-20160330041536-7834afc9e8cd
github.com/btcsuite/snappy-go v0.0.0-20151229074030-0bdef8d06723
github.com/btcsuite/websocket v0.0.0-20150119174127-31079b680792
github.com/btcsuite/winsvc v1.0.0
github.com/buger/jsonparser v0.0.0-20181115193947-bf1c66bbce23
github.com/campoy/embedmd v1.0.0
github.com/cenkalti/backoff/v4 v4.3.0
github.com/census-instrumentation/opencensus-proto v0.2.1
github.com/cespare/xxhash v1.1.0
github.com/cespare/xxhash/v2 v2.3.0
github.com/chzyer/readline v1.5.1
github.com/cilium/ebpf v0.9.1
github.com/client9/misspell v0.3.4
github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f
github.com/containerd/cgroups v1.1.0
github.com/coreos/go-semver v0.3.0
github.com/coreos/go-systemd v0.0.0-20181012123002-c6f51f82210d
github.com/coreos/go-systemd/v22 v22.5.0
github.com/cpuguy83/go-md2man/v2 v2.0.0
github.com/crackcomm/go-gitignore v0.0.0-20231225121904-e25f5bc08668
github.com/cskr/pubsub v1.0.2
github.com/davecgh/go-spew v1.1.1
github.com/davidlazar/go-crypto v0.0.0-20200604182044-b73af7476f6c
github.com/decred/dcrd/crypto/blake256 v1.0.1
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.3.0
github.com/dgraph-io/badger v1.6.2
github.com/dgraph-io/ristretto v0.0.2
github.com/docker/go-units v0.5.0
github.com/dustin/go-humanize v1.0.1
github.com/elastic/gosigar v0.14.3
github.com/envoyproxy/go-control-plane v0.9.4
github.com/envoyproxy/protoc-gen-validate v0.1.0
github.com/felixge/httpsnoop v1.0.4
github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568
github.com/flynn/noise v1.1.0
github.com/francoispqt/gojay v1.2.13
github.com/frankban/quicktest v1.14.6
github.com/fsnotify/fsnotify v1.4.7
github.com/gabriel-vasile/mimetype v1.4.4
github.com/ghodss/yaml v1.0.0
github.com/gliderlabs/ssh v0.1.1
github.com/go-errors/errors v1.0.1
github.com/go-fonts/liberation v0.3.2
github.com/go-kit/log v0.2.1
github.com/go-latex/latex v0.0.0-20231108140139-5c1ce85aa4ea
github.com/go-logfmt/logfmt v0.5.1
github.com/go-logr/logr v1.4.2
github.com/go-logr/stdr v1.2.2
github.com/go-pdf/fpdf v0.9.0
github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572
github.com/go-task/slim-sprig/v3 v3.0.0
github.com/go-yaml/yaml v2.1.0+incompatible
github.com/goccmack/gocc v0.0.0-20230228185258-2292f9e40198
github.com/godbus/dbus/v5 v5.1.0
github.com/gogo/protobuf v1.3.2
github.com/golang/freetype v0.0.0-20170609003504-e2365dfdc4a0
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da
github.com/golang/lint v0.0.0-20180702182130-06c8688daad7
github.com/golang/mock v1.6.0
github.com/golang/protobuf v1.5.3
github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db
github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c
github.com/google/go-cmp v0.6.0
github.com/google/go-github v17.0.0+incompatible
github.com/google/go-querystring v1.0.0
github.com/google/gopacket v1.1.19
github.com/google/martian v2.1.0+incompatible
github.com/google/pprof v0.0.0-20240727154555-813a5fbdbec8
github.com/google/renameio v0.1.0
github.com/google/uuid v1.6.0
github.com/googleapis/gax-go v2.0.0+incompatible
github.com/googleapis/gax-go/v2 v2.0.3
github.com/gopherjs/gopherjs v0.0.0-20190430165422-3e4dfb77656c
github.com/gorilla/mux v1.8.1
github.com/gorilla/websocket v1.5.3
github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7
github.com/grpc-ecosystem/grpc-gateway v1.5.0
github.com/grpc-ecosystem/grpc-gateway/v2 v2.20.0
github.com/gxed/hashland/keccakpg v0.0.1
github.com/gxed/hashland/murmur3 v0.0.1
github.com/hashicorp/errwrap v1.1.0
github.com/hashicorp/go-multierror v1.1.1
github.com/hashicorp/golang-lru v1.0.2
github.com/hashicorp/golang-lru/arc/v2 v2.0.7
github.com/hashicorp/golang-lru/v2 v2.0.7
github.com/hpcloud/tail v1.0.0
github.com/huin/goupnp v1.3.0
github.com/ianlancetaylor/demangle v0.0.0-20240312041847-bd984b5ce465
github.com/ipfs/bbloom v0.0.4
github.com/ipfs/boxo v0.21.0
github.com/ipfs/go-bitfield v1.1.0
github.com/ipfs/go-block-format v0.2.0
github.com/ipfs/go-blockservice v0.5.2
github.com/ipfs/go-cid v0.4.1
github.com/ipfs/go-cidutil v0.1.0
github.com/ipfs/go-datastore v0.6.0
github.com/ipfs/go-detect-race v0.0.1
github.com/ipfs/go-ds-badger v0.3.0
github.com/ipfs/go-ds-leveldb v0.5.0
github.com/ipfs/go-ipfs-blockstore v1.3.1
github.com/ipfs/go-ipfs-blocksutil v0.0.1
github.com/ipfs/go-ipfs-delay v0.0.1
github.com/ipfs/go-ipfs-ds-help v1.1.1
github.com/ipfs/go-ipfs-exchange-interface v0.2.1
github.com/ipfs/go-ipfs-pq v0.0.3
github.com/ipfs/go-ipfs-redirects-file v0.1.1
github.com/ipfs/go-ipfs-util v0.0.3
github.com/ipfs/go-ipld-cbor v0.1.0
github.com/ipfs/go-ipld-format v0.6.0
github.com/ipfs/go-ipld-legacy v0.2.1
github.com/ipfs/go-log v1.0.5
github.com/ipfs/go-log/v2 v2.5.1
github.com/ipfs/go-merkledag v0.11.0
github.com/ipfs/go-metrics-interface v0.0.1
github.com/ipfs/go-peertaskqueue v0.8.1
github.com/ipfs/go-unixfs v0.4.5
github.com/ipfs/go-unixfsnode v1.9.0
github.com/ipfs/go-verifcid v0.0.3
github.com/ipld/go-car v0.6.2
github.com/ipld/go-car/v2 v2.13.1
github.com/ipld/go-codec-dagpb v1.6.0
github.com/ipld/go-ipld-prime v0.21.0
github.com/jackpal/go-nat-pmp v1.0.2
github.com/jbenet/go-cienv v0.1.0
github.com/jbenet/go-temp-err-catcher v0.1.0
github.com/jbenet/goprocess v0.1.4
github.com/jellevandenhooff/dkim v0.0.0-20150330215556-f50fe3d243e1
github.com/jessevdk/go-flags v0.0.0-20141203071132-1679536dcc89
github.com/jpillora/backoff v1.0.0
github.com/jrick/logrotate v1.0.0
github.com/json-iterator/go v1.1.12
github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024
github.com/jtolds/gls v4.20.0+incompatible
github.com/julienschmidt/httprouter v1.3.0
github.com/kisielk/errcheck v1.5.0
github.com/kisielk/gotool v1.0.0
github.com/kkdai/bstream v0.0.0-20161212061736-f391b8402d23
github.com/klauspost/compress v1.17.9
github.com/klauspost/cpuid/v2 v2.2.8
github.com/koron/go-ssdp v0.0.4
github.com/kr/pretty v0.3.1
github.com/kr/pty v1.1.3
github.com/kr/text v0.2.0
github.com/libp2p/go-buffer-pool v0.1.0
github.com/libp2p/go-cidranger v1.1.0
github.com/libp2p/go-doh-resolver v0.4.0
github.com/libp2p/go-flow-metrics v0.1.0
github.com/libp2p/go-libp2p v0.36.1
github.com/libp2p/go-libp2p-asn-util v0.4.1
github.com/libp2p/go-libp2p-core v0.20.1
github.com/libp2p/go-libp2p-crypto v0.1.0
github.com/libp2p/go-libp2p-kad-dht v0.25.2
github.com/libp2p/go-libp2p-kbucket v0.6.3
github.com/libp2p/go-libp2p-noise v0.5.0
github.com/libp2p/go-libp2p-peerstore v0.8.0
github.com/libp2p/go-libp2p-pubsub v0.11.0
github.com/libp2p/go-libp2p-record v0.2.0
github.com/libp2p/go-libp2p-routing-helpers v0.7.4
github.com/libp2p/go-libp2p-testing v0.12.0
github.com/libp2p/go-libp2p-xor v0.1.0
github.com/libp2p/go-libp2p-yamux v0.10.0
github.com/libp2p/go-msgio v0.3.0
github.com/libp2p/go-nat v0.2.0
github.com/libp2p/go-netroute v0.2.1
github.com/libp2p/go-openssl v0.1.0
github.com/libp2p/go-reuseport v0.4.0
github.com/libp2p/go-yamux/v3 v3.1.1
github.com/libp2p/go-yamux/v4 v4.0.1
github.com/libp2p/zeroconf/v2 v2.2.0
github.com/lunixbochs/vtclean v1.0.0
github.com/mailru/easyjson v0.0.0-20190312143242-1de009706dbe
github.com/marten-seemann/tcp v0.0.0-20210406111302-dfbc87cc63fd
github.com/mattn/go-isatty v0.0.20
github.com/mattn/go-pointer v0.0.1
github.com/matttproud/golang_protobuf_extensions v1.0.4
github.com/microcosm-cc/bluemonday v1.0.1
github.com/miekg/dns v1.1.61
github.com/mikioh/tcp v0.0.0-20190314235350-803a9b46060c
github.com/mikioh/tcpinfo v0.0.0-20190314235526-30a79bb1804b
github.com/mikioh/tcpopt v0.0.0-20190314235656-172688c1accc
github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1
github.com/minio/sha256-simd v1.0.1
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
github.com/modern-go/reflect2 v1.0.2
github.com/mr-tron/base58 v1.2.0
github.com/multiformats/go-base32 v0.1.0
github.com/multiformats/go-base36 v0.2.0
github.com/multiformats/go-multiaddr v0.13.0
github.com/multiformats/go-multiaddr-dns v0.3.1
github.com/multiformats/go-multiaddr-fmt v0.1.0
github.com/multiformats/go-multibase v0.2.0
github.com/multiformats/go-multicodec v0.9.0
github.com/multiformats/go-multihash v0.2.3
github.com/multiformats/go-multistream v0.5.0
github.com/multiformats/go-varint v0.0.7
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822
github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f
github.com/neelance/astrewrite v0.0.0-20160511093645-99348263ae86
github.com/neelance/sourcemap v0.0.0-20151028013722-8c68805598ab
github.com/nxadm/tail v1.4.11
github.com/onsi/ginkgo v1.16.5
github.com/onsi/ginkgo/v2 v2.19.1
github.com/onsi/gomega v1.34.0
github.com/opencontainers/runtime-spec v1.2.0
github.com/opentracing/opentracing-go v1.2.0
github.com/openzipkin/zipkin-go v0.4.3
github.com/pbnjay/memory v0.0.0-20210728143218-7b4eea64cf58
github.com/petar/GoLLRB v0.0.0-20210522233825-ae3b015fd3e9
github.com/pion/datachannel v1.5.8
github.com/pion/dtls/v2 v2.2.12
github.com/pion/ice/v2 v2.3.32
github.com/pion/interceptor v0.1.29
github.com/pion/logging v0.2.2
github.com/pion/mdns v0.0.12
github.com/pion/randutil v0.1.0
github.com/pion/rtcp v1.2.14
github.com/pion/rtp v1.8.8
github.com/pion/sctp v1.8.20
github.com/pion/sdp/v3 v3.0.9
github.com/pion/srtp/v2 v2.0.20
github.com/pion/stun v0.6.1
github.com/pion/transport/v2 v2.2.9
github.com/pion/transport/v3 v3.0.6
github.com/pion/turn/v2 v2.1.6
github.com/pion/webrtc/v3 v3.2.50
github.com/pkg/errors v0.9.1
github.com/pmezard/go-difflib v1.0.0
github.com/polydawn/refmt v0.89.0
github.com/prometheus/client_golang v1.19.1
github.com/prometheus/client_model v0.6.1
github.com/prometheus/common v0.55.0
github.com/prometheus/procfs v0.15.1
github.com/quic-go/qpack v0.4.0
github.com/quic-go/qtls-go1-20 v0.3.2
github.com/quic-go/quic-go v0.45.2
github.com/quic-go/webtransport-go v0.8.0
github.com/raulk/go-watchdog v1.3.0
github.com/rodrigocfd/windigo v0.0.0-20230809154420-8faa606d9f5f
github.com/rogpeppe/go-internal v1.10.0
github.com/russross/blackfriday v1.5.2
github.com/russross/blackfriday/v2 v2.0.1
github.com/samber/lo v1.39.0
github.com/sclevine/agouti v3.0.0+incompatible
github.com/sergi/go-diff v1.0.0
github.com/shurcooL/component v0.0.0-20170202220835-f88ec8f54cc4
github.com/shurcooL/events v0.0.0-20181021180414-410e4ca65f48
github.com/shurcooL/github_flavored_markdown v0.0.0-20181002035957-2122de532470
github.com/shurcooL/go v0.0.0-20180423040247-9e1955d9fb6e
github.com/shurcooL/go-goon v0.0.0-20170922171312-37c2f522c041
github.com/shurcooL/gofontwoff v0.0.0-20180329035133-29b52fc0a18d
github.com/shurcooL/gopherjslib v0.0.0-20160914041154-feb6d3990c2c
github.com/shurcooL/highlight_diff v0.0.0-20170515013008-09bb4053de1b
github.com/shurcooL/highlight_go v0.0.0-20181028180052-98c3abbbae20
github.com/shurcooL/home v0.0.0-20181020052607-80b7ffcb30f9
github.com/shurcooL/htmlg v0.0.0-20170918183704-d01228ac9e50
github.com/shurcooL/httperror v0.0.0-20170206035902-86b7830d14cc
github.com/shurcooL/httpfs v0.0.0-20171119174359-809beceb2371
github.com/shurcooL/httpgzip v0.0.0-20180522190206-b1c53ac65af9
github.com/shurcooL/issues v0.0.0-20181008053335-6292fdc1e191
github.com/shurcooL/issuesapp v0.0.0-20180602232740-048589ce2241
github.com/shurcooL/notifications v0.0.0-20181007000457-627ab5aea122
github.com/shurcooL/octicon v0.0.0-20181028054416-fa4f57f9efb2
github.com/shurcooL/reactions v0.0.0-20181006231557-f2e0b4ca5b82
github.com/shurcooL/sanitized_anchor_name v1.0.0
github.com/shurcooL/users v0.0.0-20180125191416-49c67e49c537
github.com/shurcooL/webdavfs v0.0.0-20170829043945-18c3829fa133
github.com/sirupsen/logrus v1.8.1
github.com/smartystreets/assertions v1.2.0
github.com/smartystreets/goconvey v1.7.2
github.com/sourcegraph/annotate v0.0.0-20160123013949-f4cad6c6324d
github.com/sourcegraph/syntaxhighlight v0.0.0-20170531221838-bd320f5d308e
github.com/spacemonkeygo/openssl v0.0.0-20181017203307-c2dcc5cca94a
github.com/spacemonkeygo/spacelog v0.0.0-20180420211403-2296661a0572
github.com/spaolacci/murmur3 v1.1.0
github.com/stretchr/objx v0.5.2
github.com/stretchr/testify v1.9.0
github.com/syndtr/goleveldb v1.0.0
github.com/tarm/serial v0.0.0-20180830185346-98f6abe2eb07
github.com/ucarion/urlpath v0.0.0-20200424170820-7ccc79b76bbb
github.com/urfave/cli v1.22.10
github.com/viant/assertly v0.4.8
github.com/viant/toolbox v0.24.0
github.com/warpfork/go-testmark v0.12.1
github.com/warpfork/go-wish v0.0.0-20220906213052-39a1cc7a02d0
github.com/whyrusleeping/base32 v0.0.0-20170828182744-c30ac30633cc
github.com/whyrusleeping/cbor v0.0.0-20171005072247-63513f603b11
github.com/whyrusleeping/cbor-gen v0.1.2
github.com/whyrusleeping/chunker v0.0.0-20181014151217-fe64bd25879f
github.com/whyrusleeping/go-keyspace v0.0.0-20160322163242-5b898ac5add1
github.com/wlynxg/anet v0.0.3
github.com/xhit/go-str2duration/v2 v2.1.0
github.com/yuin/goldmark v1.4.13
go.opencensus.io v0.24.0
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.52.0
go.opentelemetry.io/otel v1.28.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.27.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.27.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.27.0
go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.27.0
go.opentelemetry.io/otel/exporters/zipkin v1.27.0
go.opentelemetry.io/otel/metric v1.28.0
go.opentelemetry.io/otel/sdk v1.27.0
go.opentelemetry.io/otel/trace v1.28.0
go.opentelemetry.io/proto/otlp v1.3.1
go.uber.org/atomic v1.11.0
go.uber.org/dig v1.17.1
go.uber.org/fx v1.22.1
go.uber.org/goleak v1.3.0
go.uber.org/mock v0.4.0
go.uber.org/multierr v1.11.0
go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee
go.uber.org/zap v1.27.0
go4.org v0.0.0-20180809161055-417644f6feb5
golang.org/x/build v0.0.0-20190111050920-041ab4dc3f9d
golang.org/x/crypto v0.25.0
golang.org/x/exp v0.0.0-20240719175910-8a7402abbf56
golang.org/x/image v0.14.0
golang.org/x/lint v0.0.0-20200302205851-738671d3881b
golang.org/x/mod v0.19.0
golang.org/x/net v0.27.0
golang.org/x/oauth2 v0.21.0
golang.org/x/perf v0.0.0-20180704124530-6e6d33e29852
golang.org/x/sync v0.7.0
golang.org/x/sys v0.22.0
golang.org/x/telemetry v0.0.0-20240521205824-bda55230c457
golang.org/x/term v0.22.0
golang.org/x/text v0.16.0
golang.org/x/time v0.5.0
golang.org/x/tools v0.23.0
golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028
gonum.org/v1/gonum v0.15.0
gonum.org/v1/plot v0.14.0
google.golang.org/api v0.1.0
google.golang.org/appengine v1.6.7
google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013
google.golang.org/genproto/googleapis/api v0.0.0-20240617180043-68d350f18fd4
google.golang.org/genproto/googleapis/rpc v0.0.0-20240617180043-68d350f18fd4
google.golang.org/grpc v1.64.0
google.golang.org/protobuf v1.34.2
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c
gopkg.in/errgo.v2 v2.1.0
gopkg.in/fsnotify.v1 v1.4.7
gopkg.in/inf.v0 v0.9.1
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7
gopkg.in/yaml.v2 v2.4.0
gopkg.in/yaml.v3 v3.0.1
grpc.go4.org v0.0.0-20170609214715-11d0a25b4919
honnef.co/go/tools v0.0.1-2019.2.3
lukechampine.com/blake3 v1.3.0
rsc.io/pdf v0.1.1
sourcegraph.com/sourcegraph/go-diff v0.5.0
sourcegraph.com/sqs/pbtypes v0.0.0-20180604144634-d3ebe8f20ae4
@MarcoPolo
Copy link
Collaborator

Thanks for the detailed report @elecbug! Have you noticed this issue in real networks, or just localhost/low-latency home networks?

My first suspicion is that this is related to TCP simultaneous open feature, since that causes confusion as to who is the server or client. Could you try running your repro with a small change for me to disprove this theory?

Please add these lines before your host.Connect

order := (bytes.Compare([]byte(host.ID()), []byte(found)))
if order == -1 {
    time.Sleep(100*time.Millisecond)
}
// existing code below
err = host.Connect(ctx, found)

This basically has one side wait a bit if its peer id is lexicographically less. This should remove any chance for simultaneous open.

If the issue still happens after that change, please report back and I'll dive deeper.

@elecbug
Copy link
Author

elecbug commented Aug 22, 2024

The experimental environment was a dockers warm network built with about eight servers.

First of all, it seems that the error no longer occurs in the way you suggested. host.Connect trying at the same time seems to have been a problem.

One request, then can I know the code area where the near-simultaneous call of host.Connect between the two peers is causing the problem?

@MarcoPolo
Copy link
Collaborator

One request, then can I know the code area where the near-simultaneous call of host.Connect between the two peers is causing the problem?

The "problem" is that TCP allows two endpoints to establish a connection if "two TCP simultaneously initiate the procedure"
(https://datatracker.ietf.org/doc/html/rfc793#section-3.4). As far as I know, there's no way to detect if you're in this state or not using just Go's runtime + stdlib.

May I suggest using the QUIC stack instead? You won't have these issues there.

@MarcoPolo MarcoPolo changed the title go-libp2p noise protocol problem Failure with TCP simultaneous open in a local setting Aug 22, 2024
@MarcoPolo
Copy link
Collaborator

Closing this as it's only an issue in local environments, and there isn't a great fix.

@Stebalien
Copy link
Member

Closing this as it's only an issue in local environments, and there isn't a great fix.

Note: This was fixed in multistream (not the best fix, but the best way to do it without breaking things), but then the fix was removed.

@MarcoPolo
Copy link
Collaborator

Context: #2330

@cpeliciari
Copy link
Contributor

I had the same problem in my project, basically this occurs because you are using mDNS, when after you create the peer if you wait a while before starting mDNS this does not happen.

@Stebalien
Copy link
Member

Really, the main issue is reuseport. An alternative is to use a different source port for outbound connections than the inbound port, but that makes it more difficult to detect port mappings in NATs, IIRC.

@MarcoPolo
Copy link
Collaborator

That's correct. reusing the port for dialing out lets us learn about our port mapping and our observed address as seen by our peer.

Maybe we should change the circumstances that trigger this. For example, in mDNS we can randomly delay publishing ourselves to avoid the simultaneous open in local environments.

@Stebalien
Copy link
Member

It's not just MDNS. Any scenario causing two nodes to try to connect at the same time will trigger it, especially if those nodes are far apart (e.g., synchronized reconnect timers, etc.).

@MarcoPolo
Copy link
Collaborator

Sorry for not being clear. I did not say this is a problem in just mDNS. I outlined a way to avoid this problem when peers learn about each other at the same time via mDNS.

I'm not sure how common this over the internet. I've not found any data to suggest this happens frequently. And we have some data that shows we don't see this (libp2p/specs#573). My intuition is that this is mainly a problem in networks with low latency such as the loopback interface and local networks, as peers can learn about each other quickly in those environments and send SYN packets to each other quickly.

If we accept the assumption that this is only an issue in local networks, we could not reuse the listen port for those networks. However, even if we do that, it would still be better to avoid having two connections when one would suffice. Therefore, changes to protocols that cause nodes to simultaneously connect would still be preferred.

@marten-seemann
Copy link
Contributor

You can detect simultaneous open in the security protocol, for example when a TLS client receives a TLS ClientHello. crypto/tls emits a really specific error. Then you can do a randomized backoff. Given how rare simultaneous open is in the wild, this doesn't cost you a lot.

Or just use QUIC, and avoid the entire simultaneous open nonsense...

@cpeliciari
Copy link
Contributor

You can detect simultaneous open in the security protocol, for example when a TLS client receives a TLS ClientHello. crypto/tls emits a really specific error. Then you can do a randomized backoff. Given how rare simultaneous open is in the wild, this doesn't cost you a lot.

Or just use QUIC, and avoid the entire simultaneous open nonsense...

Is it possible to implement quic-go with pnet?
I would really like to know about this.

@marten-seemann
Copy link
Contributor

No, there's no way to do that, see #1432.

@Stebalien
Copy link
Member

My intuition is that this is mainly a problem in networks with low latency such as the loopback interface and local networks, as peers can learn about each other quickly in those environments and send SYN packets to each other quickly.

It is more common in networks with high latency because SYN packets are more likely to cross on the wire.

Of course, you need a simultaneous trigger. But that can happen if an application has non-randomized reconnect timers and/or some central server signals two nodes to connect to each-other at the same time.

You can detect simultaneous open in the security protocol, for example when a TLS client receives a TLS ClientHello.

Yeah, this is the right way to fix it (that or use QUIC). And now that we have the holepunching logic, it should "just work" (return a retry error, backoff some amount of time (possibly even based on the peer IDs) and short-circuit if we see a new incoming connection).

@Stebalien
Copy link
Member

Is it possible to implement quic-go with pnet?

@cpeliciari I'd love to hear why you use PNET here: #2942.

@MarcoPolo
Copy link
Collaborator

MarcoPolo commented Aug 29, 2024

You can detect simultaneous open in the security protocol...

I like this idea. Let's do it. We also have to handle the case for when the failure happens in the multistream select layer, but it should be straightforward.

Reopening to track this fix.

@MarcoPolo MarcoPolo reopened this Aug 29, 2024
@MarcoPolo MarcoPolo changed the title Failure with TCP simultaneous open in a local setting Detect failures related to TCP simultaneous open and retry Aug 29, 2024
@MarcoPolo MarcoPolo self-assigned this Aug 29, 2024
@MarcoPolo MarcoPolo linked a pull request Sep 4, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants