No vswitch connectivity when vpp is using multiple cores

[itailev]

deployment:

• OS: ubuntu 18.04
• simple deployment (1 x master, 1 x worker), static IPs, no STN
• dedicated Mellanox NIC for data plane
• test containers on different nodes communicate over contiv-vpp data plane

Issue:
with the config below everything seems to work as expected, however when I start the vpp with multiple cores for example by using "corelist-workers 2,3", the vswitch containers are not communicating any more with each other (ping 192.168.16.X is not working any more), and interfaces counters showing drops.

contiv-vpp.yaml:

    NodeConfig:
    - nodeName: "k8s-master"
      mainVPPInterface:
        interfaceName: "HundredGigabitEthernet7/0/0"
        ip: "192.168.16.1/24"
      gateway: "192.168.16.100"
    - nodeName: "k8s-worker1"
      mainVPPInterface:
        interfaceName: "HundredGigabitEthernet7/0/0"
        ip: "192.168.16.2/24"
      gateway: "192.168.16.100"

contiv-vswitch.conf:

unix {
   nodaemon
   cli-listen /run/vpp/cli.sock
   cli-no-pager
   log /tmp/vpp.log
   poll-sleep-usec 100
}
nat {
   endpoint-dependent
   translation hash buckets 1048576
   translation hash memory 268435456
   user hash buckets 1024
   max translations per user 10000
}
acl-plugin {
   use tuple merge 0
}
api-trace {
    on
    nitems 5000
}
cpu {
  corelist-workers 2
  main-core 1
}
dpdk {
   dev 0000:07:00.0
}

get-pods output:

NAME                                 READY   STATUS    RESTARTS   AGE     IP             NODE          NOMINATED NODE   READINESS GATES
contiv-crd-4ml9p                     1/1     Running   0          77m     10.7.208.212   k8s-master    <none>           <none>
contiv-etcd-0                        1/1     Running   0          77m     10.7.208.212   k8s-master    <none>           <none>
contiv-ksr-8rhbc                     1/1     Running   0          77m     10.7.208.212   k8s-master    <none>           <none>
contiv-vswitch-g9jjq                 1/1     Running   0          77m     10.7.208.212   k8s-master    <none>           <none>
contiv-vswitch-mpjwn                 1/1     Running   1          77m     10.7.208.207   k8s-worker1   <none>           <none>
coredns-86c58d9df4-h7j5z             1/1     Running   0          76m     10.1.1.4       k8s-master    <none>           <none>
coredns-86c58d9df4-hd8t6             1/1     Running   0          76m     10.1.2.3       k8s-worker1   <none>           <none>
etcd-k8s-master                      1/1     Running   0          4h24m   10.7.208.212   k8s-master    <none>           <none>
kube-apiserver-k8s-master            1/1     Running   0          4h24m   10.7.208.212   k8s-master    <none>           <none>
kube-controller-manager-k8s-master   1/1     Running   0          4h24m   10.7.208.212   k8s-master    <none>           <none>
kube-proxy-s5wbx                     1/1     Running   0          4h18m   10.7.208.207   k8s-worker1   <none>           <none>
kube-proxy-trh5t                     1/1     Running   0          4h25m   10.7.208.212   k8s-master    <none>           <none>
kube-scheduler-k8s-master            1/1     Running   0          4h24m   10.7.208.212   k8s-master    <none>           <none>

VPP interfaces and IPs (on master node):

vpp# show int
              Name               Idx    State  MTU (L3/IP4/IP6/MPLS)     Counter          Count     
HundredGigabitEthernet7/0/0       1      up          9000/0/0/0     rx packets               2846420
                                                                    rx bytes              4305779163
                                                                    tx packets                986407
                                                                    tx bytes               115507026
                                                                    drops                         29
                                                                    ip4                      2846404
                                                                    ip6                            2
local0                            0     down          0/0/0/0
loop0                             2      up          9000/0/0/0
loop1                             6      up          9000/0/0/0     rx packets               2846384
                                                                    rx bytes              4123602867
                                                                    tx packets               1972786
                                                                    tx bytes               132373324
                                                                    ip4                      2846384
tap0                              3      up          1450/0/0/0     rx packets                  3396
                                                                    rx bytes                 1428440
                                                                    tx packets                  3389
                                                                    tx bytes                  254751
                                                                    drops                         13
                                                                    ip4                         3383
                                                                    ip6                            1
tap1                              4      up          1450/0/0/0     rx packets                  2000
                                                                    rx bytes                  156658
                                                                    tx packets                  2002
                                                                    tx bytes                  742350
                                                                    drops                          1
                                                                    ip4                         1999
                                                                    ip6                            1
tap2                              7      up          1450/0/0/0     rx packets                985012
                                                                    rx bytes                65501146
                                                                    tx packets               2845006
                                                                    tx bytes              4163354584
                                                                    ip4                       985012
vxlan_tunnel0                     5      up           0/0/0/0       rx packets               2846384
                                                                    rx bytes              4163452243
                                                                    tx packets                986393
                                                                    tx bytes               101696810
vpp#
vpp# show int addr
HundredGigabitEthernet7/0/0 (up):
  L3 192.168.16.1/24
local0 (dn):
loop0 (up):
  L3 10.1.1.1/24 ip4 table-id 1 fib-idx 1
loop1 (up):
  L2 bridge bd-id 1 idx 1 shg 1 bvi
  L3 192.168.30.1/24 ip4 table-id 1 fib-idx 1
tap0 (up):
  L3 172.30.1.1/24
tap1 (up):
  unnumbered, use loop0
  L3 10.1.1.1/24 ip4 table-id 1 fib-idx 1
tap2 (up):
  unnumbered, use loop0
  L3 10.1.1.1/24 ip4 table-id 1 fib-idx 1
vxlan_tunnel0 (up):
  L2 bridge bd-id 1 idx 1 shg 

[itailev]

any one was able to run contiv vpp vswitch over multiple cores?

[itailev]

an update:
when disabling vxlan (useL2Interconnect: true), vswitch containers running on multiple cores were able to communicate. seems like the issue is related to vxlan connectivity with multiple cores.

[rastislavs]

Hi @itaimlx
thanks for reporting this. Yes, we know that there might be some issues with using multiple cores on VPP, especially with the NAT plugin. Single-thread is the only supported mode for now.

[rastislavs]

The issue will be addressed with the upcoming VPP 19.08 release.