using BatchLookupAndDelete for PerCPU hashmap

[msherif1234]

Hi Team:
I am using latest cilium code base and trying to take advantage of BatchAPIs for PerCPU hashmap
I tried to follow sample test example

ebpf/map_test.go

Line 2167 in f95957d

b.Run("BatchLookupAndDelete", func(b *testing.B) {

by doing something like this

        possibleCPU := ebpf.MustPossibleCPU()
	var ids = make([]BpfFlowId, m.cacheMaxSize)
	var metrics = make([]BpfFlowMetrics, m.cacheMaxSize*possibleCPU)
	var cursor = ebpf.BatchCursor{}

	iterator := flowMap.Iterate()
	if iterator.Next(&id, &metrics) {
		count, err := flowMap.BatchLookupAndDelete(&cursor, ids, metrics, nil)
		if errors.Is(err, ebpf.ErrKeyNotExist) && count > 0 {
			break;
		}
		if err != nil {
		    log.Debugf("failed to use BatchLookupAndDelete api: %v fall back to use iterate and delete api", err)
                }
	}

but I am getting this error
per-CPU slice has incorrect length, expected 12, got 60000
do I have to size the slice to exactly match what is the map currently ?

maybe I should wait for #1271 ?

[lmb]

You're mixing MapIterator with the batch API. The two are mutually exclusive. The core loop you want is this:

ebpf/map_test.go

Lines 2181 to 2190 in f95957d

	var cursor BatchCursor
	for {
	_, err := m.BatchLookupAndDelete(&cursor, k, v, nil)
	if errors.Is(err, ErrKeyNotExist) {
	break
	}
	if err != nil {
	b.Fatal(err)
	}
	}

[msherif1234]

@lmb have u done any profiling for the new perCPU back api

(pprof) top10 -cum
Showing nodes accounting for 70ms, 3.14% of 2230ms total
Dropped 56 nodes (cum <= 11.15ms)
Showing top 10 nodes out of 92
      flat  flat%   sum%        cum   cum%
         0     0%     0%     1770ms 79.37%  github.com/netobserv/netobserv-ebpf-agent/pkg/flow.(*MapTracer).evictFlows
         0     0%     0%     1770ms 79.37%  github.com/netobserv/netobserv-ebpf-agent/pkg/flow.(*MapTracer).evictionSynchronization
         0     0%     0%     1760ms 78.92%  github.com/netobserv/netobserv-ebpf-agent/pkg/ebpf.(*FlowFetcher).LookupAndDeleteMap
         0     0%     0%     1620ms 72.65%  github.com/cilium/ebpf.(*Map).BatchLookupAndDelete (inline)
         0     0%     0%     1620ms 72.65%  github.com/cilium/ebpf.(*Map).batchLookup
         0     0%     0%     1620ms 72.65%  github.com/cilium/ebpf.(*Map).batchLookupPerCPU
      40ms  1.79%  1.79%     1510ms 67.71%  github.com/cilium/ebpf/internal/sysenc.Unmarshal
      30ms  1.35%  3.14%     1350ms 60.54%  encoding/binary.Read
         0     0%  3.14%     1190ms 53.36%  github.com/cilium/ebpf.unmarshalBatchPerCPUValue
         0     0%  3.14%     1180ms 52.91%  github.com/cilium/ebpf.unmarshalPerCPUValue
(pprof) 

the unmarshal and read are very slow for large tables is that expected doing single delete is much better performance wise

if I am doing delete in a loop

(pprof) top10 -cum
Showing nodes accounting for 30ms, 10.71% of 280ms total
Showing top 10 nodes out of 91
      flat  flat%   sum%        cum   cum%
         0     0%     0%      150ms 53.57%  github.com/netobserv/netobserv-ebpf-agent/pkg/flow.(*MapTracer).evictFlows
         0     0%     0%      150ms 53.57%  github.com/netobserv/netobserv-ebpf-agent/pkg/flow.(*MapTracer).evictionSynchronization
         0     0%     0%      150ms 53.57%  runtime.systemstack
         0     0%     0%      110ms 39.29%  runtime.gcBgMarkWorker
         0     0%     0%      110ms 39.29%  runtime.gcBgMarkWorker.func2
         0     0%     0%      110ms 39.29%  runtime.gcDrain
         0     0%     0%       80ms 28.57%  github.com/netobserv/netobserv-ebpf-agent/pkg/ebpf.(*FlowFetcher).LookupAndDeleteMap
      30ms 10.71% 10.71%       80ms 28.57%  runtime.scanobject
         0     0% 10.71%       70ms 25.00%  runtime.mallocgc
         0     0% 10.71%       60ms 21.43%  runtime.makeBucketArray

[ti-mo]

@msherif1234 I don't think the second benchmark is doing much at all. If you want to compare anything, please include a reproducer.

[msherif1234]

when I ran the benchmark testing it seems really good compared to iterate and delete for PerCPU

go test -exec sudo -bench=BenchmarkIterate -benchmem -count 5 -run=^#
goos: linux
goarch: amd64
pkg: github.com/cilium/ebpf
cpu: Intel(R) Core(TM) i7-10850H CPU @ 2.70GHz
BenchmarkIterate/PerCPUHash/MapIteratorDelete-12         	      22	  50287739 ns/op	 1729870 B/op	   24009 allocs/op
BenchmarkIterate/PerCPUHash/MapIteratorDelete-12         	      22	  50960934 ns/op	 1729932 B/op	   24010 allocs/op
BenchmarkIterate/PerCPUHash/MapIteratorDelete-12         	      22	  51824886 ns/op	 1729887 B/op	   24009 allocs/op
BenchmarkIterate/PerCPUHash/MapIteratorDelete-12         	      21	  49716260 ns/op	 1729937 B/op	   24010 allocs/op
BenchmarkIterate/PerCPUHash/MapIteratorDelete-12         	      22	  51736339 ns/op	 1729928 B/op	   24010 allocs/op
BenchmarkIterate/PerCPUHash/BatchLookupAndDelete-12      	     126	   9474336 ns/op	  962112 B/op	    8004 allocs/op
BenchmarkIterate/PerCPUHash/BatchLookupAndDelete-12      	     128	   9240052 ns/op	  962112 B/op	    8004 allocs/op
BenchmarkIterate/PerCPUHash/BatchLookupAndDelete-12      	     127	   9409226 ns/op	  962112 B/op	    8004 allocs/op
BenchmarkIterate/PerCPUHash/BatchLookupAndDelete-12      	     126	   9409235 ns/op	  962112 B/op	    8004 allocs/op
BenchmarkIterate/PerCPUHash/BatchLookupAndDelete-12      	     127	   9430533 ns/op	  962112 B/op	    8004 allocs/op
PASS
ok  	github.com/cilium/ebpf	29.737s

but unfortuntely that isn't what I see in my application netobserv/netobserv-ebpf-agent#256 I even changed the bench mark to use larger tables in case that may change things but it seems to be fine, I will keep digging probably I am doing something incorrect
Thanks @ti-mo

[msherif1234]

Hi @ti-mo can this netobserv/netobserv-ebpf-agent@4597d54
considered as reproducer
if I did it correctly there isn't major improvement with batch APIs

$ go test ./pkg/ebpf/ -exec sudo -bench=BenchmarkFlowFetcher_LookupAndDeleteMap -benchmem -count 5 -run=^#
goos: linux
goarch: amd64
pkg: github.com/netobserv/netobserv-ebpf-agent/pkg/ebpf
cpu: Intel(R) Core(TM) i7-10850H CPU @ 2.70GHz
BenchmarkFlowFetcher_LookupAndDeleteMap/BatchLookupAndDeleteWithGC-12         	     873	   1235421 ns/op	  696408 B/op	    2356 allocs/op
BenchmarkFlowFetcher_LookupAndDeleteMap/BatchLookupAndDeleteWithGC-12         	     835	   1317482 ns/op	  698811 B/op	    2379 allocs/op
BenchmarkFlowFetcher_LookupAndDeleteMap/BatchLookupAndDeleteWithGC-12         	     874	   1258660 ns/op	  696294 B/op	    2355 allocs/op
BenchmarkFlowFetcher_LookupAndDeleteMap/BatchLookupAndDeleteWithGC-12         	     858	   1272001 ns/op	  697334 B/op	    2365 allocs/op
BenchmarkFlowFetcher_LookupAndDeleteMap/BatchLookupAndDeleteWithGC-12         	     825	   1305501 ns/op	  699497 B/op	    2385 allocs/op
BenchmarkFlowFetcher_LookupAndDeleteMap/BatchLookupAndDeleteWithOutGC-12      	    1053	    997212 ns/op	  686220 B/op	    2268 allocs/op
BenchmarkFlowFetcher_LookupAndDeleteMap/BatchLookupAndDeleteWithOutGC-12      	    1076	   1014321 ns/op	  685333 B/op	    2259 allocs/op
BenchmarkFlowFetcher_LookupAndDeleteMap/BatchLookupAndDeleteWithOutGC-12      	    1035	   1158018 ns/op	  687043 B/op	    2275 allocs/op
BenchmarkFlowFetcher_LookupAndDeleteMap/BatchLookupAndDeleteWithOutGC-12      	     991	   1226809 ns/op	  689077 B/op	    2294 allocs/op
BenchmarkFlowFetcher_LookupAndDeleteMap/BatchLookupAndDeleteWithOutGC-12      	    1142	   1141634 ns/op	  682797 B/op	    2235 allocs/op
BenchmarkFlowFetcher_LookupAndDeleteMap/IterateLookupAndDelete-12             	     784	   1371729 ns/op	  538124 B/op	    3777 allocs/op
BenchmarkFlowFetcher_LookupAndDeleteMap/IterateLookupAndDelete-12             	     853	   1331418 ns/op	  533304 B/op	    3731 allocs/op
BenchmarkFlowFetcher_LookupAndDeleteMap/IterateLookupAndDelete-12             	     800	   1356575 ns/op	  537021 B/op	    3766 allocs/op
BenchmarkFlowFetcher_LookupAndDeleteMap/IterateLookupAndDelete-12             	     817	   1292504 ns/op	  535344 B/op	    3753 allocs/op
BenchmarkFlowFetcher_LookupAndDeleteMap/IterateLookupAndDelete-12             	     886	   1280091 ns/op	  530824 B/op	    3711 allocs/op
PASS
ok  	github.com/netobserv/netobserv-ebpf-agent/pkg/ebpf	75.327s