Integrating vegeta

[rsevilla87]

Type of change

• Refactor
• New feature
• Bug fix
• Optimization
• Documentation Update

Description

Adding vegeta to ingress-perf can be useful in the future. Have to perform more exhaustive tests with this tool.
Some of its benefits:

• Native JSON output.
• Support for not keepalived connections
• Contant throughput support. This is quite useful for latency focused benchmarks.
• It returns a summary of the returned status codes
• Possibility of integrating the plot functionality for debugging purposes in the future.

Downsides

• More resource intensive as compared to wrk, it requires more clients to reach a similar performance, this shouldn't be a problem, in most of our scenarios there're free CPU resources in the worker nodes
• Needs more testing if we want to transition to this tool

Switching wrk <--> vegeta should be transparent as the indexed documents follow the same format and structure.

Related Tickets & Documents

• Related Issue #
• Closes #

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.

Testing

• Please describe the System Under Test.
• Please provide detailed steps to perform tests related to this code change.
• How were the fix/results from this change verified? Please provide relevant screenshots or results.

[jtaleric]

Can we numerate the issues with wrk?

Maybe why we should possibly switch from wrk -> vegeta?

If this is to allow more options, great! But you mention switching to vegeta.

[krishvoor]

Multi-arch container might break here, as we are always pulling linux-amd64

[krishvoor]

By the looks of is vegeta is available for amd64/arm64 and not for other archs.

[rsevilla87]

Yeah, added a note about the supported archs. Only amd64 for now, we can work to integrate others in the future

[krishvoor]

Yeah, added a note about the supported archs. Only amd64 for now, we can work to integrate others in the future

Ack, I missed the README.md.

I do not want us to publish/ship multi-architecture images with a linux_amd64 binary,
can we please update this?
https://github.com/cloud-bulldozer/ingress-perf/blob/main/Makefile#L38

[krishvoor]

Yeah, added a note about the supported archs. Only amd64 for now, we can work to integrate others in the future

Ack, I missed the README.md.

I do not want us to publish/ship multi-architecture images with a linux_amd64 binary,
can we please update this?
https://github.com/cloud-bulldozer/ingress-perf/blob/main/Makefile#L38

[rsevilla87]

Can we numerate the issues with wrk?

Maybe why we should possibly switch from wrk -> vegeta?

If this is to allow more options, great! But you mention switching to vegeta.

One of the main reasons is to consolidate into a single tool. At the moment, wrk is great to measure throughput on HAProxy, however it lacks of some features like constant throughput configuration (available in wrk2 though), this feature turned out to be useful to perform latency focused, we need this kind of constant throughput to measure latency with more accuracy.

Other nice features vegeta brings is:

• Non keepalive connections support. This will allow us to define more benchmark scenarios
• More debugging options, the plot feature can be useful to perform debugging of regressions.

[jtaleric]

Can we numerate the issues with wrk?
Maybe why we should possibly switch from wrk -> vegeta?
If this is to allow more options, great! But you mention switching to vegeta.

One of the main reasons is to consolidate into a single tool. At the moment, wrk is great to measure throughput on HAProxy, however it lacks of some features like constant throughput configuration (available in wrk2 though), this feature turned out to be useful to perform latency focused, we need this kind of constant throughput to measure latency with more accuracy.

Other nice features vegeta brings is:

• Non keepalive connections support. This will allow us to define more benchmark scenarios
• More debugging options, the plot feature can be useful to perform debugging of regressions.

From the above it begs the question, why not just use wrk2? My main beef with wrk2 is that it seems to be a stale project - however if it gives us the performance and functionality, why switch to Vegeta?

I don't have an issue w/ Vegeta - but I am concerned if we switch, teams using our tooling will report a false flag bug due to the workload changing, and the harness not able to produce the same throughput.

PR looks fine BTW :)

[rsevilla87]

the harness not able to produce the same throughput.

I'm working on a config towards getting similar throughput using vegeta, so we can make a soft transition.

[rsevilla87]

I'm hitting some sporadic issues while validating vegeta, reported at tsenart/vegeta#651

[rsevilla87]

I'm hitting some sporadic issues while validating vegeta, reported at tsenart/vegeta#651

Rolling back to 12.9.0 fixed the issue

[rsevilla87]

I can't achieve a good performance on the passthrough termination, it seems like the connection is not using keepalive (I have seen a high number of sockets in TIME-WAIT state) for some reason.