Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create Sysbench tool and testcases using it for CPU/Memory/FileIO performance #3075

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

smit-gardhariya
Copy link
Collaborator

This PR has 2 commits.

  1. Create Sysbench Tool in LISA
  2. Create testcases for CPU/Memory/FileIO performance using Sysbench tool

@smit-gardhariya smit-gardhariya force-pushed the sgardhariya/sysbench_tool branch 3 times, most recently from 4f150a9 to 225ce70 Compare November 28, 2023 12:49
@smit-gardhariya smit-gardhariya force-pushed the sgardhariya/sysbench_tool branch 4 times, most recently from 7caff91 to 7bf8623 Compare November 28, 2023 14:07
lisa/tools/sysbench.py Outdated Show resolved Hide resolved
@smit-gardhariya smit-gardhariya force-pushed the sgardhariya/sysbench_tool branch 10 times, most recently from e7a3d69 to 30a9480 Compare November 30, 2023 08:27
lisa/tools/sysbench.py Outdated Show resolved Hide resolved
lisa/tools/sysbench.py Outdated Show resolved Hide resolved
lisa/tools/sysbench.py Outdated Show resolved Hide resolved
lisa/tools/sysbench.py Outdated Show resolved Hide resolved
lisa/tools/sysbench.py Outdated Show resolved Hide resolved
@smit-gardhariya smit-gardhariya force-pushed the sgardhariya/sysbench_tool branch 2 times, most recently from 0130a17 to 739a5e9 Compare December 6, 2023 12:16
lisa/messages.py Outdated
@@ -125,6 +125,22 @@ class PerfMessage(MessageBase):
role: str = ""
test_result_id: str = ""

# Sysbench common params
Copy link
Member

@squirrelsc squirrelsc Dec 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PerfMessage is the base class for all perf messages. Please either move them into existing perf message, or new perf messages. The fields shouldn't be tool oriented, they should be metrics oriented.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are the common one for sysbench. I put them here as i dont want to have redundant params for all child PerfMessage classes.

I created new perf message dataclass now for sysbench for all those common params and inherited it for Disk/CPU/Memory perf dataclasses

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The perf messages should be metrics oriented, instead of tool oriented. It's easy to create dashboard and explain by metrics concepts. It's the same to sysbench common fields. It looks many common fields are not help a lot as a "common" field, the tested metrics are the key information of a perf message. We need to abstract the values to perf metrics.

For example, it's easy to understand x_latency_ms in network tests, but hard to understand in CPU tests, and I'm not sure what does it mean in the mem tests. The "events" number is highly depends on the test scenarios and settings.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you look at the sysbench output it will always give latency metrics like min/max/avg. That was the reason to put it under common sysbench metrics

lisa/tools/sysbench.py Outdated Show resolved Hide resolved
test_name,
other_fields,
)
print(subtest_msg)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove the debug lines. You can add console notifier to print all messages in debug level.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's still here, not be removed.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done now. I added it later again and still pushed it.

lisa/messages.py Outdated
@@ -168,6 +190,38 @@ class DiskPerformanceMessage(PerfMessage):
randwrite_iops: Decimal = Decimal(0)
randwrite_lat_usec: Decimal = Decimal(0)

# Sysbench FileIO params
read_io_per_sec: Decimal = Decimal(0)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please try best to reuse above fields, like read_io_per_sec should be read_iops above. So the data from different tools can be compared together.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check all other fields too.

  1. Make sure it's a common concept, instead of sysbench concept.
  2. Create it in the Disk message, if it doesn't exist. But not group them like "sysbench params".

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will check on this

lisa/messages.py Outdated

@dataclass
class MemoryPerformanceMessage(PerfMessage):
total_operations: Decimal = Decimal(0)
Copy link
Member

@squirrelsc squirrelsc Dec 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

operations_per_sec is easier to understand, if it includes the time unit. And similar to other fields.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are 2 field for memory perf message. One is total_operation and other is operations_per_second.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If so, the total_operation doesn't need here. It's hard to compare cross different settings. We don't need to save all details in sysbench output.

Copy link
Collaborator Author

@smit-gardhariya smit-gardhariya Dec 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed it now

lisa/messages.py Outdated
@@ -149,6 +165,12 @@ class PerfMessage(MessageBase):
)


@dataclass
class CPUPerformanceMessage(PerfMessage):
cpu_max_prime: Decimal = Decimal(0)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it a test parameter to calculate max prime? If so, use a field like "benchmark", and value is "prime". So it's easy to extend to other perf methodologies.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is max number till which sysbench will try to get the prime numbers to generate CPU stress. Default is 10000 i.e. sysbench cpu run would try to get prime number upto 10000.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a test approach. If it's set as a column name, it's hard to extend other tests in the same CPU table. So, the columns should be like below.

benchmark: prime_10000
max_cpu_speed: ???

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

lisa/tools/sysbench.py Outdated Show resolved Hide resolved
threads: int = 1,
events: int = 0,
time_limit: int = 10,
memory_block_size_in_kb: int = 1,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Those arguments are not set in test case level too, they should be removed, unless they are used. Similar for fild and cpu.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is correct that we are not using them as of now as we are running with default values. There is plan to run the CPU/Memory perf test using sysbench in near future for our validation stack which would required to run sysbech to be run with different parameter.

Those args are necessary to expose all command-line parameter of sysbench.

I would suggest if we can keep this so we dont need or need minimal changes at tool level. Please let me know your thoughts on this

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add them back in "near future", when it's used. That would be clearer than add "all" now.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@smit-gardhariya smit-gardhariya force-pushed the sgardhariya/sysbench_tool branch 4 times, most recently from 03671cc to 6c5d058 Compare December 11, 2023 10:53
memory_access_mode: List[str] = ["seq", "rnd"]
sysbench = node.tools[Sysbench]

for op in memory_operation:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

similar to fileio, please move the loop into the tool.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

raise LisaException(
f"Invalid memory_scope. Valid memory_scope: {valid_mem_scope}"
)
elif memory_oper not in valid_mem_operation:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar with fileio, use 3 if, instead of elif on unrelated checks.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@dataclass
class MemoryPerformanceMessage(PerfMessage, SysbenchPerfMessage):
total_operations: Decimal = Decimal(0)
total_mib_transferred: int = 0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should it be total_mib_transferred_per_sec? It's easy to cross check on different test settings.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have 2 fields for this. total_mib_transferred and mib_per_second

Here is the reference output. We are fetching both the figure under above metrics.

64190.27 MiB transferred (6417.03 MiB/sec)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks similar to total operations, it's not a metrics to measure perf. We don't need to collect all information. The comparable metrics is important. Other data can be found in log for troubleshooting.

This commit has changes to add sysbench as tool to lisa
This is needed for CPU/Memory/Fileio perf test using
sysbench tool.

Signed-off-by: Smit Gardhariya <[email protected]>
This commit has changes to add testcases for CPU/Memory/FileIO
using sysbench tool of LISA.

Signed-off-by: Smit Gardhariya <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants