About extending FIO output with combined mean and std dev latencies

[perezjosibm]

Hello All,

I joined the Ceph dev team at the start of this year, I've been particularly involved in performance testing for Crimson, using FIO. Recently, I've been extending some scripts to post-process the FIO JSON output to generate some response latency charts, like the following:

I've been using a single FIO client, with fixed number of jobs, increasing the iodepth value accordingly. (The y-error bars above represent the completion latency std dev as provided by FIO).

When running a FIO with multiple jobs, the resulting JSON contains a list of dictionaries, one per job. I will be extending my scripts to use the combined mean and std dev techniques (as in this reference) to combine all the jobs results into a single data point that I could chart as per the example.

I'd like to ask the following please:

• whether the combined mean and std dev technique would be worth considering to implemented as part of the output of FIO for such multi jobs cases, and if so, where in the FIO source code should I start looking for to extend?

• I noticed a more general tool in https://github.com/axboe/fio/blob/master/doc/fio-histo-log-pctiles.pdf and was wondering how to contrast with the afore mentioned technique.

Many thanks in advance.

[vincentkfu]

If group_reporting=1 then a single fio instance running multiple jobs can merge data from separate jobs and report combined means, std devs, latencies, etc. If you do not need data for separate jobs, then it might be easier to simply use this option.

I don't have a sufficiently complete understand of what you are doing to know if this will work for you.

The fio-histo-log-pctiles script seems to focus only on latency percentiles.

[perezjosibm]

Hi @vincentkfu thanks for your response, sorry I've been slow to come up with a counterexample: attached is the .json output from FIO multijob, each job exercising oits own RBD volume. I'm using the group_reporting and new_group options in the .fio worload (attached as well). What is interesting is that the results are presented separated per job. That's why I asked the question above, since it does not look like one get a combined result (latency) for all the jobs, but rather per individual job.
fio_crimson_1osd_default_fio_unresti_mj_4job_16io_4k_randwrite_p0.json
rbd_mj_randwrite.txt

[vincentkfu]

You have new_group in your [global] section. This nullifies the group_reporting setting. Try running your job with new_group removed.

[perezjosibm]

Thank you @vincentkfu, I've verified and indeed is the case 👍

[perezjosibm]

Closing