Graphical results

[ottapav]

Hi,
The results generated by this benchmark are pretty extensive already. With minimal additional effort, is it possible to add a plotting feature, e.g., similar to that for Maros Meszaros benchmark in https://github.com/Simple-Robotics/proxqp_benchmark?

[stephane-caron]

Good point, which I think was not mentioned before in the benchmark discussions. The plot you are mentioning here is a performance profile. We chose not to plot those in the benchmark for reasons detailed here: #49

There is however the beginning of a plotting tool with the hist command, which focuses on a single metric (metrics are runtime, primal residual, duality gap, ...). To take a use case close to a performance profile, we can look at runtime:

This plot being generated by this command line:

./benchmark.py maros_meszaros_dense hist runtime high_accuracy --solvers proxqp osqp scs --bins 42

[ottapav]

Good idea! Possible improvement: To show the total number of evaluated problems as the upper bound on the y-axis. It cannot be seen if e.g. proxqp solved all the problems eventually from this figure.

[ottapav]

I saw #39, but it is a little bit confusing that in the figure above proxqp solved one problem more than the others.

[stephane-caron]

You're right, this was actually an issue with the underlying histogram calculation. I implemented a simpler solution in #50 that fixes this.

Good idea! Possible improvement: To show the total number of evaluated problems as the upper bound on the y-axis. It cannot be seen if e.g. proxqp solved all the problems eventually from this figure.

Done 👍 You can pull the latest commit from main to get the fix.

Note that I've changed the command name from hist to just plot:

python benchmark.py maros_meszaros_dense plot runtime high_accuracy --solvers proxqp osqp scs

The result is now:

Thanks for your feedback!

[ottapav]

@stephane-caron Hi Stephan, when you make some changes in the plotting feature, please consider adding a unit for the x-axis (e.g., [s]). And could you make lines distinguishable in the grayscale (maybe by using some markers)? I'm finalizing a paper ;)

[stephane-caron]

For adjustments your best bet for now is to edit plot_metric.py directly. In general it would be better in benchmark code to separate data aggregation from plotting, which would make it easier to customize plots to taste. Feel free to open a feature request about this.

In the meantime, note that we can also customize the plots after the fact when passing the savefig argument:

$ ipython -i ./benchmark.py maros_meszaros_dense plot runtime default -- --savefig runtime.pdf
Python 3.8.10 (default, Mar 13 2023, 10:26:41) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.22.0 -- An enhanced Interactive Python. Type '?' for help.
[2023-05-02 10:26:29,479] [info] Loading existing results from ./results/maros_meszaros_dense.csv (results.py:80)

In [1]: from matplotlib import pyplot as plt

In [2]: plt.xlabel("Runtime [s]")
Out[2]: Text(0.5, 18.922222222222217, 'Runtime [s]')

In [3]: plt.show()

[ottapav]

Hi Stephane, Just for your information: Currently, I have finished the Python interface and started (local) integration to the qpsolvers to be able to run the benchmark with our solver. Pavel Otta čt 9. 2. 2023 v 18:03 odesílatel Stéphane Caron ***@***.***> napsal:

…

You're right, this was actually an issue with the underlying histogram calculation. I implemented a simpler solution in #50 <#50> that fixes this. Good idea! Possible improvement: To show the total number of evaluated problems as the upper bound on the y-axis. It cannot be seen if e.g. proxqp solved all the problems eventually from this figure. Done 👍 You can pull the latest commit from main to get the fix. Note that I've changed the command name from hist to just plot: python benchmark.py maros_meszaros_dense plot runtime high_accuracy --solvers proxqp osqp scs The result is now: [image: image] <https://user-images.githubusercontent.com/1189580/217885413-c3e68c24-d04c-4a76-9ba5-f859899f92e8.png> Thanks for your feedback! — Reply to this email directly, view it on GitHub <#48 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACSCH2WHXPO5QRTFCMSXQE3WWUPM5ANCNFSM6AAAAAAUVMKGZA> . You are receiving this because you authored the thread.Message ID: <stephane-caron/qpsolvers_benchmark/repo-discussions/48/comments/4922154@ github.com>

[stephane-caron]

Sounds good! Feel free to open a PR when it is ready.