Hi, thanks for bringing this up! Many people are concerned about performance changes in arrow-rs.

I have a question about codspeed: Where do these performance tests take place? Are they run on GitHub Actions or on a dedicated environment? How can we ensure stable performance metrics and avoid false positives?

I have a question about codspeed: Where do these performance tests take place? Are they run on GitHub Actions or on a dedicated environment?

They are run on GitHub Actions default runners, using https://github.com/CodSpeedHQ/action/.

How can we ensure stable performance metrics and avoid false positives?

Thanks to the action, the benchmarks are run on a simulated CPU and thus the measurements are stable (you can read more about it in our docs).

On the existing benchmarks, you can see that the results are quite consistent:

• On a sample PR https://codspeed.io/CodSpeedHQ/arrow-rs/branches/fork-6139-Rafferty97/simplify-struct-array
• On a specific benchmark (link to the page):
  

The links above is the CodSpeed integration of a forked arrow-rs.

Thanks @adriencaccia I'm strongly in favor of adding codspeed. I discussed this with @alamb and I think he was broadly in favor.

We've been using codspeed on pydantic-core since 2022, and it's absolutely key to maintaining and improving performance. See for example pydantic/pydantic-core#1155 (comment).

Thanks to the action, the benchmarks are run on a simulated CPU and thus the measurements are stable (you can read more about it in our docs).

That's great, thanks a lot for the link.