Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add expected results tests #2361

Merged
merged 9 commits into from
Jul 24, 2023
Merged

Add expected results tests #2361

merged 9 commits into from
Jul 24, 2023

Conversation

timcassell
Copy link
Collaborator

Resolves #2356.

@timcassell timcassell force-pushed the expected-results-tests branch 2 times, most recently from 6af9151 to 85fc35a Compare July 9, 2023 10:01
@AndreyAkinshin AndreyAkinshin merged commit 2936272 into master Jul 24, 2023
12 checks passed
@AndreyAkinshin AndreyAkinshin deleted the expected-results-tests branch July 24, 2023 11:15
@AndreyAkinshin AndreyAkinshin added this to the v0.13.7 milestone Jul 24, 2023
@AndreyAkinshin
Copy link
Member

@timcassell Thanks for implementing these tests! I decided to merge it, but the tests require more polishing to ensure stability in different environments (at the moment, they are flaky). I will improve them by myself (although, it would take some time).

@timcassell
Copy link
Collaborator Author

@AndreyAkinshin I'm not surprised they're flaky. It's a very delicate measurement. I had tried to set RealTime priority to try to make it better, but I reverted that due to #2366. Maybe it will be less flaky with the change in #2374?

@AndreyAkinshin
Copy link
Member

First of all, I want to introduce more reliable asserts. I will do it as soon as we bump the used Perfolizer version.

toddbaert pushed a commit to open-feature/dotnet-sdk that referenced this pull request Aug 14, 2024
@renovate
chore(deps): update dependency benchmarkdotnet to v0.14.0 (#293)
aec222f 
[![Mend
Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [BenchmarkDotNet](https://togithub.com/dotnet/BenchmarkDotNet) |
`0.13.1` -> `0.14.0` |
[![age](https://developer.mend.io/api/mc/badges/age/nuget/BenchmarkDotNet/0.14.0?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![adoption](https://developer.mend.io/api/mc/badges/adoption/nuget/BenchmarkDotNet/0.14.0?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![passing](https://developer.mend.io/api/mc/badges/compatibility/nuget/BenchmarkDotNet/0.13.1/0.14.0?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![confidence](https://developer.mend.io/api/mc/badges/confidence/nuget/BenchmarkDotNet/0.13.1/0.14.0?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|

---

### Release Notes

<details>
<summary>dotnet/BenchmarkDotNet (BenchmarkDotNet)</summary>

###
[`v0.14.0`](https://togithub.com/dotnet/BenchmarkDotNet/releases/tag/v0.14.0):
0.14.0

Full changelog: https://benchmarkdotnet.org/changelog/v0.14.0.html

#### Highlights

- Introduce `BenchmarkDotNet.Diagnostics.dotMemory`
[#&#8203;2549](https://togithub.com/dotnet/BenchmarkDotNet/pull/2549):
memory allocation profile of your benchmarks using
[dotMemory](https://www.jetbrains.com/dotmemory/), see
[@&#8203;BenchmarkDotNet](https://togithub.com/BenchmarkDotNet).Samples.IntroDotMemoryDiagnoser
- Introduce `BenchmarkDotNet.Exporters.Plotting`
[#&#8203;2560](https://togithub.com/dotnet/BenchmarkDotNet/pull/2560):
plotting via [ScottPlot](https://scottplot.net/) (initial version)
-   Multiple bugfixes
- The default build toolchains have been updated to pass
`IntermediateOutputPath`, `OutputPath`, and `OutDir` properties to the
`dotnet build` command. This change forces all build outputs to be
placed in a new directory generated by BenchmarkDotNet, and fixes many
issues that have been reported with builds. You can also access these
paths in your own `.csproj` and `.props` from those properties if you
need to copy custom files to the output.

#### Bug fixes

- Fixed multiple build-related bugs including passing MsBuildArguments
and .Net 8's `UseArtifactsOutput`.

#### Breaking Changes

- `DotNetCliBuilder` removed `retryFailedBuildWithNoDeps` constructor
option.
- `DotNetCliCommand` removed `RetryFailedBuildWithNoDeps` property and
`BuildNoRestoreNoDependencies()` and `PublishNoBuildAndNoRestore()`
methods (replaced with `PublishNoRestore()`).

###
[`v0.13.12`](https://togithub.com/dotnet/BenchmarkDotNet/releases/tag/v0.13.12):
0.13.12

Full changelog: https://benchmarkdotnet.org/changelog/v0.13.12.html

#### Highlights

The biggest highlight of this release if our new VSTest Adapter, which
allows to run benchmarks as unit tests in your favorite IDE!
The detailed guide can be found
[here](https://benchmarkdotnet.org/articles/features/vstest.html).

This release also includes to a minor bug fix that caused incorrect job
id generation: fixed job id generation
([#&#8203;2491](https://togithub.com/dotnet/BenchmarkDotNet/pull/2491)).

Also, the target framework in the BenchmarkDotNet templates was bumped
to .NET 8.0.

###
[`v0.13.11`](https://togithub.com/dotnet/BenchmarkDotNet/releases/tag/v0.13.11):
0.13.11

Full changelog: https://benchmarkdotnet.org/changelog/v0.13.11.html

In the
[v0.13.11](https://togithub.com/dotnet/BenchmarkDotNet/issues?q=milestone:v0.13.11)
scope, 4 issues were resolved and 8 pull requests were merged. This
release includes 28 commits by 7 contributors.

#### Resolved issues (4)

-
[#&#8203;2060](https://togithub.com/dotnet/BenchmarkDotNet/issues/2060)
NativeAOT benchmark started from .Net Framework host doesn't have all
intrinsics enabled (assignee:
[@&#8203;adamsitnik](https://togithub.com/adamsitnik))
-
[#&#8203;2233](https://togithub.com/dotnet/BenchmarkDotNet/issues/2233)
Q: Include hardware counters in XML output (assignee:
[@&#8203;nazulg](https://togithub.com/nazulg))
-
[#&#8203;2388](https://togithub.com/dotnet/BenchmarkDotNet/issues/2388)
Include AVX512 in listed HardwareIntrinsics
-
[#&#8203;2463](https://togithub.com/dotnet/BenchmarkDotNet/issues/2463)
Bug. Native AOT .NET 7.0 doesn't work. System.NotSupportedException:
X86Serialize (assignee:
[@&#8203;adamsitnik](https://togithub.com/adamsitnik))

#### Merged pull requests (8)

- [#&#8203;2412](https://togithub.com/dotnet/BenchmarkDotNet/pull/2412)
Add HardwareIntrinsics AVX-512 info (by
[@&#8203;nietras](https://togithub.com/nietras))
- [#&#8203;2458](https://togithub.com/dotnet/BenchmarkDotNet/pull/2458)
Adds Metrics Columns to Benchmark Report Output (by
[@&#8203;nazulg](https://togithub.com/nazulg))
- [#&#8203;2459](https://togithub.com/dotnet/BenchmarkDotNet/pull/2459)
Enable MemoryDiagnoser on Legacy Mono (by
[@&#8203;MichalPetryka](https://togithub.com/MichalPetryka))
- [#&#8203;2462](https://togithub.com/dotnet/BenchmarkDotNet/pull/2462)
update SDK to .NET 8 (by
[@&#8203;adamsitnik](https://togithub.com/adamsitnik))
- [#&#8203;2464](https://togithub.com/dotnet/BenchmarkDotNet/pull/2464)
Use "native" for .NET 8, don't use "serialize" for .NET 7 (by
[@&#8203;adamsitnik](https://togithub.com/adamsitnik))
- [#&#8203;2465](https://togithub.com/dotnet/BenchmarkDotNet/pull/2465)
fix NativeAOT toolchain and tests (by
[@&#8203;adamsitnik](https://togithub.com/adamsitnik))
- [#&#8203;2468](https://togithub.com/dotnet/BenchmarkDotNet/pull/2468)
Add OperationsPerSecondAttribute (by
[@&#8203;DarkWanderer](https://togithub.com/DarkWanderer))
- [#&#8203;2475](https://togithub.com/dotnet/BenchmarkDotNet/pull/2475)
Fix some tests (by
[@&#8203;timcassell](https://togithub.com/timcassell))

#### Commits (28)

-
[bb55e6](https://togithub.com/dotnet/BenchmarkDotNet/commit/bb55e6b067829c74e04838255e96d949857d5731)
Set next BenchmarkDotNet version: 0.13.11 (by
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[db4d8b](https://togithub.com/dotnet/BenchmarkDotNet/commit/db4d8b6d8a652db4bb1e4b1b4b0cd9df917e9584)
Adds Metrics Columns to Benchmark Report Output
([#&#8203;2458](https://togithub.com/dotnet/BenchmarkDotNet/issues/2458))
(by [@&#8203;nazulg](https://togithub.com/nazulg))
-
[e93b2b](https://togithub.com/dotnet/BenchmarkDotNet/commit/e93b2b1b332fc90da4934025e2edba7d67a15b54)
Use "native" for .NET 8, don't use "serialize" for .NET 7
([#&#8203;2464](https://togithub.com/dotnet/BenchmarkDotNet/issues/2464))
(by [@&#8203;adamsitnik](https://togithub.com/adamsitnik))
-
[127157](https://togithub.com/dotnet/BenchmarkDotNet/commit/127157924014afe2d0b58398d682381a855d7c34)
\[build] Fix spellcheck-docs workflow (by
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[8a02ec](https://togithub.com/dotnet/BenchmarkDotNet/commit/8a02ec28d55529f9be0ea66d843049738b2be8fa)
\[build] Use our .NET SDK on Windows (by
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[1b39e8](https://togithub.com/dotnet/BenchmarkDotNet/commit/1b39e8e6d5437bdbf0bb62986e680e54b19cc873)
Suppress NU1903 in IntegrationTests.ManualRunning.MultipleFrameworks (by
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[e90311](https://togithub.com/dotnet/BenchmarkDotNet/commit/e90311539d78e4bf9d90c6aeae9f40219b31a4ac)
update SDK to .NET 8
([#&#8203;2462](https://togithub.com/dotnet/BenchmarkDotNet/issues/2462))
(by [@&#8203;adamsitnik](https://togithub.com/adamsitnik))
-
[fc7afe](https://togithub.com/dotnet/BenchmarkDotNet/commit/fc7afeddcff7a52ccee165ac99ba216e8eb138ab)
Enable MemoryDiagnoser on Legacy Mono
([#&#8203;2459](https://togithub.com/dotnet/BenchmarkDotNet/issues/2459))
(by [@&#8203;MichalPetryka](https://togithub.com/MichalPetryka))
-
[630622](https://togithub.com/dotnet/BenchmarkDotNet/commit/630622b6df3192f766ffa03ff07b5086e70cb264)
fix NativeAOT toolchain and tests
([#&#8203;2465](https://togithub.com/dotnet/BenchmarkDotNet/issues/2465))
(by [@&#8203;adamsitnik](https://togithub.com/adamsitnik))
-
[536a28](https://togithub.com/dotnet/BenchmarkDotNet/commit/536a28e0ff2196255fb120aa0d39e40bdbde454a)
Add HardwareIntrinsics AVX-512 info
([#&#8203;2412](https://togithub.com/dotnet/BenchmarkDotNet/issues/2412))
(by [@&#8203;nietras](https://togithub.com/nietras))
-
[3fa045](https://togithub.com/dotnet/BenchmarkDotNet/commit/3fa0456495cac82b536902b101a2972c62c3e4a8)
Add OperationsPerSecondAttribute
([#&#8203;2468](https://togithub.com/dotnet/BenchmarkDotNet/issues/2468))
(by [@&#8203;DarkWanderer](https://togithub.com/DarkWanderer))
-
[0583cb](https://togithub.com/dotnet/BenchmarkDotNet/commit/0583cb90739b3ee4b8258f807ef42cdc3243f82f)
Bump Microsoft.NETCore.Platforms: 5.0.0->6.0.0 (by
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[2e62b9](https://togithub.com/dotnet/BenchmarkDotNet/commit/2e62b9b0a8c80255914e9e11d06d92871df40f85)
Remove netcoreapp2.0;net461 from TFMs for
IntegrationTests.ManualRunning.Mult... (by
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[92fa3f](https://togithub.com/dotnet/BenchmarkDotNet/commit/92fa3f834e0519d32fd8fc97e26aa82f9626b241)
Bump xunit: 2.5.0->2.6.2 (by
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[01e220](https://togithub.com/dotnet/BenchmarkDotNet/commit/01e2201c826dd44e089a22c40d8c3abecba320fa)
Bump xunit.runner.visualstudio: 2.5.0->2.5.4 (by
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[29a94c](https://togithub.com/dotnet/BenchmarkDotNet/commit/29a94ce301dac6121d1e0d1a0d783a6491c27703)
Bump Verify.Xunit: 20.3.2->20.8.2 (by
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[538e0e](https://togithub.com/dotnet/BenchmarkDotNet/commit/538e0e1771be037ef587b08cb52515ce6daf5c0e)
Bump Microsoft.NET.Test.SDK: 17.6.2->17.8.0 (by
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[136e4b](https://togithub.com/dotnet/BenchmarkDotNet/commit/136e4bb3f18a419df40c18a5430a29243ab57fd8)
Remove explicit Microsoft.NETFramework.ReferenceAssemblies reference in
Bench... (by
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[423b84](https://togithub.com/dotnet/BenchmarkDotNet/commit/423b8473d02d5bd59617675190660222198bf7d0)
\[build] Bump Docfx.App: 2.71.1->2.74.0 (by
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[718953](https://togithub.com/dotnet/BenchmarkDotNet/commit/718953674a83da4de6563368f38776048024f0d3)
\[build] Bump Octokit: 7.0.0->9.0.0 (by
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[0cce91](https://togithub.com/dotnet/BenchmarkDotNet/commit/0cce9120bd717e31a4a6a4a396faa8f38fd3cc08)
\[build] Bump Cake.Frosting: 3.2.0->4.0.0 (by
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[4d5dc9](https://togithub.com/dotnet/BenchmarkDotNet/commit/4d5dc9ca13072d384cabf565bc3622f8de5626d7)
Fix Newtonsoft.Json v13.0.1 in BenchmarkDotNet.IntegrationTests (by
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[c7ec60](https://togithub.com/dotnet/BenchmarkDotNet/commit/c7ec60ad6d4e54a99463eb46a0307196cc040940)
Enable UserCanSpecifyCustomNuGetPackageDependency test on Linux (by
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[a572db](https://togithub.com/dotnet/BenchmarkDotNet/commit/a572db119798fb58b24437ccef6a364efd59e836)
Bump C# LangVersion: 11.0->12.0 (by
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[b4ac9d](https://togithub.com/dotnet/BenchmarkDotNet/commit/b4ac9df9f7890ca9669e2b9c8835af35c072a453)
Nullability cleanup (2023-11-26) (by
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[5557ae](https://togithub.com/dotnet/BenchmarkDotNet/commit/5557aee0638bda38001bd6c2000164d9b96d315a)
\[build] Bump Docfx.App: 2.74.0->2.74.1 (by
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[b987b9](https://togithub.com/dotnet/BenchmarkDotNet/commit/b987b99ed37455e5443ed03169890998c3152ae9)
Fixed some tests.
([#&#8203;2475](https://togithub.com/dotnet/BenchmarkDotNet/issues/2475))
(by [@&#8203;timcassell](https://togithub.com/timcassell))
-
[05eb00](https://togithub.com/dotnet/BenchmarkDotNet/commit/05eb00f3536061ca624bab3d9a4ca2f3c0be5922)
Prepare v0.13.11 changelog (by
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))

#### Contributors (7)

-   Adam Sitnik ([@&#8203;adamsitnik](https://togithub.com/adamsitnik))
- Andrey Akinshin
([@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
- Michał Petryka
([@&#8203;MichalPetryka](https://togithub.com/MichalPetryka))
-   Nazul Grimaldo ([@&#8203;nazulg](https://togithub.com/nazulg))
-   nietras ([@&#8203;nietras](https://togithub.com/nietras))
- Oleg V. Kozlyuk
([@&#8203;DarkWanderer](https://togithub.com/DarkWanderer))
-   Tim Cassell ([@&#8203;timcassell](https://togithub.com/timcassell))

Thank you very much!

###
[`v0.13.10`](https://togithub.com/dotnet/BenchmarkDotNet/releases/tag/v0.13.10):
0.13.10

Full changelog: https://benchmarkdotnet.org/changelog/v0.13.10.html

#### Highlights

Initial support of .NET 9 and minor bug fixes.

#### Details

In the
[v0.13.10](https://togithub.com/dotnet/BenchmarkDotNet/issues?q=milestone:v0.13.10)
scope,
2 issues were resolved and 3 pull requests were merged.
This release includes 10 commits by 4 contributors.

#### Resolved issues (2)

-
[#&#8203;2436](https://togithub.com/dotnet/BenchmarkDotNet/issues/2436)
BenchmarkDotNet Access Denied Error on WSL2 when Writing to
'/mnt/c/DumpStack.log.tmp' (assignee:
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[#&#8203;2455](https://togithub.com/dotnet/BenchmarkDotNet/issues/2455)
.NET 9 support (assignee:
[@&#8203;adamsitnik](https://togithub.com/adamsitnik))

#### Merged pull requests (3)

- [#&#8203;2447](https://togithub.com/dotnet/BenchmarkDotNet/pull/2447)
Add support for wasm/net9.0 (by
[@&#8203;radical](https://togithub.com/radical))
- [#&#8203;2453](https://togithub.com/dotnet/BenchmarkDotNet/pull/2453)
feat: set RuntimeHostConfigurationOption on generated project (by
[@&#8203;workgroupengineering](https://togithub.com/workgroupengineering))
- [#&#8203;2456](https://togithub.com/dotnet/BenchmarkDotNet/pull/2456)
implement full .NET 9 support (by
[@&#8203;adamsitnik](https://togithub.com/adamsitnik))

#### Commits (10)

-
[c27152](https://togithub.com/dotnet/BenchmarkDotNet/commit/c27152b9d7b6391501abcf7e8edcb2804999622f)
Set next BenchmarkDotNet version: 0.13.10 (by
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[2e96d2](https://togithub.com/dotnet/BenchmarkDotNet/commit/2e96d29453a804cfc1b92fffeea94c866522167a)
Don't show AssemblyInformationalVersion metadata in BDN BrandVersion (by
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[d17c6a](https://togithub.com/dotnet/BenchmarkDotNet/commit/d17c6ad0bd8ac15d83ced0a7522de7dd51526ad4)
Support Windows 11 23H2 (10.0.22631) in OsBrandStringHelper (by
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[af9c5c](https://togithub.com/dotnet/BenchmarkDotNet/commit/af9c5c6013b4e661cda0ff8fed40a50ae62d5a74)
Exception handling in DotNetCliGenerator.GetRootDirectory, fix
[#&#8203;2436](https://togithub.com/dotnet/BenchmarkDotNet/issues/2436)
(by [@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[e11136](https://togithub.com/dotnet/BenchmarkDotNet/commit/e11136897bdf26c004076bcbe812bb4ae60f8859)
\[build] Bump Docfx.App: 2.71.0->2.71.1 (by
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[7b342f](https://togithub.com/dotnet/BenchmarkDotNet/commit/7b342f5cfb63c73708f3e69dde33d7430a3c0401)
Add support for wasm/net9.0
([#&#8203;2447](https://togithub.com/dotnet/BenchmarkDotNet/issues/2447))
(by [@&#8203;radical](https://togithub.com/radical))
-
[e17068](https://togithub.com/dotnet/BenchmarkDotNet/commit/e170684208103ca5ba4212ad8dc7c2aad5cf02d4)
Adjust 'Failed to set up high priority' message (by
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[0a734a](https://togithub.com/dotnet/BenchmarkDotNet/commit/0a734a94a13733c2950d7edbac08499c6f2c108a)
feat: set RuntimeHostConfigurationOption on generated project
([#&#8203;2453](https://togithub.com/dotnet/BenchmarkDotNet/issues/2453))
(by
[@&#8203;workgroupengineering](https://togithub.com/workgroupengineering))
-
[ae4914](https://togithub.com/dotnet/BenchmarkDotNet/commit/ae49148a92c358676190772803fe0ed532814ce3)
implement full .NET 9 support
([#&#8203;2456](https://togithub.com/dotnet/BenchmarkDotNet/issues/2456))
(by [@&#8203;adamsitnik](https://togithub.com/adamsitnik))
-
[40c414](https://togithub.com/dotnet/BenchmarkDotNet/commit/40c4142734ce68bdfcbccf7086ed2b724e9428bc)
Prepare v0.13.10 changelog (by
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))

#### Contributors (4)

-   Adam Sitnik ([@&#8203;adamsitnik](https://togithub.com/adamsitnik))
- Andrey Akinshin
([@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-   Ankit Jain ([@&#8203;radical](https://togithub.com/radical))
- workgroupengineering
([@&#8203;workgroupengineering](https://togithub.com/workgroupengineering))

Thank you very much!

###
[`v0.13.9`](https://togithub.com/dotnet/BenchmarkDotNet/releases/tag/v0.13.9):
0.13.9

Full changelog: https://benchmarkdotnet.org/changelog/v0.13.9.html

In the
[v0.13.9](https://togithub.com/dotnet/BenchmarkDotNet/issues?q=milestone:v0.13.9)
scope, 3 issues were resolved and 7 pull requests were merged. This
release includes 26 commits by 5 contributors.

#### Resolved issues (3)

-
[#&#8203;2054](https://togithub.com/dotnet/BenchmarkDotNet/issues/2054)
Custom logging/visualization during the benchmark run (assignee:
[@&#8203;caaavik-msft](https://togithub.com/caaavik-msft))
-
[#&#8203;2404](https://togithub.com/dotnet/BenchmarkDotNet/issues/2404)
Using `DisassemblyDiagnoser` in GitHub Actions (assignee:
[@&#8203;timcassell](https://togithub.com/timcassell))
-
[#&#8203;2432](https://togithub.com/dotnet/BenchmarkDotNet/issues/2432)
Something went wrong with outliers when using `--launchCount` (assignee:
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))

#### Merged pull requests (7)

- [#&#8203;1882](https://togithub.com/dotnet/BenchmarkDotNet/pull/1882)
use coalesce instead of join (by
[@&#8203;askazakov](https://togithub.com/askazakov))
- [#&#8203;2413](https://togithub.com/dotnet/BenchmarkDotNet/pull/2413)
Fix linux crash from disassembler (by
[@&#8203;timcassell](https://togithub.com/timcassell))
- [#&#8203;2420](https://togithub.com/dotnet/BenchmarkDotNet/pull/2420)
Add event processor functionality (by
[@&#8203;caaavik-msft](https://togithub.com/caaavik-msft))
- [#&#8203;2421](https://togithub.com/dotnet/BenchmarkDotNet/pull/2421)
More nullability warnings fixes (by
[@&#8203;alinasmirnova](https://togithub.com/alinasmirnova))
- [#&#8203;2433](https://togithub.com/dotnet/BenchmarkDotNet/pull/2433)
Fix build errors with latest sdk (by
[@&#8203;timcassell](https://togithub.com/timcassell))
- [#&#8203;2434](https://togithub.com/dotnet/BenchmarkDotNet/pull/2434)
Fix Event Processors not being copied in ManualConfig.Add (by
[@&#8203;caaavik-msft](https://togithub.com/caaavik-msft))
- [#&#8203;2435](https://togithub.com/dotnet/BenchmarkDotNet/pull/2435)
Treat warnings not as errors in manual test project (by
[@&#8203;timcassell](https://togithub.com/timcassell))

#### Commits (26)

-
[ece5cc](https://togithub.com/dotnet/BenchmarkDotNet/commit/ece5ccfc91d92b610338b05da73d2a91508e2837)
Set next BenchmarkDotNet version: 0.13.9 (by
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[ad9376](https://togithub.com/dotnet/BenchmarkDotNet/commit/ad937654174e521741aac620e16635a8ff14b1c9)
Add event processor functionality
([#&#8203;2420](https://togithub.com/dotnet/BenchmarkDotNet/issues/2420))
(by [@&#8203;caaavik-msft](https://togithub.com/caaavik-msft))
-
[8227bb](https://togithub.com/dotnet/BenchmarkDotNet/commit/8227bbfa5f4d22c51f9c3856576d3680d8fc0a92)
Address PR feedback
([#&#8203;2434](https://togithub.com/dotnet/BenchmarkDotNet/issues/2434))
(by [@&#8203;caaavik-msft](https://togithub.com/caaavik-msft))
-
[46b3c0](https://togithub.com/dotnet/BenchmarkDotNet/commit/46b3c0171709c48f58966fdf2665b5f292ff6467)
Fix linux crash from disassembler
([#&#8203;2413](https://togithub.com/dotnet/BenchmarkDotNet/issues/2413))
(by [@&#8203;timcassell](https://togithub.com/timcassell))
-
[967a97](https://togithub.com/dotnet/BenchmarkDotNet/commit/967a975773ebd7a9744f3875220c7db8fa647957)
Fix build errors with latest sdk.
([#&#8203;2433](https://togithub.com/dotnet/BenchmarkDotNet/issues/2433))
(by [@&#8203;timcassell](https://togithub.com/timcassell))
-
[dd7a9b](https://togithub.com/dotnet/BenchmarkDotNet/commit/dd7a9b7cd132e522951eeb6916a3aa27a24ebf59)
Treat warnings not as errors in manual test project
([#&#8203;2435](https://togithub.com/dotnet/BenchmarkDotNet/issues/2435))
(by [@&#8203;timcassell](https://togithub.com/timcassell))
-
[583874](https://togithub.com/dotnet/BenchmarkDotNet/commit/58387457bd67c62fda9c831329401fe0de4ae86f)
Print full stacktrace for GenerateException, see
[#&#8203;2436](https://togithub.com/dotnet/BenchmarkDotNet/issues/2436)
(by [@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[6e3a15](https://togithub.com/dotnet/BenchmarkDotNet/commit/6e3a159d3d3ae0d7eecc759c23a7bb0124e673df)
Support WSL detection in RuntimeInformation (by
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[8986e0](https://togithub.com/dotnet/BenchmarkDotNet/commit/8986e053c2fbc0befdef7d6e1a116a7bc83da282)
Update myget url in README (by
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[516bd6](https://togithub.com/dotnet/BenchmarkDotNet/commit/516bd68238c38bb6f622f71039d7b91f3f33776d)
Enabled nullability for BenchmarkDotNet.Diagnostics.dotTrace.csproj (by
[@&#8203;alinasmirnova](https://togithub.com/alinasmirnova))
-
[5428eb](https://togithub.com/dotnet/BenchmarkDotNet/commit/5428ebdb8b6e9a847bb8ae6cf129b7dd9d784454)
Fixed nullability warnings in methods signatures (by
[@&#8203;alinasmirnova](https://togithub.com/alinasmirnova))
-
[7fbbc9](https://togithub.com/dotnet/BenchmarkDotNet/commit/7fbbc9f506cee0048f2ea6e7af15fbe1aa0bd7f7)
Removed CanBeNull attribute (by
[@&#8203;alinasmirnova](https://togithub.com/alinasmirnova))
-
[9d7350](https://togithub.com/dotnet/BenchmarkDotNet/commit/9d7350c21b30c2655705ede68929243846b8a407)
Fixed warnings on null assignments (by
[@&#8203;alinasmirnova](https://togithub.com/alinasmirnova))
-
[b43d28](https://togithub.com/dotnet/BenchmarkDotNet/commit/b43d280f1673526dff865f5fbfc1848c846eacdd)
Fixed warnings in EngineEventLogParser (by
[@&#8203;alinasmirnova](https://togithub.com/alinasmirnova))
-
[148165](https://togithub.com/dotnet/BenchmarkDotNet/commit/148165baf92233a3e3e67efc552e7528edb2fc78)
Removed an unnecessary check (by
[@&#8203;alinasmirnova](https://togithub.com/alinasmirnova))
-
[465aaf](https://togithub.com/dotnet/BenchmarkDotNet/commit/465aaf196a43d21b516edf6e9028c672c39937b9)
Fixed empty catch warning (by
[@&#8203;alinasmirnova](https://togithub.com/alinasmirnova))
-
[9a7bb7](https://togithub.com/dotnet/BenchmarkDotNet/commit/9a7bb7d5d6c72a01f991d869b9106364c26b1fce)
\[build] Bump: Microsoft.DocAsCode.App 2.67.5 -> Docfx.App 2.71.0 (by
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[9dd7b6](https://togithub.com/dotnet/BenchmarkDotNet/commit/9dd7b6f4b2511bbd30ba0f6d4999f7f58cf161a6)
Fix license badge link in README (by
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[134b8e](https://togithub.com/dotnet/BenchmarkDotNet/commit/134b8edd09ad7dad0a17728eae4e9f50e16d3fe0)
\[build] Automatic NextVersion evaluation (by
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[8090d9](https://togithub.com/dotnet/BenchmarkDotNet/commit/8090d995e847c3c3d84db1fd5acbee312a75cf81)
Suppress NETSDK1138 (TFM out of support warning) (by
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[af610e](https://togithub.com/dotnet/BenchmarkDotNet/commit/af610eec251bfa74f7317eaec915df9b905c979b)
Bump .NET SDK: 7.0.305->7.0.401 (by
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[8838ed](https://togithub.com/dotnet/BenchmarkDotNet/commit/8838ed4bf74377642d32774c558c0955e67c0faf)
\[build] Fix docfx build warnings (by
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[2d379b](https://togithub.com/dotnet/BenchmarkDotNet/commit/2d379b37310983dbe645a2129066d9af65d9e0d7)
Remove outlier consistency check, fix
[#&#8203;2432](https://togithub.com/dotnet/BenchmarkDotNet/issues/2432)
(by [@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[00628a](https://togithub.com/dotnet/BenchmarkDotNet/commit/00628ab31b79a78e1c22c298ca0086bdf28065a7)
use coalesce instead of join (by
[@&#8203;askazakov](https://togithub.com/askazakov))
-
[411a6e](https://togithub.com/dotnet/BenchmarkDotNet/commit/411a6e7594c45c9ffa55b0b6caecb7f6ed1b3081)
Prepare v0.13.9 changelog (by
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-
[228a46](https://togithub.com/dotnet/BenchmarkDotNet/commit/228a464e8be6c580ad9408e98f18813f6407fb5a)
Rollback docfx.json (by
[@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))

#### Contributors (5)

- Alina Smirnova
([@&#8203;alinasmirnova](https://togithub.com/alinasmirnova))
- Andrey Akinshin
([@&#8203;AndreyAkinshin](https://togithub.com/AndreyAkinshin))
-   askazakov ([@&#8203;askazakov](https://togithub.com/askazakov))
- Cameron Aavik
([@&#8203;caaavik-msft](https://togithub.com/caaavik-msft))
-   Tim Cassell ([@&#8203;timcassell](https://togithub.com/timcassell))

Thank you very much!

###
[`v0.13.8`](https://togithub.com/dotnet/BenchmarkDotNet/releases/tag/v0.13.8):
0.13.8

Full changelog: https://benchmarkdotnet.org/changelog/v0.13.8.html

#### Highlights

This release contains important bug fixes.

#### What's Changed

- Issue2394 multiple markdown exporters not possible even with different
names by [@&#8203;bstordrup](https://togithub.com/bstordrup) in
[dotnet/BenchmarkDotNet#2395
- Make MarkdownExporter ctor and Dialect protected by
[@&#8203;nietras](https://togithub.com/nietras) in
[dotnet/BenchmarkDotNet#2407
- Refactor out base TextLogger from StreamLogger by
[@&#8203;nietras](https://togithub.com/nietras) in
[dotnet/BenchmarkDotNet#2406
- - update the templates install command to reflect dotnet cli updates
by [@&#8203;baywet](https://togithub.com/baywet) in
[dotnet/BenchmarkDotNet#2415
- Update stub decoding for .NET 8 for disassemblers by
[@&#8203;janvorli](https://togithub.com/janvorli) in
[dotnet/BenchmarkDotNet#2416
- Enable nullability for BenchmarkDotNet.Annotations by
[@&#8203;alinasmirnova](https://togithub.com/alinasmirnova) in
[dotnet/BenchmarkDotNet#2418
- Nullability In BenchmarkDotNet project by
[@&#8203;alinasmirnova](https://togithub.com/alinasmirnova) in
[dotnet/BenchmarkDotNet#2419
- feat: add text justification style by
[@&#8203;Vahdanian](https://togithub.com/Vahdanian) in
[dotnet/BenchmarkDotNet#2410
- Default to RoslynToolchain by
[@&#8203;timcassell](https://togithub.com/timcassell) in
[dotnet/BenchmarkDotNet#2409

#### New Contributors

- [@&#8203;bstordrup](https://togithub.com/bstordrup) made their first
contribution in
[dotnet/BenchmarkDotNet#2395
- [@&#8203;baywet](https://togithub.com/baywet) made their first
contribution in
[dotnet/BenchmarkDotNet#2415
- [@&#8203;Vahdanian](https://togithub.com/Vahdanian) made their first
contribution in
[dotnet/BenchmarkDotNet#2410

**Full Changelog**:
dotnet/BenchmarkDotNet@v0.13.7...v0.13.8

###
[`v0.13.7`](https://togithub.com/dotnet/BenchmarkDotNet/releases/tag/v0.13.7):
BenchmarkDotNet v0.13.7

This release contains a bunch of important bug fixes.

Full changelog: https://benchmarkdotnet.org/changelog/v0.13.7.html

#### What's Changed

- Improve build for mono aot by
[@&#8203;radical](https://togithub.com/radical) in
[dotnet/BenchmarkDotNet#2367
- IComparable fallback for Tuple/ValueTuple by
[@&#8203;mrahhal](https://togithub.com/mrahhal) in
[dotnet/BenchmarkDotNet#2368
- Don't copy `PackageReference` in csproj by
[@&#8203;timcassell](https://togithub.com/timcassell) in
[dotnet/BenchmarkDotNet#2365
- Fix regression in parsing arguments with spaces Closes
[#&#8203;2373](https://togithub.com/dotnet/BenchmarkDotNet/issues/2373)
by [@&#8203;kant2002](https://togithub.com/kant2002) in
[dotnet/BenchmarkDotNet#2375
- `AggressiveOptimization` in `InProcess` toolchains by
[@&#8203;timcassell](https://togithub.com/timcassell) in
[dotnet/BenchmarkDotNet#2335
- Add expected results tests by
[@&#8203;timcassell](https://togithub.com/timcassell) in
[dotnet/BenchmarkDotNet#2361
- \[chore]: fix error message by
[@&#8203;BurakTaner](https://togithub.com/BurakTaner) in
[dotnet/BenchmarkDotNet#2379
- Cancel old jobs on push by
[@&#8203;timcassell](https://togithub.com/timcassell) in
[dotnet/BenchmarkDotNet#2380
- Support `--cli` argument for `CsProjClassicNetToolchain` by
[@&#8203;timcassell](https://togithub.com/timcassell) in
[dotnet/BenchmarkDotNet#2381
- Rebuild .Net Framework projects by
[@&#8203;timcassell](https://togithub.com/timcassell) in
[dotnet/BenchmarkDotNet#2370
- Fix missing import on Debug build by
[@&#8203;caaavik-msft](https://togithub.com/caaavik-msft) in
[dotnet/BenchmarkDotNet#2385
- perfcollect: don't restore symbols for local builds by
[@&#8203;adamsitnik](https://togithub.com/adamsitnik) in
[dotnet/BenchmarkDotNet#2384
- Fix PlatformNotSupportedException thrown on Android in ConsoleTitler
by [@&#8203;Adam--](https://togithub.com/Adam--) in
[dotnet/BenchmarkDotNet#2390

#### New Contributors

- [@&#8203;BurakTaner](https://togithub.com/BurakTaner) made their first
contribution in
[dotnet/BenchmarkDotNet#2379
- [@&#8203;caaavik-msft](https://togithub.com/caaavik-msft) made their
first contribution in
[dotnet/BenchmarkDotNet#2385
- [@&#8203;Adam--](https://togithub.com/Adam--) made their first
contribution in
[dotnet/BenchmarkDotNet#2390

**Full Changelog**:
dotnet/BenchmarkDotNet@v0.13.6...v0.13.7

###
[`v0.13.6`](https://togithub.com/dotnet/BenchmarkDotNet/releases/tag/v0.13.6):
BenchmarkDotNet v0.13.6

#### Highlights

- New
[BenchmarkDotNet.Diagnostics.dotTrace](https://www.nuget.org/packages/BenchmarkDotNet.Diagnostics.dotTrace)
NuGet package. Once this package is installed, you can annotate your
benchmarks with the `[DotTraceDiagnoser]` and get a
[dotTrace](https://www.jetbrains.com/profiler/) performance snapshot at
the end of the benchmark run.
[#&#8203;2328](https://togithub.com/dotnet/BenchmarkDotNet/pull/2328)
- Updated documentation website. We migrated to
[docfx](https://dotnet.github.io/docfx/) 2.67 and got the refreshed
modern template based on bootstrap 5 with dark/light theme switcher.
- Updated
[BenchmarkDotNet.Templates](https://www.nuget.org/packages/BenchmarkDotNet.Templates).
Multiple issues were resolved, now you can create new benchmark projects
from terminal or your favorite IDE.
[#&#8203;1658](https://togithub.com/dotnet/BenchmarkDotNet/issues/1658)
[#&#8203;1881](https://togithub.com/dotnet/BenchmarkDotNet/issues/1881)
[#&#8203;2149](https://togithub.com/dotnet/BenchmarkDotNet/issues/2149)
[#&#8203;2338](https://togithub.com/dotnet/BenchmarkDotNet/pull/2338)
- Response file support. Now it's possible to pass additional arguments
to BenchmarkDotNet using `@filename` syntax.
[#&#8203;2320](https://togithub.com/dotnet/BenchmarkDotNet/pull/2320)
[#&#8203;2348](https://togithub.com/dotnet/BenchmarkDotNet/pull/2348)
- Custom runtime support.
[#&#8203;2285](https://togithub.com/dotnet/BenchmarkDotNet/pull/2285)
- Introduce CategoryDiscoverer, see
[`IntroCategoryDiscoverer`](xref:BenchmarkDotNet.Samples.IntroCategoryDiscoverer).
[#&#8203;2306](https://togithub.com/dotnet/BenchmarkDotNet/issues/2306)
[#&#8203;2307](https://togithub.com/dotnet/BenchmarkDotNet/pull/2307)
-   Multiple bug fixes.

Full changelog: https://benchmarkdotnet.org/changelog/v0.13.6.html

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR was generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View the
[repository job
log](https://developer.mend.io/github/open-feature/dotnet-sdk).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOC4xOC4xNyIsInVwZGF0ZWRJblZlciI6IjM4LjE4LjE3IiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6W119-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@AndreyAkinshin AndreyAkinshin AndreyAkinshin approved these changes

Assignees
No one assigned
Labels
Area:DevOps
Projects
None yet
Milestone
v0.13.7
Development

Successfully merging this pull request may close these issues.

Add tests for expected results
2 participants
@timcassell @AndreyAkinshin