Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a benchmark for SKData #2129

Closed
wants to merge 7 commits into from
Closed

Add a benchmark for SKData #2129

wants to merge 7 commits into from

Conversation

mattleibow
Copy link
Contributor

@mattleibow mattleibow commented Jun 27, 2022

Description of Change

This PR just updates the benchmarks project and build tooling so that it is easier to run benchmarks.

Benchmarks can be run from the IDE or via the CLI:

cd benchmarks/SkiaSharp.Benchmarks
dotnet run -c Release -f net6.0

I ran on CI: https://github.com/mattleibow/SkiaSharpBenchmarks/runs/7082633214
Download ETL + Artifacts (226MB): https://github.com/mattleibow/SkiaSharpBenchmarks/suites/7116242957/artifacts/282140555
libSkiaSharp.dll Symbols: libSkiaSharp.pdb.zip

The tests are simple and compare all the complexity of managed types and the handle registration with the raw p/invokes and I find the p/invokes are slower... somehow slower than all the managed stuff that also calls the very same p/invokes since the managed stuff wraps the p/invokes...

// * Summary *

BenchmarkDotNet=v0.13.1, OS=Windows 10.0.20348
Intel Xeon Platinum 8272CL CPU 2.60GHz, 1 CPU, 2 logical and 2 physical cores
.NET SDK=6.0.301
  [Host]   : .NET 6.0.6 (6.0.622.26707), X64 RyuJIT
  .NET 6.0 : .NET 6.0.6 (6.0.622.26707), X64 RyuJIT

Job=.NET 6.0  Runtime=.NET 6.0  

|      Method | SizeKB |       Mean |     Error |    StdDev |     Median |     Gen 0 |    Allocated |
|------------ |------- |-----------:|----------:|----------:|-----------:|----------:|-------------:|
|    Seekable |      1 | 177.338 ms | 1.2944 ms | 1.2107 ms | 177.187 ms | 1666.6667 | 35,200,469 B |
| NonSeekable |      1 | 249.793 ms | 1.1976 ms | 1.1202 ms | 249.815 ms | 2000.0000 | 45,604,072 B |
|     PInvoke |      1 |  33.169 ms | 0.2254 ms | 0.1998 ms |  33.073 ms |         - |         32 B |  <-
|    Seekable |    100 |   8.631 ms | 0.0325 ms | 0.0304 ms |   8.643 ms |   15.6250 |    352,014 B |
| NonSeekable |    100 |  20.535 ms | 0.7700 ms | 2.2216 ms |  21.214 ms |         - |    456,034 B |
|     PInvoke |    100 |  23.658 ms | 1.5165 ms | 4.4476 ms |  25.085 ms |         - |         15 B |  <-
|    Seekable |   1000 |  19.620 ms | 0.1443 ms | 0.1280 ms |  19.612 ms |         - |     35,214 B |
| NonSeekable |   1000 |  26.144 ms | 1.3387 ms | 3.8411 ms |  24.046 ms |         - |     45,702 B |
|     PInvoke |   1000 |  20.989 ms | 0.3592 ms | 0.7255 ms |  20.883 ms |         - |         34 B |  <-
|    Seekable |  10000 |  76.404 ms | 0.8505 ms | 0.7540 ms |  76.370 ms |         - |      3,721 B |
| NonSeekable |  10000 | 112.895 ms | 2.2408 ms | 5.3688 ms | 112.395 ms |         - |      4,764 B |
|     PInvoke |  10000 |  94.687 ms | 1.8322 ms | 2.2501 ms |  95.006 ms |         - |        120 B |  <-

// * Warnings *
MultimodalDistribution
  SKDataCreateFromStream.NonSeekable: .NET 6.0 -> It seems that the distribution can have several modes (mValue = 2.95)

// * Hints *
Outliers
  SKDataCreateFromStream.PInvoke: .NET 6.0     -> 1 outlier  was  removed (33.62 ms)
  SKDataCreateFromStream.NonSeekable: .NET 6.0 -> 4 outliers were removed, 12 outliers were detected (13.21 ms..13.42 ms, 22.22 ms..22.95 ms)
  SKDataCreateFromStream.PInvoke: .NET 6.0     -> 1 outlier  was  removed, 11 outliers were detected (10.40 ms..10.59 ms, 25.82 ms)
  SKDataCreateFromStream.Seekable: .NET 6.0    -> 1 outlier  was  removed (20.01 ms)
  SKDataCreateFromStream.NonSeekable: .NET 6.0 -> 5 outliers were removed (50.09 ms..52.60 ms)
  SKDataCreateFromStream.PInvoke: .NET 6.0     -> 15 outliers were removed (28.03 ms..32.39 ms)
  SKDataCreateFromStream.Seekable: .NET 6.0    -> 1 outlier  was  removed (78.93 ms)

// * Legends *
  SizeKB    : Value of the 'SizeKB' parameter
  Mean      : Arithmetic mean of all measurements
  Error     : Half of 99.9% confidence interval
  StdDev    : Standard deviation of all measurements
  Median    : Value separating the higher half of all measurements (50th percentile)
  Gen 0     : GC Generation 0 collects per 1000 operations
  Allocated : Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)
  1 ms      : 1 Millisecond (0.001 sec)
// * Summary *

BenchmarkDotNet=v0.13.1, OS=Windows 10.0.22000
Intel Core i9-9980HK CPU 2.40GHz, 1 CPU, 16 logical and 8 physical cores
.NET SDK=6.0.400-preview.22301.10
  [Host]   : .NET 6.0.6 (6.0.622.26707), X64 RyuJIT
  .NET 6.0 : .NET 6.0.6 (6.0.622.26707), X64 RyuJIT

Job=.NET 6.0  Runtime=.NET 6.0

|      Method | SizeKB |       Mean |     Error |     StdDev |     Median |     Gen 0 |    Allocated |
|------------ |------- |-----------:|----------:|-----------:|-----------:|----------:|-------------:|
|    Seekable |      1 | 151.756 ms | 3.0298 ms |  6.9614 ms | 151.189 ms | 4000.0000 | 35,200,204 B |
| NonSeekable |      1 | 208.893 ms | 4.3076 ms | 12.4285 ms | 205.653 ms | 5000.0000 | 45,600,816 B |
|     PInvoke |      1 |  27.417 ms | 0.5226 ms |  0.5592 ms |  27.318 ms |         - |         15 B |  <-
|    Seekable |    100 |   7.288 ms | 0.0847 ms |  0.1215 ms |   7.302 ms |   31.2500 |    352,013 B |
| NonSeekable |    100 |  16.017 ms | 0.4614 ms |  1.3459 ms |  16.053 ms |   46.8750 |    456,013 B |
|     PInvoke |    100 |  17.501 ms | 1.1390 ms |  3.2310 ms |  18.604 ms |         - |         15 B |  <-
|    Seekable |   1000 |  12.377 ms | 0.2014 ms |  0.1682 ms |  12.343 ms |         - |     35,214 B |
| NonSeekable |   1000 |  18.203 ms | 1.0855 ms |  3.1144 ms |  16.843 ms |         - |     45,663 B |
|     PInvoke |   1000 |  16.223 ms | 0.3213 ms |  0.6984 ms |  16.112 ms |         - |         15 B |  <-
|    Seekable |  10000 |  34.093 ms | 0.4628 ms |  0.4103 ms |  34.059 ms |         - |      3,571 B |
| NonSeekable |  10000 |  50.349 ms | 0.9412 ms |  1.6971 ms |  49.599 ms |         - |      4,662 B |
|     PInvoke |  10000 |  49.853 ms | 0.8517 ms |  0.9466 ms |  49.715 ms |         - |        120 B |  <-

// * Hints *
Outliers
  SKDataCreateFromStream.Seekable: .NET 6.0    -> 1 outlier  was  removed (186.02 ms)
  SKDataCreateFromStream.NonSeekable: .NET 6.0 -> 4 outliers were removed (246.96 ms..276.37 ms)
  SKDataCreateFromStream.Seekable: .NET 6.0    -> 9 outliers were removed (8.34 ms..9.08 ms)
  SKDataCreateFromStream.NonSeekable: .NET 6.0 -> 2 outliers were removed, 8 outliers were detected (11.15 ms..11.92 ms, 18.40 ms, 19.63 ms)
  SKDataCreateFromStream.PInvoke: .NET 6.0     -> 7 outliers were removed, 21 outliers were detected (9.55 ms..10.51 ms, 20.88 ms..23.54 ms)
  SKDataCreateFromStream.Seekable: .NET 6.0    -> 2 outliers were removed (13.08 ms, 15.96 ms)
  SKDataCreateFromStream.NonSeekable: .NET 6.0 -> 5 outliers were removed (27.88 ms..31.10 ms)
  SKDataCreateFromStream.PInvoke: .NET 6.0     -> 8 outliers were removed (19.61 ms..20.83 ms)
  SKDataCreateFromStream.Seekable: .NET 6.0    -> 1 outlier  was  removed (36.37 ms)
  SKDataCreateFromStream.NonSeekable: .NET 6.0 -> 12 outliers were removed (58.64 ms..66.31 ms)
  SKDataCreateFromStream.PInvoke: .NET 6.0     -> 4 outliers were removed (54.43 ms..57.34 ms)

// * Legends *
  SizeKB    : Value of the 'SizeKB' parameter
  Mean      : Arithmetic mean of all measurements
  Error     : Half of 99.9% confidence interval
  StdDev    : Standard deviation of all measurements
  Median    : Value separating the higher half of all measurements (50th percentile)
  Gen 0     : GC Generation 0 collects per 1000 operations
  Allocated : Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)
  1 ms      : 1 Millisecond (0.001 sec)

@mattleibow mattleibow changed the title Update the Benchmarks project Add a benchmark for SKData Jun 28, 2022
@mattleibow mattleibow changed the base branch from main to dev/bench June 28, 2022 00:21
@mattleibow mattleibow marked this pull request as draft June 28, 2022 00:23
Base automatically changed from dev/bench to main June 28, 2022 01:36
@mattleibow
Copy link
Contributor Author

Closing this as it does not seem to make a difference.

@mattleibow mattleibow closed this Feb 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant