Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

heap-buffer-overflow running address sanitizer on Min with float16 #21558

Open
adamreeve opened this issue Jul 30, 2024 · 2 comments · May be fixed by #22135
Open

heap-buffer-overflow running address sanitizer on Min with float16 #21558

adamreeve opened this issue Jul 30, 2024 · 2 comments · May be fixed by #22135
Labels
core runtime issues related to core runtime stale issues that have not been addressed in a while; categorized by a bot

Comments

@adamreeve
Copy link
Contributor

adamreeve commented Jul 30, 2024 •
edited
Loading

Describe the issue

I ran into asan errors when adding tests of the min and max operators with the CPU provider and float16 data with NaNs: #21492

Not being able to add tests for this blocked fixing of NaN propagation in the Min and Max operators with float16 on CPU and GPU (see #19984 (comment) too).

I originally assumed this was related to the NaNs in the input, but it looks like it was actually due to the shape of the data used, so possibly the Min and Max NaN propagation with float16 can be fixed and tests added that have different data shapes to work around this. This is still a concerning bug though, and also caused crashes of the web builds.

To reproduce

Add a new test that runs Min with 3x3 float16 data (adamreeve@37839ba):

diff --git a/onnxruntime/test/providers/cpu/math/element_wise_ops_test.cc b/onnxruntime/test/providers/cpu/math/element_wise_ops_test.cc
index bd3d21d492..c606be75b7 100644
--- a/onnxruntime/test/providers/cpu/math/element_wise_ops_test.cc
+++ b/onnxruntime/test/providers/cpu/math/element_wise_ops_test.cc
@@ -1787,6 +1787,25 @@ TEST(MathOpTest, Min_12_MLFloat16_Scalar1) {
   test.Run(OpTester::ExpectResult::kExpectSuccess, "", {kTensorrtExecutionProvider});  // TensorRT: Input batch size is inconsistent
 }
 
+TEST(MathOpTest, Min_12_MLFloat16_AsanRepro) {
+  OpTester test("Min", 12);
+  test.AddInput<MLFloat16>("data_2", {3, 3},
+                           MakeMLFloat16({1.0f, 1.0f, 1.0f,
+                                          -0.5f, 0.0f, -2.0f,
+                                          0.5f, 0.0f, 2.0f}));
+  test.AddInput<MLFloat16>("data_1", {3, 1},
+                           MakeMLFloat16({0.0f, -1.0f, 1.0f}));
+  test.AddOutput<MLFloat16>("min", {3, 3},
+                            MakeMLFloat16({0.0f, 0.0f, 0.0f,
+                                           -1.0f, -1.0f, -2.0f,
+                                           0.5f, 0.0f, 1.0f}));
+  if (nullptr != DefaultCpuExecutionProvider()) {
+    std::vector<std::unique_ptr<IExecutionProvider>> execution_providers;
+    execution_providers.push_back(DefaultCpuExecutionProvider());
+    test.Run(OpTester::ExpectResult::kExpectSuccess, "", {}, nullptr, &execution_providers);
+  }
+}
+
 TEST(MathOpTest, Max_6) {
   OpTester test("Max", 6);
   std::vector<int64_t> dims{3, 3};

Build with address sanitizer enabled:

./build.sh --update --build --config Debug --build_shared_lib --parallel --compile_no_warning_as_error --skip_submodule_sync --enable_address_sanitizer

Run the new test with asan:

LD_PRELOAD=/usr/lib64/libasan.so.8 GTEST_FILTER="MathOpTest.Min_12_MLFloat16_AsanRepro" ctest --verbose --test-dir build/Linux/Debug -R 'onnxruntime_test_all'

Output is:

Internal ctest changing into directory: /home/adam/dev/gross/onnxruntime/build/Linux/Debug
UpdateCTestConfiguration  from :/home/adam/dev/gross/onnxruntime/build/Linux/Debug/DartConfiguration.tcl
Parse Config file:/home/adam/dev/gross/onnxruntime/build/Linux/Debug/DartConfiguration.tcl
UpdateCTestConfiguration  from :/home/adam/dev/gross/onnxruntime/build/Linux/Debug/DartConfiguration.tcl
Parse Config file:/home/adam/dev/gross/onnxruntime/build/Linux/Debug/DartConfiguration.tcl
Test project /home/adam/dev/gross/onnxruntime/build/Linux/Debug
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 1
    Start 1: onnxruntime_test_all

1: Test command: /home/adam/dev/gross/onnxruntime/build/Linux/Debug/onnxruntime_test_all "--gtest_output=xml:/home/adam/dev/gross/onnxruntime/build/Linux/Debug/onnxruntime_test_all.Debug.results.xml"
1: Working Directory: /home/adam/dev/gross/onnxruntime/build/Linux/Debug
1: Test timeout computed to be: 7200
1: Note: Google Test filter = MathOpTest.Min_12_MLFloat16_AsanRepro
1: [==========] Running 1 test from 1 test suite.
1: [----------] Global test environment set-up.
1: [----------] 1 test from MathOpTest
1: [ RUN      ] MathOpTest.Min_12_MLFloat16_AsanRepro
1: =================================================================
1: ==312770==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x509000000052 at pc 0x0000038a0b01 bp 0x7fffbcce7620 sp 0x7fffbcce7618
1: READ of size 2 at 0x509000000052 thread T0
1:     #0 0x38a0b00 in Eigen::internal::mapbase_evaluator<Eigen::Map<Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> >, Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> const>::coeff(long) const (/home/adam/dev/gross/onnxruntime/build/Linux/Debug/onnxruntime_test_all+0x38a0b00) (BuildId: 0885f9f0665127b4a0fbd8529a0f0d31a8497999)
1:     #1 0x389e108 in Eigen::internal::binary_evaluator<Eigen::CwiseBinaryOp<Eigen::internal::scalar_min_op<Eigen::half, Eigen::half, 0>, Eigen::Map<Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> > const, Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op<Eigen::half>, Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> > const>, Eigen::internal::IndexBased, Eigen::internal::IndexBased, Eigen::half, Eigen::half>::coeff(long) const (/home/adam/dev/gross/onnxruntime/build/Linux/Debug/onnxruntime_test_all+0x389e108) (BuildId: 0885f9f0665127b4a0fbd8529a0f0d31a8497999)
1:     #2 0x3899566 in Eigen::internal::generic_dense_assignment_kernel<Eigen::internal::evaluator<Eigen::Map<Eigen::Array<Eigen::half, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> > >, Eigen::internal::evaluator<Eigen::CwiseBinaryOp<Eigen::internal::scalar_min_op<Eigen::half, Eigen::half, 0>, Eigen::Map<Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> > const, Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op<Eigen::half>, Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> > const> >, Eigen::internal::assign_op<Eigen::half, Eigen::half>, 0>::assignCoeff(long) /home/adam/dev/gross/onnxruntime/build/Linux/Debug/_deps/eigen-src/Eigen/src/Core/AssignEvaluator.h:660
1:     #3 0x3893f1f in Eigen::internal::dense_assignment_loop<Eigen::internal::generic_dense_assignment_kernel<Eigen::internal::evaluator<Eigen::Map<Eigen::Array<Eigen::half, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> > >, Eigen::internal::evaluator<Eigen::CwiseBinaryOp<Eigen::internal::scalar_min_op<Eigen::half, Eigen::half, 0>, Eigen::Map<Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> > const, Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op<Eigen::half>, Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> > const> >, Eigen::internal::assign_op<Eigen::half, Eigen::half>, 0>, 1, 0>::run(Eigen::internal::generic_dense_assignment_kernel<Eigen::internal::evaluator<Eigen::Map<Eigen::Array<Eigen::half, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> > >, Eigen::internal::evaluator<Eigen::CwiseBinaryOp<Eigen::internal::scalar_min_op<Eigen::half, Eigen::half, 0>, Eigen::Map<Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> > const, Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op<Eigen::half>, Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> > const> >, Eigen::internal::assign_op<Eigen::half, Eigen::half>, 0>&) (/home/adam/dev/gross/onnxruntime/build/Linux/Debug/onnxruntime_test_all+0x3893f1f) (BuildId: 0885f9f0665127b4a0fbd8529a0f0d31a8497999)
1:     #4 0x388884d in void Eigen::internal::call_dense_assignment_loop<Eigen::Map<Eigen::Array<Eigen::half, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> >, Eigen::CwiseBinaryOp<Eigen::internal::scalar_min_op<Eigen::half, Eigen::half, 0>, Eigen::Map<Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> > const, Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op<Eigen::half>, Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> > const>, Eigen::internal::assign_op<Eigen::half, Eigen::half> >(Eigen::Map<Eigen::Array<Eigen::half, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> >&, Eigen::CwiseBinaryOp<Eigen::internal::scalar_min_op<Eigen::half, Eigen::half, 0>, Eigen::Map<Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> > const, Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op<Eigen::half>, Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> > const> const&, Eigen::internal::assign_op<Eigen::half, Eigen::half> const&) (/home/adam/dev/gross/onnxruntime/build/Linux/Debug/onnxruntime_test_all+0x388884d) (BuildId: 0885f9f0665127b4a0fbd8529a0f0d31a8497999)
1:     #5 0x3875dec in Eigen::internal::Assignment<Eigen::Map<Eigen::Array<Eigen::half, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> >, Eigen::CwiseBinaryOp<Eigen::internal::scalar_min_op<Eigen::half, Eigen::half, 0>, Eigen::Map<Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> > const, Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op<Eigen::half>, Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> > const>, Eigen::internal::assign_op<Eigen::half, Eigen::half>, Eigen::internal::Dense2Dense, void>::run(Eigen::Map<Eigen::Array<Eigen::half, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> >&, Eigen::CwiseBinaryOp<Eigen::internal::scalar_min_op<Eigen::half, Eigen::half, 0>, Eigen::Map<Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> > const, Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op<Eigen::half>, Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> > const> const&, Eigen::internal::assign_op<Eigen::half, Eigen::half> const&) (/home/adam/dev/gross/onnxruntime/build/Linux/Debug/onnxruntime_test_all+0x3875dec) (BuildId: 0885f9f0665127b4a0fbd8529a0f0d31a8497999)
1:     #6 0x38568f0 in void Eigen::internal::call_assignment_no_alias<Eigen::Map<Eigen::Array<Eigen::half, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> >, Eigen::CwiseBinaryOp<Eigen::internal::scalar_min_op<Eigen::half, Eigen::half, 0>, Eigen::Map<Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> > const, Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op<Eigen::half>, Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> > const>, Eigen::internal::assign_op<Eigen::half, Eigen::half> >(Eigen::Map<Eigen::Array<Eigen::half, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> >&, Eigen::CwiseBinaryOp<Eigen::internal::scalar_min_op<Eigen::half, Eigen::half, 0>, Eigen::Map<Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> > const, Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op<Eigen::half>, Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> > const> const&, Eigen::internal::assign_op<Eigen::half, Eigen::half> const&) (/home/adam/dev/gross/onnxruntime/build/Linux/Debug/onnxruntime_test_all+0x38568f0) (BuildId: 0885f9f0665127b4a0fbd8529a0f0d31a8497999)
1:     #7 0x384fdf0 in void Eigen::internal::call_assignment<Eigen::Map<Eigen::Array<Eigen::half, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> >, Eigen::CwiseBinaryOp<Eigen::internal::scalar_min_op<Eigen::half, Eigen::half, 0>, Eigen::Map<Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> > const, Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op<Eigen::half>, Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> > const>, Eigen::internal::assign_op<Eigen::half, Eigen::half> >(Eigen::Map<Eigen::Array<Eigen::half, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> >&, Eigen::CwiseBinaryOp<Eigen::internal::scalar_min_op<Eigen::half, Eigen::half, 0>, Eigen::Map<Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> > const, Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op<Eigen::half>, Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> > const> const&, Eigen::internal::assign_op<Eigen::half, Eigen::half> const&, Eigen::internal::enable_if<!Eigen::internal::evaluator_assume_aliasing<Eigen::CwiseBinaryOp<Eigen::internal::scalar_min_op<Eigen::half, Eigen::half, 0>, Eigen::Map<Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> > const, Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op<Eigen::half>, Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> > const>, Eigen::internal::evaluator_traits<Eigen::CwiseBinaryOp<Eigen::internal::scalar_min_op<Eigen::half, Eigen::half, 0>, Eigen::Map<Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> > const, Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op<Eigen::half>, Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> > const> >::Shape>::value, void*>::type) /home/adam/dev/gross/onnxruntime/build/Linux/Debug/_deps/eigen-src/Eigen/src/Core/AssignEvaluator.h:858
1:     #8 0x38467e3 in void Eigen::internal::call_assignment<Eigen::Map<Eigen::Array<Eigen::half, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> >, Eigen::CwiseBinaryOp<Eigen::internal::scalar_min_op<Eigen::half, Eigen::half, 0>, Eigen::Map<Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> > const, Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op<Eigen::half>, Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> > const> >(Eigen::Map<Eigen::Array<Eigen::half, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> >&, Eigen::CwiseBinaryOp<Eigen::internal::scalar_min_op<Eigen::half, Eigen::half, 0>, Eigen::Map<Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> > const, Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op<Eigen::half>, Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> > const> const&) /home/adam/dev/gross/onnxruntime/build/Linux/Debug/_deps/eigen-src/Eigen/src/Core/AssignEvaluator.h:836
1:     #9 0x3826d8c in Eigen::Map<Eigen::Array<Eigen::half, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> >& Eigen::DenseBase<Eigen::Map<Eigen::Array<Eigen::half, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> > >::operator=<Eigen::CwiseBinaryOp<Eigen::internal::scalar_min_op<Eigen::half, Eigen::half, 0>, Eigen::Map<Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> > const, Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op<Eigen::half>, Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> > const> >(Eigen::DenseBase<Eigen::CwiseBinaryOp<Eigen::internal::scalar_min_op<Eigen::half, Eigen::half, 0>, Eigen::Map<Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> > const, Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op<Eigen::half>, Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> > const> > const&) (/home/adam/dev/gross/onnxruntime/build/Linux/Debug/onnxruntime_test_all+0x3826d8c) (BuildId: 0885f9f0665127b4a0fbd8529a0f0d31a8497999)
1:     #10 0x37e1f2a in onnxruntime::MinMaxMLFloat16<true>(onnxruntime::OpKernel const&, onnxruntime::OpKernelContext*)::{lambda(onnxruntime::BroadcastHelper&)#2}::operator()(onnxruntime::BroadcastHelper&) const (/home/adam/dev/gross/onnxruntime/build/Linux/Debug/onnxruntime_test_all+0x37e1f2a) (BuildId: 0885f9f0665127b4a0fbd8529a0f0d31a8497999)
1:     #11 0x37e1fd3 in onnxruntime::MinMaxMLFloat16<true>(onnxruntime::OpKernel const&, onnxruntime::OpKernelContext*)::{lambda(onnxruntime::BroadcastHelper&)#2}::_FUN(onnxruntime::BroadcastHelper&) (/home/adam/dev/gross/onnxruntime/build/Linux/Debug/onnxruntime_test_all+0x37e1fd3) (BuildId: 0885f9f0665127b4a0fbd8529a0f0d31a8497999)
1:     #12 0x37e878e in void onnxruntime::BroadcastLooper<onnxruntime::BroadcastHelper>(onnxruntime::BroadcastHelper&, onnxruntime::ProcessBroadcastSpanFuncs const&) /home/adam/dev/gross/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.h:1014
1:     #13 0x37cb8e0 in UntypedBroadcastVariadic /home/adam/dev/gross/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.cc:2037
1:     #14 0x37e2679 in onnxruntime::common::Status onnxruntime::MinMaxMLFloat16<true>(onnxruntime::OpKernel const&, onnxruntime::OpKernelContext*) (/home/adam/dev/gross/onnxruntime/build/Linux/Debug/onnxruntime_test_all+0x37e2679) (BuildId: 0885f9f0665127b4a0fbd8529a0f0d31a8497999)
1:     #15 0x37bd0c0 in onnxruntime::Min_8::Compute(onnxruntime::OpKernelContext*) const /home/adam/dev/gross/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.cc:810
1:     #16 0x4e7aa9a in onnxruntime::ExecuteKernel(onnxruntime::StreamExecutionContext&, unsigned long, unsigned long, bool const&, onnxruntime::SessionScope&) /home/adam/dev/gross/onnxruntime/onnxruntime/core/framework/sequential_executor.cc:495
1:     #17 0x4da3834 in onnxruntime::LaunchKernelStep::Execute(onnxruntime::StreamExecutionContext&, unsigned long, onnxruntime::SessionScope&, bool const&, bool&) /home/adam/dev/gross/onnxruntime/onnxruntime/core/framework/execution_steps.cc:73
1:     #18 0x4f325a3 in onnxruntime::RunSince(unsigned long, onnxruntime::StreamExecutionContext&, onnxruntime::SessionScope&, bool const&, unsigned long) /home/adam/dev/gross/onnxruntime/onnxruntime/core/framework/stream_execution_context.cc:222
1:     #19 0x4e7b538 in operator() /home/adam/dev/gross/onnxruntime/onnxruntime/core/framework/sequential_executor.cc:589
1:     #20 0x4e7f3d1 in __invoke_impl<void, onnxruntime::ExecuteThePlan(const SessionState&, gsl::span<int const>, gsl::span<const OrtValue>, gsl::span<int const>, std::vector<OrtValue>&, const std::unordered_map<long unsigned int, std::function<common::Status(const TensorShape&, const OrtDevice&, OrtValue&, bool&)> >&, const logging::Logger&, const DeviceStreamCollection*, bool const&, bool, bool)::<lambda()>&> /usr/include/c++/14/bits/invoke.h:61
1:     #21 0x4e7f17f in __invoke_r<void, onnxruntime::ExecuteThePlan(const SessionState&, gsl::span<int const>, gsl::span<const OrtValue>, gsl::span<int const>, std::vector<OrtValue>&, const std::unordered_map<long unsigned int, std::function<common::Status(const TensorShape&, const OrtDevice&, OrtValue&, bool&)> >&, const logging::Logger&, const DeviceStreamCollection*, bool const&, bool, bool)::<lambda()>&> /usr/include/c++/14/bits/invoke.h:111
1:     #22 0x4e7e68e in _M_invoke /usr/include/c++/14/bits/std_function.h:290
1:     #23 0x17a0a9d in std::function<void ()>::operator()() const /usr/include/c++/14/bits/std_function.h:591
1:     #24 0x1790bce in onnxruntime::concurrency::ThreadPool::Schedule(onnxruntime::concurrency::ThreadPool*, std::function<void ()>) /home/adam/dev/gross/onnxruntime/include/onnxruntime/core/platform/threadpool.h:233
1:     #25 0x4e7bdba in onnxruntime::ExecuteThePlan(onnxruntime::SessionState const&, gsl::span<int const, 18446744073709551615ul>, gsl::span<OrtValue const, 18446744073709551615ul>, gsl::span<int const, 18446744073709551615ul>, std::vector<OrtValue, std::allocator<OrtValue> >&, std::unordered_map<unsigned long, std::function<onnxruntime::common::Status (onnxruntime::TensorShape const&, OrtDevice const&, OrtValue&, bool&)>, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<std::pair<unsigned long const, std::function<onnxruntime::common::Status (onnxruntime::TensorShape const&, OrtDevice const&, OrtValue&, bool&)> > > > const&, onnxruntime::logging::Logger const&, onnxruntime::DeviceStreamCollection const*, bool const&, bool, bool) /home/adam/dev/gross/onnxruntime/onnxruntime/core/framework/sequential_executor.cc:588
1:     #26 0x4f88136 in ExecuteGraphImpl /home/adam/dev/gross/onnxruntime/onnxruntime/core/framework/utils.cc:659
1:     #27 0x4f89692 in onnxruntime::utils::ExecuteGraph(onnxruntime::SessionState const&, onnxruntime::FeedsFetchesManager&, gsl::span<OrtValue const, 18446744073709551615ul>, std::vector<OrtValue, std::allocator<OrtValue> >&, ExecutionMode, bool const&, onnxruntime::logging::Logger const&, onnxruntime::DeviceStreamCollectionHolder&, bool, onnxruntime::Stream*) /home/adam/dev/gross/onnxruntime/onnxruntime/core/framework/utils.cc:756
1:     #28 0x4f898d7 in onnxruntime::utils::ExecuteGraph(onnxruntime::SessionState const&, onnxruntime::FeedsFetchesManager&, gsl::span<OrtValue const, 18446744073709551615ul>, std::vector<OrtValue, std::allocator<OrtValue> >&, ExecutionMode, OrtRunOptions const&, onnxruntime::DeviceStreamCollectionHolder&, onnxruntime::logging::Logger const&) /home/adam/dev/gross/onnxruntime/onnxruntime/core/framework/utils.cc:783
1:     #29 0x30c597c in onnxruntime::InferenceSession::Run(OrtRunOptions const&, gsl::span<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 18446744073709551615ul>, gsl::span<OrtValue const, 18446744073709551615ul>, gsl::span<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 18446744073709551615ul>, std::vector<OrtValue, std::allocator<OrtValue> >*, std::vector<OrtDevice, std::allocator<OrtDevice> > const*) /home/adam/dev/gross/onnxruntime/onnxruntime/core/session/inference_session.cc:2590
1:     #30 0x30ca211 in onnxruntime::InferenceSession::Run(OrtRunOptions const&, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, OrtValue, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, OrtValue> > > const&, gsl::span<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 18446744073709551615ul>, std::vector<OrtValue, std::allocator<OrtValue> >*) /home/adam/dev/gross/onnxruntime/onnxruntime/core/session/inference_session.cc:2798
1:     #31 0x1bb2893 in void onnxruntime::test::BaseTester::ExecuteModel<onnxruntime::InferenceSession>(onnxruntime::Model&, onnxruntime::InferenceSession&, onnxruntime::test::BaseTester::ExpectResult, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, OrtRunOptions const*, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, OrtValue, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, OrtValue> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) /home/adam/dev/gross/onnxruntime/onnxruntime/test/providers/base_tester.cc:332
1:     #32 0x1bab51f in onnxruntime::test::BaseTester::ExecuteModelForEps(std::vector<std::unique_ptr<onnxruntime::IExecutionProvider, std::default_delete<onnxruntime::IExecutionProvider> >, std::allocator<std::unique_ptr<onnxruntime::IExecutionProvider, std::default_delete<onnxruntime::IExecutionProvider> > > >&&, onnxruntime::Model&, onnxruntime::SessionOptions, onnxruntime::test::BaseTester::ExpectResult, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, OrtRunOptions const*, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, OrtValue, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, OrtValue> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::shared_ptr<onnxruntime::CustomRegistry>, std::allocator<std::shared_ptr<onnxruntime::CustomRegistry> > > const*, bool, bool, unsigned long*, unsigned long*) /home/adam/dev/gross/onnxruntime/onnxruntime/test/providers/base_tester.cc:839
1:     #33 0x1ba725d in onnxruntime::test::BaseTester::RunWithConfig(unsigned long*, unsigned long*) /home/adam/dev/gross/onnxruntime/onnxruntime/test/providers/base_tester.cc:622
1:     #34 0x1ba5e8d in onnxruntime::test::BaseTester::Run(onnxruntime::SessionOptions, onnxruntime::test::BaseTester::ExpectResult, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::unordered_set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, OrtRunOptions const*, std::vector<std::unique_ptr<onnxruntime::IExecutionProvider, std::default_delete<onnxruntime::IExecutionProvider> >, std::allocator<std::unique_ptr<onnxruntime::IExecutionProvider, std::default_delete<onnxruntime::IExecutionProvider> > > >*, onnxruntime::Graph::ResolveOptions const&, unsigned long*, unsigned long*) /home/adam/dev/gross/onnxruntime/onnxruntime/test/providers/base_tester.cc:572
1:     #35 0x1ba5c43 in onnxruntime::test::BaseTester::Run(onnxruntime::test::BaseTester::ExpectResult, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::unordered_set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, OrtRunOptions const*, std::vector<std::unique_ptr<onnxruntime::IExecutionProvider, std::default_delete<onnxruntime::IExecutionProvider> >, std::allocator<std::unique_ptr<onnxruntime::IExecutionProvider, std::default_delete<onnxruntime::IExecutionProvider> > > >*, ExecutionMode, onnxruntime::Graph::ResolveOptions const&) /home/adam/dev/gross/onnxruntime/onnxruntime/test/providers/base_tester.cc:532
1:     #36 0x1e2942c in onnxruntime::test::MathOpTest_Min_12_MLFloat16_AsanRepro_Test::TestBody() /home/adam/dev/gross/onnxruntime/onnxruntime/test/providers/cpu/math/element_wise_ops_test.cc:1805
1:     #37 0x7fdbadbaaa89 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) [clone .constprop.0] (/lib64/libgtest.so.1.14.0+0x55a89) (BuildId: 460980516317b81e65f4725621047b239b7e4fcc)
1:     #38 0x7fdbadb9552d in testing::Test::Run() (/lib64/libgtest.so.1.14.0+0x4052d) (BuildId: 460980516317b81e65f4725621047b239b7e4fcc)
1:     #39 0x7fdbadb95734 in testing::TestInfo::Run() (/lib64/libgtest.so.1.14.0+0x40734) (BuildId: 460980516317b81e65f4725621047b239b7e4fcc)
1:     #40 0x7fdbadb9593e in testing::TestSuite::Run() (/lib64/libgtest.so.1.14.0+0x4093e) (BuildId: 460980516317b81e65f4725621047b239b7e4fcc)
1:     #41 0x7fdbadba216f in testing::internal::UnitTestImpl::RunAllTests() (/lib64/libgtest.so.1.14.0+0x4d16f) (BuildId: 460980516317b81e65f4725621047b239b7e4fcc)
1:     #42 0x7fdbadba0d3f in testing::UnitTest::Run() (/lib64/libgtest.so.1.14.0+0x4bd3f) (BuildId: 460980516317b81e65f4725621047b239b7e4fcc)
1:     #43 0x2f38732 in RUN_ALL_TESTS() /usr/include/gtest/gtest.h:2317
1:     #44 0x2f39152 in main /home/adam/dev/gross/onnxruntime/onnxruntime/test/unittest_main/test_main.cc:94
1:     #45 0x7fdbace39087 in __libc_start_call_main (/lib64/libc.so.6+0x2a087) (BuildId: 8f53abaad945a669f2bdcd25f471d80e077568ef)
1:     #46 0x7fdbace3914a in __libc_start_main_alias_2 (/lib64/libc.so.6+0x2a14a) (BuildId: 8f53abaad945a669f2bdcd25f471d80e077568ef)
1:     #47 0x4a6924 in _start (/home/adam/dev/gross/onnxruntime/build/Linux/Debug/onnxruntime_test_all+0x4a6924) (BuildId: 0885f9f0665127b4a0fbd8529a0f0d31a8497999)
1: 
1: 0x509000000052 is located 0 bytes after 18-byte region [0x509000000040,0x509000000052)
1: allocated by thread T0 here:
1:     #0 0x7fdbad4f6f8d in posix_memalign (/usr/lib64/libasan.so.8+0xf6f8d) (BuildId: c1431025b5d8af781c22c9ceea71f065c547d32d)
1:     #1 0x4d0bcbd in onnxruntime::AllocatorDefaultAlloc(unsigned long) /home/adam/dev/gross/onnxruntime/onnxruntime/core/framework/allocator.cc:89
1:     #2 0x4d0bda7 in onnxruntime::CPUAllocator::Alloc(unsigned long) /home/adam/dev/gross/onnxruntime/onnxruntime/core/framework/allocator.cc:107
1:     #3 0x11dda10 in onnxruntime::test::DummyArena::Alloc(unsigned long) /home/adam/dev/gross/onnxruntime/onnxruntime/test/framework/TestAllocatorManager.cc:26
1:     #4 0x4f38e3e in onnxruntime::Tensor::Tensor(onnxruntime::DataTypeImpl const*, onnxruntime::TensorShape const&, std::shared_ptr<onnxruntime::IAllocator>) /home/adam/dev/gross/onnxruntime/onnxruntime/core/framework/tensor.cc:94
1:     #5 0x31ba592 in std::__detail::_MakeUniq<onnxruntime::Tensor>::__single_object std::make_unique<onnxruntime::Tensor, onnxruntime::DataTypeImpl const*&, onnxruntime::TensorShape const&, std::shared_ptr<onnxruntime::IAllocator> >(onnxruntime::DataTypeImpl const*&, onnxruntime::TensorShape const&, std::shared_ptr<onnxruntime::IAllocator>&&) /usr/include/c++/14/bits/unique_ptr.h:1076
1:     #6 0x4f396c2 in onnxruntime::Tensor::InitOrtValue(onnxruntime::DataTypeImpl const*, onnxruntime::TensorShape const&, std::shared_ptr<onnxruntime::IAllocator>, OrtValue&) /home/adam/dev/gross/onnxruntime/onnxruntime/core/framework/tensor.cc:108
1:     #7 0x1814bef in void onnxruntime::test::BaseTester::AddData<onnxruntime::MLFloat16>(std::vector<onnxruntime::test::BaseTester::Data, std::allocator<onnxruntime::test::BaseTester::Data> >&, char const*, std::variant<std::vector<long, std::allocator<long> >, absl::InlinedVector<long, 6ul, std::allocator<long> > > const&, onnxruntime::MLFloat16 const*, long, bool, bool, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const*, float, float, bool) /home/adam/dev/gross/onnxruntime/onnxruntime/test/providers/base_tester.h:709
1:     #8 0x1905379 in void onnxruntime::test::BaseTester::AddInput<onnxruntime::MLFloat16>(char const*, std::initializer_list<long>, std::vector<onnxruntime::MLFloat16, std::allocator<onnxruntime::MLFloat16> > const&, bool, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const*) /home/adam/dev/gross/onnxruntime/onnxruntime/test/providers/base_tester.h:70
1:     #9 0x1e28e53 in onnxruntime::test::MathOpTest_Min_12_MLFloat16_AsanRepro_Test::TestBody() /home/adam/dev/gross/onnxruntime/onnxruntime/test/providers/cpu/math/element_wise_ops_test.cc:1792
1:     #10 0x7fdbadbaaa89 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) [clone .constprop.0] (/lib64/libgtest.so.1.14.0+0x55a89) (BuildId: 460980516317b81e65f4725621047b239b7e4fcc)
1:     #11 0x7fdbadb9552d in testing::Test::Run() (/lib64/libgtest.so.1.14.0+0x4052d) (BuildId: 460980516317b81e65f4725621047b239b7e4fcc)
1:     #12 0x7fdbadb95734 in testing::TestInfo::Run() (/lib64/libgtest.so.1.14.0+0x40734) (BuildId: 460980516317b81e65f4725621047b239b7e4fcc)
1:     #13 0x7fdbadb9593e in testing::TestSuite::Run() (/lib64/libgtest.so.1.14.0+0x4093e) (BuildId: 460980516317b81e65f4725621047b239b7e4fcc)
1:     #14 0x7fdbadba216f in testing::internal::UnitTestImpl::RunAllTests() (/lib64/libgtest.so.1.14.0+0x4d16f) (BuildId: 460980516317b81e65f4725621047b239b7e4fcc)
1:     #15 0x7fdbadba0d3f in testing::UnitTest::Run() (/lib64/libgtest.so.1.14.0+0x4bd3f) (BuildId: 460980516317b81e65f4725621047b239b7e4fcc)
1:     #16 0x2f38732 in RUN_ALL_TESTS() /usr/include/gtest/gtest.h:2317
1:     #17 0x2f39152 in main /home/adam/dev/gross/onnxruntime/onnxruntime/test/unittest_main/test_main.cc:94
1:     #18 0x7fdbace39087 in __libc_start_call_main (/lib64/libc.so.6+0x2a087) (BuildId: 8f53abaad945a669f2bdcd25f471d80e077568ef)
1:     #19 0x7fdbace3914a in __libc_start_main_alias_2 (/lib64/libc.so.6+0x2a14a) (BuildId: 8f53abaad945a669f2bdcd25f471d80e077568ef)
1:     #20 0x4a6924 in _start (/home/adam/dev/gross/onnxruntime/build/Linux/Debug/onnxruntime_test_all+0x4a6924) (BuildId: 0885f9f0665127b4a0fbd8529a0f0d31a8497999)
1: 
1: SUMMARY: AddressSanitizer: heap-buffer-overflow (/home/adam/dev/gross/onnxruntime/build/Linux/Debug/onnxruntime_test_all+0x38a0b00) (BuildId: 0885f9f0665127b4a0fbd8529a0f0d31a8497999) in Eigen::internal::mapbase_evaluator<Eigen::Map<Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> >, Eigen::Array<Eigen::half, -1, 1, 0, -1, 1> const>::coeff(long) const
1: Shadow bytes around the buggy address:
1:   0x508ffffffd80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1:   0x508ffffffe00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1:   0x508ffffffe80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1:   0x508fffffff00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1:   0x508fffffff80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1: =>0x509000000000: fa fa fa fa fa fa fa fa 00 00[02]fa fa fa fa fa
1:   0x509000000080: fa fa fa fa fa fa fa fa 06 fa fa fa fa fa fa fa
1:   0x509000000100: fa fa fa fa fa fa fa fa 00 00 02 fa fa fa fa fa
1:   0x509000000180: fa fa fa fa fa fa fa fa 00 00 02 fa fa fa fa fa
1:   0x509000000200: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
1:   0x509000000280: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
1: Shadow byte legend (one shadow byte represents 8 application bytes):
1:   Addressable:           00
1:   Partially addressable: 01 02 03 04 05 06 07 
1:   Heap left redzone:       fa
1:   Freed heap region:       fd
1:   Stack left redzone:      f1
1:   Stack mid redzone:       f2
1:   Stack right redzone:     f3
1:   Stack after return:      f5
1:   Stack use after scope:   f8
1:   Global redzone:          f9
1:   Global init order:       f6
1:   Poisoned by user:        f7
1:   Container overflow:      fc
1:   Array cookie:            ac
1:   Intra object redzone:    bb
1:   ASan internal:           fe
1:   Left alloca redzone:     ca
1:   Right alloca redzone:    cb
1: ==312770==ABORTING
1/1 Test #1: onnxruntime_test_all .............***Failed    1.16 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) =   1.16 sec

The following tests FAILED:
	  1 - onnxruntime_test_all (Failed)
Errors while running CTest
Output from these tests are in: /home/adam/dev/gross/onnxruntime/build/Linux/Debug/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.

Urgency

No response

Platform

Linux

OS Version

Fedora 40

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

5d78b9a

ONNX Runtime API

C++

Architecture

X64

Execution Provider

Default CPU

Execution Provider Library Version

No response
@adamreeve
Copy link
Contributor Author

adamreeve commented Jul 30, 2024 •
edited
Loading

It turns out the NaNs in the input were a red herring, the error can be reproduced without any NaNs and is related to the shape of the inputs used in this test. I've edited the description to remove NaNs from the example code.

From testing various shapes, it seems like any 2D inputs cause the error but 1D is fine.

@adamreeve adamreeve changed the title heap-buffer-overflow running address sanitizer on Min with float16 NaNs heap-buffer-overflow running address sanitizer on Min with float16 Jul 30, 2024
@snnn snnn added the core runtime issues related to core runtime label Jul 30, 2024
@github-actions GitHub Actions
Copy link
Contributor

This issue has been automatically marked as stale due to inactivity and will be closed in 30 days if no further activity occurs. If further support is needed, please provide an update and/or more details.

@github-actions github-actions bot added the stale issues that have not been addressed in a while; categorized by a bot label Aug 31, 2024
@adamreeve adamreeve linked a pull request Sep 18, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
core runtime issues related to core runtime stale issues that have not been addressed in a while; categorized by a bot
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix memory access violations in the CPU float16 min and max operators adamreeve/onnxruntime
2 participants
@adamreeve @snnn