[WIP] Out-Tree EP feature #21450

jslhcl · 2024-07-23T00:36:08Z

Description

Out-Tree EP feature.

Motivation and Context

Decouple ExecutionProvider from ONNXRuntime and make it a standalone class which will benefit 3rd party EP authors to write their own EP.
Binary compatibility is required for this feature.

samples/outTreeEp/out_tree_ep.cc

samples/c_test/test.cpp

samples/outTreeEp/out_tree_ep.h

@@ -0,0 +1,35 @@
+#pragma once


onnxruntime/core/framework/provider_adapter.h

@@ -0,0 +1,14 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/c_test/test.cpp

onnxruntime/core/session/inference_session.cc

jslhcl · 2024-07-23T22:11:31Z

samples/outTreeEp/out_tree_ep.cc

+#ifdef __cplusplus
+extern "C" {
+#endif
+OrtExecutionProviderFactory* RegisterCustomEp() {


return Status instead

… EP as graph API is not exported by ORT. Need to put these graph API into ortapi structure

samples/outTreeEp/out_tree_ep.cc

samples/c_test/test.cpp

…roviderAdapter::Compile()

adrianlizarraga · 2024-07-30T21:24:49Z

include/onnxruntime/core/session/onnxruntime_c_api.h

+} OrtMetaDef;
+
+typedef struct OrtIndexedSubGraph {
+  OrtMetaDef* meta_def; // TODO(leca): how to define a nested structure pointer?


Does this have to be a pointer to an OrtMetaDef? It may be simpler if this meta_def is contained by value instead.

It looks we will check the pointer is null or not to distinguish between single node mode and fused node mode (See base class IExecutionProvider::GetCapability() which does not set this pointer and TryAssignSingleNode() which will check this pointer)

adrianlizarraga · 2024-07-30T21:34:37Z

samples/outTreeEp/out_tree_ep.cc

+
+OutTreeEp::OutTreeEp(const char* ep_type, const OutTreeEpInfo& ep_info) : info(ep_info) {
+    type = ep_type;
+    OrtExecutionProvider::GetCapability = [](const OrtExecutionProvider* this_, const OrtGraphViewer* graph, size_t* cnt, OrtIndexedSubGraph*** indexed_sub_graph) {


If I'm understanding correctly, the type of the OrtIndexedSubGraph*** indexed_sub_graph parameter is essentially asking the EP to fill out an array of pointers to OrtIndexedSubGraph objects.

Would it be simpler to change this to OrtIndexedSubgraph** indexed_sub_graph so that the EP fills out an array of OrtIndexedSubGraph objects directly? Each OrtIndexedSubgraph struct is a simple POD that can be created on the stack and copied around. It seems like it would result in less pointer tracking.

Also, who is responsible for freeing this memory and when? If the EP allocates an array, then the EP should free it. The currently example leaks the allocations.

Edit: one possibility is to have onnxruntime call a new EP function (e.g., ReleaseOrtIndexedSubGraph()) so the the EP can free the memory. onnxruntime would call this once it is done using the indexed_sub_graph.

The problem is that we don't know how many OrtIndexedSubGraph would be before we call GetCapability() function. I will fix the leak issue in the coming commits

samples/outTreeEp/out_tree_ep.cc

onnxruntime/core/framework/provider_factory_adapter.h

@@ -0,0 +1,21 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/outTreeEp_kernel/kernel_ep.cc

@@ -0,0 +1,89 @@
+#include "kernel_ep.h"


samples/outTreeEp_kernel/kernel_ep.h

@@ -0,0 +1,36 @@
+#pragma once


include/onnxruntime/core/framework/ort_type_constraints.h

@@ -0,0 +1,15 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


onnxruntime/core/framework/ort_type_constraints.cc

@@ -0,0 +1,14 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/qnnEp/builder/qnn_model_wrapper.cc

@@ -0,0 +1,627 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/qnnEp/builder/qnn_model_wrapper.h

@@ -0,0 +1,285 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/qnnEp/builder/qnn_quant_params_wrapper.cc

@@ -0,0 +1,266 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/qnnEp/builder/qnn_quant_params_wrapper.h

@@ -0,0 +1,147 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/qnnEp/builder/qnn_utils.cc

@@ -0,0 +1,557 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/qnnEp/builder/qnn_utils.h

@@ -0,0 +1,110 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/qnnEp/qnn_execution_provider.cc

@@ -0,0 +1,54 @@
+#include "qnn_execution_provider.h"


samples/qnnEp/qnn_execution_provider.h

@@ -0,0 +1,33 @@
+#pragma once


…es in Compile function's lambda

samples/outTreeEp/out_tree_ep.cc

@@ -0,0 +1,110 @@
+#include "out_tree_ep.h"


samples/tensorRTEp/tensorrt_execution_provider_utils.h

add GetCapability for tensorrt ep

samples/tensorRTEp/onnx_ctx_model_helper.cc

jslhcl · 2024-08-27T18:29:05Z

include/onnxruntime/core/session/onnxruntime_c_api.h

+
+  ORT_API2_STATUS(OrtGraph_GetModelPath, const OrtGraphViewer* graph, _Outptr_ const void** path);
+
+  ORT_API2_STATUS(OrtGraph_GetOrtGraph, const OrtGraphViewer* graph_viewer, _Outptr_ const OrtGraph** graph);


I don't think we need to expose OrtGraph. We can keep them here and will see

samples/tensorRTEp/onnx_ctx_model_helper.h

@@ -0,0 +1,64 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/tensorRTEp/tensorrt_execution_provider_utils.h

@@ -0,0 +1,403 @@
+#pragma once


samples/tensorRTEp/tensorrt_execution_provider_info.cc

samples/tensorRTEp/tensorrt_execution_provider_info.h

Only implemented two simple CUDA allocators (without BFC) for now. Note: several TODO.

samples/tensorRTEp/tensorrt_cuda_allocator.h

@@ -0,0 +1,82 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/tensorRTEp/tensorrt_execution_provider_info.cc

@@ -0,0 +1,340 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/tensorRTEp/tensorrt_execution_provider_info.h

@@ -0,0 +1,67 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


…ph C API

…untime into leca/outOfTreeEP

…nted out in GetSupportedList()

…stPoneRegisterResourceFor()

… trt type to tensorrtEp

### Description  ### Motivation and Context  --------- Co-authored-by: guyang3532 <[email protected]>

samples/c_test/test.cpp

@@ -0,0 +1,216 @@
+#include "core/session/onnxruntime_c_api.h"


… to make graph partition the same, and improve test

jslhcl added 2 commits July 17, 2024 20:39

opaque pointer for graph

0e6a80c

ORT C API RegisterOrtExecutionProviderLibrary work

c30a639

jslhcl requested review from souptc, adrianlizarraga, jywu-msft and chilo-ms July 23, 2024 00:36

github-advanced-security bot found potential problems Jul 23, 2024

View reviewed changes

ORT C-API SessionOptionsAppendOrtExecutionProvider work

7bfe57e

github-advanced-security bot found potential problems Jul 23, 2024

View reviewed changes

jslhcl commented Jul 23, 2024

View reviewed changes

onnxruntime/core/session/inference_session.cc Outdated Show resolved Hide resolved

jslhcl commented Jul 23, 2024

View reviewed changes

Test Relu with compile based EP, build work, runtime error of loading…

8e7d28d

… EP as graph API is not exported by ORT. Need to put these graph API into ortapi structure

github-advanced-security bot found potential problems Jul 26, 2024

View reviewed changes

samples/outTreeEp/out_tree_ep.cc Fixed Show fixed Hide fixed

samples/c_test/test.cpp Fixed Show fixed Hide fixed

jslhcl added 2 commits July 29, 2024 17:49

prototype works with hardcode node_compute_info's index in ExecutionP…

808bfc3

…roviderAdapter::Compile()

prototype works without hardcode

49e396c

adrianlizarraga reviewed Jul 30, 2024

View reviewed changes

samples/outTreeEp/out_tree_ep.cc Outdated Show resolved Hide resolved

jslhcl added 2 commits July 31, 2024 20:38

fix comments for Compile function

e790105

add provider_factory_adapter.h

92f529d

github-advanced-security bot found potential problems Aug 1, 2024

View reviewed changes

onnxruntime/core/framework/provider_factory_adapter.h

@@ -0,0 +1,21 @@

// Copyright (c) Microsoft Corporation. All rights reserved.

Check warning

Code scanning / lintrunner

CLANGFORMAT/format Warning

See https://clang.llvm.org/docs/ClangFormat.html.
Run lintrunner -a to apply this patch.

jslhcl added 2 commits August 5, 2024 19:27

fix crash after introducing kernel based EP

3d83ed1

kernel based EP work with type constraint check commented out

e29499a

github-advanced-security bot found potential problems Aug 6, 2024

View reviewed changes

add kernel type constraints from out tree EP

f3678c4

github-advanced-security bot found potential problems Aug 7, 2024

View reviewed changes

jslhcl added 2 commits August 7, 2024 16:27

add API ReleaseOrtTypeConstraints

ac5ae0a

introduce qnn ep

0cc78e8

github-advanced-security bot found potential problems Aug 12, 2024

View reviewed changes

more graph/node C API

740a687

Add extra parameters in OrtExecutionProvider to avoid capture variabl…

1f10c28

…es in Compile function's lambda

github-advanced-security bot found potential problems Aug 22, 2024

View reviewed changes

samples/outTreeEp/out_tree_ep.cc

@@ -0,0 +1,110 @@

#include "out_tree_ep.h"

Check warning

Code scanning / lintrunner

CLANGFORMAT/format Warning

See https://clang.llvm.org/docs/ClangFormat.html.
Run lintrunner -a to apply this patch.

add OrtGraph_SerializeToArray

5e46d0f

github-advanced-security bot found potential problems Aug 23, 2024

View reviewed changes

samples/tensorRTEp/tensorrt_execution_provider_utils.h Fixed Show fixed Hide fixed

jslhcl and others added 3 commits August 24, 2024 00:17

finish Compile function

85c168d

add override function implementation and cudart dependency for tensorrt

7bdb36a

add outOfTree tensorrt ep.1 (#21830)

7d915b7

add GetCapability for tensorrt ep

github-advanced-security bot found potential problems Aug 27, 2024

View reviewed changes

samples/tensorRTEp/onnx_ctx_model_helper.cc Fixed Show fixed Hide fixed

jslhcl commented Aug 27, 2024

View reviewed changes

GetSupportedList

4aea94b

github-advanced-security bot found potential problems Aug 28, 2024

View reviewed changes

GetSubGraph and TensorrtExecutionProviderInfo

865a17f

github-advanced-security bot found potential problems Aug 29, 2024

View reviewed changes

samples/tensorRTEp/tensorrt_execution_provider_info.cc Fixed Show fixed Hide fixed

samples/tensorRTEp/tensorrt_execution_provider_info.h Fixed Show fixed Hide fixed

Add simple CUDA allocators for TRT EP (#21901)

2811541

Only implemented two simple CUDA allocators (without BFC) for now. Note: several TODO.

github-advanced-security bot found potential problems Aug 29, 2024

View reviewed changes

samples/tensorRTEp/tensorrt_cuda_allocator.h

@@ -0,0 +1,82 @@

// Copyright (c) Microsoft Corporation. All rights reserved.

Check warning

Code scanning / lintrunner

CLANGFORMAT/format Warning

See https://clang.llvm.org/docs/ClangFormat.html.
Run lintrunner -a to apply this patch.

add constructor for tensorrt ep and refine GetCapability (#21914)

c97b19f

github-advanced-security bot found potential problems Aug 29, 2024

View reviewed changes

jslhcl and others added 11 commits August 29, 2024 18:11

relu can work on out tree TRT now

36f97b5

rebuild graph proto from scratch with the information needed from gra…

2fc7aac

…ph C API

complete the GetCapability (#21956)

4ad6993

Chi's fix and reorder ep for registering shared resource

53c736f

complete the GetSubGraph (#21998)

5fcb972

run resnet18v1_7, crash on GetSubGraph()

c3bb437

Merge branch 'leca/outOfTreeEP' of https://github.com/microsoft/onnxr…

d1c657c

…untime into leca/outOfTreeEP

resnet18-v1-7 works for TRT EP, with next_nodes_list assignment comme…

3efac97

…nted out in GetSupportedList()

test cases for decoder and fast_rcnn, delete dynamic_cast in ShouldPo…

766fec9

…stPoneRegisterResourceFor()

add tensorrt home in CMakeLists, add trt and CUDA ep for test, change…

ea2465c

… trt type to tensorrtEp

github-advanced-security bot found potential problems Sep 18, 2024

View reviewed changes

samples/c_test/test.cpp

@@ -0,0 +1,216 @@

#include "core/session/onnxruntime_c_api.h"

Check warning

Code scanning / lintrunner

CLANGFORMAT/format Warning test

See https://clang.llvm.org/docs/ClangFormat.html.
Run lintrunner -a to apply this patch.

use parameter ExecutionOrder::PRIORITY_BASED for GraphViewerToProto()…

330cdb6

… to make graph partition the same, and improve test

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Out-Tree EP feature #21450

[WIP] Out-Tree EP feature #21450

jslhcl commented Jul 23, 2024 •

edited

Loading

jslhcl Jul 23, 2024

adrianlizarraga Jul 30, 2024 •

edited

Loading

jslhcl Jul 30, 2024

adrianlizarraga Jul 30, 2024

adrianlizarraga Jul 30, 2024 •

edited

Loading

jslhcl Jul 31, 2024 •

edited

Loading

jslhcl Aug 27, 2024

		@@ -0,0 +1,14 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

		@@ -0,0 +1,21 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

		@@ -0,0 +1,15 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

		@@ -0,0 +1,627 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

		@@ -0,0 +1,285 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

		@@ -0,0 +1,266 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

		@@ -0,0 +1,147 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

		@@ -0,0 +1,557 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

		@@ -0,0 +1,110 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.


		ORT_API2_STATUS(OrtGraph_GetModelPath, const OrtGraphViewer* graph, _Outptr_ const void** path);

		ORT_API2_STATUS(OrtGraph_GetOrtGraph, const OrtGraphViewer* graph_viewer, _Outptr_ const OrtGraph** graph);

		@@ -0,0 +1,64 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

		@@ -0,0 +1,82 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

		@@ -0,0 +1,340 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

		@@ -0,0 +1,67 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

		@@ -0,0 +1,216 @@
		#include "core/session/onnxruntime_c_api.h"

[WIP] Out-Tree EP feature #21450

Are you sure you want to change the base?

[WIP] Out-Tree EP feature #21450

Conversation

jslhcl commented Jul 23, 2024 • edited Loading

Description

Motivation and Context

jslhcl Jul 23, 2024

Choose a reason for hiding this comment

adrianlizarraga Jul 30, 2024 • edited Loading

Choose a reason for hiding this comment

jslhcl Jul 30, 2024

Choose a reason for hiding this comment

adrianlizarraga Jul 30, 2024

Choose a reason for hiding this comment

adrianlizarraga Jul 30, 2024 • edited Loading

Choose a reason for hiding this comment

jslhcl Jul 31, 2024 • edited Loading

Choose a reason for hiding this comment

jslhcl Aug 27, 2024

Choose a reason for hiding this comment

jslhcl commented Jul 23, 2024 •

edited

Loading

adrianlizarraga Jul 30, 2024 •

edited

Loading

adrianlizarraga Jul 30, 2024 •

edited

Loading

jslhcl Jul 31, 2024 •

edited

Loading