Integrating ONNXRuntime + QNN EP with UWP applications

[rgsousa88]

Hi,
I'm trying to integrate ONNXRuntime+QNN EP+Qualcomm SDK dependencies in a UWP application but I'm couldn't perform inference in HTP device yet. The steps done are described below.

1 - ONNXRuntime with QNN EP was built successfully according steps described here;
2 - It was created a nuget file with all required includes and libs from ONNXRuntime + QNN EP. It also was included Qualcomm SDK binaries .dll and .so files as described here.
3 - This nuget was included to an UWP application.
4 - Inside the application code, by following this sample code, it was possible to call methods related to session creation and execution, including call to QNN execution provider appending. This application was built without any error, so it was possible to confirm that headers included in nuget were visible to application.

But when I tried to execute the application to perform inference in a HTP device using a quantized onnx model, I've verified that the model was executed in CPU instead of HTP. I've noticed that QnnHtp.dll couldn't be loaded during the backend setup as shown in ONNXRuntime log message: QNN SetupBackend failed qnn_backend_manager.cc:56 onnxruntime::qnn::QnnBackendManager::GetQnnInterfaceProvider Unable to load backend, error: load library failed.
Even thought this library was present in AppX folder, it could not be found by the QNN backend manager. It also was verified that all library are correctly referenced in project files.

In an attempt to execute the model, the full AppX folder path was included to backend library path. In this case, the backend could be created which indicates that this backend setup error is related to library searching mechanism performed by QNN EP running inside a UWP application.
But, backend setup was not enough to perform inference in HTP device. Again, an issue related to library searching by QNN EP prevent the creation of device. At this point, the QNN EP couldn't find libQnnHtpV73Skel.so which was also present in AppX folder. In this case, it was not possible to set the path of this library due to call or search for it be performed by methods inside QNN EP which are not accessible.

In this sense, I would like to know how to integrate ONNXRuntime+QNN EP and the Qualcomm SDK dependencies to an UWP application in such way that inference could be performed in an HTP device. Is there any additional setup that should be done in order that required Qualcomm SDK libraries become visible to UWP application at execution time?

[hariharans29]

Tagging @jywu-msft and @HectorSVC for this query

[HectorSVC]

I doubt that QNN can support UWP. Any reason you have to use UWP?

[rgsousa88]

Hi @HectorSVC ,

It is project requirement that the application should be in UWP. In order to perform model inference using ONNXRuntime, a library was written in C++ and this library, and all its dependencies (in this case, all QNN SDK .dll) are included in UWP application via a nuget package.

Regarding this part,

But, backend setup was not enough to perform inference in HTP device. Again, an issue related to library searching by QNN EP prevent the creation of device. At this point, the QNN EP couldn't find libQnnHtpV73Skel.so which was also present in AppX folder. In this case, it was not possible to set the path of this library due to call or search for it be performed by methods inside QNN EP which are not accessible.

It was observed that loading of libQnnHtpV73Skel.so is performed by QnnHtp.dll. For some still unknown reason, even thought this .so file being in the same folder (in this case, AppX folder) of QnnHtp.dll, the loading fails. But it hasn't occurred when running the C++ sample application. Do you have any ideia why this is happening?

[HectorSVC]

UWP environment is quite different. We need to confirm with QC. I don't think QNN support UWP yet.