Is it possible to use OpenCL backend on AMD GPUs?

[biergaizi]

According to the documentation of oneAPI DPC++, currently the "official" way of using DPC++ on AMD GPUs is using the ROCm HIP backend. However, so far AMD only supports a few selected datacenter and workstation model in ROCm. From the documentation alone, it appears that there's no way to port DPC++ programs to desktop AMDGPU models.

However, from a previous question from 2021, OpenCL backend on Nvidia GPU, I realized DPC++ in fact still has a OpenCL backend as a fallback, although it requires OpenCL 2.1.

The questions are:

1. Is the OpenCL backend still usable in today's DPC++? Or has it already been deprecated?
2. If I have a AMD GPU with OpenCL 2.1, is it possible to run DPC++ code without ROCm HIP?

Thanks.

[al42and]

Hi! Not an official answer, but I tried using oneAPI with AMD consumer hardware recently:

OpenCL backend of DPC++ uses SPIR-V, which is not supported in any current AMD driver. E.g., here's an old official AMD reply: https://community.amd.com/t5/opencl/amd-rx5700xt-opencl-2-1-suport-for-clcreateprogramwithil/td-p/423303; I haven't seen any related activity since then.

That said, while ROCm HIP does not officially support consumer devices, it tends to work fine on them, as does DPC++.

Note: since recently, DPC++ decided to use an explicit of supported architectures, and some consumer devices might be missing there, despite working with ROCm HIP; that is a bit annoying, but not too hard to work around.

[AlexeySachkov]

Hi @biergaizi,

Is the OpenCL backend still usable in today's DPC++? Or has it already been deprecated?

Yes, OpenCL is still one of the supported backends by this SYCL implementation and I haven't heard about any plans to deprecate it.

If I have a AMD GPU with OpenCL 2.1, is it possible to run DPC++ code without ROCm HIP?

Yes, it should be possible. Strictly speaking, you don't even need OpenCL 2.1, but you need support for SPIR-V. It could also be achieved by supporting cl_khr_il_program by OpenCL runtimes of an older version or it could be OpenCL 3.0 which returns SPIR-V through CL_DEVICE_IL_VERSION (or other similar queries).

At DPC++ side, you would need to compile with spir64 target (which is default one) to instruct the compiler generate SPIR-V. The rest will be automatically done by DPC++ RT assuming that such OpenCL device is properly configured (I mean discoverable through OpenCL ICD Loader).

Unfortunately, I don't have any practical experience with AMD devices (regardless of the backend). Please also see reply from @al42and with more practical data points.

[al42and]

Got curious and did a quick test with Mesa 23.1: with recent IntelLLVM from this repo, the devices are detected, but both iris and radeonsi backend fail when launching a simple kernel 😿:

$ RUST_BACKTRACE=full RUSTICL_ENABLE=radeonsi ./hello_sycl 
Running on AMD Radeon RX 6400 (navi24, LLVM 15.0.7, DRM 3.49, 6.2.8-060208-generic)
thread '<unnamed>' panicked at 'called `Option::unwrap()` on a `None` value', ../src/gallium/frontends/rusticl/core/program.rs:589:13
stack backtrace:
   0:     0x7ff05e2f75a3 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hf15e8f9e6884dd5f
   1:     0x7ff05e369b7c - core::fmt::write::he42254d9e3c27115
   2:     0x7ff05e2c8575 - <unknown>
   3:     0x7ff05e2d1d04 - <unknown>
   4:     0x7ff05e2d1944 - std::panicking::default_hook::h27aa44be03b01ac8
   5:     0x7ff05e2d2513 - std::panicking::rust_panic_with_hook::he7013d2ea706cde0
   6:     0x7ff05e2f7b69 - <unknown>
   7:     0x7ff05e2f76e4 - <unknown>
   8:     0x7ff05e2d21f2 - rust_begin_unwind
   9:     0x7ff05dfd1d53 - core::panicking::panic_fmt::hbfca70bd08ba6212
  10:     0x7ff05dfd1c1d - core::panicking::panic::hc1bbfa9fa05e63d3
  11:     0x7ff05e0addae - <unknown>
  12:     0x7ff05e059c37 - <unknown>
  13:     0x7ff05e01dcfa - <unknown>
  14:     0x7ff05e01ee8a - <unknown>
  15:     0x7ff05e038482 - <unknown>
  16:     0x7ff05e0b8d5c - <unknown>
  17:     0x7ff071be693e - piKernelCreate
  18:     0x7ff06ec71e38 - _ZNK4sycl3_V16detail6plugin12call_nocheckILNS1_9PiApiKindE52EJP11_pi_programPKcPP10_pi_kernelEEE10_pi_resultDpT0_
  19:     0x7ff06ec885a4 - _ZN4sycl3_V16detail14ProgramManager17getOrCreateKernelElRKSt10shared_ptrINS1_12context_implEERKS3_INS1_11device_implEERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPKNS1_12program_implE
  20:     0x7ff06ecd4031 - _ZN4sycl3_V16detail16enqueueImpKernelERKSt10shared_ptrINS1_10queue_implEERNS1_8NDRDescTERSt6vectorINS1_7ArgDescESaISA_EERKS2_INS1_18kernel_bundle_implEERKS2_INS1_11kernel_implEERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKlRS9_IP9_pi_eventSaISX_EEPSX_RKSt8functionIFPvPNS1_16AccessorImplHostEEE23_pi_kernel_cache_config
  21:     0x7ff06ecd641f - _ZN4sycl3_V16detail13ExecCGCommand10enqueueImpEv
  22:     0x7ff06ecc5b28 - _ZN4sycl3_V16detail7Command7enqueueERNS1_14EnqueueResultTENS1_9BlockingTERSt6vectorIPS2_SaIS7_EE
  23:     0x7ff06ece4141 - _ZN4sycl3_V16detail9Scheduler14GraphProcessor14enqueueCommandEPNS1_7CommandERSt11shared_lockISt18shared_timed_mutexERNS1_14EnqueueResultTERSt6vectorIS5_SaIS5_EES5_NS1_9BlockingTE
  24:     0x7ff06ece1029 - _ZN4sycl3_V16detail9Scheduler19enqueueCommandForCGESt10shared_ptrINS1_10event_implEERSt6vectorIPNS1_7CommandESaIS8_EENS1_9BlockingTE
  25:     0x7ff06ece318c - _ZN4sycl3_V16detail9Scheduler5addCGESt10unique_ptrINS1_2CGESt14default_deleteIS4_EERKSt10shared_ptrINS1_10queue_implEE
  26:     0x7ff06ed22490 - _ZN4sycl3_V17handler8finalizeEv
  27:     0x7ff06ed4647b - _ZN4sycl3_V16detail10queue_impl11submit_implERKSt8functionIFvRNS0_7handlerEEERKSt10shared_ptrIS2_ESD_SD_RKNS1_13code_locationEPKS3_IFvbbRNS0_5eventEEE
  28:     0x7ff06ed46860 - _ZN4sycl3_V15queue11submit_implESt8functionIFvRNS0_7handlerEEERKNS0_6detail13code_locationE
  29:           0x4042e6 - _ZN4sycl3_V15queue6submitIZ4mainEUlRNS0_7handlerEE_EENS0_5eventET_RKNS0_6detail13code_locationE
  30:           0x403bf6 - main
  31:     0x7ff06e429d90 - __libc_start_call_main
                               at ./csu/../sysdeps/nptl/libc_start_call_main.h:58:16
  32:     0x7ff06e429e40 - __libc_start_main_impl
                               at ./csu/../csu/libc-start.c:392:3
  33:           0x403685 - _start
  34:                0x0 - <unknown>
Segmentation fault (core dumped)

[al42and]

There is also https://github.com/MathiasMagnus/spirv2bin, which aims to add SPIR-V support to AMD's official OpenCL library. Does not look production-ready yet.

So, overall, using HIP PI is the only usable option today, despite the lack of official support.

[biergaizi]

Got curious and did a quick test with Mesa 23.1: with recent IntelLLVM from this repo, the devices are detected, but both iris and radeonsi backend fail when launching a simple kernel crying_cat_face:

If it's not known already, it would be useful to report this problem to the upstream.

[biergaizi]

Got curious and did a quick test with Mesa 23.1: with recent IntelLLVM from this repo, the devices are detected, but both iris and radeonsi backend fail when launching a simple kernel crying_cat_face:

If it's not known already, it would be useful to report this problem to the upstream.

I took a look and found a similar Mesa OpenCL bug here, not sure if it's the same one (update: it was not!):
https://gitlab.freedesktop.org/mesa/mesa/-/issues/7553

It apparently was an LLVM bug in the AMDGPU backend, the fix was only landed LLVM 16. If you have spare time, perhaps try rebuilding Mesa with LLVM 16 next time.

[biergaizi]

I also replicated the same problem on my machine, with LLVM 16 (my previously "identified fix" was for another unrelated issue). Using a debug build, I managed to find an assertion error within Mesa's SPIR-V parser.

I've filed an upstream bug report:

https://gitlab.freedesktop.org/mesa/mesa/-/issues/9037

Update: around 10 problems have already been identified by us within 3 days. Thus, currently SYCL in Mesa should be considered very-early alpha. We're tracking all problems using the following link. It's not surprising, Mesa's Rusticl OpenCL engine is only 2 months old. But perhaps, Mesa may become usable in 2024.

https://gitlab.freedesktop.org/mesa/mesa/-/issues/9061