If pika tries to use completion modes and the user did not setup a pool,
the default pool must be returned to prevent accessing one that has not
been created.

Ensure polling is started and handle MPI errors more generically

Remove return type from mpi transform

Changes to mpi internals broke the RAII polling setup.

To test continuations, we need to add 16 to the numeric mpi modes
so detecting the presence at configure time is necessary

…tion

mpi transform tests using set_error are triggering stack corruption
errors that might or might not be real, but disabling them (for now)
will clean up the dashboard

Co-authored-by: Mikael Simberg <[email protected]>

… thread use

The existing mode uses suspend/resume which can be problematic
if threads suspend and block, but in principle we should directly
execute continuations as function calls which this new mode does

Callbacks to sender set_value/error should not be triggered directly
on a polling thread as continuations will be called on the polling
thread. Instead the execution should be tranferred to a pika
thread/task.

Adding a transfer adaptor costs task creation, plus enqueuing/dequeuing
of the task, so the bypass scheduler supports inline execution, but
with the creation of a task wrapper around the callback so that it is
safe to suspend/resume etc and no raw scheduler/polling threads are
running user code.

Bypass scheduler must never be run on top of an existing pika task,
otherwise coroutine self pointer is clobberred (thread local).

Rearrange transfers in transform_mpi to better route inline
continuations through the bypass scheduler and skip it when not needed

We do not need the RAII state restorer object as our thread creation is
always on the executing thread and certain race conditions are not valid

Improve comments and remove unsupported API features

The bypass needs to be added to the cuda polling, but this termporary
fix inserts and extra bypass before issuing mpi requests if the thread
id is invalid.

Also, if the bypass scheduler is invoked on a thread that is already a
task, it skips task creation and becomes a simple pass through
execution.