Running sway via X11 - send M/K input to Wayland session?

[falk4243]

I am experimenting with starting a sway session based on a short tutorial for headless Steam streaming. Rather surprisingly, this works quite well for the most part - kind of like a poor man's multi-seat solution with a single GPU, something which normally doesn't work under Linux without sacrificing full OpenGL / Vulkan acceleration.

The only issue I have encountered is that while the gamepad correctly sends input to apps started via the sway session, the keyboard and mouse control the terminal / the underlying session instead. I have tested this from Android and iOS, both with touch input and physical devices.

Is there something that could be done on the Sunshine end to enable similar behavior to Steam and wayvnc? Or is there a setting on the OS side (udev rule?) that would do the trick? I looked at "udevadm monitor --property", but it only ever shows gamepad events. If anybody has an idea how to make this work, I'd love to hear it. If you need any logs, let me know.

As for the Sunshine and sway setup, here are the relevant configs -

Sunshine:

hevc_mode = 2
resolutions = [1280x720,1920x1080,2560x1440,3840x2160 ]
min_log_level = 2
output_name = 2
encoder = vaapi
fps = [30,60]
gamepad = x360
upnp = disabled

Sway (default from /etc/sway/config, plus):

exec --no-startup-id dbus-update-activation-environment --all
#exec pipewire (necessary for Steam)
exec pulseaudio -D
exec "sh -c 'sleep 3;exec /usr/libexec/xdg-desktop-portal-gtk -r'"
exec "sh -c 'sleep 6;exec /usr/libexec/xdg-desktop-portal -rv'"
exec "sh -c 'sleep 10;exec sunshine'"
#exec "sh -c 'sleep 15;exec flatpak run com.valvesoftware.Steam'" [or really any other app]

[falk4243]

In the meantime, a related issue has been opened as #137 and I am wondering if nothing can be done to send keyboard and mouse events from the client to the sway session instead of the terminal or X11 desktop it was started from. The gamepad registers perfectly fine under /devices/virtual/input and controls the sway session, but nothing at all is shown for other devices. It's a classical case of "being so close and yet so far" away from the solution. Does anybody have an idea or is this something that would need to be added to Sunshine?

[falk4243]

Had a brief chat with the sway devs and it's clear now why the controller works: sway does not manage those at all and apps connect to them directly.

That leaves the question: Would it be possible for Sunshine to make any devices sending input via Moonlight available to the sway session or is that out of scope for the project? Replicating Steam input might be a bit much to ask after all :p

[ReenigneArcher]

The project is open source, anyone is free to contribute improvements. It's not a guarantee that changes will get merged, but if it improves the experience for the end user there's a good chance.

[falk4243]

Thank you for your answer and while I'd love to contribute code to the project, embarrassingly enough I never learned how to program ... doesn't mean I can't tinker around though and I've found a solution that at least partly covers what I have been asking for. In case anybody else is interested (see below, or above) -

[falk4243]

Starting sway with WLR_BACKENDS=headless,libinput sway on another console (tty1, 2, ...) than the graphical session makes all devices connected to the host PC and client device available to that session as long as the console is selected. You can then just start any program (yuzu, RPCS3, etc.) via mouse and keyboard. As the apps connect to game controllers directly, even when switching the focus away from the console (to the X11 session for example), gamepad input stays with the Sunshine session, while M/K input changes focus to the selected console. This allows one person to play controller-enabled games while somebody else uses the machine for work or other tasks that require mouse and keyboard input; or use M/K in the sway / Sunshine session - I've found that KMS streaming works with this setup, while I never managed to get it running via X11.

I don't see an obvious way to split input devices between the two sessions without resorting to containers or cgroups (not sure if changing the owner on devices would work ...), but am quite happy with this setup, so case closed as far as I am concerned. Feel free to ask any questions regarding the setup if you're interested in getting this to work :)

[falk4243]

PS. With only sway / Wayland, it might be possible to improve this answer using sway-input seats. I'll give it a shot eventually, but am not quite ready to use Wayland as my daily driver yet, so this might take a while. In case anybody ist interested in trying.