Problems generating builtin DLLs as Rust

[evmar]

As part of an upcoming change I'll generate win32 DLL files. Currently they are assembly and compiled via clang, but it might be nice to write them in Rust. That would make it so the whole project can be built with a single toolchain, and also it would allow for writing more code in them as Rust instead of as assembly.

I investigated writing them as Rust and found the following issues, some with workarounds described here. Ultimately, the function folding problems means I won't pursue this, so this bug is just collecting my notes.

File layout

I need one .dll file per Windows library, e.g. kernel32/user32/etc. A given Cargo.toml can only produce a single DLL, so this might require one directory per library, and then for each directory we also need a build.rs etc for them. And to build them all, there's no way to give them all to cargo to build in parallel -- dependencies on cdylibs don't do this -- so we have to use Make or something for that.

One hack I found (from someone else who had a very similar problem!) was to use Rust's "examples" feature, which lets us bundle them all in one Cargo.toml and share a build.rs, though it feels like a serious hack.

Linking

The DLLs will link against retrowin32.dll's retrowin32_syscall symbol. I tried .lib file but I couldn't figure out how to make Rust find the symbol because Rust wants to stdcall symbols to have a @0 stdcall suffix. There's a clang hack where you write #[link_name = "\x01retrowin32_syscall"] and the embedded 1 byte disables mangling(!)

Alternatively, Rust can generate its own import lib, which isn't so bad and looks like

#[link(
    name = "retrowin32",
    kind = "raw-dylib",
    import_name_type = "undecorated"
)]
extern "stdcall" {
    fn retrowin32_syscall();
}

Function merging

The body of each export is just a call to retrowin32_syscall, which means the functions look like

#[no_mangle]
pub unsafe extern "stdcall" fn RegCloseKey(_: u32) {
    retrowin32_syscall();
}

#[no_mangle]
pub unsafe extern "stdcall" fn RegOpenKey(_: u32) {
    retrowin32_syscall();
}

But the compiler does a function-merging pass, folding these into one pointer in the DLL, which breaks our ability to differentiate them. I saw the Rust tree had some hacks for working around this in the past.

Note this is different(!) than the ICF pass in the linker, which we can disable with a flag in build.rs. Toggling this is -Zmerge-functions=disabled which requires nightly Rust, which so far I've avoided.

Vtables

The other thing the DLLs need is to export vtables. I had to do these tricks to embed a function pointer in an exported array, but otherwise I think it works.

// Have to wrap the pointers in a struct to impl Sync.
// Have to impl Sync to use in a static.
// The other option is a "static mut" but that creates a .data section we don't otherwise need.
#[repr(transparent)]
pub struct VTableEntry(*const fn());
unsafe impl Sync for VTableEntry {}

#[no_mangle]
pub static vtab: [VTableEntry; 4] = [
    VTableEntry(core::ptr::null()),
    VTableEntry(RegCloseKey as _),
    VTableEntry(core::ptr::null()),
    VTableEntry(RegOpenKey as _),
];

Code

PR containing a semi-working DLL: #65