How to use Windows ConPTY API from a process whose output has been redirected?

[ForNeVeR]

I am working on an application which creates a Windows pseudoconsole (using the ConPTY API, according to the official manual which is really good).

It starts a PowerShell session (for test), and shows everything ConPTY sends from that session via the pseudoconsole, and everything works really well. Until I redirect the parent program output to anywhere.

Here's my full code listing for reference.

#include <string>
#include <stdexcept>
#include <iostream>
#include <vector>

#include <Windows.h>

using namespace std::string_literals;

void throwLastError(int number)
{
	throw std::runtime_error("Error "s + std::to_string(number) + ": "s + std::to_string(GetLastError()));
}

STARTUPINFOEXW prepareStartupInformation(HPCON hPCon)
{
	STARTUPINFOEXW startupInfo{sizeof(STARTUPINFOEXW)};
	SIZE_T bytesRequired = 0;
	if (InitializeProcThreadAttributeList(nullptr, 1, 0, &bytesRequired))
		throw std::runtime_error("InitializeProcThreadAttributeList wasn't expected to succeed at that time.");

	const auto threadAttributeList = static_cast<LPPROC_THREAD_ATTRIBUTE_LIST>(calloc(bytesRequired, 1));
	startupInfo.lpAttributeList = threadAttributeList;
	
	if (!InitializeProcThreadAttributeList(threadAttributeList, 1, 0, &bytesRequired))
		throwLastError(4);

	if (!UpdateProcThreadAttribute(
			threadAttributeList,
			0,
			PROC_THREAD_ATTRIBUTE_PSEUDOCONSOLE,
			hPCon,
			sizeof(HPCON),
			nullptr,
			nullptr)
	)
		throwLastError(5);

	return startupInfo;
}

PROCESS_INFORMATION startProcess(STARTUPINFOEXW startupInfo, const std::wstring &commandLine)
{
	std::vector<wchar_t> cmdLineBuffer;
	for (auto x : commandLine)
		cmdLineBuffer.push_back(x);
	cmdLineBuffer.push_back(L'\0');
	
	PROCESS_INFORMATION processInfo{nullptr};
	if (!CreateProcessW(
		nullptr,
		cmdLineBuffer.data(),
		nullptr,
		nullptr,
		FALSE,
		EXTENDED_STARTUPINFO_PRESENT,
		nullptr,
		nullptr,
		&startupInfo.StartupInfo,
		&processInfo)
	)
		throwLastError(6);

	return processInfo;
}

int main()
{
	auto h = GetStdHandle(STD_OUTPUT_HANDLE);
	auto type = GetFileType(h);
	std::cout << "STD_OUTPUT_HANDLE " << h << " " << type << std::endl;
	
	try
	{
		HANDLE inPipeRead = nullptr, inPipeWrite = nullptr;
		if (!CreatePipe(&inPipeRead, &inPipeWrite, nullptr, 0))
			throwLastError(1);
	
		HANDLE outPipeRead = nullptr, outPipeWrite = nullptr;
		if (!CreatePipe(&outPipeRead, &outPipeWrite, nullptr, 0))
			throwLastError(2);
	
		HPCON hPCon = nullptr;
		if (CreatePseudoConsole(COORD { 80, 25 }, inPipeRead, outPipeWrite, 0, &hPCon) != S_OK)
			throwLastError(3);

		CloseHandle(inPipeRead);
		CloseHandle(outPipeWrite);
	
		auto startupInfo = prepareStartupInformation(hPCon);

		auto processInfo = startProcess(startupInfo, L"powershell.exe");
	
		std::cout << "PID: " << processInfo.dwProcessId << "\nTID: " << processInfo.dwThreadId << "\n";

		char buffer[1024];
		DWORD readBytes = 0;
		while (ReadFile(
			outPipeRead,
			buffer,
			sizeof buffer - 1,
			&readBytes,
			nullptr
		))
		{
			buffer[readBytes] = 0;
			std::cout << "Read bytes: " << readBytes << "\n";
			std::cout << buffer << "\n";
		}
	}
	catch (const std::runtime_error &x)
	{
		std::cerr << x.what() << "\n";
	}
}

So, if I start this program from terminal or from an IDE, it works (and tells how much data it has read from the PTY pipe). But if I start it with output redirected (e.g. myprogram.exe > C:\Temp\somefile.txt, or even myprogram.exe | out-host in pwsh), then it stops working: PowerShell then somehow inherits the stdin and stdout, doesn't write anything to the PTY pipe and uses the stdout/stdin instead.

For diagnostics, I've added output of GetFileType(GetStdHandle(STD_OUTPUT_HANDLE)), which helps to detect if the output has been redirected (since it's not always obvious in practice). For pristine stdout, it should write 2, and other numbers for other output modes.

How can I overcome this issue? Is it possible for my program to work even if its own stdout/stderr were redirected somewhere? I thought one of the points of PTY was to create a separate isolated environment, which won't interfere with the parent one.

[ForNeVeR]

I've been able to reproduce the same issue even on the test C# GUI application. Steps to reproduce:

1. Open GUIConsole.WPF.csproj from this repo, and replace <OutputType>WinExe</OutputType> with <OutputType>Exe</OutputType>, to make it a console application with redirectable output (aka /SUBSYSTEM:CONSOLE):
   

   terminal/samples/ConPTY/GUIConsole/GUIConsole.WPF/GUIConsole.WPF.csproj

   Line 8 in 0a7310d

   <OutputType>WinExe</OutputType>

2. Compile it and start GUIConsole.WPF.exe, observe that it works well.
3. Start GUIConsole.WPF.exe > C:\Temp\file.txt, observe how it breaks (no console output, the window is empty).
   

[DHowett]

Thanks for expressing appreciation for that document! You know, this is tickling a memory of handle inheritance causing us some significant trouble in the past.

While I go look for the change we made on the OS intended to make this better, could you try setting startupInfo.StartupInfo.dwFlags |= STARTF_USESTDHANDLES?

Enabling the use of hStdIn... et al but leaving them set to null should force the new process to not inherit the redirected default handles.

[DHowett]

A-ha! We did do some work to improve handle inheritance, but only if the inherited handles were console handles (so: the spawned application would connect to its parent's stdin/stdout even if it was supposed to be using a pseudoconsole)

    //
    // If we are establishing a Pseudoterminal Connection, close
    // any in/out/err handles that may have been copied from the parent.
    // We will re-make new connections to the given PTY instead of
    // using the automatically inherited handles.
    //
        
    if ((ProcessParameters->ConsoleFlags & CONSOLE_USING_PTY_REFERENCE) != 0)
    {
        ConsoleCloseIfConsoleHandle(&ProcessParameters->StandardInput);
        ConsoleCloseIfConsoleHandle(&ProcessParameters->StandardOutput);
        ConsoleCloseIfConsoleHandle(&ProcessParameters->StandardError);
    }

We didn't do the same for all other inherited handles, so... I'm going to file a bug on the docs to make sure we document the need for STARTF_USESTDHANDLES.

[eryksun]

We didn't do the same for all other inherited handles, so... I'm going to file a bug on the docs to make sure we document the need for STARTF_USESTDHANDLES.

It's not handle inheritance. CreateProcess is called with bInheritHandles as FALSE. What happens is that the standard handles get duplicated to the child, not inherited (i.e. the handle values are not the same). It's an old hack to make spawning console applications magically work when the parent's console is inherited but handle inheritance is disabled, and the standard handles aren't explicitly set. Specifically, this occurs in the following case:

• the target executable is flagged as a console application
• inheritance is disabled
• the startup info doesn't include the flag STARTF_USESTDHANDLES
• the creation flags do not specify CREATE_NEW_CONSOLE, CREATE_NO_WINDOW, or DETACHED_PROCESS

Maybe I'm missing something, but it seems to me that the pseudoconsole case is like the latter three scenarios (i.e. a new console or no console). If so, then the implementation of NtCreateUserProcess in the kernel should be updated to skip duplicating the parent's standard handles to the child, and to set the standard handle values in the child's ProcessParameters to NULL.

[ForNeVeR]

@DHowett, thanks for your answer!

Yes, your suggestion seems to work. I'll perform some experiments this week to see if it covers my use case, and let you know if there are any additional questions.

For now, I guess we could close the issue? Or would you like for it to stay open until the documentation changes are made?

[eryksun]

Or would you like for it to stay open until the documentation changes are made?

That's a disappointing resolution. If a child process isn't getting spawned with the same console as the parent, the system simply shouldn't duplicate the parent's standard I/O handles to the child. It should work the same way as CREATE_NEW_CONSOLE, CREATE_NO_WINDOW, and DETACHED_PROCESS.