RFC 1665: windows-subsystem

tools (platform | attributes | linkage)

Summary

Rust programs compiled for Windows will always allocate a console window on startup. This behavior is controlled via the SUBSYSTEM parameter passed to the linker, and so can be overridden with specific compiler flags. However, doing so will bypass the Rust-specific initialization code in libstd, as when using the MSVC toolchain, the entry point must be named WinMain.

This RFC proposes supporting this case explicitly, allowing libstd to continue to be initialized correctly.

Motivation

The WINDOWS subsystem is commonly used on Windows: desktop applications typically do not want to flash up a console window on startup.

Currently, using the WINDOWS subsystem from Rust is undocumented, and the process is non-trivial when targeting the MSVC toolchain. There are a couple of approaches, each with their own downsides:

Define a WinMain symbol

A new symbol pub extern "system" WinMain(...) with specific argument and return types must be declared, which will become the new entry point for the program.

This is unsafe, and will skip the initialization code in libstd.

The GNU toolchain will accept either entry point.

Override the entry point via linker options

This uses the same method as will be described in this RFC. However, it will result in build scripts also being compiled for the WINDOWS subsystem, which can cause additional console windows to pop up during compilation, making the system unusable while a build is in progress.

Detailed design

When an executable is linked while compiling for a Windows target, it will be linked for a specific subsystem. The subsystem determines how the operating system will run the executable, and will affect the execution environment of the program.

In practice, only two subsystems are very commonly used: CONSOLE and WINDOWS, and from a user's perspective, they determine whether a console will be automatically created when the program is started.

New crate attribute

This RFC proposes two changes to solve this problem. The first is adding a top-level crate attribute to allow specifying which subsystem to use:

#![windows_subsystem = "windows"]

Initially, the set of possible values will be {windows, console}, but may be extended in future if desired.

The use of this attribute in a non-executable crate will result in a compiler warning. If compiling for a non-Windows target, the attribute will be silently ignored.

Additional linker argument

For the GNU toolchain, this will be sufficient. However, for the MSVC toolchain, the linker will be expecting a WinMain symbol, which will not exist.

There is some complexity to the way in which a different entry point is expected when using the WINDOWS subsystem. Firstly, the C-runtime library exports two symbols designed to be used as an entry point:

mainCRTStartup
WinMainCRTStartup

LINK.exe will use the subsystem to determine which of these symbols to use as the default entry point if not overridden.

Each one performs some unspecified initialization of the CRT, before calling out to a symbol defined within the program (main or WinMain respectively).

The second part of the solution is to pass an additional linker option when targeting the MSVC toolchain: /ENTRY:mainCRTStartup

This will override the entry point to always be mainCRTStartup. For console-subsystem programs this will have no effect, since it was already the default, but for WINDOWS subsystem programs, it will eliminate the need for a WinMain symbol to be defined.

This command line option will always be passed to the linker, regardless of the presence or absence of the windows_subsystem crate attribute, except when the user specifies their own entry point in the linker arguments. This will require rustc to perform some basic parsing of the linker options.

Drawbacks

Alternatives

Unresolved questions

None