libs (io)
io_safetyClose a hole in encapsulation boundaries in Rust by providing users of
AsRawFd and related traits guarantees about their raw resource handles, by
introducing a concept of I/O safety and a new set of types and traits.
Rust's standard library almost provides I/O safety, a guarantee that if one
part of a program holds a raw handle privately, other parts cannot access it.
FromRawFd::from_raw_fd is unsafe, which prevents users from doing things
like File::from_raw_fd(7), in safe Rust, and doing I/O on a file descriptor
which might be held privately elsewhere in the program.
However, there's a loophole. Many library APIs use AsRawFd/IntoRawFd to
accept values to do I/O operations with:
pub fn do_some_io<FD: AsRawFd>(input: &FD) -> io::Result<()> {
some_syscall(input.as_raw_fd())
}
AsRawFd doesn't restrict as_raw_fd's return value, so do_some_io can end
up doing I/O on arbitrary RawFd values. One can even write do_some_io(&7),
since RawFd itself implements AsRawFd.
This can cause programs to access the wrong resources, or even break encapsulation boundaries by creating aliases to raw handles held privately elsewhere, causing spooky action at a distance.
And in specialized circumstances, violating I/O safety could even lead to
violating memory safety. For example, in theory it should be possible to make
a safe wrapper around an mmap of a file descriptor created by Linux's
memfd_create system call and pass &[u8]s to safe Rust, since it's an
anonymous open file which other processes wouldn't be able to access. However,
without I/O safety, and without permanently sealing the file, other code in
the program could accidentally call write or ftruncate on the file
descriptor, breaking the memory-safety invariants of &[u8].
This RFC introduces a path to gradually closing this loophole by introducing:
from_raw_fd/from_raw_handle/from_raw_socket explaining why
they're unsafe in terms of I/O safety, addressing a question that has
come up a few times.Rust's standard library has low-level types, RawFd on Unix-like platforms,
and RawHandle/RawSocket on Windows, which represent raw OS resource
handles. These don't provide any behavior on their own, and just represent
identifiers which can be passed to low-level OS APIs.
These raw handles can be thought of as raw pointers, with similar hazards.
While it's safe to obtain a raw pointer, dereferencing a raw pointer could
invoke undefined behavior if it isn't a valid pointer or if it outlives the
lifetime of the memory it points to. Similarly, it's safe to obtain a raw
handle, via AsRawFd::as_raw_fd and similar, but using it to do I/O could
lead to corrupted output, lost or leaked input data, or violated encapsulation
boundaries, if it isn't a valid handle or it's used after the close of its
resource. And in both cases, the effects can be non-local, affecting otherwise
unrelated parts of a program. Protection from raw pointer hazards is called
memory safety, so protection from raw handle hazards is called I/O safety.
Rust's standard library also has high-level types such as File and
TcpStream which are wrappers around these raw handles, providing high-level
interfaces to OS APIs.
These high-level types also implement the traits FromRawFd on Unix-like
platforms, and FromRawHandle/FromRawSocket on Windows, which provide
functions which wrap a low-level value to produce a high-level value. These
functions are unsafe, since they're unable to guarantee I/O safety. The type
system doesn't constrain the handles passed in:
use std::fs::File;
use std::os::unix::io::FromRawFd;
// Create a file.
let file = File::open("data.txt")?;
// Construct a `File` from an arbitrary integer value. This type checks,
// however 7 may not identify a live resource at runtime, or it may
// accidentally alias encapsulated raw handles elsewhere in the program. An
// `unsafe` block acknowledges that it's the caller's responsibility to
// avoid these hazards.
let forged = unsafe { File::from_raw_fd(7) };
// Obtain a copy of `file`'s inner raw handle.
let raw_fd = file.as_raw_fd();
// Close `file`.
drop(file);
// Open some unrelated file.
let another = File::open("another.txt")?;
// Further uses of `raw_fd`, which was `file`'s inner raw handle, would be
// outside the lifetime the OS associated with it. This could lead to it
// accidentally aliasing other otherwise encapsulated `File` instances,
// such as `another`. Consequently, an `unsafe` block acknowledges that
// it's the caller's responsibility to avoid these hazards.
let dangling = unsafe { File::from_raw_fd(raw_fd) };
Callers must ensure that the value passed into from_raw_fd is explicitly
returned from the OS, and that from_raw_fd's return value won't outlive the
lifetime the OS associates with the handle.
I/O safety is new as an explicit concept, but it reflects common practices.
Rust's std will require no changes to stable interfaces, beyond the
introduction of some new types and traits and new impls for them. Initially,
not all of the Rust ecosystem will support I/O safety though; adoption will
be gradual.
OwnedFd and BorrowedFd<'fd>These two types are conceptual replacements for RawFd, and represent owned
and borrowed handle values. OwnedFd owns a file descriptor, including closing
it when it's dropped. BorrowedFd's lifetime parameter says for how long
access to this file descriptor has been borrowed. These types enforce all of
their I/O safety invariants automatically.
For Windows, similar types, but in Handle and Socket forms.
These types play a role for I/O which is analogous to what existing types in Rust play for memory:
| Type | Analogous to |
|---|---|
OwnedFd | Box<_> |
BorrowedFd<'a> | &'a _ |
RawFd | *const _ |
One difference is that I/O types don't make a distinction between mutable and immutable. OS resources can be shared in a variety of ways outside of Rust's control, so I/O can be thought of as using interior mutability.
AsFd, Into<OwnedFd>, and From<OwnedFd>These three are conceptual replacements for AsRawFd::as_raw_fd,
IntoRawFd::into_raw_fd, and FromRawFd::from_raw_fd, respectively,
for most use cases. They work in terms of OwnedFd and BorrowedFd, so
they automatically enforce their I/O safety invariants.
Using these, the do_some_io example in the motivation can avoid the
original problems. Since AsFd is only implemented for types which properly
own or borrow their file descriptors, this version of do_some_io doesn't
have to worry about being passed bogus or dangling file descriptors:
pub fn do_some_io<FD: AsFd>(input: &FD) -> io::Result<()> {
some_syscall(input.as_fd())
}
For Windows, similar traits, but in Handle and Socket forms.
I/O safety and the new types and traits wouldn't need to be adopted immediately; adoption could be gradual:
First, std adds the new types and traits with impls for all the relevant
std types. This is a backwards-compatible change.
After that, crates could begin to use the new types and implement the new traits for their own types. These changes would be small and semver-compatible, without special coordination.
Once the standard library and enough popular crates implement the new traits, crates could start to switch to using the new traits as bounds when accepting generic arguments, at their own pace. These would be semver-incompatible changes, though most users of APIs switching to these new traits wouldn't need any changes.
In addition to the Rust language's memory safety, Rust's standard library also
guarantees I/O safety. An I/O operation is valid if the raw handles
(RawFd, RawHandle, and RawSocket) it operates on are values
explicitly returned from the OS, and the operation occurs within the lifetime
the OS associates with them. Rust code has I/O safety if it's not possible
for that code to cause invalid I/O operations.
While some OS's document their file descriptor allocation algorithms, a handle value predicted with knowledge of these algorithms isn't considered "explicitly returned from the OS".
Functions accepting arbitrary raw I/O handle values (RawFd, RawHandle,
or RawSocket) should be unsafe if they can lead to any I/O being
performed on those handles through safe APIs.
OwnedFd and BorrowedFd<'fd>OwnedFd and BorrowedFd are both repr(transparent) with a RawFd value
on the inside, and both can use niche optimizations so that Option<OwnedFd>
and Option<BorrowedFd<'_>> are the same size, and can be used in FFI
declarations for functions like open, read, write, close, and so on.
When used this way, they ensure I/O safety all the way out to the FFI boundary.
These types also implement the existing AsRawFd, IntoRawFd, and FromRawFd
traits, so they can interoperate with existing code that works with RawFd
types.
AsFd, Into<OwnedFd>, and From<OwnedFd>These types provide as_fd, into, and from functions similar to
AsRawFd::as_raw_fd, IntoRawFd::into_raw_fd, and FromRawFd::from_raw_fd,
respectively.
All of the above is prototyped here:
https://github.com/sunfishcode/io-lifetimes
The README.md has links to documentation, examples, and a survey of existing crates providing similar features.
Crates with APIs that use file descriptors, such as nix and mio, would
need to migrate to types implementing AsFd, or change such functions to be
unsafe.
Crates using AsRawFd or IntoRawFd to accept "any file-like type" or "any
socket-like type", such as socket2's SockRef::from, would need to
either switch to AsFd or Into<OwnedFd>, or make these functions unsafe.
Rust historically drew a line in the sand, stating that unsafe would only
be for memory safety. A famous example is std::mem::forget, which was
once unsafe, and was changed to safe. The conclusion stating that unsafe
only be for memory safety observed that unsafe should not be for “footguns”
or for being “a general deterrent for "should be avoided" APIs”.
Memory safety is elevated above other programming hazards because it isn't just about avoiding unintended behavior, but about avoiding situations where it's impossible to bound the set of things that a piece of code might do.
I/O safety is also in this category, for two reasons.
I/O safety errors can lead to memory safety errors in the presence of
safe wrappers around mmap (on platforms with OS-specific APIs allowing
them to otherwise be safe).
I/O safety errors can also mean that a piece of code can read, write, or delete data used by other parts of the program, without naming them or being given a reference to them. It becomes impossible to bound the set of things a crate can do without knowing the implementation details of all other crates linked into the program.
Raw handles are much like raw pointers into a separate address space; they can dangle or be computed in bogus ways. I/O safety is similar to memory safety; both prevent spooky-action-at-a-distance, and in both, ownership is the main foundation for robust abstractions, so it's natural to use similar safety concepts.
The main alternative would be to say that raw handles are plain data, with no concept of I/O safety and no inherent relationship to OS resource lifetimes. On Unix-like platforms at least, this wouldn't ever lead to memory unsafety or undefined behavior.
However, most Rust code doesn't interact with raw handles directly. This is a good thing, independently of this RFC, because resources ultimately do have lifetimes, so most Rust code will always be better off using higher-level types which manage these lifetimes automatically and which provide better ergonomics in many other respects. As such, the plain-data approach would at best make raw handles marginally more ergonomic for relatively uncommon use cases. This would be a small benefit, and may even be a downside, if it ends up encouraging people to write code that works with raw handles when they don't need to.
The plain-data approach also wouldn't need any code changes in any crates. The
I/O safety approach will require changes to Rust code in crates such as
socket2, nix, and mio which have APIs involving AsRawFd and
RawFd, though the changes can be made gradually across the ecosystem rather
than all at once.
IoSafe trait (and OwnsRaw before it)Earlier versions of this RFC proposed an IoSafe trait, which was meant as a
minimally intrusive fix. Feedback from the RFC process led to the development
of a new set of types and traits. This has a much larger API surface area,
which will take more work to design and review. And it and will require more
extensive changes in the crates ecosystem over time. However, early indications
are that the new types and traits are easier to understand, and easier and
safer to use, and so are a better foundation for the long term.
Earlier versions of IoSafe were called OwnsRaw. It was difficult to find a
name for this trait which described exactly what it does, and arguably this is
one of the signs that it wasn't the right trait.
Most memory-safe programming languages have safe abstractions around raw
handles. Most often, they simply avoid exposing the raw handles altogether,
such as in C#, Java, and others. Making it unsafe to perform I/O through
a given raw handle would let safe Rust have the same guarantees as those
effectively provided by such languages.
There are several crates on crates.io providing owning and borrowing file descriptor wrappers. The io-lifetimes README.md's Prior Art section describes these and details how io-lifetimes' similarities and differences with these existing crates in detail. At a high level, these existing crates share the same basic concepts that io-lifetimes uses. All are built around Rust's lifetime and ownership concepts, and confirm that these concepts are a good fit for this problem.
Android has special APIs for detecting improper closes; see
rust-lang/rust#74860 for details. The motivation for these APIs also applies
to I/O safety here. Android's special APIs use dynamic checks, which enable
them to enforce rules across source language boundaries. The I/O safety
types and traits proposed here are only aiming to enforce rules within Rust
code, so they're able to use Rust's type system to enforce rules at
compile time rather than run time.
This RFC doesn't define a formal model for raw handle ownership and lifetimes. The rules for raw handles in this RFC are vague about their identity. What does it mean for a resource lifetime to be associated with a handle if the handle is just an integer type? Do all integer types with the same value share that association?
The Rust reference defines undefined behavior for memory in terms of LLVM's pointer aliasing rules; I/O could conceivably need a similar concept of handle aliasing rules. This doesn't seem necessary for present practical needs, but it could be explored in the future.
Some possible future ideas that could build on this RFC include:
Clippy lints warning about common I/O-unsafe patterns.
A formal model of ownership for raw handles. One could even imagine extending Miri to catch "use after close" and "use of bogus computed handle" bugs.
A fine-grained capability-based security model for Rust, built on the fact that, with this new guarantee, the high-level wrappers around raw handles are unforgeable in safe Rust.
There are a few convenience features which can be implemented for types
that implement AsFd, Into<OwnedFd>, and/or From<OwnedFd>:
from_into_fd function which takes a Into<OwnedFd> and converts it
into a From<OwnedFd>, allowing users to perform this common sequence
in a single step.as_filelike_view::<T>() function returns a View, which contains a
temporary instance of T constructed from the contained file descriptor,
allowing users to "view" a raw file descriptor as a File, TcpStream,
and so on.Portability for simple use cases. Portability in this space isn't easy,
since Windows has two different handle types while Unix has one. However,
some use cases can treat AsFd and AsHandle similarly, while some other
uses can treat AsFd and AsSocket similarly. In these two cases, trivial
Filelike and Socketlike abstractions could allow code which works in
this way to be generic over Unix and Windows.
Similar portability abstractions could apply to From<OwnedFd> and
Into<OwnedFd>.
Thanks to Ralf Jung (@RalfJung) for leading me to my current understanding of this topic, for encouraging and reviewing drafts of this RFC, and for patiently answering my many questions!