Rust 2020

I was thinking of not writing a blog post this year. I didn't think I'd have much to say. But I thought that I should, I had some nagging feelings that we needed to do some things differently. Once I started writing, I found plenty to talk about. This blog post is long, sorry (in the words of Mark Twain: "if I had more time, I would write a shorter blog post").

I'll start by arguing in favour of an edition in 2021. Then I will indulge in some rumination on the nature and necessity of roadmaps. Finally I'll make some probably obvious points in a long-winded way. Tl;dr, money is a thing we need to think about, async programming and WASM are important, and the compiler should be faster. For good measure, I will re-open a can of worms concerning the standard library.

I suggest you get a drink.

A 2021 edition?

The 2018 edition made a lot of changes and was hard work for a lot of people. There is a (reasonable) worry that we don't have the capacity to launch another edition, and/or that we don't have enough changes to make it worthwhile.

My hope is that a 2021 edition will be much easier than in 2018: we've done it before so we have experience, we have two years rather than one to prepare, we have fewer breaking changes in mind, and much less lofty goals.

But should we have a 2021 edition? I think we should. I don't think we should wait until we have a huge number of breaking changes, even if there are none, then there are benefits to having a version as a rallying point. Furthermore, if we wait until we do have a lot of breaking changes then it is too late - we won't have enough time to make the edition without another whole load of stress. I'm also confident that given the opportunity, we'll be able to come up with a bunch of breaking changes next year :-)

But my main reason for wanting a 2021 edition is async/await. I think the impact of async/await is going to be HUGE. I've been using it recently and it despite not being polished it is incredibly powerful, and it really feels like a different language - there is a different mental model, different libraries, different ways of debugging, etc. There is an opportunity that a polished async/await implementation and supporting ecosystem of libraries and tools could cause a revolution in high-performance server programming.

By the time the dust has settled on language changes, we make the extended changes to things like the futures library, and the ecosystem catches up with all that, we'll be close to edition time.

In other words, even if there is no other reason, I believe that async/await is reason enough by itself to make a 2021 edition worthwhile.

Getting to the edition

If we are going to have an edition launch towards the end of 2021, what should 2020 and 2021 look like? I think that the end of 2020 should be a hard deadline for breaking changes for the 2021 edition (and for planning the scope of the edition in other ways). That gives us most of 2021 to work on bug fixing, tooling, documentation, marketing, etc.

I don't think we should think about orientating 2020 around edition changes, however. We should treat editions like releases - there will be another one soon enough. Let's do what is necessary in 2020, and at the end of the year plan and scope the 2021 edition based on what we have.

Should we even have a roadmap?

There are some benefits in having roadmaps - it is good to think about the big picture and assess where we are relative to where we wanted to be. It is a great way to get community feedback and involvement on the project's goals. We must have some way to focus our decision making, especially in the RFC process.

On the other hand, roadmaps are overly prescriptive, anti-iteration, and (even if constructed with team's input) take autonomy away from teams. Roadmaps try to describe the output of work, not the outcome. To give an example, we might care that compile times are too slow, and we should include this in our 'roadmap', but we should not care whether that comes from micro-optimisations or incremental compilation, that should be up to the compiler team. Putting that implementation detail in the roadmap only restricts them.

Intrinsically, a roadmap is about deadlines - these are the things that we want to get done this year. However, estimating the duration of software engineering tasks is famously difficult; even more so in an OSS project where most people are contributing their time voluntarily.

We don't really know what will work and what won't in advance. In software engineering iterative is usually best, but a detailed roadmap discourages iteration in favour of up-front planning. Worse, it makes it harder to change course during the year if that is necessary.

Instead of a roadmap, I propose that we take input from the community's blog posts, then derive a vision for where we want Rust to be in approximately one year. That might sound like what we do at the moment, but the difference is that we describe where we want to be, not how we get there. Furthermore, we explicitly acknowledge that when we get there is basically unknown, some things will happen sooner, and some much later.

With that in mind, the rest of this blog post proposes what I think are some important parts of a vision for Rust at roughly the end of 2020.

2020

It might be my change of viewpoint, but I believe the Rust project is in a categorically different place from where it was a year ago. The difference feels greater than pre- and post-1.0. We've had a step-change in maturity and we're solving very different problems from last year. Looking back at last year's blog posts and especially the year before, I feel that priorities for the project have changed a huge amount.

The changes we are making are smaller and less urgent. When I look at the current RFC proposals, I don't see any that I think need to be done right now or which will significantly change Rust. In fact, the most interesting ones to me are are best described as 'polishing'.

To be clear, I don't think this difference is bad, it reflects the popularity and maturity of Rust. I think the language, the standard library, and core tools are in a really good place. And at this stage in life, stability is a nice thing to have (although of course we must watch out for stagnation).

Secondly, we've had a lot of leadership changes - key people leaving or changing role, people moving in or out of full-time commitment to Rust, new people, and people in new roles.

I think these two differences coming at a similar time make me feel like we're dealing with something new.

So in 2020, I think we need to take a step back, reflect, reset, and refocus. Let's find what is important and double-down on it. Let's make sure our organisation allows us to perform in the ways we need to, and let's make sure we are doing the right things for the long-term.

Leadership, governance, and money

We should look closely at our organisation and governance structures and think about if we're doing the right things. I believe we may need some changes to better serve the project.

One of the big challenges over the past year, and I think more so going forward, is people doing the grunt work of leadership. Beyond writing fancy blog posts, there is a tonne of work that needs doing all over the project which is essentially management (of one kind or another). It is always difficult to get this work done in open source and on a volunteer basis. The tasks which need work are not as glamorous or exciting as they used to be.

At the same time, Rust is being used more and more by serious companies with very deep pockets. Rust is bringing these companies (and many others now and in the future) millions of dollars in value.

In the past we have preferred to matchmake - point sponsors to people in the community doing essential work so they can work together - rather than be directly involved with money. I think that has worked in some cases, mostly with clearly defined engineering projects. But I don't think it is a sustainable model for the project in the future.

So in 2020 we should think seriously about money. We have shied away from this in the past because it is really, really hard. The legal issues, the administration, the kind of work that needs to be done in talking to potential funders. It's all really hard and the kind of thing most in the project don't have experience with. But I think we must do it. We need to be able to pay people proper money to work on boring stuff in Rust over the long-term. We can't rely on the goodwill of a few companies and short-term contracts to do that.

This will take time and work. I expect it will occupy the core team and others almost entirely in 2020. But I believe we must do that work at some point, and sooner or later it will be too late to do it. If we can contract a person for a year to lead this (rather than to work on engineering problems), I think it would be the best investment we could make.

Refocus

It used to be easy to find goals to rally behind. That is not so true any more (in the core project and in the wider ecosystem). Most work is incremental and long-term. Writing X in Rust is less revolutionary than it used to be. There is much less scope for implementing a foundational crate. At the same time, many more people are using Rust to do things, some exciting, some mundane. There are far more projects than the core project can keep track of, let alone actively support.

I propose that for 2020 and beyond we refocus in a few ways:

  • We cut back on direct support for non-core projects. Let's focus on a couple of core domains, rather than trying to push into many. I think the working group concept has been useful, but now it is better if such groups are self-organising and community based. The best way for the core project to help in 2020 is to mentor leaders to run their own WGs, rather than running WGs or setting up new WGs ourselves. We should help some of the existing WGs to exist independently.
  • We don't start any new, major initiatives in any areas, instead focussing on maintaining and incrementally improving what we currently have.
  • We should do less work to actively promote Rust and directly support companies who want to use Rust. Instead, we should make sure resources are available to help people evangelise Rust in their workplaces.
  • We focus on promoting Rust in domains where we are uniquely positioned to succeed (Rust is a fantastic language for X, rather than Rust is a safe and fast language).

I don't think Rust is unsuited to a wide range of domains; I think there are lots of places where it is the best language to use. I hope that the community and potential users will continue to make Rust a well-supported language. It is because Rust has an amazing community, a good reputation, and a great value proposition that I believe the core project can focus on the niches where we are most likely to succeed without losing steam in other areas.

But which areas to focus on? We need to pick areas where Rust has a clear advantage: that means that both speed (ideally, latency) and safety are paramount. We also need to pick areas which are not burdened with the 'network effects' of large legacy code bases or change-averse communities (e.g., OS dev, game dev). I think we should focus on areas where we are already somewhat established, rather than areas where we are still working on getting users.

The two domains I think best fit this description are server-side programming (async, networking, etc.) and WASM. In both domains we could become the go-to language.

Async and network programming

As I said above, I think async programming is going to be both a huge change and opportunity for Rust. Async/await is almost in a stable release, what needs to be done? As the saying goes, we've done the first 90% and now we only have the last 90% to go. There is a whole ecosystem of libraries and tools to implement, things will need changing in the language design and the compiler, we're going to need a lot more documentation, and so forth. I think this is going to be a lot of work and touch every part of Rust. But if we do it right, it could be hugely positive.

Along with async programming in general, I think we need to look at our networking landscape too. We've put off a lot of the work here, waiting for futures and async/await to be done. We're still not done, but I think we're close enough that we need to actually do that work now. There is a lot of promising work in this area and the community should iterate and improve, as well as try some new things.

WASM

WASM is a good match for Rust and Rust seems to be doing well in the space. I think it makes sense to continue to push Rust here. I'm afraid I know nothing of the details.

Essential libraries

One question we've wrestled with since before 1.0 is how 'batteries included' our standard library should be. We've ended up with a fairly minimal (but high quality) standard library. I think that is a good decision, but it is unsatisfactory in some ways too. There are a bunch of crates which are de facto standardised and which would be part of a 'batteries included' standard library. But these get no input from the core Rust org, can be hard for new users to discover, and contribute to our long compile times.

I think we should accept that it is time for some of these libraries to be moved or adapted into std (lazy static is a great example). There is a second group of crates (e.g., regex, Serde) that should be better supported. There have been lots of proposals for solutions.

My current favourite is:

  • we have a very high bar for selection, including a 1.0 release; initially perhaps only three or four libraries.
  • We build the most recent, stable version of each library, and distribute that as a Rustup component (part of the default profile, so that all new users get it).
  • Cargo and rustc would treat these libraries like std - i.e., no need to mention in Cargo.toml, just need to use them.
  • Each release would take the most recent version, up to breaking changes which would only happen at an edition.
  • We would never remove crates from this collection.
  • If users explicitly depend on a version via Cargo that overrides the distributed version.
  • We would provide some documentation in the book and would include the crates' Rustdoc at rust-lang.org, not just docs.rs.

The benefits would be better discoverability, and improved initial compile times. It would also extend the range of programs that can be written by beginners before introducing Cargo or crates.io in tutorials. There are probably huge flaws in this plan. I present it as a straw-man proposal to demonstrate that there are things we could do in this space.

The usual suspects

There are some ongoing projects that require continued attention, I think it is important to mention them in the roadmap because they are valuable, and that value should be pointed out explicitly. The following is an incomplete list.

Maintenance: both technical (bug fixing, addressing performance issues, etc.) and non-technical (moderation and keeping the community healthy in other ways).

Compile times: the optimal compile time is 0. We are a long way off that and we must keep pushing to improve. We've focussed on this in the past and we've made some huge gains, but we are still not 'there' yet. This will take sustained effort over years.

Tooling: can always be better and while we are in a good place in many ways, compared to other languages we are lagging.

Procedural macros: these have huge potential and are already carrying weight, but there is some distance to go until the feature is close to complete.

Conclusion

In a few years time I want to see Rust with a thriving community and a reputation for maturity, stability, and being well-run. I want Rust to be the obvious choice for anyone working on high performance code in browsers or servers. Async programming in Rust should be the standard other languages dream of achieving.

I think to get there we must reform our organisational structure and processes, including our year-to-year planning, as soon as possible. In 2020, we need to focus on the domains where we are most likely to succeed, and we need to prioritise keeping the project healthy over new development.