Cargo's next few years

The Cargo team have been thinking about and discussing long-term plans for Cargo. In this post I'll talk about what we hope Cargo will look like around the time of the next edition (assuming there is another edition and that it happens in about three years, neither of which is confirmed). There will be another post soon on more concrete plans for this year, including some kind of roadmap.

Cargo makes 'systems' programming accessible by eliminating barriers around building, running, and distributing Rust code. That at least is what Cargo is to me. But that is kinda underselling it; Cargo is a force multiplier for a project: it lowers barriers to contribution, meaning more people can get involved more easily, and it makes it easy to try out software in a consistent manner. For example, if you've heard of this cool project ripgrep (which is like grep, but faster and better) and you want to try it out, then you just run cargo install ripgrep and you're ready to go. And if you think it is great and you want to contribute, then after git clone, all you have to know is to run cargo build or cargo test to get started.

Note that I haven't talked about build systems or about package management. These are things which Cargo does, but they are not the problem that is most important to solve.

Cargo in 2022

Cargo is a great tool. Over the next few years I don't want to prioritise doing more things or doing things differently. Instead I want to focus on doing the same things for more people. I want us to bring the Cargo magic to people who use Rust within a larger project with its own build system, or cross-compile (such as to WASM). I want Cargo to work where a project has resources which need to be provided or processed, and so forth.

I won't pretend that is a complete vision for the next three years. I think it's a starting point and that over the next year we'll work to make it more concrete, in particular working out what it means to "bring the Cargo magic" to these new scenarios.

The edition cycle

We just released the 2018 edition (well, 8 weeks ago now) and the next edition will probably happen in about three years time. The 2018 edition was awesome, but it was hard work and there was a lot of change. Some people have been calling for less change or a slower pace of change in its wake.

I think that a good rhythm is to focus on experimentation and paying down technical and organisational debt in the first year, focus on design and implementation in the second year, and focus on consolidation, 'finishing things', and deprecating/removing things in the third year.

It's also worth noting that Cargo did not change much for the edition, so I think there is more appetite for development and experimentation here than in other areas.

How we get there

We need to focus our work. I think we should focus on one group of users at a time with a small, coordinated set of features. We should make sure that they integrate well with the rest of Cargo, and that the end result will not be a bloated mess. As mentioned above, we should also think about timing, especially with respect to the edition cycle.

I propose we focus on the following domains:

  • 2019: cross-compilation, including wasm and embeddded,
  • 2020: embedding Cargo inside build systems and IDEs,
  • 2021: customising end-user workflow.

Let's dig-in to what each of those mean, and why I've picked this timing

2019: cross-compilation, including wasm and embeddded

Building for a target system which is not the same as your host is a common systems-programming workflow. Applications include mobile development, embedded systems, multi-platform development, and with the arrival of WASM, the web. Embedded and WASM are key domains for Rust and with the domain working groups have seen lots of interest. By targeting cross-compilation we can bring many existing and potential users into a standard Cargo workflow.

As well as broad impact, I think we should focus on cross-compilation this year because:

  • tools such as Xargo, WASM Pack, and Rustup already exist, so there are fewer open questions,
  • the work dovetails with several other desirable Cargo features, such as distributing and using binaries,
  • WASM is a huge and time-critical opportunity for Rust,
  • there are several related goals for the next few years which do require some research and experimentation and are related to some of the features which support cross-complilation (e.g., general binary distribution, custom workflows). I think it is a good idea to get this work done now to support that later work.

The goal is that by the end of this year, cross-compilation should be no more complicated than regular compilation. I don't want to specify exactly what that means, but roughly you should be able to clone a Git repo and cargo build for a target such as WASM. Note that I expect there to be some setup involved for the user, such as installing a plugin, but that should be easy.

I believe the main risk with this goal is that it is too large to complete in one year and/or it grows to encompass too many new features. To mitigate that risk we should keep a close eye on feature creep, and be willing to re-scope the work mid year to something small and tightly defined, for example only supporting std-aware builds.

2020: embedding Cargo inside build systems and IDEs

Cargo works really well for pure Rust projects and projects with limited amounts of other code (which can be handled by a small build script). However, when a Rust component is embedded in a larger project and that project uses a different build system, Cargo cannot offer so much. As Rust grows, these use cases will become more common and more important; supporting such projects is essential for Rust's continued growth.

If you squint, then an IDE is another kind of build system, and here too Cargo could provide a better experience (large parts of Cargo are emulated in the RLS, for example). If we continue towards a highly incremental compiler to improve the RLS, then better Cargo integration will be even more important (it's possible we'll change path and use something like Rust Analyzer or Lark, in which case we'll re-evaluate this goal next year).

In 2020 I think we should focus on improving how Cargo works inside build systems and IDEs. I think this is hugely important work, but there are still many open questions around the architecture and design of such integration. So I think that this year we should spend some time refining requirements and considering architectural issues so that we can make fast progress on design and implementation in 2020. We will have time after 2020 to iterate, consolidate, and stabilise before the next edition.

The main risk here is not being able to adequately provide a solution, or at least without unpalatable changes to Cargo. There is also a risk that while we wait to focus on this area, sub-optimal, ad hoc solutions will start to become standard (on the other hand, this is sometimes how great solutions emerge).

2021: customising end-user workflow

Cargo provides a few essential commands: build, run, test, install, publish, etc. However, many users want additional functionality, for example to upgrade dependencies to the latest version, or to run a test inside a debugger, etc. For other users, the existing commands are not flexible enough - for example, many projects want to be able to install resources as part of cargo install or generate custom files as part of cargo new. I'm lumping these things (and the desire for more scripting, such as post-build scripts) together as end-user workflow issues. These are diverse problems and will likely have diverse solutions; the high-level similarity is making common actions more consistent and automatic between projects.

I propose that we focus on plugins this year (see below) to immediately address the need for extra functionality, and to bring improvements to large numbers of users as soon as possible. In 2021 we will consider which workflows are essential to all users and consider moving them into Cargo, we can also apply lessons learnt from plugin development. That is a fairly light load for the edition year, which should give us space for edition-related changes and overflow from the prior two years.

The risk to this approach is that we don't make plugins feel 'first class', and users are dissatisfied about not having some of the key workflows built-in to Cargo. There is also a risk that requiring plugins for some projects causes the ecosystem to fragment and we lose the magic of just being able to run cargo build everywhere.

2019 again: plugins

Cargo plugins are awesome! For example, you can type cargo install tree, and cargo tree Just Works! (Well, usually, sometimes there are issues due to API changes in Cargo, which I hope to address). However, they are not well-known. There is not a lot of effort to make users aware of them. Writing plugins is also sub-optimal: although it is easy to make a plugin (just name your binary cargo-foo) there is very little support in the form of libraries or APIs.

I propose improving the situation in two ways: make plugins more discoverable and even easier to install, and make plugins easier to write. The second goal will primarily be driven by extracting crates from Cargo so that functionality is shared between plugins and Cargo itself. Some examples of this are stabilising Cargo metadata and providing a library to read and process it, providing a toml parser optimised for Cargo.toml and Cargo.lock, and verifying crates for publication.

The end goal should be that there is little difference between developing a new Cargo command as a plugin or in-tree. In the limit, even cargo build should be easy to implement as a plugin.

In the process of doing this I hope we can find ways to improve Cargo's architecture and discover good APIs which can be used by tools and build systems, as well as plugins.

Other risks and challenges

Whilst I think focusing on the above work is the right thing to do, there are risks to being over-focused. We need to keep an eye on technical and organsiational debt (e.g., lack of design, incoherent sets of commands and arguments, poor documentation, not on-boarding and up-skilling contributors), and we must make sure that we are meeting the needs of our users, especially if requirements shift over time (we should not be afraid to radically change the plans laid out here if necessary).

Other stuff

It's worth mentioning that there is quite a lot of other stuff that should be planned for Cargo, including: features in service of other goals for the Rust project (e.g., reducing compile times), paying down technical debt, polishing existing features, stabilisations, and improving documentation.