More on the RLS and a 1.0 release

In my last post, I announced a release candidate for the RLS 1.0. There has been a lot of feedback (and quite a lot of that was negative on the general idea), so I wanted to expand on what 1.0 means for the RLS, and why I think it is ready. I also want to share some of my vision for the future of the RLS, in particular changes that might warrant a major version release.

The general thrust of a lot of the feedback was that the RLS is not ready for a 1.0 release. Where a specific reason was given it was usually regarding code completion. I'll drill deeper in a minute. There were also a few comments along the lines of the RLS architecture being wrong. I'm not going to argue against that one, certainly if we started over I would do some things differently (hindsight is great). However, I don't think that at this point changing the architecture (which basically means starting again from scratch) is going to get us to an excellent product any quicker than continuing with the current architecture. I'm also confident that the current design can evolve over time.

Some clarifications

What does 1.0 mean for the RLS?

There is no backwards compatibility guarantee for the RLS, and no obvious quality cut-off. This is what I meant by "arbitrary" in my last post.

The way I have been thinking of 1.0 is as a signal that the RLS is worth using for a majority of users. It is that signal which is much more important than the 1.0 label. The Rust roadmap has 'a high quality IDE experience' as a key goal, and that is what I want to deliver, regardless of the version number. In terms of "ticking things off on a Powerpoint presentation", it is the 'high quality IDE experience' we're going for, not the 1.0 label.

I believe that for the RLS, quality can be measured along four axes: stability - how often does the RLS crash, fail to respond, or give incorrect results; completeness - how many cases are covered (e.g., code completion not working for some types, type info missing for macro-generated identifiers, etc.); features - what can do the RLS do (e.g., code completion, refactoring); and performance.

I have mainly been thinking in terms of stability for 1.0 - it is essential that for nearly all users, crashes and incorrect results are rare. I believe that we have achieved that (or that we can achieve that by the end of the 'release candidate' phase). Most of the negative feedback around the RC announcement focused on completeness. Features and performance can always be improved; my opinion is that there is minimum core that is required for 1.0 and we have that, though of course there is plenty of room for improvement.

Code completion

Code completion is powered by Racer. Long-term we'd like to drive code completion with the compiler, but that is a long way off. Racer needs to be able to determine the type of the expression with fairly limited input. Given the type, it can be pretty smart about finding implemented traits and providing methods. However, this is not perfect.

There are two ways that code completion is incomplete: we can't give any suggestions where we can't find the type, and even if we can find the type, then sometimes we do not suggest all the things that we should. One could consider incomplete suggestions for a type to be incorrect as well as incomplete; I think that depends on how suggestions are used. Although I haven't observed it, I believe Racer can rarely make incorrect suggestions.

What can be done?

The RLS has three kinds of information source: the compiler, Racer, and external tools (Rustfmt, Clippy, etc.). The latter are not really relevant here. The compiler processes a program semantically, i.e., it knows what the program means. It takes a long time to do that. Racer processes a program syntactically, it only knows what a program looks like. It runs quickly. We use the compiler for 'goto def', showing types and documentation, etc. We use Racer mostly for code completion.

There are some fundamental limitations. The compiler has to be able to build your program, which prevents it working with individual files and some very complex systems like the Rust project. For large programs, it is only feasible to use it 'on save', rather than 'live' as the user types. We can make this better, but it is a very long-term process. In particular, getting the compiler fast enough to do code completion will take at least a year, probably more.

Racer cannot understand every expression in a program. We can't hope for Racer to have perfect completion for all expressions.

Fully complete (i.e., compiler-powered) code completion will take years. Even when completion is powered by the compiler, there will be work to do to understand macros and so forth. Excluding the most complex cases, we're still looking at more than a year for complete code completion.

On the compiler side, there are a few things which could improve things, but none are trivial. My opinion is that here we have a 'good enough' base and we can incrementally improve that.

One new idea is to pass data from the compiler to Racer, to improve Racer's completeness. This is a promising angle to investigate, and might get us some big wins to the completeness of code completion.

What should we do?

Rolling back the 1.0 RC is definitely an option at this stage. I do believe that although the RLS is lacking in completeness, for most users, the cost/benefit ratio is favourable and therefore it should be widely recommended (if you use the RLS and don't agree, I'd be keen to hear why). One question is how to signal that.

For how to handle the RC and 1.0 release, some options I see (I'd be happy to hear more ideas):

  • Rollback the 1.0 RC. Postpone 1.0 until we have compiler-supported code completion. Indicate stability by promoting the RLS, but avoid a 1.0 release. Estimated 1.0 release: early 2020.
  • Pause the 1.0 RC. Address some of the lower-hanging fruit (e.g., compiler support for path prefixes: the foo in foo::bar). RC again when ready. Estimated 1.0 release: mid 2019.
  • Keep the 1.0 RC, but don't release an actual 1.0 until some of the above things are implemented (i.e., have a long RC period). Estimated 1.0 release: mid 2019 or later.
  • Go ahead with the current plan on the understanding that 2.0 and further major version increments are on the way, and the 1.0 label applies to stability rather than completeness. DO a better job of setting expectations. Estimated 1.0 release: late 2018.

When thinking about the quality requirement for a 1.0 it is good to consider some other examples. Rust itself set a pretty high bar for it's 1.0 release. However, there was plenty which was missing, incomplete, or imperfect; the focus was on backwards compatibility. Other IDEs have been around a long time and have seen a lot of improvement (e.g., IntelliJ was first released in 2001), I would avoid comparing VSCode and the RLS to IntelliJ or similar tools for that reason.

In the wider Rust ecosystem, there are a lot of crates which have yet to announce a 1.0 release. However, the feeling among the core and library teams is that the community is often over-conservative here; that the ecosystem would be better served by announcing 1.0 releases and not being so keen to avoid a major version increment (in particular, the current situation can give the impression that Rust is 'not ready').

The future

There are a lot of things that could be added or improved in the RLS. I'll try and list some of the more significant things here, in particular things that might warrant a major version increment.

For each work item, I've indicated the priority (further discussion about priorities in particular is needed), status, and sometimes an expected version. A status of "-" means that it could be started straight away, but hasn't been started yet. A status of "planning" means that we need to work out how the feature will be implemented. A version number indicates that the feature should block the given version (assuming 1.0 RC becomes 1.0).

Enhancing existing features

  • working with other build systems (p-med, WIP)
  • symbol search improvements (p-med, planning)

Implementation improvements

  • enhancing Racer with type info (p-high, planning)
  • better testing of the RLS (p-high, -)
  • help for users to report errors (p-high, -)
  • more complete type information (e.g., path prefixes, macros) (p-high/-med, -)
  • share code with Cargo (p-med, planning)
  • incremental save-analysis update? (p-?, need to evaluate cost/benefit)
  • 'online' interaction with the compiler via queries (p-med, blocked on compiler work, 3.0)
  • compiler-driven completion (p-high, blocked on compiler work and online interaction, 3.0)

New features

  • Cargo.toml support (e.g., updating crates, checking for more recent versions) (p-high, prototypes exist)
  • LSP feature work (p-med, -)
  • debugging support (p-med, -, 2.0)
  • full auto-imports (i.e., without having to use a suggestion) (p-med, planning, 2.0)
  • refactoring support (p-low, planning, 2.0)

What do you think?

Tell me!