Macros in Rust pt5

I've previously laid out the current state of affairs in Rust (0, 1, 2, 3, 4). In this post I want to highlight what I think are issues with the current system and if/when I think we should fix them. I plan to do some focused design work on this leading to an RFC and to work on the implementation as well. Help with both will be greatly appreciated, let me know if you're interested.

I'd like to gather the community's thoughts on this before getting to the RFC stage (and I'm only really requirements gathering at this stage, not coming up with a plan). To help with that I have opened a thread on internals.rust-lang, please comment there if you have thoughts.

RFC issue 440 describes many of the issues, I hope I have collected all of those which are still relevant here.

high-level motivation

The most pressing issue is the instability of procedural macros. We've seen people do really cool things with them, they are an extremely powerful mechanism, and one of the most requested features for stability. However, there are many doubts about the current system. We would like to have a system of procedural macros we are happy with and move towards stability.

There are many issues with macro_rules macros which we would have liked to address before 1.0, but couldn't due to time constraints and prioritisation. Many of the 'rough edges' would be breaking changes to fix. It is unclear how difficult such changes would be to cope with and if it is worth making such changes. In particular, we can have the best macro system in the world, but if the old one has momentum and no-one uses the new one, then that would not be a good investment.

Finally, the two macro systems interact a lot. We need to make sure that any decisions we make in one area will leave us the freedom to do what we want in the other.

macro_rules issues

Our hygiene story should be more complete. See blog post 3 for details of what is not covered by Rust's hygiene implementation. In particular type and lifetimes variables should be covered, and we should have unsafety and privacy hygiene. One open question here is how items introduced by macro expansion should be scoped.
Modularisation of macros needs work. It would be nice if macros could be named in the same way as other items and importing worked the same way for macros as for other items. This will require some modifications to our name resolution implementation (which is probably a good thing, it is kinda ugly at the moment).
Having to use absolute paths for naming is a drag. It would be better if naming respected the scopes in which macros are defined.
Ordering of macro definitions and uses shouldn't matter, in the same way that it doesn't for other items. E.g.,

foo!();
macro_rules! foo { ... }

should work.

Ordering of uses shouldn't matter, e.g.,

macro_rules! foo {  
    () => {
        baz!();
    }
}
macro_rules! bar {
    () => {
        macro_rules! baz {  
            () => {
                ...
            }
        }
    }
}

foo!();
bar!();

should work.

There are some significant-sounding bugs that should be addressed, or at least assessed. E.g., around parsing macro arguments: 6994, 3232, and 27832

stability

The big question is whether we can adapt the current system with minimal breakage, or whether we need to define a macros 2.0 system and if so, how to do that. I believe that even if we go the macros 2.0 path, it must be close enough to the current system that the majority of macros continue to work unchanged, otherwise changing will be too painful for the new system to be successful. That obviously limits some of the things we can do.

procedural macro issues

The current breakdown of procedural macros into different kinds using traits is a bit ad hoc. We should try to rationalise this breakdown. Hopefully we can merge ItemDecorator and ItemModifier, remove IdentTT, and make MacroRulesTT more of an implementation detail. We should also find a simpler interface than the current mess of traits and data structures.
All kinds of macros should take and produce token trees, rather than ASTs. However, there should be good facilities for working with the token trees as ASTs. Either provided by the compiler or from external libraries.
There should be powerful and easy to use libraries for working with hygiene and spans.
The plugin registry system is not ideal, instead we should allow scoped attributes and support modularisation of procedural macros.
Having to build procedural macros as separate crates is not ideal, it would be nice to be able to have procedural macros defined inline like macro_rules macros. I imagine we would still allow something like the current system, but also support inline macros as a kind of sugar (nesting these inside macro_rules macros will make things fun).

stability

Currently procedural macros have access to all of the compiler. If we were to stabilise macros like this, then we could never change the compiler or language. We need to present APIs to procedural macros that allow the language to evolve whilst the remain stable. Various ways have been suggested for this, such as working only with quasi-quoting, relying on external libraries, some kind of extensible AST API, extensible enums, and a more flexible token tree structure.

other stuff

Scoped attributes were mentioned for procedural macros, it would be nice to allow tools to make use of these too. This could allow more precise interaction with the various attribute lints (unknown attribute, etc.).
Concatenating idents is supported by concat_idents, however, it is not very useful due to hygiene constraints. I believe we can do much better. (See tracking issue). We might also want to allow macros in ident position.
Proper modularisation of macros would require moving some of name resolution to the macro expansion phase. This would be a big change. However, name resolution is due for some serious refactoring and we could move what is left to the AST->HIR lowering step or even later and make it 'on-demand' during type checking. The latter would allow us to use the same code for associated items as for plain names.
We seem to be paying the price for some unification/orthogonality without getting a great deal of benefit. It would be nice to either make macro_rules macros more built-in and/or be able to do more compiler stuff as pure procedural macros (e.g., all the built-in macros, cfg, maybe even macro_rules macros).

out of scope?

There are some things which are touched on here or in the last few blog posts, but which I don't want to think about at the moment. I think most of them are orthogonal to the main focus here (modularisation, hygiene, stabilising procedural macros) and backwards compatible. Things I can think of:

other plugin registry stuff - the plugin registry is pretty ugly, I'd rather it disappeared completely. Eliminating the registration of procedural macros is a step towards that, but I don't want to think about lints, LLVM passes, etc. for now.
Macro uses in more positions (I mentioned ident position above, I'm not sure how important it is to consider that now. I'm not sure if there are other positions we should consider) - we can add more if there is demand, but it doesn't intersect with much else here.
Expansion stacks and other implementation stuff - we can probably do better, but I think it is mostly orthogonal to the high level design.
Tooling - we need better tooling around macros, but it's pretty orthogonal to this work
Compiler as a library.
gensym in macro_rules.
Allow matching of + and * in macro patterns .
External deriving implementations - hopefully this will extend the procdural macros work, but I don't think it affects the design at this stage.
Dealing with repetition in sequences - i.e., generating n of something in a macro, counting the number of repetitions in a pattern.