RFC 2457: non-ascii-idents

lang (syntax | non-ascii-idents)

Summary

Allow non-ASCII letters (such as accented characters, Cyrillic, Greek, Kanji, etc.) in Rust identifiers.

Motivation

Writing code using domain-specific terminology simplifies implementation and discussion as opposed to translating words from the project requirements. When the code is only intended for a limited audience such as with in-house projects or in teaching it can be beneficial to write code in the group's language as it boosts communication and helps people not fluent in English to participate and write Rust code themselves.

The rationale from PEP 3131 nicely explains it:

Python Rust code is written by many people in the world who are not familiar with the English language, or even well-acquainted with the Latin writing system. Such developers often desire to define classes and functions with names in their native languages, rather than having to come up with an (often incorrect) English translation of the concept they want to name. By using identifiers in their native language, code clarity and maintainability of the code among speakers of that language improves.

For some languages, common transliteration systems exist (in particular, for the Latin-based writing systems). For other languages, users have larger difficulties to use Latin to write their native words.

Additionally some math oriented projects may want to use identifiers closely resembling mathematical writing.

Guide-level explanation

Identifiers include variable names, function and trait names and module names. They start with a letter or an underscore and may be followed by more letters, digits and some connecting punctuation.

Examples of valid identifiers are:

Examples of invalid identifiers are:

Composed characters like those used in the word ḱṷṓn can be represented in different ways with Unicode. These different representations are all the same identifier in Rust.

To disallow any Unicode identifiers in a project (for example to ease collaboration or for security reasons) limiting the accepted identifiers to ASCII add this lint to the lib.rs or main.rs file of your project:

#![forbid(non_ascii_idents)]

Some Unicode character look confusingly similar to each other or even identical like the Latin A and the Cyrillic А. The compiler may warn you about names that are easy to confuse with keywords, names from the same crate and imported items. If needed (but not recommended) this warning can be silenced with a #[allow(confusable_idents)] annotation on the enclosing function or module.

Usage notes

All code written in the Rust Language Organization (rustc, tools, std, common crates) will continue to only use ASCII identifiers and the English language.

For open source crates it is suggested to write them in English and use ASCII-only. An exception can be made if the application domain (e.g. math) benefits from Unicode and the target audience (e.g. for a crate interfacing with Russian passports) is comfortable with the used language and characters. Additionally crates should consider to provide an ASCII-only API.

Private projects can use any script and language the developer(s) desire. It is still a good idea (as with any language feature) not to overdo it.

Reference-level explanation

Identifiers in Rust are based on the Unicode® Standard Annex #31 Unicode Identifier and Pattern Syntax.

Note: The supported Unicode version should be stated in the documentation.

The lexer defines identifiers as:

Lexer:
IDENTIFIER_OR_KEYWORD:
   XID_Start XID_Continue*
   | _ XID_Continue*

IDENTIFIER :
IDENTIFIER_OR_KEYWORD Except a [strict] or [reserved] keyword

XID_Start and XID_Continue are used as defined in the aforementioned standard. The definition of identifiers is forward compatible with each successive release of Unicode as only appropriate new characters are added to the classes but none are removed. We effectively are using UAX 31's default definition of valid identifier, with a tailoring that underscores are included with XID_Start. (Note that this allows bare underscores to be identifiers, that is currently also the case with _ in identifier contexts being a reserved keyword)

Rust lexers normalize identifiers to NFC. Every API accepting identifiers as strings (such as proc_macro::Ident::new normalizes them to NFC and APIs returning them as strings (like proc_macro::Ident::to_string) return the normalized form. Procedural and declarative macros receive normalized identifiers in their input as well. This means two identifiers are equal if their NFC forms are equal.

A non_ascii_idents lint is added to the compiler. This lint is allow by default. The lint checks if any identifier in the current context contains a codepoint with a value equal to or greater than 0x80 (outside ASCII range). Not only locally defined identifiers are checked but also those imported from other crates and modules into the current context.

Remaining ASCII-only names

Only ASCII identifiers are allowed within an external block and in the signature of a function declared #[no_mangle]. Otherwise an error is reported.

Note: These functions interface with other programming languages and these may allow different characters or may not apply normalization to identifiers. As this is a niche use-case it is excluded from this RFC. A future RFC may lift the restriction.

This RFC keeps out-of-line modules without a #[path] attribute ASCII-only. The allowed character set for names on crates.io is not changed.

Note: This is to avoid dealing with file systems on different systems right now. A future RFC may allow non-ASCII characters after the file system issues are resolved.

Confusable detection

Rust compilers should detect confusingly similar Unicode identifiers and warn the user about it.

Note: This is not a mandatory for all Rust compilers as it requires considerable implementation effort and is not related to the core function of the compiler. It rather is a tool to detect accidental misspellings and intentional homograph attacks.

A new confusable_idents lint is added to the compiler. The default setting is warn.

Note: The confusable detection is set to warn instead of deny to enable forward compatibility. The list of confusable characters will be extended in the future and programs that were once valid would fail to compile.

The confusable detection algorithm is based on Unicode® Technical Standard #39 Unicode Security Mechanisms Section 4 Confusable Detection. For every distinct identifier X execute the function skeleton(X). If there exist two distinct identifiers X and Y in the same crate where skeleton(X) = skeleton(Y) report it. The compiler uses the same mechanism to check if an identifier is too similar to a keyword.

Note: A fast way to implement this is to compute skeleton for each identifier once and place the result in a hashmap as a key. If one tries to insert a key that already exists check if the two identifiers differ from each other. If so report the two confusable identifiers.

Exotic codepoint detection

A new less_used_codepoints lint is added to the compiler. The default setting is to warn.

The lint is triggered by identifiers that contain a codepoint that is not part of the set of "Allowed" codepoints as described by Unicode® Technical Standard #39 Unicode Security Mechanisms Section 3.1 General Security Profile for Identifiers.

Note: New Unicode versions update the set of allowed codepoints. Additionally the compiler authors may decide to allow more codepoints or warn about those that have been found to cause confusion.

For reference, a list of all the code points allowed by this lint can be found here, with the script group mentioned on the right.

There are some specific interesting code points that we feel necessary to call out here:

Adjustments to the "bad style" lints

Rust RFC 0430 establishes naming conventions for Rust ASCII identifiers. The rustc compiler includes lints to promote these recommendations.

The following names refer to Unicode character categories:

These are the three different naming conventions and how their corresponding lints are specified to accommodate non-ASCII codepoints:

Note: Scripts with upper- and lowercase variants ("bicameral scripts") behave similar to ASCII. Scripts without this distinction ("unicameral scripts") are also usable but all identifiers look the same regardless if they refer to a type, variable or constant. Underscores can be used to separate words in unicameral scripts even in UpperCamelCase contexts.

Mixed script confusables lint

We keep track of the script groups in use in a document using the comparison heuristics in Unicode® Technical Standard #39 Unicode Security Mechanisms Section 5.2 Restriction-Level Detection.

We identify lists of code points which are Allowed by UTS 39 section 3.1 (i.e., code points not already linted by less_used_codepoints) and are "exact" confusables between code points from other Allowed scripts. This is stuff like Cyrillic о (confusable with Latin o), but does not include things like Hebrew ס which is somewhat distinguishable from Latin o. This list of exact confusables can be modified in the future.

We expect most of these to be between Cyrillic-Latin-Greek and some in Ethiopic-Armenian, but a proper review can be done before stabilization. There are also confusable modifiers between many script.

In a code base, if the only code points from a given script group (aside from Latin, Common, and Inherited) are such exact confusables, lint about it with mixed_script_confusables (lint name can be finalized later).

As an implementation note, it may be worth dealing with confusable modifiers via a separate lint check -- if a modifier is from a different (non-Common/Inherited) script group from the thing preceding it. This has some behaviorial differences but should not increase the chance of false positives.

The exception for Latin is made because the standard library is Latin-script. It could potentially be removed since a code base using the standard library (or any Latin-using library) is likely to be using enough of it that there will be non-confusable characters in use. (This is in unresolved questions)

Reusability

The code used for implementing the various lints and checks will be released to crates.io. This includes:

Confusables detection works well when there are other identifiers to compare against, but in some cases there's only one instance of an identifier in the code, and it's compared with user-supplied strings. For example we have crates that use proc macros to expose command line options or REST endpoints. Crates that do things like these can use such algorithms to ensure better error handling; for example if we accidentally end up having an /арр endpoint (in Cyrillic) because of a #[annotation] fn арр(), visiting /app (in Latin) may show a comprehensive error (or pass-through, based on requirements)

Conformance Statement

Drawbacks

Rationale and alternatives

As stated in Motivation allowing Unicode identifiers outside the ASCII range improves Rusts accessibility for developers not working in English. Especially in teaching and when the application domain vocabulary is not in English it can be beneficial to use names from the native language. To facilitate this it is necessary to allow a wide range of Unicode character in identifiers. The proposed implementation based on the Unicode TR31 is already used by other programming languages and is implemented behind the non_ascii_idents in rustc but lacks the NFC normalization proposed.

NFC normalization was chosen over NFKC normalization for the following reasons:

Possible variants:

  1. Require all identifiers to be already in NFC form.
  2. Two identifiers are only equal if their codepoints are equal.
  3. Perform NFKC mapping instead of NFC mapping for identifiers.
  4. Only a number of common scripts could be supported.
  5. A restriction level is specified allowing only a subset of scripts and limit script-mixing within an identifier.

An alternative design would use Immutable Identifiers as done in C++. In this case a list of Unicode codepoints is reserved for syntax (ASCII operators, braces, whitespace) and all other codepoints (including currently unassigned codepoints) are allowed in identifiers. The advantages are that the compiler does not need to know the Unicode character classes XID_Start and XID_Continue for each character and that the set of allowed identifiers never changes. It is disadvantageous that all not explicitly excluded characters at the time of creation can be used in identifiers. This allows developers to create identifiers that can't be recognized as such. It also impedes other uses of Unicode in Rust syntax like custom operators if they were not initially reserved.

It always a possibility to do nothing and limit identifiers to ASCII.

It has been suggested that Unicode identifiers should be opt-in instead of opt-out. The proposal chooses opt-out to benefit the international Rust community. New Rust users should not need to search for the configuration option they may not even know exists. Additionally it simplifies tutorials in other languages as they can omit an annotation in every code snippet.

Confusable detection

The current design was chosen because the algorithm and list of similar characters are already provided by the Unicode Consortium. A different algorithm and list of characters could be created. I am not aware of any other programming language implementing confusable detection. The confusable detection was primarily included because homoglyph attacks are a huge concern for some members of the community.

Instead of offering confusable detection the lint forbid(non_ascii_idents) is sufficient to protect a project written in English from homoglyph attacks. Projects using different languages are probably either written by students, by a small group or inside a regional company. These projects are not threatened as much as large open source projects by homoglyph attacks but still benefit from the easier debugging of typos.

Alternative mixed script lints

These are previously-proposed lints attempting to prevent problems caused by mixing scripts, which were ultimately replaced by the current mixed script confusables lint.

Mixed script detection

A new mixed_script_idents lint would be added to the compiler. The default setting is to warn.

The lint is triggered by identifiers that do not qualify for the "Moderately Restrictive" identifier profile specified in Unicode® Technical Standard #39 Unicode Security Mechanisms Section 5.2 Restriction-Level Detection.

Note: The definition of "Moderately Restrictive" can be changed by future versions of the Unicode standard to reflect changes in the natural languages used or for other reasons.

Global mixed script detection with confusables

As an additional measure, we would try to detect cases where a codebase primarily using a certain script has identifiers from a different script confusable with that script.

During mixed_script_idents computation, keep track of how often identifiers from various script groups crop up. If an identifier is from a less-common script group (say, <1% of identifiers), and it is entirely confusable with the majority script in use (e.g. the string "арр" or "роре" in Cyrillic)

This can trigger confusable_idents, mixed_script_idents, or a new lint.

We identify sets of characters which are entirely confusable: For example, for Cyrillic-Latin, we have а, е, о, р, с, у, х, ѕ, і, ј, ԛ, ԝ, ѐ, ё, ї, ӱ, ӧ, ӓ, ӕ, ӑ amongst the lowercase letters (and more amongst the capitals). This list likely can be programmatically derived from the confusables data that Unicode already has. It may be worth filtering for exact confusables. For example, Cyrillic, Greek, and Latin have a lot of confusables that are almost indistinguishable in most fonts, whereas ھ and ס are noticeably different-looking from o even though they're marked as a confusables.

The main confusable script pairs we have to worry about are Cyrillic/Latin/Greek, Armenian/Ethiopic, and a couple Armenian characters mapping to Greek/Latin. We can implement this lint conservatively at first by dealing with a blacklist of known confusables for these script pairs, and expand it if there is a need.

There are many confusables within scripts -- Arabic has a bunch of these as does Han (both with other Han characters and with kana), but since these are within the same language group this is outside the scope of this RFC. Such confusables are equivalent to l vs I being confusable in some fonts.

For reference, a list of all possible Rust identifier characters that do not trip less_used_codepoints but have confusables can be found here, with their confusable skeleton and script group mentioned on the right. Note that in many cases the confusables are visually distinguishable, or are diacritic marks.

Prior art

"Python PEP 3131: Supporting Non-ASCII Identifiers" is the Python equivalent to this proposal. The proposed identifier grammar XID_Start XID_Continue* is identical to the one used in Python 3. While Python uses KC normalization this proposes to use normalization form C.

JavaScript supports Unicode identifiers based on the same Default Identifier Syntax but does not apply normalization.

The CPP reference describes the allowed Unicode identifiers it is based on the immutable identifier principle.

Java also supports Unicode identifiers. Character must belong to a number of Unicode character classes similar to XID_start and XID_continue used in Python. Unlike in Python no normalization is performed.

The Go language allows identifiers in the form Letter (Letter | Number)* where Letter is a Unicode letter and Number is a Unicode decimal number. This is more restricted than the proposed design mainly as is does not allow combining characters needed to write some languages such as Hindi.

Unresolved questions

Copyright 2022 Nick Cameron and the RFC authors.