RFC 2141: alternative-registries

tools (cargo | registry)

Feature Name: cargo_alternative_registries
Start Date: 2017-09-06
RFC PR: rust-lang/rfcs#2141
Rust Issue: rust-lang/rust#44931

Summary

This RFC proposes the addition of the support for alternative crates.io servers to be used alongside the public crates.io server. This would allow users to publish crates to their own private instance of crates.io, while still able to use the public instance of crates.io.

Motivation

Cargo currently has support for getting crates from a public server, which works well for open source projects using Rust, however is problematic for closed source code. A workaround for this is to use Git repositories to specify the packages, but that means that the helpful versioning and discoverability that Cargo and crates.io provides is lost. We would like to change this such that it is possible to have a local crates.io server which crates can be pushed to, while still making use of the public crates.io server.

Guide-level explanation

Registry definition specification

We need a way to define what registries are valid for Cargo to pull from and publish to. For this purpose, we propose that users would be able to define multiple registries in a .cargo/config file. This allows the user to specify the locations of registries in one place, in a parent directory of all projects, rather than needing to configure the registry location within each project's Cargo.toml. Once a registry has been configured with a name, each Cargo.toml can use the registry name to refer to that registry.

Another benefit of using .cargo/config is that these files are not typically checked in to the projects' source control. The registries might have credentials associated with them, which should not be checked in. Separating the URLs and the use of the URLs in this way encourages good security practices of not checking in credentials.

In order to tell Cargo about a registry other than crates.io, you can specify and name it in a .cargo/config as follows, under the registries key:

[registries]
choose-a-name = "https://my-intranet:8080/index"

Instead of choose-a-name, place the name you'd like to use to refer to this registry in your Cargo.toml files. The URL specified should contain the location of the registry index for this registry; the registry format is specified in the Registry Index Format Specification section.

Alternatively, you can specify each registry as follows:

[registries.choose-a-name]
index = "https://my-intranet:8080/index"

If you need to specify authentication information such as a username or password to access a registry's index, those should be specified in a .cargo/credentials file since it has more restrictive file permissions than .cargo/config. Adding a username and password to .cargo/credentials for a registry named my-registry would look like this:

[registries.my-registry]
username = "myusername"
password = "mypassword"

CI

Because this system discourages checking in the registry configuration, the registry configuration won't be immediately available to continuous integration systems like TravisCI. However, Cargo currently supports configuring any key in .cargo/config using environment variables instead:

Cargo can also be configured through environment variables in addition to the TOML syntax above. For each configuration key above of the form foo.bar the environment variable CARGO_FOO_BAR can also be used to define the value. For example the build.jobs key can also be defined by CARGO_BUILD_JOBS.

To configure TravisCI to use an alternate registry named my-registry for example, you can use Travis' encrypted environment variables feature to set:

CARGO_REGISTRIES_MY_REGISTRY_INDEX=https://my-intranet:8080/index

Using a dependency from another registry

Note: this syntax will initially be implemented as an unstable cargo feature available in nightly cargo only and stabilized as it becomes ready.

Once you've configured a registry (with a name, for example, my-registry) in .cargo/config, you can specify that a dependency comes from an alternate registry by using the registry key:

[dependencies]
secret-crate = { version = "1.0", registry = "my-registry" }

Publishing to another registry; preventing unwanted publishes

Today, Cargo allows you to add a key publish = false to your Cargo.toml to indicate that you do not want to publish a crate anywhere. In order to specify that a crate should only be published to a particular set of registries, this key will be extended to accept a list of registries that are allowed with cargo publish:

publish = ["my-registry"]

If you run cargo publish without specifying an --index argument pointing to an allowed registry, the command will fail. This prevents accidental publishes of private crates to crates.io, for example.

Not having a publish key is equivalent to specifying publish = true, which means publishing to crates.io is allowed. publish = [] is equivalent to publish = false, meaning that publishing to anywhere is disallowed.

Running a minimal registry

The most minimal form of a registry that Cargo can use will consist of:

A registry in the format specified in the Registry index format specification section, which contains a pointer to:
A location containing the .crate files for the crates in the registry.

Running a fully-featured registry

This RFC does not attempt to standardize or specify any of crates.io's APIs, but it should be possible to take crates.io's codebase and run it along with a registry index in order to provide crates.io's functionality as an alternate registry.

Crates.io

Because crates.io's purpose is to be a reliable host for open source crates, crates that have dependencies from registries other than crates.io will be rejected at publish time. Crates.io cannot make availability guarantees about alternate registries, so much like git dependencies today, publishing with dependencies from other registries won't be allowed.

In crates.io's codebase, we will add a configuration option that specifies a list of approved alternate registry locations that dependencies may use. For private registries run using crates.io's code, this will likely include the private registry itself plus crates.io, so that private crates are allowed to depend on open source crates. Any crates with dependencies from registries not specified in this configuration option will be rejected at publish time.

Interaction with existing features

This RFC is not proposing any changes to the way source replacement and cargo-vendor work; everything proposed here should be compatible with those.

Mirrors will still be required to serve exactly the same files (matched checksums) as the source they're mirroring.

Reference-level explanation

Registry index format specification

Cargo needs to be able to get a registry index containing metadata for all crates and their dependencies available from an alternate registry in order to perform offline version resolution. The registry index for crates.io is available at https://github.com/rust-lang/crates.io-index, and this section aims to specify the format of this registry index so that other registries can provide their own registry index that Cargo will understand.

This is version 1 of the registry index format specification. There may be other versions of the specification someday. Along with a new specification version will be a plan for supporting registries using the older specification and a migration plan for registries to upgrade the specification version their index is using.

A valid registry index meets the following criteria:

The registry index is stored in a git repository so that Cargo can efficiently fetch incremental updates to the index.
There will be a file at the top level named config.json. This file will be a valid JSON object with the following keys:
```
{
  "dl": "https://my-crates-server.com/api/v1/crates/{crate}/{version}/download",
  "api": "https://my-crates-server.com/",
  "allowed-registries": ["https://github.com/rust-lang/crates.io-index", "https://my-intranet:8080/index"]
}
```
The dl key is required and specifies where Cargo can download the tarballs containing the source files of the crates listed in the registry. It is templated by the strings {crate} and {version} which are replaced with the name and version of the crate to download, respectively.

The api key is optional and specifies where Cargo can find the API server that provides the same API functionality that crates.io does today, such as publishing and searching. Without the api key, these features will not be available. This RFC is not attempting to standardize crates.io's API in any way, although that could be a future enhancement.

The allowed-registries key is optional and specifies the other registries that crates in this index are allowed to have dependencies on. The default will be nothing, which will mean only crates that depend on other crates in the current registry are allowed. This is currently the case for crates.io and will remain the case for crates.io going forward. Alternate registries will probably want to add crates.io to this list.
There will be a number of directories in the git repository.
- 1/ - holds files for all crates whose names have one letter.
- 2/ - holds files for all crates whose names have two letters.
- 3/a etc - for all crates whose names have three letters, their files will be in a directory named 3, then a subdirectory named with the first letter of their name.
- aa/aa/ etc - for all crates whose names have four or more letters, their files will be in a directory named with the first and second letters of their name, then in a subdirectory named with the third and fourth letters of their name. For example, a file for a crate named sample would be found in sa/mp/.
For each crate in the registry, there will be a file with the name of that crate in the directory structure as specified above. The file will contain metadata about each version of the crate, with one version per line. Each line will be valid JSON with, minimally, the keys as shown. More keys may be added, but Cargo may ignore them. The contents of one line are pretty-printed here for readability.
```
{
    "name": "my_serde",
    "vers": "1.0.11",
    "deps": [
        {
            "name": "serde",
            "req": "^1.0",
            "registry": "https://github.com/rust-lang/crates.io-index",
            "features": [],
            "optional": true,
            "default_features": true,
            "target": null,
            "kind": "normal"
        }
    ],
    "cksum": "f7726f29ddf9731b17ff113c461e362c381d9d69433f79de4f3dd572488823e9",
    "features": {
        "default": [
            "std"
        ],
        "derive": [
            "serde_derive"
        ],
        "std": [

        ],
    },
    "yanked": false
}
```
The top-level keys for a crate are:
- name: the name of the crate
- vers: the version of the crate this row is describing
- deps: a list of all dependencies of this crate
- cksum: a SHA256 checksum of the tarball downloaded
- features: a list of the features available from this crate
- yanked: whether or not this version has been yanked
Within the deps list, each dependency should be listed as an item in the deps array with the following keys:
- name: the name of the dependency
- req: the semver version requirement string on this dependency
- registry: New to this RFC: the registry from which this crate is available
- features: a list of the features available from this crate
- optional: whether this dependency is optional or not
- default_features: whether the parent uses the default features of this dependency or not
- target: on which target this dependency is needed
- kind: can be normal, build, or dev to be a regular dependency, a build-time dependency, or a development dependency. Note: this is a required field, but a small number of entries exist in the crates.io index with either a missing or null kind field due to implementation bugs.

If a dependency's registry is not specified, Cargo will assume the dependency can be located in the current registry. By specifying the registry of a dependency in the index, cargo will have the information it needs to fetch crate files from the registry indices involved without needing to involve an API server.

New command: `cargo generate-index-metadata`

Currently, the knowledge of how to create a file in the registry index format is spread between Cargo and crates.io. This RFC proposes the addition of a Cargo command that would generate this file locally for the current crate so that it can be added to the git repository using a mechanism other than a server running crates.io's codebase.

Related issues

In order to make working with multiple registries more convenient, we would also like to support:

Adding a cargo add-registry command that could prompt for index URL and authentication information and place the right information in the right format in the right files to make setup for each user easier.
Being able to specify the API location rather than the index location, so that, for example, you could specify https://host.company.com/api/cargo/private-repo rather than https://github.com/host-company/cargo-index. We do not want to require specifying the API location, since some registries will choose not to have an API at all and only supply an index and a location for crate files. This would require the API to have a way to tell Cargo where the associated registry index is located.
Being able to save multiple tokens in .cargo/credentials, one per registry, so that people publishing to multiple registries don't need to log in over and over or specify tokens on every publish.
Being able to specify --registry registry-name for all Cargo commands that currently take --index
Being able to use a dependency under a different name. Alternate registries that are not mirrors should be allowed to have crates with the same name as crates in any other registry, including crates.io. In order to allow a crate to depend on both, say, the http crate from crates.io and the http crate from a private registry, at least one will need to be renamed when listed as a dependency in Cargo.toml. RFC 2126 proposes this change as follows:

Cargo will provide a new crate key for aliasing dependencies, so that e.g. users who want to use the rand crate but call it random instead can now write random = { version = "0.3", crate = "rand" }.
Being able to use environment variables to specify values in .cargo/credentials in the same way that you can use environment variables to specify values in .cargo/config
For registries that don't require any authentication to access, such as public registries or registries only accessible within a firewall, we could support a shorthand where the index location (or API location when that is supported) is specified entirely within a crate dependency:
```
[dependencies]
my-crate = { version = "1.0", registry = "http://crate-mirror.org/index" }
```
In order to discourage/disallow credentials checked in to Cargo.toml, if the URL contains a username or password, Cargo will deliberately remove it. If the registry is then inaccessible, the error message will mention that usernames and passwords in URLs in Cargo.toml are not allowed.

Drawbacks

Supporting alternative registries, and having multiple public registries, could fracture the ecosystem. However, we feel that supporting private registries, and the Rust adoption that could enable, outweighs the potential downsides of having multiple public registries.

Rationale and Alternatives

A previous RFC proposed having the registry information completely defined within Cargo.toml rather than using .cargo/config. This requires repeating the same information multiple times for multiple projects, and encourages checking in credentials that might be needed to access the registries. That RFC also didn't specify the format for the registry index, which needs to be shared among all registries.

An alternative design could be to support specifying the registry URL in either .cargo/config or Cargo.toml. This has the downsides of creating more choices for the user and potentially encouraging poor practices such as checking credentials into a project's source control. The implementation of this feature would also be more complex. The upside would be supporting configuration in ways that would be more convenient in various situations.

Unresolved questions

Are the names of everything what we want?
- cargo generate-index-metadata?
- registry = my-registry?
- publish-registries = []?
What kinds of authentication parameters do we need to support in .cargo/credentials?