Private hosting of Rust crates

For a long time, systems programming in general and embedded programming in particular, have been dominated by C, and to some extent C++. The reason is clear: the programmer has control over the system, and the overhead is minimal. At the same time, the type system is weak, handling dependencies is hard, and it takes a lot of experience and discipline to do things correctly. Of course, there exist other languages trying to solve these shortcomings (Ada, Java Embedded), but they have stayed niche or faded away into obscurity.

 

In recent years, the programming language Rust has seen a rise in popularity. Rust promises to combine three essential features:

  • Performant: abstractions comes with no or minimal costs, overhead is always opt-in (no garbage collection or other runtime as part of the language).
  • Reliable: a rich and strict type system gives high safety; correctly used, many bugs can be found at compile time.
  • Developer friendly: great documentation, a thriving ecosystem and excellent tooling.

When building a satellite terminal, neither performance nor safety may be neglected. For this reason, we at Satcube are moving to Rust for new software development of our embedded systems.

Managing projects

One of the benefits of Rust is the package manager Cargo that ships with the compiler. It is mainly used to collect the dependencies of the project and build it. It can also package the code and upload it to the central package registry crates.io. Often (for embedded systems, after some initial setup) building an executable can be as simple as stating the dependencies, writing the code and issuing  cargo build --release.

When a project grows, we would like to break out parts of the code base into libraries, and have Cargo handle this for us. For obvious reasons, we do not want to publish all of these in a public registry. It is possible to specify dependencies from git repositories, and sometimes this may work out just fine. The drawback is that we need to either fix a specific commit/tag or follow a specific branch. In the first case, we will be certain that the dependency doesn’t change, meaning a build is always reproducible, but we will not be able to pull in new versions, that may contain critical fixes, unless we manually update the specification. In the second case, we will always fetch the latest version of the dependency, including potentially breaking changes. When depending on libraries (or crates as it is named in the Rust lingua), semantic versioning is used to handle this: we receive patches but will not upgrade major, api-breaking changes unless we manually opt in to them.

In order to use these benefits, and still keep code private, we need to set up a private registry! The remainder of this text will explain the steps we took to accomplish this.

Setting up a private crate registry

The steps needed to implement a private crate registry are described in RFC 2141. This includes having a git repository that hold the actual index and some location from where the actual packages can be served. Keeping this in sync may be done manually, but would be tiresome and error prone as the registry grows. Luckily, there exists a software that takes care of these things, and additionally provides an API to handle crates, users, roles etc: Meuse. The steps needed to get a working registry using Meuse are:

  • Create a git repository that should hold the index. We host our repo at GitLab, but any host should do.
  • Commit the file config.json to the repository, containing
{
  "dl": "http://localhost:8855/api/v1/crates",
  "api": "http://localhost:8855",
  "allowed-registries": [ "https://github.com/rust-lang/crates.io-index" ] 
}

 

When moving from local hosting, remember to change the file accordingly.

  • Create some means for the registry to authenticate with the git host; GitLab has deploy keys that can be used to give the service access to the specific repo, but creating a designed user should also do. The following setup assumes you have a public/private key pair, with the public key registred at the git host.
  • Clone our private-crate-registry repository.
  • Adapt the config files to suit your needs. Pay special attention to
    • Dockerfile – set the ARGuments accordingly
    • meuse-config.yml – see the Meuse documentation
  • Create database.env containing the following keys, that should be in accordance with the values set in meuse-config.yml:

 

POSTGRES_USER=meuse
POSTGRES_PASSWORD=meuse
POSTGRES_DB=meuse

 

  • Build the docker image (Docker 18.09 or higher is needed), using the private key registry-rsa:

 

DOCKER_BUILDKIT=1 docker build \ 
  --secret id=private_key,src=./registry-rsa \
  --build-arg USER=my_user \
  --build-arg GIT_USER_NAME="My user" \
  --build-arg GIT_USER_MAIL="myuser@mycompany.com" \
  --build-arg GIT_HOST="gitlab.com" \
  --build-arg GIT_REGISTRY_URL="git@gitlab.com:mycompany/registry" \
  -t my-registry \
  .

 

  • Start the service:

 

MEUSE_IMAGE=my-registry \
  GIT_REGISTRY_URL="git@gitlab.com:mycompany/registry" \
  docker-compose up

 

Using the private registry

Once the registry is set up, it can be used in a crate by adding the file .cargo/config.toml:

 

[registries]
my_registry = { index = "ssh://git@gitlab.com/company/crates/registry" }

[registry] default = "my_registry"

 

Also, in order to ensure that the publishing of the crate is going to the private registry, and not to crates.io, Cargo.toml needs to be adapted like so:

 

[package]
# ...
publish = ["my_registry"]

 

In upstream projects, the crate may then be used by specifying the registry in the dependecies section in Cargo.toml (assuming it as well has a .cargo/config.toml as above, defining the registry):

 

[dependencies]
my_crate = { version = "0.1", registry = "my_registry" }
1 Comment
  • Minecraft Hosting

    2 December, 2020 at 14:54

    Pretty nice post. I simply stumbled upon your weblog andd wished to mention that I’ve truly loved brolwsing your weblog posts.
    In anyy case Iwilll be subscribing to your rss feed and
    I’m hoping you write once more soon!

Post a Comment