[Bug 1957932] Re: [MIR] rustc, cargo

[Simon Chopin]

Status of the rustc part of the MIR:

There's a package available at
https://launchpad.net/~schopin/+archive/ubuntu/rustc-mir/+packages
(still building...). I'm aware that there are issues with the armhf
build and am still investigating them. Barring that, I feel the package
is ready for security review.

I'll walk through the items from cpaelzer review.

I'll leave #1 for a later date (and MIR :D), but I do have the issue in
mind.

#2 has been resolved by reverting the bundling of LLVM. We expect to be
more proactive in the Foundations team regarding rustc and LLVM, and the
latter needs regular upgrades for other components of our releases as
well (kernel and mesa)

#3: I've shipped Cargo.lock in /usr/share/doc/rustc/Cargo.lock, the
exact location can be revisited. I've checked, it is indeed generated at
build time and so is up to date. Note that src:rustc does *not* use dh-
cargo so the related points aren't applicable.

#4: Still no action on MIRing dh-cargo at this time.

#5: I've added a self-build autopkgtest to rustc. It actually already
caught a small regression in their linkchecker tool that impacts us
because we de-bundle the JS and fonts from the doc.

#6: The testsuite is run on s390x, it "just" has a higher failure
tolerance than other architectures, with 40 failures allowed (8 for
amd64)

#7, #8: I still don't know how to do the exclude definition :-/. As
stated earlier, I think the scope of the MIR should be for the rustc and
libstd-rust-* packages, with the -doc and all ancillary tooling
remaining in universe as they're not essential to the packaging of Rust
applications.

#9: LTO has been explicitly disabled in d/rules, with a comment to
actually revisit this decision to see if it's still needed.

#10: We'll go with separate LLVM, I'll update the bug description.

#11: Tests have been re-enabled using Debian's "not too many errors"
rules as a first step, and I plan on cataloging failures to narrow the
policy, but it won't be possible in the near future.

#12: foundations-bugs is now subscribed to both src:rustc and src:cargo

#13: I worked on 1.58.1 directly, as I needed to do some repacking
anyway to remove some vendored dependencies that didn't need to be in
there (e.g. the curl and openssl crates)


Regarding the MANY warnings during the build, those warnings are
basically linting for constructs that used to be necessary but aren't
anymore as the compiler gets smarter, and, at least when compiling with
the "right" rustc version (meaning the N-1 version), are all explicitly
allowed. However, since we need to be able to compile with the *current*
version, we've had to lift the blanket deny instruction for all *other*
warnings, as new warnings introduced by the version compiled won't be in
this allowed list, as we've experienced when working on 1.57. Going
forward, an axis of improvement would be to see if it would be possible
to actually suppress all the benign warnings to make the new ones stand
out.


Regarding armhf:

When mentioning on IRC that the armhf build had some issue, a question
was raised of whether we should enable rustc on this architecture at
all, especially since rustc tends towards heavy memory consumption.

The armhf platform is currently considered upstream as "Tier 2 with host
tools", which means that we should be able to build *and* use rustc and
cargo on the platform.

Also, while I understand the sentiment, not having a native rustc on
armhf means we'd need to have rock-solid cross-build support for quite a
few packages in the archive. I'm not familiar with this particular part
of the packaging experience, but I'd wager it's not trivial to set up?

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to rustc in Ubuntu.
https://bugs.launchpad.net/bugs/1957932

Title:
  [MIR] rustc, cargo

Status in cargo package in Ubuntu:
  Incomplete
Status in rustc package in Ubuntu:
  Incomplete

Bug description:
  [Availability]
  The packages rustc and cargo are already in Ubuntu universe.
  The packages build for the architectures they are designed to work on,
  and are also built on platform with lesser upstream support, see
  https://doc.rust-lang.org/nightly/rustc/platform-support.html for details.

  They currently build and works for architectures:
    * amd64
    * arm64
    * armhf
    * i386
    * ppc64el
    * riscv64
    * s390x

  Link to packages:
  https://launchpad.net/ubuntu/+source/rustc
  https://launchpad.net/ubuntu/+source/cargo

  [Rationale]
  The packages rustc and cargo are required in Ubuntu main as the Rust programming
  language is gaining in popularity, and those two packages are, respectively, its
  main compiler implementation and its dedicated build tool (and dependency manager).

  There are a few packages in main already that have partially switched to Rust
  as their implementation language, and so rustc and cargo will be needed to keep
  us in sync with their upstream. See for instance
  https://bugs.launchpad.net/ubuntu/+source/mdevctl/+bug/1942394 and
  https://lists.debian.org/debian-python/2021/12/msg00000.html
  (python-cryptography is in main)

  Note that the huge majority of our users will not use these packages, their
  purpose is to be a build-dependency for other packages. In particular, it is
  not particularly expected at this stage that those of our users that are Rust
  developers, which usually rely on their toolchain being managed in their $HOME
  by the `rustup` tool.

  [Security]
  cargo and rustc had 19 recorded security issues in the past, mostly in the Rust standard library
  (1 affecting cargo):

  https://nvd.nist.gov/vuln/search/results?form_type=Advanced&results_type=overview&search_type=all&isCpeNameSearch=false&cpe_vendor=cpe%3A%2F%3Arust-
  lang&cpe_product=cpe%3A%2F%3A%3Arust

  All issues are usually handled promptly by the Rust team. However, the fixes
  are rarely (if ever) backported to previous releases besides an occasional
  1.X.1 point release for the latest stable.

  There is an official Rust Security working group that curates a database of security
  issues within the Rust ecosystem, including rustc/cargo:

  https://github.com/rustsec/advisory-db

  - no `suid` or `sgid` binaries
  - no executables in `/sbin` and `/usr/sbin`
  - Package does not install services, timers or recurring jobs
  - Packages does not open privileged ports (ports < 1024)
  - Packages does not contain extensions to security-sensitive software
  (filters, scanners, plugins, UI skins, ...)

  Note however that in typical use, building a project with cargo+rustc involves
  executing code that has been downloaded from crates.io: cargo builds and executes
  the `build.rs` file for any pre-compilation task (a bit like a Makefile), and any use
  of a proc macro dependency basically implies running arbitrary code (the macro) within
  the execution context of rustc.

  [Quality assurance - function/usage]
  The packages work well right after install, one can easily create a simple Rust project
  and run it.

  [Quality assurance - maintenance]
  The packages do not deal with exotic hardware we cannot support

  [Quality assurance - testing]
  The packages both run a test suite on build time. However, the rustc test suite
  does NOT make the build fail. The reason is that there are always a few tests that fail, and it was a tradeoff made due to limited resources. Please note that Debian has a strategy of only failing the build if there are *too many* errors. As the Foundations team commits more resources on this toolchain, this policy will probably be revisited.

  Neither package has any autopkgtests.

  TODO: cargo autopkgtest that creates a new crate, adds a simple dependency
  (anyhow is a good candidate, no transitive deps), vendors it, builds the binary
  from the vendored crate, and runs it.

  [Quality assurance - packaging]
  debian/watch is present and works

  rustc yields quite a bit of lintian output, but they seem mostly harmless.
  https://lintian.debian.org/sources/rustc

  There are lintian overrides present, but those are usually justified in the override
  files themselves.

  cargo is a bit more tame on the warning fronts
  https://lintian.debian.org/sources/cargo

  It has an override file for the source package, for relatively minor
  warnings.

  These packages does not rely on obsolete or about to be demoted
  packages.

  The packages will not be installed by default

  The packaging is fairly complex, especially in the case of rustc. The crux of
  the complexity can be attributed to the boostrapping issue, as well as the need
  to deal with vendored dependencies. There is extensive documentation within the
  debian/ directories to help with the difficulties, and a lot of things have been
  automated in scripts living in debian/scripts/*

  Note that originally, the upstream tarball for rustc includes the sources for
  cargo as well as its vendored dependencies, but the Debian Rust team chose to
  split it out in its own package.

  [UI standards]
  I do not believe there's a need for translation for these applications given the
  stated purpose for having them in main.

  [Dependencies]
  The packages have quite a few vendored dependencies. However, their non-vendored
  dependencies are all in main, as well as the build-dependencies, assuming for the purpose
  of this analysis that src:rustc is in main.

  [Standards compliance]
  The packages violate the Debian Policy on vendored
  dependencies, with a copy of LLVM in rustc, and a copy of libgit2 (as a delta to Debian) in cargo.

  Note that one can also find the sources for libcurl in src:rustc, but
  that's a leftover of the dependencies of cargo, which has been split
  out of src:rustc.

  [Maintenance/Owner]
  Owning Team will be Foundations
  Team is not yet, but will subscribe to the package before promotion

  Both packages use static linking for the Rust dependencies, as well as any
  vendored C dependency (i.e. libgit2 and LLVM).
  The copy of LLVM is there because rustc depends on relatively recent version of LLVM, and has been upgraded regularly in our LTS releases due to firefox regularly bumping its minimal rust version requirements. Removing the embedded copy would probably mean having to have those more recent versions of LLVM in the archive as well.

  rustc uses vendored rust code tracked in Cargo.lock as shipped, in the package,
  refreshing that code works via `cargo update ...` + cargo vendor

  cargo has a different process, documented in debian/README.source.
  TODO: actually try to update a dependency in cargo using the debian/README.source,
  which are not that clear.

  [Background information]
  The Package descriptions explains the package well
  Upstream Name is developed by the Rust Compiler team and the Cargo team, under
  the umbrella of the Rust Foundation
  Link to upstream project: https://www.rust-lang.org/

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/cargo/+bug/1957932/+subscriptions