rustc/debian

Document by Luca Bruno & Sylvestre Ledru

This source package is unfortunately quite tricky and with
several cutting edges, due to the complexity of rust-lang
bootstrapping system and the high rate of language changes
still ongoing.

We try to describe here inner packaging details and the
reasons behind them.


Import of a new upstream version
================================

$ uscan
$ gbp import-orig --upstream-branch=upstream/experimental --debian-branch=debian/experimental ../rustc_X.YY.0+dfsg1.orig.tar.xz
$ dch -v X.YY.0+dfsg1-1
$ debian/rules update-version
# might also need to bump the version of the cargo Build-Depends


Embedded libraries
==================

This source package embeds several external libraries (foeked and managed
by rust upstream as git submodules).
In early stages, many more libraries were forked/emebedded but we are steadily
progressing in splitting them out.

Here below the remaining ones, with the technical reasons.

 * jemalloc from https://github.com/rust-lang/jemalloc
   -> system-wide one can't be used due to rust using a "je_" prefix.

   This is intentional upstream design and won't change soon, see:
    - https://github.com/rust-lang/rust/pull/18678
    - http://smallcultfollowing.com/babysteps/blog/2014/11/14/allocators-in-rust/

 * compiler-rt from https://github.com/rust-lang/compiler-rt
   -> system-wide compiler-rt fails during linkage

   Bug reported upstream, still to be fixed, see:
    - https://github.com/rust-lang/rust/issues/15054
    - https://github.com/rust-lang/rust/issues/15708

As a summary, we plan to:
 * keep embedding jemalloc (probably forever)
 * work with upstream to fix compiler-rt linkage soon.

 -- Sylvestre Ledru <sylvestre@debian.org>  Sat, 06 May 2017 13:26:08 +0200


Building from source
====================

The Debian rustc package will use the system rustc to bootstrap itself from.
The system rustc has to be either the previous or the same version as the rustc
being built; the build will fail if this is not the case.

    sudo apt-get build-dep ./
    dpkg-buildpackage
    # Or, to directly use what's in the Debian FTP archive
    sudo apt-get build-dep rustc
    apt-get source --compile rustc

Alternatively, you may give the "pkg.rustc.dlstage0" DEB_BUILD_PROFILE to
instead use the process defined by Rust upstream. This downloads the "official"
stage0 compiler for the version being built from rust-lang.org. At the time of
writing "official" means "the previous stable version".

    sudo apt-get build-dep -P pkg.rustc.dlstage0 ./
    dpkg-buildpackage
    # Or, to directly use what's in the Debian FTP archive
    sudo apt-get build-dep -P pkg.rustc.dlstage0 rustc
    apt-get source --compile -P pkg.rustc.dlstage0 rustc

After [1] is fixed, both of these should in theory give identical results.

If neither of these options are acceptable to you, e.g. because your distro
does not have rustc already and your build process cannot access the network,
see "Bootstrapping" below.

[1] https://github.com/rust-lang/rust/issues/34902


Bootstrapping
=============

To bootstrap rustc on a distro that does not have it or cargo available on any
architecture (so cross-compiling is not an option) you can run `debian/rules
source_orig-stage0`. This creates a .dsc that does not Build-Depend on rustc or
cargo. Instead, it includes an extra orig-stage0 source tarball that contains
the official stage0 compiler, pre-downloaded from rust-lang.org so that your
build daemons don't need to access the network during the build.

    debian/rules source_orig-stage0
    # Follow the final manual instructions that it outputs. Then:
    sbuild ../rustc_*.dsc && dput ../rustc_*.dsc

To only bootstrap specific architectures, run this instead:

    upstream_bootstrap_arch="arm64 armhf" debian/rules source_orig-stage0

This way, other architectures will be omitted from the orig-stage0 tarball. You
might want to do this e.g. if these other architectures are already present in
your distro, but the $upstream_bootstrap_arch ones are not yet present.

Notes
-----

The approach bundles the upstream bootstrapping binaries inside the Debian
source package. This is a nasty hack that stretches the definition of "source
package", but has a few advantages explained below.

The traditional Debian way of bootstrapping compilers - and other distros have
similar approaches - is some variant of the following:

1. A developer locally installs some upstream bootstrapping binaries.
2. They locally build a Debian package, using these binaries as undeclared
   build dependencies.
3. They upload these binary packages to Debian, which can be used as declared
   Build-Depends in the future, including by the same package.

The problem with this is, Debian does not have any policy nor infrastructure
that can try to reproduce what this developer supposedly did.

Using bootstrapping binary blobs *at some point of the process* is unavoidable.
Rather than pretending we didn't do this, it is better to record *which blobs*
we used, so it can be audited later. If we bundle non-Debian build-dependencies
inside the source package, then we can do a *source-only upload*, and the
building of the binary packages can be done by the normal build infrastructure.

If the build process is reproducible [1] then we can be sure that *you* (as the
developer that prepared the source-only upload) didn't backdoor the binaries,
nor did the build daemons even if they were compromised during the build.

The bootstrapping binaries may still have been backdoored, but this is true in
both scenarios. So our arrangement is still a strict improvement in security,
because it reduces the set of "things that may have been backdoored". Also,
more people use the upstream binaries than the "magical original Debian
package", so backdoors have a greater chance of being detected in the former.

In the long run, this process is laying the foundations for doing Diverse
Double-Compilation [2], where we use *many independent* bootstrapping binaries
to reproduce bit-for-bit identical output compilers, giving confidence that
nothing was backdoored along the way.

[1] The build process for rustc is currently *not* reproducible but we're
    working towards it. https://github.com/rust-lang/rust/issues/34902
[2] http://www.dwheeler.com/trusting-trust/