code:
Tor Browser Build
=================

Installing build dependencies
-----------------------------

To build Tor Browser, you need a recent Linux distribution with support
for user_namespaces(7) (such as Debian Buster, Ubuntu 16.04, Fedora 30,
etc ...). You will need to install the uidmap package, providing the
newuidmap and newgidmap commands.

The sources of most components are downloaded using git, which needs to
be installed.

You also need a few perl modules installed:
- YAML::XS
- File::Basename
- Getopt::Long
- Template
- IO::Handle
- IO::CaptureOutput
- JSON
- File::Temp
- Path::Tiny
- File::Path
- File::Slurp
- File::Copy::Recursive
- String::ShellQuote
- Sort::Versions
- Digest::SHA
- Data::UUID
- Data::Dump
- DateTime

If you are running Debian or Ubuntu, you can install them with:

# apt-get install libyaml-libyaml-perl libtemplate-perl libdatetime-perl \
                  libio-handle-util-perl libio-all-perl \
                  libio-captureoutput-perl libjson-perl libpath-tiny-perl \
                  libstring-shellquote-perl libsort-versions-perl \
                  libdigest-sha-perl libdata-uuid-perl libdata-dump-perl \
                  libfile-copy-recursive-perl libfile-slurp-perl git \
                  uidmap

The build system is based on rbm, which is included as a git submodule
in the rbm/ directory. You can fetch the rbm git submodule by running
'make submodule-update'.

The build uses user_namespaces(7), which are disabled by default on Debian.
To enable them you can use the following command as root:

  # sysctl -w kernel.unprivileged_userns_clone=1

You can enable them permanently by adding the setting to /etc/sysctl.d/

The user you use to build needs to have a range of subordinate uids and
gids in /etc/subuid and /etc/subgid. Most of the time they are added by
default when the user is created. If it was not the case, you can use
usermod(8) with the --add-subuids and --add-subgids options. See also
the subuid(5) and subgid(5) man pages.

Starting a build
----------------

To start a build, run one of the following commands, depending on the
channel you want to build:

  $ make torbrowser-release
  $ make torbrowser-alpha
  $ make torbrowser-nightly

You can find the build result in the directory release/unsigned/$version
or alpha/unsigned/$version for release or alpha builds. The result of
nightly can be found in the nightly/$version directory.

If you want to build for a specific platform only, append the platform
name to the makefile target:

  $ make torbrowser-nightly-linux-x86_64
  $ make torbrowser-nightly-linux-i686
  $ make torbrowser-nightly-windows-i686
  $ make torbrowser-nightly-windows-x86_64
  $ make torbrowser-nightly-osx-x86_64
  $ make torbrowser-nightly-android-armv7
  $ make torbrowser-nightly-android-aarch64
  $ make torbrowser-nightly-android-x86
  $ make torbrowser-nightly-android-x86_64

When you want to quickly do a build to test a change, you can use the
testbuild makefile target, and find the build in the testbuild directory.
The build will be the same as regular alpha builds, except that in order
to make the build faster, only the en-US locale will be built, and no
mar file will be created. If you want to base your testbuild on the latest
nightly code insted, rename rbm.local.conf.example to rbm.local.conf
and adapt the torbrowser-testbuild option accordingly.

Similar makefile targets exist for building Base Browser instead of
Tor Browser. To build Base Browser, replace `torbrowser` by `basebrowser`
in the target name.

Updating git sources
--------------------

You can run `make fetch` to fetch the latest sources from git for all
components included in Tor Browser. You should run this if you want to
make a nightly build with the latest commits, and you disabled automatic
fetching of new commits for nightly builds in rbm.local.conf.

Number of make processes
------------------------

By default `nproc` is used to determine the number of processes to run
simultaneously (with make -jN where N is the number returned by `nproc`).
If you want to change the number of processes used, you can set the
RBM_NUM_PROCS environment variable:

  $ export RBM_NUM_PROCS=8

You can also set the num_procs option in rbm.local.conf.

Automated builds
----------------

If the build fails, a shell will automatically open in the build
container to help you debug the problem. You probably want to disable
this if you want to do automated builds. To disable this, set
the RBM_NO_DEBUG environment variable to 1:

   export RBM_NO_DEBUG=1

Or set the debug option to 0 in the rbm.local.conf file.

If you want to select the output directory, you can use rbm's --output-dir
option. You can look at the Makefile to find the rbm command for what
you want to build, and add the --output-dir option. For example, if you
want to build Tor Browser nightly for linux-x86_64:

   ./rbm/rbm build release --output-dir=/var/builds/nightly/2020-05-23 \
                        --target nightly --target torbrowser-linux-x86_64

The files will be put in the directory selected by --output-dir in a
subdirectory named as the version number (or current date for nightly).
To remove this version subdirectory, add the noversiondir target:

   ./rbm/rbm build release --output-dir=/var/builds/nightly/2020-05-23 \
                        --target nightly --target torbrowser-linux-x86_64 \
                        --target noversiondir

Automated builds using tbb-testsuite
------------------------------------

The Tor Browser testsuite scripts can also be used to do nightly builds
and publish the build logs. The recommended way to do that is to use
the ansible roles from the tools/ansible directory. See next section
for details.

Using ansible to set up a nightly build machine
-----------------------------------------------

The directory tools/ansible contains some ansible roles to set up a
nightly build machine. You can look at the playbook defined in
boklm-tbb-nightly-build.yml and variables in group_vars/boklm-tbb-nightly/
for an example of how it can be used.

Signing builds
--------------

If the environment variable RBM_SIGN_BUILD is set to 1, the
sha256sums-unsigned-build.txt and sha256sums-unsigned-build.incrementals.txt
files will be signed with gpg. You can use the RBM_GPG_OPTS environment
variable to add some options to the gpg command used to sign the file.
You can also set the var/sign_build and var/sign_build_gpg_opts options
in the rbm.local.conf file.

Cleaning obsolete files and containers images
---------------------------------------------

You can run `make clean` to clean old build files and containers that
are no longer used in current builds. Before doing that, you need to
configure the branches and build targets you are using in the
rbm.local.conf file. The cleaning script will check out all the configured
branches to create a list of used build files, and delete the files
from the 'out' directory that are not used. If you want to see the list
of files and containers that would be removed without doing it, you can
use `make clean-dry-run`.

Building without containers (Android builds only)
-------------------------------------------------

By default the build is done inside containers. Adding the no_containers
target will disable the use of containers. The following commands can
be used to build the alpha version for e.g. android-armv7:

  ./rbm/rbm build release --target no_containers --target testbuild \
                          --target torbrowser-android-armv7

Note: the logs will still show the use and creation of a container image
called "containers_disabled". This is due to the way we disable the use
of containers: the container-image project is still called, but it will
just create an empty file instead of a real container image.

The build without containers is currently only supported for the Android
builds, and will require that you run Debian Buster and install build
dependencies for all the components that are built. This can be done
with the following command:

  # apt-get install build-essential python automake libtool zip unzip \
            autoconf2.13 openjdk-8-jdk gettext-base autotools-dev \
            automake autoconf libtool autopoint libssl-dev \
            pkg-config zlib1g-dev libparallel-forkmanager-perl \
            libfile-slurp-perl bzip2 xz-utils apksigner yasm

Common Build Errors
-------------------

You can look at the file doc/BUILD_ERRORS.txt for a list of common build
errors and their solutions.

Hacking on the Tor Browser build
--------------------------------

The file doc/HACKING.txt tries to list the main things to know when
making changes to the Tor Browser build.

Description of makefile rules
-----------------------------

You can find a description of the Makefile rules in the file doc/MAKEFILE.txt.

path: README


code:
This file lists some common build errors and their solutions.

Error starting remote
---------------------

If you have an error like this:

----
Error: Error starting remote:
could not synchronise with container process: no subsystem for mount
----

You may be experiencing a similar but different problem. Anecdotally
this occured on Ubuntu 18.04 Bionic, kernel 4.15.0-24-generic. You need
to add systemd.legacy_systemd_cgroup_controller=1 to the kernel
boot commandline in /etc/default/grub (followed by `sudo update-grub`).

Error during debootstrap image creation
---------------------------------------

If the debootstrap-image-.log contains errors similar to the following:
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/zesty/InRelease
   Temporary failure resolving 'archive.ubuntu.com'

Check /etc/resolv.conf on the host to see if the nameserver is set to
127.0.0.1. This can happen when runc performs a bind mount of
/etc/resolv.conf and the host system is running systemd-resolved.

sudo systemctl disable systemd-resolved.service
sudo service systemd-resolved stop

Put the following line in the [main] section of your
/etc/NetworkManager/NetworkManager.conf:

dns=default

Delete the symlink /etc/resolv.conf

rm /etc/resolv.conf

Restart network-manager

sudo service network-manager restart

Could not find uid in /etc/subuid
---------------------------------

In some cases you can have the error:

  Error: Error starting remote:
  Error: Could not find uid in /etc/subuid
  Error: failed to set uidmap

The user you use to build needs to have a range of subordinate uids and
gids in /etc/subuid and /etc/subgid. Most of the time they are added by
default when the user is created. If it was not the case, you can use
usermod(8) with the --add-subuids and --add-subgids options. See also
the subuid(5) and subgid(5) man pages.

Note that the root user is not exempted from the requirement for a
valid /etc/subuid and /etc/subgid entry, if you are building as root.

repo_name: JeremyRand/tor

license: mit


code:
Hacking on Tor Browser Build
============================

This file tries to list the main things to know when making changes to
the Tor Browser build.

rbm documentation
-----------------

If you go to directory rbm/doc, you can find the rbm documentation. You
can build it with 'make html' or 'make man'.

Using and defining options
--------------------------

Options can be used in templates with the following syntax:

  [% c("option_name") %]

More details about the templates syntax can be found in
rbm/doc/rbm_templates.7 and [http://template-toolkit.org/docs/manual/index.html].

Some options have a specific meaning to rbm. You can see their descriptions
in rbm/doc/rbm_config.7 and rbm/doc/rbm_input_files.7.

When the option does not have a specific meaning to rbm, it is a good
idea to define it under var/ to avoid potential conflicts in option names
with a future version of rbm.

The options can be defined in different places:

- rbm.conf for options available to all components
- project/$project/config for options available to one component
- rbm.local.conf for user defined options

In each of those places, an option can be defined:

- at the root of the document
- under targets/$target/, in which case the definition only applies when
  a specific target is defined

The targets are usually used to select:

- the platform: torbrowser-linux-x86_64, torbrowser-linux-i686,
  torbrowser-windows-i686, torbrowser-windows-x86_64, torbrowser-osx-x86_64,
  torbrowser-android-armv7, torbrowser-android-aarch64, torbrowser-android-x86,
  torbrowser-android-x86_64
- the channel: release, nightly, alpha

The targets torbrowser-linux-x86_64, torbrowser-linux-i686,
torbrowser-windows-i686, torbrowser-windows-x86_64, torbrowser-osx-x86_64,
torbrowser-android-armv7, torbrowser-android-x86, torbrowser-android-aarch64,
torbrowser-android-x86_64 are special cases. They do not contain options
directly, instead they contain a list of other targets. For instance, the
torbrowser-linux-x86_64 target is pointing to the linux-x86_64 and linux
targets. You should define an option under the linux target if it applies to
Linux on both architectures, or under the linux-x86_64 if it only applies to
the x86_64 architecture.

An option that is defined at the root of rbm.conf can be overridden by
an other definition under a target, or inside projects/$project/config.
You can find the complete priority order in rbm/doc/rbm_config.7.

Defining a project's filename
-----------------------------

The filename option defines the output file or directory from the build
of a component. If the file or directory exists in the out/$component
directory, the component will not be rebuilt when it is used as a
dependency from an other component.

The filename is usually something like this:

  filename: '[% project %]-[% c("version") %]-[% c("var/osname") %]-[% c("var/build_id") %].tar.gz'

The var/build_id value is a hash that is based on the build inputs of
the component:
 - the build script (after template evaluation)
 - the container definition
 - the input files (the filename when it is a dependency on an other
   project, the filename and hash of its content otherwise)
This means that each time the build script, the container or one of the
dependencies is modified, the output filename will change and a rebuild
of the component will be required.

The version and var/osname values could be removed from the filename,
but they are useful to get an idea of what a file contains.

Adding some Linux/Windows/OSX specific changes to a build script
----------------------------------------------------------------

You can use the following template syntax in the build scripts:

  [% IF c("var/linux") -%]
    # do something for linux
  [% ELSIF c("var/windows") -%]
    # do something for windows
  [% ELSIF c("var/osx") -%]
    # do something for osx
  [% END -%]

You can also use var/linux-x86_64 and var/linux-i686 for things that
only apply to x86_64 and i686 linux builds. You can use the var/release,
var/alpha and var/nightly options to do things depending on the channel.

As an alternative you can define an option with a different value
depending on the target:

  targets:
    linux:
      var:
        do_something: 'do something for linux'
    windows-i686:
      var:
        do_something: 'do something for windows'
    osx-x86_64:
      var:
        do_something: 'do something for osx'

And in the build script, use:

  [% c("var/do_something") %]

Evaluating a component's build script
-------------------------------------

If you want to look at the build script used to build a component for a
specific platform/channel, you can use the following command:

   $ ./rbm/rbm showconf $component build --target $channel --target $platform

$component should be replaced by the name of the component.

$channel should be one of the following:

 - release

 - alpha

 - nightly

$platform should be one of the following:

 - torbrowser-linux-x86_64

 - torbrowser-linux-i686

 - torbrowser-windows-i686

 - torbrowser-windows-x86_64

 - torbrowser-osx-x86_64

 - torbrowser-android-armv7

 - torbrowser-android-aarch64

 - torbrowser-android-x86

 - torbrowser-android-x86_64

For example, to see tor's build script for linux x86_64 on the alpha
channel, you can use:

  $ ./rbm/rbm showconf tor build --target alpha --target \
    torbrowser-linux-x86_64

If the component you are looking at has many dependencies, the display
can take some time as various build_id values need to be computed. If
you don't care about the accuracy of input and output file names, you
can add '--target no_build_id' to the command line. For instance, if you
want to look at the build script for the browser component (which
has a lot of dependencies), you can use:

  $ ./rbm/rbm showconf browser build --target alpha --target \
     torbrowser-linux-x86_64 --target no_build_id

The same type of commands can be used to look at any option values,
replacing build with the name of the option you want to look at. For
instance, if you want to know the output filename of tor on linux-x86_64
on the alpha channel, you can use:

  $ ./rbm/rbm showconf tor filename --target alpha --target \
    torbrowser-linux-x86_64

Building a single component
---------------------------

When you are working on some changes to a component, before creating a
full bundle with those changes, you might want to try a build of the
component you modified only.

This can be done with the following command:

   $ ./rbm/rbm build $component --target $channel --target $platform

See the previous section "Evaluating a component's build script" for a
list of possible values for $channel and $platform.

For instance, if you want to build tor for linux-x86_64 on the alpha
channel, you can run:

   $ ./rbm/rbm build tor --target alpha --target torbrowser-linux-x86_64

To find the resulting file from the build, you can use 'ls -ltr out/tor/'
to find the file with the last modification time.

Patching Firefox (or an other component)
----------------------------------------

If you want to test a Firefox patch, the easiest way to do it is to
copy the patch file to the projects/firefox/ directory, then edit
projects/firefox/config to add the new patch to the list of input_files:

  - filename: patch-for-XXXX.patch

Then edit projects/firefox/build to add a line somewhere (probably just
before running the configure script) to apply the patch:

  patch -p1 < $rootdir/patch-for-XXXX.patch

You can now run 'make torbrowser-testbuild' (or an other build target)
to start a build with the patch.

As an alternative, if you have your patch in a git repository, you can
edit projects/firefox/config to change the git_url option to point to
your git repository, and change the git_hash option to point to the
commit you want to build. If the new git_hash option is not pointing to
a signed tag, you will also need to comment the 'tag_gpg_id: 1' line.
The git_hash option can point to a tag, a commit, a branch or anything
that is understood by git.

If you specify a branch in the git_hash option, you don't need to prefix
it with a git remote name as all branches from the git repository
selected in the git_url option are created as local branches.

If you want to work on a local git repository, it is not a good idea to
work on the git_clones/firefox repository directly. Instead you should
create a separate git repository and update the git_url option with a
file:// URL pointing to your repository.

After changing the git_url option or when new commits have been added
to a branch, you should run "make fetch" before starting a build to get
the new commits. If you want to fetch new commits automatically when
starting a new build you can add a "fetch: 1" line to
projects/firefox/config.

Remember that the git_hash option has different definitions for alpha
and nightly builds, so you should modify the right one depending on
what type of build you are doing (see also the "Types of builds" section).

The first alternative is not working in case patches contain binary diffs,
which the patch command can't handle. However, working with submodules can
make the git branch alternative tricky to use. In that case you can resort
to a third option by installing git in the build container (via var/deps in
the respective config file). Then you can change to the submodule directory
in the build script and do a

  git apply patch-for-XXXX.patch

after adding the patch to the project's config file (see above).

The Firefox mozconfig files
---------------------------

In the gitian build, we are using the mozconfig files included in the
tor-browser.git repository (the .mozconfig, .mozconfig-mac and
.mozconfig-mingw files).

In the rbm build however, we need to make some small modifications to
those files, so we are instead using mozconfig files stored in the
projects/firefox/ directory, ignoring the .mozconfig files from the
tor-browser.git repository.

This could change in the future, when we are not using Gitian anymore.

Debugging a build error
-----------------------

If the build fails, a debugging shell will automatically be opened in
the container used for the build (unless you set debug to 0 in
rbm.local.conf).

Before using the debugging shell, you can first look at the build logs
in an other terminal (the filename of the log file is indicated in the
rbm output).

In the debug shell, you can type 'bash' to get a shell with completion.
You can then look at the 'build' file, and copy paste some of its commands
to debug the problem. If you need to edit files, vim is not installed
in the build containers, but vi can be used instead.

When you press ctrl+d (twice if you opened a bash shell) the debug shell
is closed and the container containing the failed build is removed.

The path to the container should be printed on the screen in case you
want to backup its rootfs to be able to look at it later.

Manually removing old containers
--------------------------------

When a build finishes or when you exit a debugging shell, the old
container should automatically be removed. In some cases however, for
example your computer is rebooted in the middle of a build, some old
container directories may be left in the tmp directory. Some of the
files in the container directories are owned by subordinate user ids
(see the subuid man page), which will prevent you from removing them
with your normal user id. To remove them you can open a container
shell (a new User namespace) using the following command:

   $ ./rbm/container run -- /bin/bash

From this shell you should be able to remove the old containers
directories in the tmp directory.

It is also possible to pass the rm command directly without opening a
shell:

   $ ./rbm/container run -- rm -Rf ./tmp/rbm-*

Testing an rbm patch
--------------------

When you are working on a patch to rbm, you might want to try a Tor
Browser build using your patched version of rbm. You could patch the
rbm in the rbm/ directory, however your patch can be reverted if you
use any of the makefile rules that does a 'git submodule update'.
To avoid this you can clone the rbm git repository to a separate
directory, where you will apply your patch. To do a build using your
patched rbm, take the command from the makefile, but replace $(rbm)
with the path to your patched rbm.

For example, if you want to try a Linux x86_64 alpha build, you can run:
  $ /path_to_rbm/rbm build release --target alpha --target \
                                                torbrowser-linux-x86_64

Types of builds: nightly, alpha, release, and testbuild
-------------------------------------------------------

The torbrowser-testbuild makefile target allows you to do a build
quickly in the testbuild directory, skipping the generation of all the
locales and the .mar files. This is useful during development.

In the case of Android builds, we are generating a multi-locale apk,
contrary to the desktop builds where we have one bundle for each locale.
Removing locales in a multi-locale bundle does not make a significant
difference in build time, therefore we still include all the locales in
the Android testbuild. There are also no .mar files generated in the
Android builds, so currently, in the Android case, the only difference
introduced by the testbuild target is the output directory.

By default the testbuild is based on the alpha build. All the options
can have a different definition for the alpha, release and nightly builds.
Usually the git_hash option has a different definition for the nightly
builds in order to point to the main branch.

If you want your testbuild target to create builds based on nightly
rather than alpha, you can edit your rbm.local.conf file and uncomment
the targets/torbrowser-testbuild definition.


code:
Description of makefile rules
=============================

torbrowser-release
------------------
Build Tor Browser for the release channel, for all supported platforms.
The resulting build can be found in directory release/$version.

torbrowser-release-{linux-x86_64,linux-i686,windows-i686,osx-x86_64,
                    android-armv7, android-x86,android-x86_64,
                    android-aarch64,desktop,android}
--------------------------------------------------------------------
Same as torbrowser-release, for the selected platform only. The "desktop"
platform means Linux, Windows and macOS. The "android" platform means
android for all architectures we support.

torbrowser-alpha
----------------
Build Tor Browser for the alpha channel, for all supported platforms.
The resulting build can be found in directory alpha/$version.

torbrowser-alpha-{linux-x86_64,linux-i686,windows-i686,osx-x86_64,
                  android-armv7,android-x86,android-x86_64,
                  android-aarch64,desktop,android}
------------------------------------------------------------------
Same as torbrowser-alpha, for the selected platform only.

torbrowser-nightly
------------------
Build Tor Browser for the nightly channel, for all supported platforms.
Many of the components will be based on the main branch. You might
want to run "make fetch" before, to fetch the latest commits from all
components.

The resulting build can be found in directory nightly/$date

torbrowser-nightly-{linux-x86_64,linux-i686,windows-i686,osx-x86_64,
                    android-armv7,android-x86,android-x86_64,
                    android-aarch64,desktop,android}
--------------------------------------------------------------------
Same as torbrowser-nightly, for the selected platform only.

torbrowser-testbuild
--------------------
Build a Tor Browser test build. To make the build faster, only the en-US
locale is created, without mar files, and without the source tarballs.
By default it is based on the alpha channel, but you can change that by
editing rbm.local.conf. The resulting build can be found in directory
testbuild.

torbrowser-{release,alpha,nightly,testbuild}-linux-x86_64-asan
--------------------------------------------------------------
Create a build for Linux x86_64 with Address Sanitizer enabled.

torbrowser-{release,alpha,nightly,testbuild}-src
------------------------------------------------
Create source tarballs for firefox, torbutton and tor-launcher.

basebrowser-*
-------------
The same rules for building Tor Browser also exist for building
Base Browser.

fetch
-----
Fetch new commits from all components. This is useful when you want to
build the nightly channel.

clean
-----
Clean files created during previous builds that are no longer used in
current builds. Before using this command, you first need to configure
in rbm.local.conf the channel(s) that you are building.

In other software, the clean rule is used to remove all build files,
but this is not the case here, where only obsolete build files are
removed. It is usually not necessary to remove all build files before
starting a new build as the need for rebuilding components is detected
automatically. If you want to remove all build files, you can remove
the out/ directory.

clean-dry-run
-------------
Print the files that would be removed when running 'make clean'.

signtag-{release,alpha}
-----------------------
Create a git signed tag for the selected channel, using the version and
build number defined as var/torbrowser_version and var/torbrowser_build.

incrementals-{release,alpha}
----------------------------
Create incremental mar files for an unsigned build in the release or
alpha channel. The list of versions we create incrementals from is
defined as var/torbrowser_incremental_from in rbm.conf.

incrementals-nightly
--------------------
Create incremental mar files for the current nightly build. The number
of previous versions we should generate incremental mars from is defined
as var/max_torbrowser_incremental_from in rbm.conf.

If you want to create incremental mars for a version that is not the
current one, you can define the environment variable
TORBROWSER_NIGHTLY_VERSION to the version you want to generate
incremental mars for.

dmg2mar-{release,alpha)
-----------------------
Generate updated mar files for the OSX bundles, from the dmg files, then
regenerate the OSX incremental mar files. You should run this after
signing the OSX dmg files.

update_responses-{release,alpha}
--------------------------------
Create update responses xml files for a signed build in the release or
alpha channel. The files can be found in a tar in the directory
{release,alpha}/update-responses.


code:
### Updating Gradle Dependencies
If additional Android dependencies are required by the project's build, then
the Gradle build will fail due to missing dependencies. To find out what the
missing dependencies are take the following steps.

1) Updating gradle dependencies for `application-services`, `android-components`
   or `fenix`

The following makefile rules can be used:

  make get_gradle_dependencies_list-application-services
  make get_gradle_dependencies_list-android-components
  make get_gradle_dependencies_list-fenix

which should create the gradle-dependencies-list.txt files in the
corresponding out/$project directory, using nightly's branch.

Note: `android-components` and `fenix` require modified `geckoview` and
`android-components` artifacts to compile successfully. In order to generate the
necessary dependencies for them you need to point to the respective
`android-components` and `fenix` commits our patches are based upon, too, first.
That way the code as Mozilla is shipping it is used for the Gradle dependencies
fetching which should avoid issues caused by broken builds (it is not guaranteed
that all the dependencies are already fetched at the point when the build
breaks). For `fenix` dependencies there is still a manual processing required
afterwards right now as e.g. our tor-android-service related artifacts are not
picked up.

2) Updating gradle dependencies for `geckoview`, `tor-android-service`
   or `tor-onion-proxy-library`

The `geckoview`, `tor-android-service` and `tor-onion-proxy-library`
projects don't have (yet) a makefile rule to generate their
gradle-dependencies-list.txt file, so a few more steps are needed.

For the geckoview project, comment out the following line in the project's
build file:

   export GRADLE_MAVEN_REPOSITORIES="file://$gradle_repo"

Also modify the gradle flags to include the debug option so the download logs will show up:

   export GRADLE_FLAGS="--no-daemon --debug"

Otherwise, when calling gradle in the project's build script replace the
`--offline` flag with `--debug` and remove any `-Dmaven.repo.local` arguments.

Finally, allow network access during the build by setting
`container/disable_network/build` to `0` in rbm.conf, and rerun the build.

Dependent artifacts will show up as downloads in the logs. You can pull out
these dependencies into a list by passing the log file to the gradle dependency
list script in the projects/common directory:

`projects/common/gen_gradle_deps_file.sh /path/to/log/file`

3) Copying the resulting `gradle-dependencies-list.txt`

Copy the resulting `gradle-dependencies-list.txt` (from step 1. or 2.)
over the one in the respective project. Then, in the project's config
file, increment the `var/gradle_dependencies_version` and make sure to
restore the project's build file back to original if you modified it.


code:
In order to be able to do offline builds for mobile we need to collect and
distribute Glean dependencies beforehand.[1]

There are three steps involved in the process:
  1) Finding out the `glean_parser` version
  2) Downloading and verifying the dependencies, and creating the tarball
  3) Pushing it to the sources mirror

1)

Running `make list_toolchain_updates-android-components` should tell
you which version of `glean_parser` is needed.

2)

Update `steps/create_glean_deps_tarball/version` in `projects/glean/config`
to the version from step 1 above.

Run this command to generate the tarball:

  make create_glean_deps_tarball

There is no really good way (yet) to verify that `pip` gave us actually what we
wanted. We can download `glean_parser` and its dependencies using a different
network path, though, e.g. by using `torsocks`. Comparing the results of both
downloads should yield the exact same binaries.

If you want to use torsocks, you can use this command:

  make create_glean_deps_tarball-with_torsocks

3)

Upload the compressed tarball to the sources mirror and update the SHA-256 sum of
it as well as the `glean_parser` version in the respective projects.

[1] See: https://bugzilla.mozilla.org/show_bug.cgi?id=1651662 for details.
[2] https://github.com/mozilla/glean


code:
1. Go to https://gitlab.torproject.org/tpo/web/manual/-/jobs/
2. Open the latest build stage
3. Download the artifacts (they come in a .zip file).
4. Rename it to manual_$PIPELINEID.zip
5. Upload it to people.tpo
6. Update projects/manual/config
   6.1 Change the version to $PIPELINEID
   6.2 Update the hash in the input_files section
   6.3 Update the URL if it was different from the previous uploader


code:
# vim: filetype=yaml sw=2
version: '[% c("abbrev") %]'
git_url: https://github.com/sclevine/agouti
git_hash: e3f6c97a4077ad821daab49db8172cf9f6690faf #v3.0.0
filename: '[% project %]-[% c("version") %]-[% c("var/osname") %]-[% c("var/build_id") %].tar.gz'

build: '[% c("projects/go/var/build_go_lib") %]'

container:
  use_container: 1

var:
  go_lib: github.com/sclevine/agouti

input_files:
  - project: container-image
  - name: go
    project: go

path: projects/agouti/config


code:
#!/bin/bash
[% c("var/set_default_env") -%]

tar xvf [% project %]-[% c('version') %].tar.gz

mkdir -p /var/tmp/dist/[% project %]
cd /var/tmp/dist/

cp -a $rootdir/[% project %]*/[% project %]/*.py ./[% project %]/

[% c('tar', {
        tar_src => '.',
        tar_args => '-czf ' _ dest_dir _ '/' _ c('filename'),
        }) %]

path: projects/aiohttp/build


code:
# vim: filetype=yaml sw=2
version: 3.5.4
git_url: https://github.com/aio-libs/aiohttp.git
git_hash: f6f647eb828fa738610d61481f11fa51e42599e9
# TODO: This Git repo doesn't use GPG sigs.  We should pester them about that.
filename: "[% project %]-[% c('version') %]-[% c('var/build_id') %].tar.gz"
container:
  use_container: 1
input_files:
  - project: container-image

path: projects/aiohttp/config


code:
#!/bin/bash
[% c("var/set_default_env") -%]

tar xvf [% project %]-[% c('version') %].tar.gz

mkdir -p /var/tmp/dist/
cd /var/tmp/dist/

cp -a $rootdir/[% project %]*/[% project %] ./[% project %]

[% c('tar', {
        tar_src => '.',
        tar_args => '-czf ' _ dest_dir _ '/' _ c('filename'),
        }) %]


code:
# vim: filetype=yaml sw=2
version: 0.2.2
git_url: https://github.com/romis2012/aiohttp-socks.git
git_hash: 3252f4bdd37fb9a7360481977f800189cb3e3aca
# TODO: This Git repo doesn't use GPG sigs.  We should pester them about that.
filename: "[% project %]-[% c('version') %]-[% c('var/build_id') %].tar.gz"
container:
  use_container: 1
input_files:
  - project: container-image


path: projects/aiorpcx/build


code:
# vim: filetype=yaml sw=2
version: 0.18.3
git_url: https://github.com/kyuupichan/aiorpcX.git
git_hash: 4f39366e5dee3fd0a857e53f383c628807cd2715
# TODO: This Git repo doesn't use GPG sigs.  We should pester them about that.
filename: "[% project %]-[% c('version') %]-[% c('var/build_id') %].tar.gz"
container:
  use_container: 1
input_files:
  - project: container-image

path: projects/aiorpcx/config


code:
#!/bin/bash
[% c("var/set_default_env") -%]
[% pc(c('var/compiler'), 'var/setup', { compiler_tarfile => c('input_files_by_name/' _ c('var/compiler')) }) %]
distdir=/var/tmp/dist
builddir=/var/tmp/build/[% project %]
mkdir $distdir/[% project %]
mkdir /var/tmp/build

tar -C $GRADLE_HOME -xf $rootdir/[% c('input_files_by_name/gradle') %]

cat > get-moz-build-date << "EOF"
[% INCLUDE "get-moz-build-date" %]
EOF

[% IF !c("var/fetch_gradle_dependencies") %]
  tar -C $distdir -xf [% c('input_files_by_name/geckoview') %]
  gradle_repo=$rootdir/[% c('input_files_by_name/gradle-dependencies') %]
  # This overwrites the release, beta, and nightly geckoview .aar files in our
  # gradle-dependencies directory to be sure that only the one we provide is
  # getting used when building beta/release code.
  find $gradle_repo -type f -name geckoview*omni*.aar -exec cp -f $distdir/geckoview/geckoview*omni*.aar {} \;
  cp -r $gradle_repo/dl/android/maven2/* $gradle_repo
  cp -r $gradle_repo/maven2/* $gradle_repo
  cp -r $gradle_repo/m2/* $gradle_repo
  # Put the Kotlin/Native compiler at the right place, see:
  # tor-browser-build#40217.
  kotlin_dir=/home/rbm/.konan
  mkdir $kotlin_dir
  find $gradle_repo -type f -name kotlin-native-prebuilt-linux*tar.gz -exec tar -C $kotlin_dir -xaf {} \;
  tar -C $distdir -xf [% c('input_files_by_name/application-services') %]
  # XXX: We could be smarter and just copy over the projects that are needed
  # according to the Gradle dependencies list.
  cp -rf $distdir/application-services/maven/org $gradle_repo
[% END %]
tar -C /var/tmp/build -xf [% project %]-[% c('version') %].tar.gz

cd $builddir-[% c('version') %]

# Avoid call to `getGitHash()`, which will fail b/c it invokes git, which is absent in
# the build container. Instead just inject the commit hash, which we know!
sed -i 's/getGitHash()/\"[% c('abbrev') %]\"/' components/support/base/build.gradle
sed -i 's/tag = getGitHash()/tag = \"[% c('abbrev') %]\"/' publish.gradle

[% IF c("var/fetch_gradle_dependencies") %]
  # XXX: fetching deps for `assembleGeckoBeta -x lint` by using that same target
  # results in some missing dependencies for yet unknown reasons. Thus, we use
  # `assemble` instead for now.
  gradle --debug --no-daemon assemble > gradle.log 2>&1 || ( cat gradle.log; exit 1 )
  $rootdir/gen_gradle_deps_file.sh gradle.log
  mv gradle-dependencies-list.txt '[% dest_dir _ "/" _ c("filename") %]'
[% ELSE %]
  # Prepare Glean dependencies for offline build
  tar -xf $rootdir/[% c('input_files_by_name/glean') %]
  # We need to set `LC_ALL` and `LANG` to something that is not ASCII as encoding
  # otherwise `click` barfs. See: https://click.palletsprojects.com/python3/
  export LC_ALL=C.UTF-8
  export LANG=C.UTF-8
  patch -p1 < $rootdir/mavenLocal.patch

  eval $(perl $rootdir/get-moz-build-date [% pc("fenix", "var/copyright_year") %] [% pc("fenix", "var/torbrowser_version") %])
  if [ -z $MOZ_BUILD_DATE ]
  then
    echo "MOZ_BUILD_DATE is not set"
    exit 1
  fi

  # Increase the maximum memory usage, for when compiling with --disable-install-strip
  gradle_args="--offline --no-daemon -Dmaven.repo.local=$gradle_repo -Dorg.gradle.jvmargs=-Xmx8g"

  gradle $gradle_args assembleGecko -x lint
  gradle $gradle_args publish

  # We only need the archives and .pom files for now.
  find components -regex '.*[% c('var/android_components_version') %].\(aar\|jar\|pom\)' -exec cp --parents {} $distdir \;

  # Package
  cd $distdir
  # Package just the maven directory structure for easier replacement of
  # Mozilla's android-components.
  find components -type d -name maven -exec cp -rf {} [% project %] \;
  [% c('tar', {
        tar_src => [ project ],
        tar_args => '-czf ' _ dest_dir _ '/' _ c('filename'),
    }) %]
[% END %]

path: projects/android-components/build


code:
# vim: filetype=yaml sw=2
version: '[% c("abbrev") %]'
filename: '[% project %]-[% c("version") %]-[% c("var/build_id") %].tar.gz'
git_hash: '[% project %]-[% c("var/android_components_version") %]-[% c("var/torbrowser_branch") %]-1-build3'
git_url: https://gitlab.torproject.org/tpo/applications/android-components.git
tag_gpg_id: 1
gpg_keyring: torbutton.gpg
variant: '[% IF c("var/release") %]Release[% ELSE %]Beta[% END %]'
container:
  use_container: 1

var:
  android_components_version: 99.0.3
  torbrowser_branch: 11.5
  # This should be updated when the list of gradle dependencies is changed.
  gradle_dependencies_version: 35
  gradle_version: 7.0.2
  glean_parser: 5.0.1
  git_branch: '[% project %]-[% c("var/android_components_version") %]-[% c("var/torbrowser_branch") %]-1'
  arch_deps:
    - python3-venv

targets:
  nightly:
    git_hash: '[% c("var/git_branch") %]'
    tag_gpg_id: 0

input_files:
  - project: container-image
    pkg_type: build
  - name: '[% c("var/compiler") %]'
    project: '[% c("var/compiler") %]'
    pkg_type: build
  - project: gradle
    name: gradle
    pkg_type: build
  - name: geckoview
    project: geckoview
    enable: '[% !c("var/fetch_gradle_dependencies") && c("var/android_single_arch") %]'
  - name: geckoview
    project: geckoview
    pkg_type: merge_aars
    enable: '[% !c("var/fetch_gradle_dependencies") && !c("var/android_single_arch") %]'
  - name: application-services
    project: application-services
    enable: '[% !c("var/fetch_gradle_dependencies") %]'
  - filename: 'gradle-dependencies-[% c("var/gradle_dependencies_version") %]'
    name: gradle-dependencies
    exec: '[% INCLUDE "fetch-gradle-dependencies" %]'
    enable: '[% !c("var/fetch_gradle_dependencies") %]'
  - filename: mavenLocal.patch
    enable: '[% !c("var/fetch_gradle_dependencies") %]'
  - URL: '[% pc("glean", "var/glean_wheels_url/" _ c("var/glean_parser"), { error_if_undef => 1 }) %]'
    name: glean
    sha256sum: '[% pc("glean", "var/glean_wheels_sha256sum/" _ c("var/glean_parser"), { error_if_undef => 1 }) %]'
    enable: '[% !c("var/fetch_gradle_dependencies") %]'
  - filename: gen_gradle_deps_file.sh
    enable: '[% c("var/fetch_gradle_dependencies") %]'

steps:
  list_toolchain_updates:
    git_url: https://github.com/mozilla-mobile/android-components.git
    git_hash: releases/99.0
    #tag_gpg_id: 0
    container:
      use_container: 0
    var:
      get_glean_version: |
        #!/bin/bash
        read -d '' p << 'EOF' || true
        if (m/const\\sval\\smozilla_glean\\s=\\s"([^"]+)"/) {
          print $1;
          exit;
        }
        EOF
        perl -ne "$p" < buildSrc/src/main/java/Dependencies.kt
      glean_version: '[% exec(c("var/get_glean_version")) %]'
    input_files:
      - name: glean
        project: glean
        pkg_type: src

  get_gradle_dependencies_list:
    filename: 'gradle-dependencies-list-[% c("var/android_components_version") %].txt'
    get_gradle_dependencies_list: '[% INCLUDE build %]'
    git_hash: '[% c("var/git_branch") %]^{/Bug 40023: Stop PrivateNotificationService}'
    var:
      fetch_gradle_dependencies: 1

path: projects/android-components/config