code:
Display
=======

Until now, the exploration of the Allwinner A64 SoC was mainly concerned with
the Pine-A64-LTS board, which offers developer conveniences like booting over
the network, or easily accessible reset and GPIO pins.
The upcoming topics require us to switch out development workflow from the
Pine-A64-LTS board to the real deal - the PinePhone. This adjustment is
covered by a
[https://genodians.org/nfeske/2021-09-20-pine-fun-pinephone-boot - dedicated article].
With those precautions taken, it is time to turn our attention to the arguably
most challenging parts of the hardware, namely the display subsystem.

Why do I regard this part as the most challenging? The display subsystem of
a mobile device is not solely one peripheral but a conglomerate of several
devices that are (more or less) under software control and need to work
together. The complexity of the interplay and domain-specific terminology
can be quite staggering. MIPI, DSI, PLL, PHY, panel, plane, channel,
connector, encoder, regulator, mixer, CRTC, RSB, TCON, LVDS, PWM. Are you
still with me?

Driving the display with a bare-bones Linux kernel
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Not knowing much about the internal structure of the display hardware,
it is good to take Linux as a working starting point. When booting
Armbian Linux, the display works after all. Observing the Linux boot, the
following messages seem obviously be related to the display.

! [    5.936404] Console: switching to colour frame buffer device 170x48
! [    5.955920] simple-framebuffer be000000.framebuffer: fb0: simplefb registered!
! [    5.959687] mmc1: new SDHC card at address 0001
! [    5.967848] sun4i-drm display-engine: bound 1100000.mixer (ops 0xffff800010e340c0)
! [    5.979490] sun4i-drm display-engine: bound 1200000.mixer (ops 0xffff800010e340c0)
! [    5.990232] sun4i-drm display-engine: No panel or bridge found... RGB output disabled
! [    6.000377] sun4i-drm display-engine: bound 1c0c000.lcd-controller (ops 0xffff800010e2f8d0)
! [    6.012100] sun4i-drm display-engine: bound 1c0d000.lcd-controller (ops 0xffff800010e2f8d0)
! [    6.026726] sun8i-dw-hdmi 1ee0000.hdmi: Detected HDMI TX controller v1.32a with HDCP (sun8i_dw_hdmi_phy)
! [    6.117391] sun8i-dw-hdmi 1ee0000.hdmi: registered DesignWare HDMI I2C bus driver
! [    6.130875] sun4i-drm display-engine: bound 1ee0000.hdmi (ops 0xffff800010e333f8)
! [    6.200146] fb0: switching to sun4i-drm-fb from simple
! [    6.210896] Console: switching to colour dummy device 80x25
! [    6.216994] [drm] Initialized sun4i-drm 1.0.0 20150629 for display-engine on minor 0
! [    6.603061] Console: switching to colour frame buffer device 170x48
! [    6.641668] sun4i-drm display-engine: [drm] fb0: sun4i-drmdrmfb frame buffer device

Correlating those words with the device tree brings us to the device node
of the so-called display engine.

!  de: display-engine {
!   compatible = "allwinner,sun50i-a64-display-engine";
!   allwinner,pipelines = <&mixer0>, <&mixer1>;
!   status = "disabled";
!  };

The device node's 'compatible' string, in turn, draws the connection to the
part of the Linux kernel that is of interest to us.

! linux$ grep -r "allwinner,sun50i-a64-display-engine"
! drivers/gpu/drm/sun4i/sun4i_drv.c: { .compatible = "allwinner,sun50i-a64-display-engine" },

So _drivers/gpu/drm/sun4i/_ seems to be good starting point for exploration.

Having identified the driver code that want to execute for sure, we have
to answer two questions:

# What are the in-kernel dependencies of this driver code? All those
  dependencies are of interest to us because they are prerequisites.

# Which parts of the Linux kernel are unrelated to the driver functionality?
  We would like to drop those parts to narrow our view on the interesting
  driver code as much as possible.

The investigation of those two questions is an iterative process that follows
the pattern discussed in Section [Taking Linux out for a walk].
In our present case, the success criterion of our custom-built bare-bones
Linux kernel is the display of the little Tux at the top of the screen.
Our kernel won't need anything else, Tux is enough.

To find the smallest possible selection of kernel configuration parameters,
the bisecting approach that we previously used for isolating the network
driver becomes handy again. Without further ado, here comes the solution
as supplement for our
[https://github.com/genodelabs/genode-allwinner/blob/master/src/a64_linux/target.inc - target.inc].

! # framebuffer driver
! LX_ENABLE += DRM DRM_SUN4I DRM_SUN8I_MIXER DRM_SUN8I_DW_HDMI
!
! # determined by bisecting kernel configuration options (needed by fb driver)
! LX_ENABLE += CMA DMA_CMA MFD_AXP20X_RSB REGULATOR REGULATOR_AXP20X
! LX_ENABLE += PROC_FS SYSFS
!
! # to automatically set up screen mode at boot time
! LX_ENABLE += FRAMEBUFFER_CONSOLE
!
! # show Tux
! LX_ENABLE += LOGO

Don't ask how often I operated the reset button to find this global
minimum of kernel configuration parameters.

With the bare-bones Linux kernel running, we can use Busybox to
interactively poke around with the driver. It is nice to see some
response, like the display going dark.

! / # mkdir proc
! / # mkdir sys
! / # mount -tproc proc
! / # mount -tsysfs sys
! / # cd /sys/devices/platform/display-engine/graphics
! /sys/devices/platform/display-engine/graphics # cd fb0/
! /sys/devices/platform/display-engine/graphics/fb0 # echo 1 > blank
! /sys/devices/platform/display-engine/graphics/fb0 # echo 0 > blank

To further tighten our focus, the next step is the pruning of the
device tree using the DTS-extract tool discussed
in Section [Pruning device trees].
For reference, the device tree extracted with following arguments suffices
to allow Linux to drive the display. The central element is the Allwinner
[https://linux-sunxi.org/images/7/7b/Allwinner_DE2.0_Spec_V1.0.pdf - Display Engine] (DE).

! genode$ ./tool/dts/extract --select /backlight --select de --select dsi \
!                            flat_pinephone.dts

The resulting device-tree nodes at a glance:

[image de_dts]
  Device-tree nodes related to the display engine and the DSI output.

Let's not get scared. A glossary of the terminology seen the picture
should lift the clouds a bit.

* The PIO device controls general-purpose I/O pins. We explored this
  device [https://genodians.org/nfeske/2021-04-29-platform-driver - previously].
  All pins are naturally related to the PIO controller but only a few of them
  are actually relevant for the display. So we can consider the large number of
  pin nodes as just noise for the most part.

* All nodes prefixed with 'r_' belong to a certain part of the SoC that is
  referred to as "RTC" (real-time clock). Those parts are powered
  independently from the ARM application processor and are meant to be driven
  by a small microcontroller called [https://linux-sunxi.org/AR100 - AR100]
  that ought to manage power.

* The 'r_rsb' controller is a two-wire bus similar to I2C that connects the
  A64 SoC with a separate power management chip (PMIC, or AXP803). This chip
  is responsible for generating various voltages on the board. For driving
  the display, this chip is important because it provides the power for the
  LCD display part, which is off by default. So in order to power the
  display, the driver needs to talk via the RSB bus to the PMIC chip.

* TCON0 and TCON1 are two interfaces of the SoC where a display can be
  connected. So the SoC supports dual-head scenarios like driving an HDMI
  display and an LCD display at the same time.

* CCU stands for central clock unit. It controls the configuration and
  gating of all kinds of internal clocks and reset lines. The R_CCU refers
  to the clocks and reset lines associated with the RTC part of the chip.

* The backlight is a separate device. Its brightness is controlled via
  pulse-width-modulated digital signal generated by the 'r_pwm' device.

* The DPHY is responsible for the physical link of the digital connector.
  Electronics stuff.

* All these mixer nodes are related to the display engines ability to
  blend multiple images together.

When booting our bare-bones Linux kernel with the pruned device tree,
Tux shows up, and the last life signs of the kernel are the following
messages.

! sun4i-drm display-engine: No panel or bridge found... RGB output disabled
! sun4i-drm display-engine: bound 1c0c000.lcd-controller (ops 0xffffffc010272138)
! sun4i-drm display-engine: bound 1c0d000.lcd-controller (ops 0xffffffc010272138)
! sun4i-drm display-engine: bound 1ca0000.dsi (ops 0xffffffc010275810)
! [drm] Initialized sun4i-drm 1.0.0 20150629 for display-engine on minor 0
! sun4i-drm display-engine: [drm] Cannot find any crtc or sizes
! Console: switching to colour frame buffer device 90x90
! sun4i-drm display-engine: [drm] fb0: sun4i-drmdrmfb frame buffer device
! sun6i-mipi-dsi 1ca0000.dsi: Attached device xbd599
! panel-sitronix-st7703 1ca0000.dsi.0: 720x1440@55 24bpp dsi 4dl - ready

[image tux_pinephone]
  Hello my friends, nice to see you!

The logo. That's all we want from the Linux kernel for now.

A monolithic display driver running on Genode
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In Section [Networking],
we covered the path from a down-stripped bare-bones Linux kernel to a
Genode driver component using Genode's device-driver environment.
The same principle method works equally well for transplanting the display
driver,

* Selecting the relevant driver sources,
* Generating dummy implementations using the _tool/dde_linux/create_dummies_ tool,
* Taking cues from the existing DDE-Linux-based drivers for supplementing
  custom Linux emulation code,
* Using a custom run script (appropriately named _framebuffer_pinephone.run_)
  as a dedicated test bed for the driver,
* Resolving the access to device resources like memory-mapped I/O ranges and
  interrupts by enhancing the platform-driver configuration step by step.

For the test bed, the
[https://github.com/genodelabs/genode/tree/master/repos/os/src/test/framebuffer - framebuffer test]
is a handy tool. When combined with a display driver, it presents a sequence
of colors and patterns, and it nicely highlights the border of the screen to
verify the entirety of the framebuffer is indeed visible.

[image fb_test_pinephone]
  The _framebuffer_pinephone.run_ scenario.

In the test scenario, the test-framebuffer component takes the place of the
GUI server, providing a capture service. The framebuffer driver plays the
role of a mere capture client that captures the synthetic pixel data generated
by the test-framebuffer component.

For interfacing the Linux kernel code with Genode's capture session interface,
the driver uses the following kernel function declared in _linux/fb.h_
as a hook to get hold of the pixel data.

! int register_framebuffer(struct fb_info *fb_info);

This is arguably quite primitive and does not allow the use of many
driver features. However, we have to start somewhere.

The framebuffer driver interacts with device hardware through the platform
driver as introduced
in Section [One Platform driver to rule them all].
The following picture shows all the devices the driver interacts with.

[tikz img/pinephone_fb_drv]

This octopus-resembling creature raises two tricky questions.

* Is it reasonable to entrust the driver with the access to all those devices?
  Given the complexity of the ported driver code, this seems risky, doesn't
  it?

* Some of the devices seem to be relevant to other drivers, too. If we grant
  the framebuffer driver exclusive access to those devices, how can we
  combine the framebuffer driver with other drivers running on the same
  system?

We will come to solving those questions soon. For now, let's enjoy the
colorful display for a bit.

For reference, the entire commit for the monolithic framebuffer driver can
be found
[https://github.com/genodelabs/genode-allwinner/commit/c1d088ef5d9d4c82ffa761f672c33f70431dec31 - here].


code:
Pruning device trees
====================

We briefly touched the treasure trove called device trees in the
previous section. To leverage the
wealth of information for the development and porting of Genode device
drivers, this article introduces a handy new tool set.

As summarized in the previous article, device-tree files as found at
Linux source tree under _arch/<arch>/boot/dts/_ provide both
a structural description of an SoC and parametrization data for individual
device drivers. It goes without saying that this information is extremely
valuable. On the other hand, the encoding of the information in the form
of so-called _Devicetree Specification_
([https://github.com/devicetree-org/devicetree-specification/releases/download/v0.3/devicetree-specification-v0.3.pdf - PDF])
files is not ideal for us.

The authors of DTS files anticipate a monolithic kernel where a global view of
the system is natural. In contrast, Genode fosters a strict separation of
drivers from each other where each driver gets to see only a tiny part of the
picture. With a DTS file of more than 1600 lines (as for the Pine-A64-LTS)
board given, it is really hard to see to see clear lines of responsibilities
between drivers. This is where Genode's tool at _tool/dts/extract_
comes into play. Just for reference, usage information are provided by
executing the tool without arguments.

Let's assume we have generated an all-encompassing DTS file
_flat_pine64lts.dts_ for our board via the C preprocessor as
described in Section [Device-tree treasure trove].

The _tool/dts/extract_ utility allows us to generate a dot graph from the
source, which can be processed by the [https://graphviz.org/ - Graphviz] dot
tool to generate a PNG file.

! tool/dts$ ./extract --dot-graph flat_pine64lts.dts > pine64.dot
! tool/dts$ dot -Tpng pine64.dot > pine64.png

[image pine64_dts]
  Does this count as generative art?

Even though the picture presents only a tiny fraction of the information
present in the DTS file - neither any properties, nor device addresses, nor
unlabeled nodes are shown - it is too overwhelming to be useful.

Let's say we are interested in the porting of the ethernet driver. In the
previous article we already manually walked the DTS tree and spotted the
corresponding node along the way. With the '--labels' option, the _extract_
tool provides a convenient way to get an overview of the nodes present in
the tree.

! tool/dts$ ./extract --labels flat_pine64lts.dts
! ...
! uart1 /soc/serial@1c28400
! spi0_pins /soc/pinctrl@1c20800/spi0-pins
! ve_sram /soc/syscon@1c00000/sram@1d00000/sram-section@0
! reg_aldo1 /soc/rsb@1f03400/pmic@3a3/regulators/aldo1
! emac /soc/ethernet@1c30000
! uart2 /soc/serial@1c28800
! lradc /soc/lradc@1c21800
! ...

Each line presents a label accompanied with the corresponding path of the
device node. Of course, the command is best combined with grep.

! tool/dts$ ./extract --labels flat_pine64lts.dts | grep ether
! emac /soc/ethernet@1c30000
! mdio /soc/ethernet@1c30000/mdio
! ext_rgmii_phy /soc/ethernet@1c30000/mdio/ethernet-phy@1

The 'emac' label should ring a bell from the previous article.
To find out about the interaction of the emac device with the other
parts of the device tree, the extract tool allows us to generate a
new DTS tree with only a selection of devices and their dependencies
present.

! tool/dts$ ./extract --select emac flat_pine64lts.dts > emac.dts

From the more of 1600 lines of the original DTS file, the result
comprises only about 200 lines. This amount of information can be
digested without choking.

! tool/dts$ wc -l emac.dts
! 213 emac.dts
! tool/dts$ ./extract --dot-graph emac.dts > emac.dot
! tool/dts$ dot -Tpng emac.doc > emac.png

With a few final manual tweaks of the layout parameters, one can get a picture
as nice as this.

[image emac_dts]
  A sudden moment of clarity.

Finally, we can create a device-tree binary out of the pruned device-tree
source.

! tool/dts$ dtc -Idts emac.dts > emac.dtb

The device-tree compiler does not complain, which gives us the reassurance
that the tree is in a healthy state after the brutal pruning.

Test-driving Linux with the tuned device tree
---------------------------------------------

In order to successfully boot the Linux kernel, the supplied device tree
needs a few mandatory ingredients.
First, we need to supply the information about the timer to be used
by the kernel, which is provided by the '/timer' node.
Furthermore, '/chosen' node contains the
'stdout-path' property, which tells the kernel where messages should go.
In the device tree for the Pine-A64-LTS board, it is defined as
! stdout-path = "serial0:115200n8";
The 'serial0' part of the string refers to an entry of the '/aliases' node,
which is defines as follows. Note that it contains an alias referring to
our Ethernet device 'emac'.

! aliases {
!   ethernet0 = &emac;
!   serial0 = &uart0;
!   serial1 = &uart1;
!   serial2 = &uart2;
!   serial3 = &uart3;
!   serial4 = &uart4;
! };

The following command extracts a device tree featuring those mandatory nodes.
Since the 'emac' device is implicitly pulled in by the '/alias' node, we
don't need to explicitly specify the '--select emac' argument.

! tool/dts$ ./extract --select /chosen --select /aliases --select /timer \
!                     flat_pine64lts.dts

The resulting device tree, once compiled into its dtb representation, suffices
to boot the hand-crafted Linux kernel we built in the previous article. It
looks as follows.

[image emac_lx_dts]
  This fine-tuned device tree suffices for using Ethernet with Linux.

The '/timer', '/chosen', and '/aliases' nodes are not shown because the graph
omits unlabeled nodes. It still contains a few obviously unneeded parts such
as the ones relates to the 'simplefb' nodes (defined inside the '/chosen'
node) or the 'uart1' to 'uart4' nodes (referenced by the '/aliases' node). To
remove those, one may consider cutting off those dependencies by commenting
out those parts in the _flat_pine64lts.dts_ file.

Prospects
---------

Even though the primary motivation behind the new tooling is the pruning of
device trees to attain driver-specific miniature device trees to be fed to
ported Linux driver code, I already see myself using the graph feature as an
aid for understanding SoC hardware. As of now, the graph is admittedly just a
quick hack. The dot language allows for generating nicely structured images,
e.g., presenting child nodes contained in parent nodes. It's also tempting to
generate XML configuration data for Genode's platform driver from the
device-tree information.

repo_name: Fhiss/genode


code:
Introduction
############

This document complements the Genode Foundations book with
low-level hardware-related topics. It is primarily intended for integrators
and developers of device drivers. Before studying the Genode Platforms
material, it is highly recommended to give the Genode Foundations book a read.
The book can be downloaded at [https://genode.org].

In this first edition, the document features a practical guide for the steps
needed to bring Genode to a new ARM SoC. The content is based on the ongoing
Pine Fun article series at [https://genodians.org - Genodians.org].
Note that the document is not set in stone. We plan to continuously extend it
with further practical topics as we go.


code:
Kernel packaging and testing
============================

With our toes still a bit frozen from testing the waters of the user land, we
now take the remaining steps towards a cultivated Genode life, largely
automating our work flow, packaging the kernel, and testing the platform like
there is no tomorrow.

During the initial user-land bring-up described in the
previous section, the process of
building a system image, loading the image onto the board, and obtaining log
output required quite a few manual steps: Starting picocom, issuing
the 'make run/log' command, copying the system image to the TFTP directory,
resetting the board, scanning the log output with our eyes. Some parts of this
process can be streamlined.

Accelerating our run-script workflow
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

First, instead of manually instructing the run tool to produce a uImage
instead of an ELF image at the command line, we can place the following
line into our _etc/built.conf_ file.

! RUN_OPT += --include image/uboot

Second, we can let the run tool manage the execution of the 'picocom' command
instead of manually starting the it by adding the following line:

! RUN_OPT += --include log/serial

This way, we can skip the step of spawning of picocom. But more importantly,
the run tool becomes able to detect the success of run scripts automatically!
So the part about "scanning the log output with our eyes" becomes much more
relaxing.

Third, the copying of the uImage file into the TFTP directory can be
automated by adding the following lines.

! RUN_OPT += --include load/tftp
! RUN_OPT +=   --load-tftp-base-dir /var/lib/tftpboot
! RUN_OPT +=   --load-tftp-offset-dir /$(BOARD)

Upon the next execution of the 'make run/log KERNEL=hw BOARD=pine_a64lts'
command, a new symbolic link appears at _/var/lib/tftpboot_.

! $ ls -la /var/lib/tftpboot/pine_a64lts
! lrwxrwxrwx ... /var/lib/tftpboot/pine_a64lts -> /.../build/arm_v8a/var/run/log/uImage

The symlink is updated each time a run script is executed. It always points
to the most recently built system image.
By setting the U-Boot bootcmd to load _/var/lib/tftpboot/pine_a64lts_, the board
will automatically fetch the most recently built system image.

! => env edit bootcmd
! edit: bootp 10.0.0.32:/var/lib/tftpboot/pine_a64lts ; bootm
! => env save
! Saving Environment to FAT... OK

With these little tweaks, the work flow with run scripts becomes almost
fully automated. The only remaining manual steps are:

# Issuing the 'make run/...' command at the build directory.
# Once the message 'Terminal ready' appears, pressing the reset button
  on the board.

This is a perfectly acceptable level of convenience.
If you want to go even further, you may find the following two articles
inspiring.

:Remote-control your test target via power scripts:

  [https://genodians.org/chelmuth/2019-03-13-powerplug]

:Exploring Genode Base HW with Raspberry Pi - further workflow automation:

  [https://genodians.org/tomga/2019-08-13-rpi-automation]

There is no better way to celebrate the new level of efficiency than to
test-drive a few hand-picked run scripts.

Stress-testing the init component
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The log scenario that we executed so far already employed Genode's init
component, which is the first (and only) component immediately started by
core. Init constructs a subsystem of components according to a
configuration in XML form. The init configuration for the log scenario
was rather primitive. There exists comprehensive test that exercises
the entire feature set of init by running a dynamically configured
instance of init as a child of init.
The test is hosted at the _repos/os/_ repository and has the form of a
deployable package (more on that later).
You can find the test's ingredients at _recipes/pkg/test-init/_ (package with
the runtime description) _src/test/init/_ (driver for executing a test
sequence) _recipes/raw/test-init/test-init.config_ (sequence executed by the
test driver).

That's probably be a bit overwhelming. Let's better just try it out. To run a
test package, the _os/run/test.run_ script becomes handy. We can use it as
follows, passing the name of the test package as 'PKG' argument.

! build/arm_v8a$ make run/test PKG=test-init KERNEL=hw BOARD=pine_a64lts
! ...
! ...
! genode build completed
! ...
! Terminal ready

At this point, we have to press the reset button of the board.

! ...
! ... log output of more than 200 test steps
! ...
! [init -> test -> test-init] --- test complete ---
! [init -> test] child "test-init" exited with exit value 0
!
! Run script execution successful.

What else could we ask for! When examining the log output, you can get a
glimpse of the feature set at work: Addition and deletion of subsystems,
changing access-control policies in the fly, reconfiguring child components,
chaining services, balancing resources among the components, heartbeat
monitoring, and exit handling.

As another noteworthy detail, in contrast to the simple log test, the init
test employs a timer at the user level. Since the test passed, we have the
confirmation that the in-kernel timer driver and interrupt-controller driver
work in principle.

Timer accuracy test
~~~~~~~~~~~~~~~~~~~

Speaking of the timer, it is generally not enough to know that the timer
works in principle but also that it is precise, which comes down to its
correct calibration. Genode provides a ready-to-use test that compares
the notion of time as observed by the Genode system with the wall-clock time
as known on your host system. The run script for this low-level test
is located at _repos/base/run/timer_accuracy.run_.

! build/arm_v8a$ make run/timer_accuracy KERNEL=hw BOARD=pine_a64lts
! ...
! ...
! Genode 20.11-197-g635985f542 <local changes>
! 2010 MiB RAM and 64533 caps assigned to init
! [init -> test-timer_accuracy]
! [init -> test-timer_accuracy]
! [init -> test-timer_accuracy]
! [init -> test-timer_accuracy]
! [init -> test-timer_accuracy]
! [init -> test-timer_accuracy]
! [init -> test-timer_accuracy]
! [init -> test-timer_accuracy]
! [init -> test-timer_accuracy]
! [init -> test-timer_accuracy]
! Good: round 1, host measured 1000 ms, test measured 1008 ms
! Good: round 2, host measured 2000 ms, test measured 2000 ms
! Good: round 3, host measured 3000 ms, test measured 3006 ms
! Good: round 4, host measured 4000 ms, test measured 3997 ms
! Good: round 5, host measured 5000 ms, test measured 5003 ms
! Good: round 6, host measured 6000 ms, test measured 6007 ms
! Good: round 7, host measured 7000 ms, test measured 6995 ms
! Good: round 8, host measured 8000 ms, test measured 7984 ms
! Good: round 9, host measured 9000 ms, test measured 9005 ms
!
! Run script execution successful.

Be patient, the test can take up to 40 seconds.
The output looks just perfect.

Testing the dynamic linker
~~~~~~~~~~~~~~~~~~~~~~~~~~

The ldso test exercises the functionality of the dynamic linker, including
the execution of global constructors, transitive library dependencies,
exception handling across libraries, and cross-library symbol resolution.

! build/arm_v8a$ make run/test PKG=test-ldso KERNEL=hw BOARD=pine_a64lts
! ...
! ... build libc
! ...
! [init -> test] child "test-ldso" exited with exit value 123
!
! Run script execution successful.

Given that the init test succeeded, which already employed the dynamic linker,
the result is not surprising but reassuring.

Packaging the kernel
~~~~~~~~~~~~~~~~~~~~

All but the most basic run scripts leverage Genode's
package management often referred to as _depot_ and described in Section
5.5 "Package management" in the Genode Foundations book. A run script can
conveniently incorporate packaged components into a system scenario via the
'import_from_depot' function. When reviewing the various existing run scripts
in the Genode source tree for this function, one can spot the following
pattern.

! import_from_depot [depot_user]/src/[base_src] \
!                   [depot_user]/...

Each argument denotes a path of a depot archive. The 'depot_user' and
'base_src' are function calls. The 'depot_user' function returns the name of
the originator/creator of the given depot archive. It returns 'genodelabs' by
default and can be customized in the _etc/build.conf_ file.

The 'base_src' function returns the archive name of the so-called "base"
source archive for a given combination of board and kernel.
It contains the lowest-level and kernel-specific fundamentals any
system scenario relies on, namely the kernel/core, the dynamic linker, and a
timer driver. In order to execute any of the run scripts that follow this
pattern, we need to create such a depot archive for our version
of the kernel. When using the "hw" kernel, the 'base_src' function can be
found at _tool/run/boot_dir/hw_.

! $ grep -r base_src tool/run/
! ...
! run/boot_dir/hw:proc base_src { } { return "base-hw-[board]" }
! ...

In the case of our 'pine_a64lts' board, the source archive would hence be
named 'base-hw-pine_a64lts'. The _tool/depot/create_ tool can be used
to populate the depot. Even though we have not yet provided any declaration
for our base archive, let's call the tool and see how it breaks:

! $ ./tool/depot/create x/src/base-hw-pine_a64lts UPDATE_VERSIONS=1 FORCE=1
! Error: incomplete or missing recipe (x/src/base-hw-pine_a64lts)

The following things are worth noting about the command-line arguments.

* I supply 'x' as depot user, which is just a dummy name that is good
  enough while pursuing the packaging work. Once the work is finished, it
  allows me to just remove the _depot/x/_ directory and all testing artifacts
  are gone.

* The UPDATE_VERSIONS=1 argument tells the tool to automatically increase
  the version of the depot archive whenever the content differs from the
  previously packaged version. During the packaging work, I always set it
  to 1.

* The FORCE=1 argument tells the tool to perform all packaging steps from
  scratch instead of reusing artifacts from previous runs.

Now, let's address the error message. It tell us that the tool expected a
so-called recipe for the given depot archive, which does not exist yet. The
so-called recipes describe how a depot archive can be extracted from the
source tree. They are searched in the _<repo>/recipes/_ directories of all
repositories. E.g., the recipe for the base source archive for the i.MX8 EVK
board resides at _repos/base-hw/recipes/src/base-hw-imx8q_evk/_. This
is a suitable template for out pine_a64lts recipe.

! $ mkdir -p repos/allwinner/recipes/src/base-hw-pine_a64lts
! $ cp -r repos/base-hw/recipes/src/base-hw-imx8q_evk/* \
!         repos/allwinner/recipes/src/base-hw-pine_a64lts/

The directory hosts three files:

:content.mk:

  This is Makefile snippet with rules for gathering the content of the
  archive from the source tree. The copied file, however, merely includes
  rules from a file called _base-hw_content.inc_. We can keep this line.

  ! include $(GENODE_DIR)/repos/base-hw/recipes/src/base-hw_content.inc

:used_apis:

  This file contains a list of APIs required to build a binary archive
  from the source archive. The copied template contains merely two lines,
  which we can keep that way. Naturally, the base-hw kernel requires the
  definitions of the generic Genode API (base API) and the supplements
  that are specific for the base-hw kernel (base-hw API).

  ! base-hw
  ! base

:hash:

  The hash file tells the depot tools about the current version of the
  archive and draws the connection to the corresponding archive content
  by specifying a hash value.

  ! 2021-02-24 d122ddee70f0b075de8cec50a41c9f4783702e05

  The hash value is computed over the entire content of the archive. Should
  the hash of a freshly created archive deviate from the hash stored at the
  recipe, we know that the version should better be updated. Of course, it
  would be tiresome to calculate such hash values manually. Thankfully, the
  depot tools do this job for us. While keeping in mind that the hash is most
  certainly wrong for our the pine_a64lts source archive, we leave it as is
  because we don't know any better value anyway at this point.

With the new source recipe in place, let's give the package creation
another try.

! $ ./tool/depot/create x/src/base-hw-pine_a64lts UPDATE_VERSIONS=1 FORCE=1

This time, the output looks different:

! $ ./tool/depot/create x/src/base-hw-pine_a64lts UPDATE_VERSIONS=1 FORCE=1
! created x/api/base/2021-02-22
! created x/api/base-hw/2021-02-22
!
! Error: CPU architecure for board pine_a64lts undefined
!
!   missing file /.../repos/allwinner/board/pine_a64lts/arch

The tool has successfully created the API archives for the dependencies
we stated in the _used_apis_ file. However, the source-archive creation
still backs out, missing the information of the board's CPU architecture.
The CPU architecture dictates the subset of files of the base-hw repository
that are relevant for the given board. This information is expected at
the path printed by the error message. That's our call!

! $ mkdir -p repos/allwinner/board/pine_a64lts
! $ echo arm_v8a > repos/allwinner/board/pine_a64lts/arch

Upon the next attempt to create the source archive, the creation-process
succeeds.

! $ ./tool/depot/create x/src/base-hw-pine_a64lts UPDATE_VERSIONS=1 FORCE=1
! created x/api/base/2021-02-22
! created x/api/base-hw/2021-02-22
! created x/src/base-hw-pine_a64lts/2021-03-04 (new version)

The last line tells us that the tool has detected the inconsistency
of our recipe's hash file with the assembled archive content and has
automatically adjusted the version (taking the current date) and hash in the
hash file. Now it looks as follows:

! $ cat repos/allwinner/recipes/src/base-hw-pine_a64lts/hash
! 2021-03-04 4589eb3b4d816d17a2f2a539031a54eff9dd3712

You may like to have a look at the resulting archive.

! $ ls depot/x/src/base-hw-pine_a64lts/2021-03-04/
! etc  include  lib  LICENSE  src  used_apis

Finally, let us check that it is possible to create a binary archive from our
source archive by specifying 'x/bin/arm_v8a/base-hw-pine_a64lts' as
depot-archive path to the depot-create tool. To accelerate the build, we can
append '-j8' as argument to enable the use of multiple CPUs.

! $ ./tool/depot/create x/bin/arm_v8a/base-hw-pine_a64lts \
!                       UPDATE_VERSIONS=1 FORCE=1 -j8
! created x/api/base-hw/2021-02-22
! created x/api/base/2021-02-22
! created x/src/base-hw-pine_a64lts/2021-03-04
! checking library dependencies...
! ...
! ... many build steps
! ...
!     LINK     timer
! created x/bin/arm_v8a/base-hw-pine_a64lts/2021-03-04

This time, the tool was satisfied with the current hash of our source recipe,
the build process ran to completion, and we can inspect the results at
the printed location within the depot.

! $ ls -1 depot/x/bin/arm_v8a/base-hw-pine_a64lts/2021-03-04/
! bootstrap-hw-pine_a64lts.o
! core-hw-pine_a64lts.a
! ld.lib.so
! timer

Combined test suite of over 80 system scenarios
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

With the kernel packaged, a whole new world of run scripts opens up for us.
The most intriguing one is _repos/gems/run/depot_autopilot.run_. It is a
system that does not only execute a single test scenario but orchestrates the
execution of more than 80 system scenarios one after the other. The init test
we executed earlier is just one of these scenarios. Combined, the
scenarios form the comprehensive test suite for Genode's base framework
covering the following topics.

* Low-level data structures and allocators
* Parsing and generating XML, UTF-8
* Integration of Ada/SPARK with Genode's C++ API
* Publisher-subscriber mechanism
* Management of dynamic subsystems
* Fault detection mechanism
* Synthetic tests for low-level components and interfaces
  such as init, timer, VFS, block access, terminal
* VFS infrastructure
* C runtime (I/O, execve, fork, pthreads)
* Standard C++ library
* TCP/IP
* Network routing
* Tracing
* On-target deployment of depot packages

[tikz img/depot_autopilot_arch]

The diagram gives an overview of the architecture of the system scenario. It
is described in great detail in the release documentation [1].

[1] [https://genode.org/documentation/release-notes/18.11#Automated_test_infrastructure_hosted_on_top_of_Genode]

As a prerequisite for executing the depot-autopilot scenario, the depot
packages for the whole arsenal of tests must be made available. We can
instruct the build system to automatically create the depot content as needed,
by enabling the following option at the _etc/build.conf_ file:

! RUN_OPT += --depot-auto-update

Furthermore, we need to make sure to have the following repositories
enabled in the _etc/build.conf_ file.

! REPOSITORIES += $(GENODE_DIR)/repos/libports
! REPOSITORIES += $(GENODE_DIR)/repos/dde_linux
! REPOSITORIES += $(GENODE_DIR)/repos/gems

The dde_linux repository is solely needed for the TCP/IP stack ported from the
Linux kernel (lxip). The gems repository hosts the depot-autopilot. With these
precautions taken, we can kick off the _depot_autopilot.run_ script as usual.

! build/arm_v8a$ make run/depot_autopilot KERNEL=hw BOARD=pine_a64lts

You will most likely encounter an error like the following.

! Error: Ports not prepared or outdated:
!   ada-runtime dde_linux expat gcov gmp libc lwip sanitizer stdcxx
!
! You can prepare respectively update them as follows:
!   /.../tool/ports/prepare_port ada-runtime dde_linux expat gcov \
!                                gmp libc lwip sanitizer stdcxx

The printed ports of 3rd-party software are required. They can be imported
into Genode's _contrib/_ directory by executing the command as suggested
by the error message.

Once the prepare_port command has completed, we can give the
depot_autopilot.run script another try. This time, we can lay back and enjoy
tons of build output scroll by, take a nip at a cup of coffee, maybe stretch
our back a little, continue watching the build output, relax, not to forget to
keep breathing. Have I mentioned looking at the build output?

When finally loading the resulting uImage on the board, we are greeted
with a shocking message:

! TFTP from server 10.0.0.32; our IP address is 10.0.0.178
! Filename '/var/lib/tftpboot/uImage'.
! Load address: 0x42000000
! Loading: #################################################################
!          #################################################################
!          #################################################################
!          #################################################################
!          #################################################################
!          #################################################################
!          #################################################################
!          #################################################################
!          #################################################################
!          ##
!          3.5 MiB/s
! done
! Bytes transferred = 8608974 (835cce hex)
! ## Booting kernel from Legacy Image at 42000000 ...
!    Image Name:
!    Image Type:   AArch64 Linux Kernel Image (gzip compressed)
!    Data Size:    8608910 Bytes = 8.2 MiB
!    Load Address: 40010000
!    Entry Point:  40010000
!    Verifying Checksum ... OK
!    Uncompressing Kernel Image
! Error: inflate() returned -5
! Image too large: increase CONFIG_SYS_BOOTM_LEN
! Must RESET board to recover
! resetting ...

Just at the climax of our expectations, U-Boot's ELF loader went on strike.
The ELF image is apparently too large. We take the mental note to adjust
'CONFIG_SYS_BOOTM_LEN' and re-build and re-install U-Boot taking the steps of
described in Section [The U-Boot boot loader].

To satisfy our urgent need of a reward for our patience during the build of
the depot-autopilot system image, we can side step U-Boot's image-size
constraint by loading a raw binary image. Since we need an ELF image instead
of a uImage, we have to temporarily disable the '--include image/uboot'
RUN_OPT in the _etc/build.conf_ file and rebuild the image.

! build/arm_v8a$ make run/depot_autopilot KERNEL=hw BOARD=pine_a64lts
! ....

A quick look at the result reveals that the uncompressed ELF image is
quite large compared to the uImage file of 8 MiB. Here we can see the benefit
of uImage files.

! build/arm_v8a$ ls -lh var/run/depot_autopilot/boot/image.elf
! -rwxrwxr-x 1 ... 49M ... var/run/depot_autopilot/boot/image.elf

The ELF image is swiftly converted to a raw binary placed in our TFTP directory.

! build/arm_v8a$ /usr/local/genode/tool/current/bin/genode-aarch64-objcopy \
!                    -Obinary \
!                    var/run/depot_autopilot/boot/image.elf \
!                    /var/lib/tftpboot/depot_autopilot.img
! build/arm_v8a$ ls -lh /var/lib/tftpboot/depot_autopilot.img
! -rwxrwxr-x 1 ... 49M ... /var/lib/tftpboot/depot_autopilot.img

Now, we can use the following U-Boot command to load it on the board.

! => bootp 0x40010000 10.0.0.32:/var/lib/tftpboot/depot_autopilot.img
! BOOTP broadcast 1
! DHCP client bound to address 10.0.0.178 (109 ms)
! Using ethernet@1c30000 device
! TFTP from server 10.0.0.32; our IP address is 10.0.0.178
! Filename '/var/lib/tftpboot/depot_autopilot'.
! Load address: 0x40010000
! Loading: #################################################################
! ...
!          ######################################################
!          3.2 MiB/s
! done
! Bytes transferred = 51351552 (30f9000 hex)

... and run it!

! => go 0x40010000
! ## Starting application at 0x40010000 ...
!
! kernel initialized
! ...

... massive amount of log output scrolls by for about 9 minutes ...

! ...
! [init -> depot_autopilot] --- Finished after 519.179 sec ---
! [init -> depot_autopilot]
! [init -> depot_autopilot]  test-spark                      ok         0.239  log
! [init -> depot_autopilot]  test-spark_exception            ok         0.161  log
! [init -> depot_autopilot]  test-spark_secondary_stack      ok         0.518  log
! [init -> depot_autopilot]  test-block                      ok         1.124  log
! [init -> depot_autopilot]  test-block_cache                ok         0.692  log
! [init -> depot_autopilot]  test-clipboard                  ok         3.676  log
! [init -> depot_autopilot]  test-depot_query_index          ok         0.289  log
! [init -> depot_autopilot]  test-ds_ownership               ok         0.227  log
! [init -> depot_autopilot]  test-dynamic_config             ok         3.154  log
! [init -> depot_autopilot]  test-dynamic_config_loader      ok         3.211  log
! [init -> depot_autopilot]  test-dynamic_config_slave       ok         2.640  log
! [init -> depot_autopilot]  test-entrypoint                 ok        40.117  log
! [init -> depot_autopilot]  test-expat                      ok         0.361  log
! [init -> depot_autopilot]  test-fault_detection            ok         2.433  log
! [init -> depot_autopilot]  test-fs_log                     ok         0.938  log
! [init -> depot_autopilot]  test-fs_packet                  ok         1.292  log
! [init -> depot_autopilot]  test-fs_report                  ok         2.173  log
! [init -> depot_autopilot]  test-fs_rom_update              ok         6.486  log
! [init -> depot_autopilot]  test-fs_rom_update_fs           ok         6.598  log
! [init -> depot_autopilot]  test-fs_rom_update_ram          ok         6.491  log
! [init -> depot_autopilot]  test-fs_tool                    ok         1.212  log
! [init -> depot_autopilot]  test-init                       ok        30.140  log
! [init -> depot_autopilot]  test-init_loop                  ok        10.156  log
! [init -> depot_autopilot]  test-ldso                       ok         1.083  log
! [init -> depot_autopilot]  test-libc                       ok         3.694  log
! [init -> depot_autopilot]  test-libc_connect_lwip          ok        21.060  log
! [init -> depot_autopilot]  test-libc_connect_lxip          ok        21.524  log
! [init -> depot_autopilot]  test-libc_connect_vfs_server_lw ok        21.463  log
! [init -> depot_autopilot]  test-libc_connect_vfs_server_lx ok        21.961  log
! [init -> depot_autopilot]  test-libc_counter               ok        11.723  log
! [init -> depot_autopilot]  test-libc_execve                ok         6.718  log
! [init -> depot_autopilot]  test-libc_fifo_pipe             ok        13.094  log
! [init -> depot_autopilot]  test-libc_fork                  ok         3.097  log
! [init -> depot_autopilot]  test-libc_getenv                ok         0.309  log
! [init -> depot_autopilot]  test-libc_pipe                  ok         0.816  log
! [init -> depot_autopilot]  test-libc_vfs                   ok         2.037  log
! [init -> depot_autopilot]  test-libc_vfs_audit             ok         3.917  log
! [init -> depot_autopilot]  test-libc_vfs_block             ok         0.468  log
! [init -> depot_autopilot]  test-libc_vfs_counter           ok        11.809  log
! [init -> depot_autopilot]  test-libc_vfs_fs                ok         2.036  log
! [init -> depot_autopilot]  test-libc_vfs_fs_chained        ok         2.306  log
! [init -> depot_autopilot]  test-libc_vfs_ram               ok         1.719  log
! [init -> depot_autopilot]  test-log                        ok         0.265  log
! [init -> depot_autopilot]  test-lx_block                   skipped
! [init -> depot_autopilot]  test-magic_ring_buffer          ok         0.205  log
! [init -> depot_autopilot]  test-mmio                       ok         0.100  log
! [init -> depot_autopilot]  test-new_delete                 ok         0.417  log
! [init -> depot_autopilot]  test-nic_loopback               ok         1.369  log
! [init -> depot_autopilot]  test-part_block_gpt             ok         3.096  log
! [init -> depot_autopilot]  test-part_block_mbr             ok         1.726  log
! [init -> depot_autopilot]  test-pthread                    ok        28.275  log
! [init -> depot_autopilot]  test-ram_fs_chunk               ok         0.510  log
! [init -> depot_autopilot]  test-read_only_rom              ok        19.988  timeout 20 sec
! [init -> depot_autopilot]  test-reconstructible            ok         0.379  log
! [init -> depot_autopilot]  test-registry                   ok         0.177  log
! [init -> depot_autopilot]  test-report_rom                 ok         0.647  log
! [init -> depot_autopilot]  test-resource_request           ok         6.609  log
! [init -> depot_autopilot]  test-resource_yield             ok        20.824  log
! [init -> depot_autopilot]  test-rm_fault                   skipped
! [init -> depot_autopilot]  test-rm_fault_no_nox            ok         1.302  log
! [init -> depot_autopilot]  test-rm_nested                  ok         2.891  log
! [init -> depot_autopilot]  test-rm_stress                  ok         1.386  log
! [init -> depot_autopilot]  test-rom_filter                 ok         4.316  log
! [init -> depot_autopilot]  test-sanitizer                  ok         0.261  log
! [init -> depot_autopilot]  test-sequence                   ok         1.243  log
! [init -> depot_autopilot]  test-signal                     ok        24.387  log
! [init -> depot_autopilot]  test-slab                       ok        20.486  log
! [init -> depot_autopilot]  test-stack_smash                ok         0.103  log
! [init -> depot_autopilot]  test-stdcxx                     ok         0.390  log
! [init -> depot_autopilot]  test-synced_interface           ok         0.133  log
! [init -> depot_autopilot]  test-tcp_bulk_lwip              skipped
! [init -> depot_autopilot]  test-tcp_bulk_lxip              skipped
! [init -> depot_autopilot]  test-terminal_crosslink         ok         0.314  log
! [init -> depot_autopilot]  test-timer                      ok        15.809  log
! [init -> depot_autopilot]  test-tls                        ok         0.134  log
! [init -> depot_autopilot]  test-token                      ok         0.119  log
! [init -> depot_autopilot]  test-trace                      ok         7.488  log
! [init -> depot_autopilot]  test-trace_logger               ok        13.921  log
! [init -> depot_autopilot]  test-utf8                       ok         0.142  log
! [init -> depot_autopilot]  test-vfs_block                  ok         1.967  log
! [init -> depot_autopilot]  test-vfs_stress_fs              ok         2.243  log
! [init -> depot_autopilot]  test-vfs_stress_ram             ok         0.560  log
! [init -> depot_autopilot]  test-weak_ptr                   ok         2.900  log
! [init -> depot_autopilot]  test-xml_generator              ok         0.570  log
! [init -> depot_autopilot]  test-xml_node                   ok         1.028  log
! [init -> depot_autopilot]  gcov                            ok        39.064  log
! [init -> depot_autopilot]
! [init -> depot_autopilot] succeeded: 82 failed: 0 skipped: 4
! [init -> depot_autopilot]
! [init] child "depot_autopilot" exited with exit value 0

The entire test suite succeeded with no errors!

It took 519 seconds. To cross-correlate this duration with the depot-autopilot
test on the i.MX8q EVK board: The i.MX board takes 465 seconds, which makes
the Pine-A64-LTS around 10% slower than the i.MX8 EVK. This is of course no
benchmark to draw meaningful conclusions from. But the fact that both values
are in the same ballpark reassures us that nothing fundamental (like the
low-level CPU or memory configuration) went wrong with our port.


code:
Kernel skeleton
===============

Of the several kernels supported by the Genode OS framework, the so-called
base-hw kernel is our go-to microkernel for ARM-based devices. Section
7.7. "Execution on bare hardware"
of the Genode Foundations book goes into detail about its underlying software
design. This section describes the process of porting this kernel to a new
board, specifically the Pine-A64-LTS single-board computer.

Equipped with the bare-metal serial-output facility developed in the
previous section, we are eager to turn
our attention to the kernel. Before attempting the porting of the kernel to
the new board, however, it is recommended to run it first on one of the
already supported boards to have a working reference.
In the case of the Pine-A64 board, which is based on an Allwinner multi-core
64-bit ARM SoC, the closest approximation would be the NXP i.MX8Q EVK board,
which ticks the boxes ARM, multi-core, and 64-bit.
At the very least, one should give the kernel a try using Qemu's virtual
pbxa9 board, which is a 32-bit platform. Even though this board has not much
in common with ours, it is still useful for seeing how the various bits and
pieces described below are supposed to work together.

A tour through the code base
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The starting point of our line of work will be the existing board support
for the i.MX8Q EVK. To get an idea of the amount of work ahead of us, let's
examine the base-hw source tree within Genode for occurrences of the board's
name. The search pattern "imx" is a good start.

! $ find repos/base-hw -type f | grep imx8
! repos/base-hw/lib/mk/spec/arm_v8/core-hw-imx8q_evk.mk
! repos/base-hw/lib/mk/spec/arm_v8/bootstrap-hw-imx8q_evk.mk
! repos/base-hw/recipes/src/base-hw-imx8q_evk/hash
! repos/base-hw/recipes/src/base-hw-imx8q_evk/content.mk
! repos/base-hw/recipes/src/base-hw-imx8q_evk/used_apis
! repos/base-hw/src/bootstrap/board/imx8q_evk/platform.cc
! repos/base-hw/src/bootstrap/board/imx8q_evk/board.h
! repos/base-hw/src/include/hw/spec/arm_64/imx8q_evk_board.h
! repos/base-hw/src/core/board/imx8q_evk/board.h

We can ignore everything inside the _recipes/_ directory for now. This
directory contains package descriptions. We will come back to the packaging
topic later. A 'grep -v' hides these files from our view.

! $ find repos/base-hw -type f | grep imx8 | grep -v recipes
! repos/base-hw/lib/mk/spec/arm_v8/core-hw-imx8q_evk.mk
! repos/base-hw/lib/mk/spec/arm_v8/bootstrap-hw-imx8q_evk.mk
! repos/base-hw/src/bootstrap/board/imx8q_evk/platform.cc
! repos/base-hw/src/bootstrap/board/imx8q_evk/board.h
! repos/base-hw/src/include/hw/spec/arm_64/imx8q_evk_board.h
! repos/base-hw/src/core/board/imx8q_evk/board.h

On the one hand, it is nice to see such a small number of files to be
concerned about. On the other hand, those files appear quite scattered
throughout the source tree with a deep hierarchy, which is a bit confusing.
To lift the clouds, let's have a look at the source-tree structure.

[tikz img/imx8q_evk_files]

The files appearing under _lib/mk/_ are build-description files for libraries.
There are two such files, having the file extension '.mk'. They are located in
a sub directory called _spec/arm_v8/_, which means that the
[https://genode.org/documentation/genode-foundations/20.05/development/Build_system.html - build system]
considers them only when building for an instruction set architecture that
matches ARMv8.

Distinction between bootstrap and core
--------------------------------------

Given the set of files depicted above, we can immediately spot two
construction sites, namely "bootstrap" and "core". The distinction between
those two parts is illustrated in the following picture.

[tikz img/base_hw_boot_steps]

The *bootstrap* program is started by the boot loader while the CPU is running
in physical mode. The MMU is disabled at this point.
Only one CPU - usually referred to as the boot CPU - is active. Bootstrap is
tasked with all the dirty and quirky work needed in preparation to bring up
the so-called core component.
This involves board-specific trickery like tweaking clocks and voltages,
setting up the page tables for executing the core program in virtual memory,
enabling the MMU, the initialization of additional CPU cores, and the
ELF-loading of the core ELF executable. Once these steps are taken, bootstrap
passes the control to the core component and ceases to exist.

The *core* component contains the microkernel executed in privileged mode.
When using Genode on a traditional microkernel like NOVA or seL4, core
is the first user-level program started by the kernel. It is usually
called roottask. In contrast, when using base-hw as we are going to do now,
core and the kernel are one single program. Core _is_ the microkernel at the
root of Genode's component tree. Hence, in the following, the terms core and
kernel are used synonymously.

Core is executed with the MMU enabled.
It is globally mapped at the upper part of the virtual address space.
To operate as the kernel, it contains basic drivers for the interrupt
controller, kernel timer (for preemptive scheduling), cache maintenance,
and cross-CPU synchronization.
For the interplay with the user level components running on top of core,
it features code paths for exiting the kernel into the user land and,
vice versa, for entering the kernel from the user land (syscalls, exceptions,
interrupts).
Functionality-wise, it implements mechanisms for inter-component
communication, asynchronous notifications, physical-memory allocation, the
management of virtual address spaces, and the world-switching between virtual
machines (if used as a hypervisor). In short, everything a microkernel needs
to do and - more importantly - nothing a microkernel shouldn't do.

Review of the board-specific code
---------------------------------

Before starting the work on the new board support, let us briefly look into
each of the files for the existing i.MX8q EVK board. Genode's support for the
NXP i.MX family is hosted in the dedicated
[https://github.com/genodelabs/genode-imx - genode-imx repository]. Let's draw
our attention to the files named after board.

! repos/imx$ find | grep imx8q_evk

In the list of files, we spot three header files, one _board.h_ header under
_src/bootstrap/_, one _board.h_ header under _src/core/_, and one
_imx8q_evk_board.h_ header under _src/include/_. The former two files
are specific for bootstrap and core, whereas the latter contains definitions
useful for both programs. The _board.h_ files are located in directories
named after the board. With this structure, generic (board-agnostic) code
can '#include <board.h>'. The build system picks the right _board.h_ file
by adding the board-specific directory to the include-search path.

Let us start with with definitions used across bootstrap and core.

:_repos/imx/src/include/hw/spec/arm_64/imx8q_evk_board.h_:

! #include <drivers/uart/imx.h>
! #include <hw/spec/arm/boot_info.h>
!
! namespace Hw::Imx8q_evk_board {
!     using Serial = Genode::Imx_uart;
!
!     enum {
!         RAM_BASE   = 0x40000000,
!         RAM_SIZE   = 0xc0000000,
!
!         UART_BASE  = 0x30860000,
!         UART_SIZE  = 0x1000,
!         UART_CLOCK = 250000000,
!     };
!
!     namespace Cpu_mmio {
!         enum {
!             IRQ_CONTROLLER_DISTR_BASE  = 0x38800000,
!             IRQ_CONTROLLER_DISTR_SIZE  = 0x10000,
!             IRQ_CONTROLLER_VT_CPU_BASE = 0x31020000,
!             IRQ_CONTROLLER_VT_CPU_SIZE = 0x2000,
!             IRQ_CONTROLLER_REDIST_BASE = 0x38880000,
!             IRQ_CONTROLLER_REDIST_SIZE = 0xc0000,
!         };
!     };
! }

Both bootstrap and core need to know the memory-mapped device registers for
the UART device to print diagnostic messages. The UART driver
(_drivers/uart.imx.h_) is included. The 'Serial' type refers to the
concrete UART driver implementation as present on the board. Thanks to this
definition, generic code is able to rely on the UART functionality via the
type name 'Serial'.

The start and size of physical memory must be known by both bootstrap and
core. So it is defined here.

Both bootstrap and core access the interrupt controller. Whereas bootstrap
performs the one-time initializations needed in order to start secondary CPU
cores, core drives the interrupt controller at runtime.

The bootstrap-specific files concern build descriptions and actual code.
The build description looks as follows.

:_repos/imx/lib/mk/spec/arm_v8/bootstrap-hw-imx8q_evk.mk_:

! REP_INC_DIR += src/bootstrap/board/imx8q_evk
!
! SRC_CC  += bootstrap/board/imx8q_evk/platform.cc
! SRC_CC  += bootstrap/spec/arm/gicv3.cc
! SRC_CC  += bootstrap/spec/arm_64/cortex_a53_mmu.cc
! SRC_CC  += lib/base/arm_64/kernel/interface.cc
! SRC_CC  += spec/64bit/memory_map.cc
! SRC_S   += bootstrap/spec/arm_64/crt0.s
!
! NR_OF_CPUS = 4
!
! vpath spec/64bit/memory_map.cc $(call select_from_repositories,src/lib/hw)
!
! vpath bootstrap/% $(REP_DIR)/src
!
! include $(call select_from_repositories,lib/mk/bootstrap-hw.inc)

The i.MX8 SoC uses the GICv3 as interrupt-controller.
Hence, the driver gicv3.cc is included.
In contrast, as we learned from the Linux boot log, the Allwinner A64 SoC
uses the GICv2 interrupt controller.

The MMU driver differs between the various ARM versions. The
i.MX8 is based on A53 CPU cores. The Allwinner A64 uses the same.

The assembly file _arm_64/crt0.s_ contains the entry point into the program
as jumped to by the boot loader.

The NR_OF_CPUS definition is used for the static allocation of data
structures that must be present for each CPU. Hence, this value is globally
defined.

The strange looking '$(call select_from_repositories...)' is a mechanism
for accessing files across different source repositories. You can find the
mechanism described in Section 5.3. "Build system" in the Genode Foundations
book.

:_repos/imx/src/bootstrap/board/imx8q_evk/board.h_:

! #include <hw/spec/arm_64/imx8q_evk_board.h>
! #include <hw/spec/arm_64/cpu.h>
! #include <hw/spec/arm/gicv3.h>
! #include <hw/spec/arm/lpae.h>
!
! namespace Board {
!     using namespace Hw::Imx8q_evk_board;
!
!     struct Cpu : Hw::Arm_64_cpu
!     {
!         static void wake_up_all_cpus(void*);
!     };
!
!     using Hw::Pic;
! }

The Board namespace aggregates the knowledge of the board details that
matter to the bootstrap code, namely the specific interrupt controller
(gicv3.h) and the declaration of the 'wake_up_all_cpus' function.
The 'Board' namespace hosts the 'Pic' (programmable interrupt controller)
type, which allows the generic code of bootstrap to interact with the
interrupt controller without knowing the exact type of device.

:_repos/imx/src/bootstrap/board/imx8q_evk/platform.cc_:

! Bootstrap::Platform::Board::Board()
! :
!   early_ram_regions(Memory_region { ::Board::RAM_BASE, ::Board::RAM_SIZE }),
!   late_ram_regions(Memory_region { }),
!   core_mmio(Memory_region { ::Board::UART_BASE, ::Board::UART_SIZE },
!             Memory_region { ::Board::Cpu_mmio::IRQ_CONTROLLER_DISTR_BASE,
!                             ::Board::Cpu_mmio::IRQ_CONTROLLER_DISTR_SIZE },
!             Memory_region { ::Board::Cpu_mmio::IRQ_CONTROLLER_REDIST_BASE,
!                             ::Board::Cpu_mmio::IRQ_CONTROLLER_REDIST_SIZE })
! {
!     ::Board::Pic pic {};
!
!     ... incomprehensible magic spells, some gibberish about GPIO, CCM, PLL ...
! }
!
! void Board::Cpu::wake_up_all_cpus(void * ip)
! {
!     ... more magic spells, digressing into assembly code ...
! }

The 'early_ram_regions', 'late_ram_regions', and 'core_mmio' data structures
are initialized with the known ranges of physical memory and memory-mapped
I/O registers. This information is designated to be passed further to core.

The call of '::Board::Pic pic {};' performs basic interrupt-controller
initialization that is needed only once.
It is followed by a sequence of board-specific tweaks to bring the
board into a defined state for the kernel to rely on. For instance,
setting the I/O MUX configuration, default voltages, and frequencies.
The U-boot boot loader already does a fine job for establishing a
base line but it is rather conservative. The code for the i.MX8 EVK
boosts the voltages and frequencies for improving the performance.

The 'wake_up_all_cpus' call invokes a hook to enable secondary CPU cores.
The used mechanism varies from board to board, specifically depending
on the operation of the ARM Trusted Firmware. We have to brace ourself
for some investigation once we look into multi-processor support.
At the beginning, however, we will use only the boot CPU. So we can
ignore this function for now.

Finally, let's turn our attention to the core-specific files.

:_repos/imx/lib/mk/spec/arm_v8/core-hw-imx8q_evk.mk_:

! REP_INC_DIR += src/core/board/imx8q_evk
! REP_INC_DIR += src/core/spec/arm/virtualization
!
! # add C++ sources
! SRC_CC += kernel/vm_thread_on.cc
! SRC_CC += spec/arm/gicv3.cc
! SRC_CC += spec/arm_v8/virtualization/kernel/vm.cc
! SRC_CC += spec/arm/virtualization/platform_services.cc
! SRC_CC += spec/arm/virtualization/vm_session_component.cc
! SRC_CC += vm_session_common.cc
! SRC_CC += vm_session_component.cc
!
! #add assembly sources
! SRC_S += spec/arm_v8/virtualization/exception_vector.s
!
! NR_OF_CPUS = 4
!
! # include less specific configuration
! include $(call select_from_repositories,lib/mk/spec/arm_v8/core-hw.inc)

Core needs to know the type of the interrupt controller because
it processes interrupts at runtime. Here, the GICv3 driver is incorporated.

Similar to bootstrap, a few data structures within core are statically
allocated for each CPU, hence the NR_OF_CPUS must be specified here as well.

We can ignore the files with 'vm_*' and 'virtualization' in their names
for now. They are important for hosting virtual machines. Since the
virtualization support is a generic feature of the ARM CPU, we don't
have to take board-specific precautions.

:_repos/imx/src/core/board/imx8q_evk/board.h_:

! #include <hw/spec/arm_64/imx8q_evk_board.h>
! #include <spec/arm/generic_timer.h>
! #include <spec/arm/virtualization/gicv3.h>
! #include <spec/arm_v8/cpu.h>
! #include <spec/arm_64/cpu/vm_state_virtualization.h>
! #include <spec/arm/virtualization/board.h>
!
! namespace Board {
!     using namespace Hw::Imx8q_evk_board;
!
!     enum {
!         TIMER_IRQ           = 14 + 16,
!         VT_TIMER_IRQ        = 11 + 16,
!         VT_MAINTAINANCE_IRQ = 9  + 16,
!         VCPU_MAX            = 16
!     };
! }

In addition to the aggregation of headers matching the board and SoC - like
the generic timer driver - we see the definitions of just the few interrupt
numbers that are important to core. The kernel is completely oblivious
about all other peripheral devices.

The VCPU_MAX definition is solely used for the dimensioning of an
array that keeps the state of virtual CPUs for virtual machine.
It is not important for now.

A new home for the board support
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The easiest way to add support for a new board is the mirroring of the
files introduced above.
We could march forward with adding new files and directories to a
new branch of the Genode repository.
Alternatively, the Genode build system allows us to host our custom
board-specific files in a dedicated source repository that we can
maintain independently from the Genode main repository.
The latter approach has the following advantages.

First, it reinforces a clean separation between board-specific code from
generic Genode code. In particular, the _segregation of code_ constricts the
working set of files relevant for a given board, keeping only important code
in view.

Operationally, it allows the decoupling of _code ownership_ in terms of
responsibility, quality assurance, licensing hygiene, development
process, and the choice of source hosting.

Finally, it alleviates the pressure to agree on one big joint code base,
_removing_ potential points of _friction_ between developers.

In the following, we will put our code into a new repository named
_allwinner_.

! mkdir repos/allwinner

In principle, the directory can be anywhere but I find it practical to
host it under the _repos_ directory of the Genode source tree.
One may also opt to use a symlink, e.g., _repos/allwinner_ pointing to
_~/src/genode-allwinner.git_.

We need to come up with with a concise name for our board support.
Throughout Genode, we follow certain *naming conventions*. In particular, we
use underscore '_' for tightly coupled words, and minus '-' for loosely
coupled terms. For example, in the file name _core-hw-imx8q_evk.mk_,
"imx8q_evk" belong closely together whereas the words "core" and "hw" are used
as some kind of category (read: the "core" component for the "hw" kernel for
the "imx8q_evk" board). With the background of these conventions, the board
name *pine_a64lts* seems sensible. Specific enough while still concise.

For the initial content from our new _allwinner_ repository be blatantly
mirror the files of the base-hw repository.

[tikz img/mirror_imx8_pine]

At the current stage, we are concerned about getting the build process right.
To concentrate at this one thing at a time, let us pretend that the
Pine-A64-LTS board works equal to the i.MX8 EVK. We don't mind that the
technicalities copied from the existing board don't match our new board until
we run the code on the board. That said, as the build-description files (those
with the 'mk' suffix) steer the build process, they must be made consistent
with our directory structure. So we have to revisit those files while looking
out for the pattern 'imx8q_evk'.

A look into _lib/mk/spec/arm_v8/bootstrap-hw-pine_a64lts.mk_ reveals the
following line:
! REP_INC_DIR += src/bootstrap/board/imx8q_evk
We have to replace it with
! REP_INC_DIR += src/bootstrap/board/pine_a64lts

Similarly, _allwinner/lib/mk/spec/arm_v8/core-hw-pine_a64lts.mk_ contains the
line:
! REP_INC_DIR += src/core/board/imx8q_evk
This must be changed to
! REP_INC_DIR += src/core/board/pine_a64lts

System-integration dry-run
--------------------------

Let us see how the Genode build system swallows - or chokes on - our new board
support. First, we need a build directory for the ARMv8 architecture.

! $ ./tool/create_builddir arm_v8a
! Successfully created build directory at /.../genode/build/arm_v8a.
! Please adjust /.../genode/build/arm_v8a/etc/build.conf according to your needs.

As suggested, we open _build/etc/build.conf_ in our favorite text editor.
Normally, I enable parallel builds by uncommenting the corresponding line
right at the beginning of the file. But for now, let us keep it disabled
until the skeleton builds successfully. The steps of the build system are
easier to follow if it operates deterministically.

We need to extend the 'REPOSITORIES' variable with the path to our custom
repository. For the _allwinner_ repository, that would be following line:

! REPOSITORIES += $(GENODE_DIR)/repos/allwinner

Note that the order of REPOSITORIES defines the search order of the build
system for files.
If the _allwinner_ repository should be able to override content
of the other repositories, specifically base-hw, the above line should appear
before the others.

With these changes in place, we can issue the build of bootstrap for new
board.

! $ cd build/arm_v8a
! $ make bootstrap/hw KERNEL=hw BOARD=pine_a64lts
!  ...
!  Library bootstrap-hw-pine_a64lts
!    ...
!    MERGE    bootstrap-hw-pine_a64lts.lib.a
!  Program bootstrap/hw/bootstrap_hw_pine_a64lts

The result can be found in the sub directory _bootstrap/hw/_. We find
a single object file named _bootstrap-hw-pine_a64lts.o_ along with a stripped
version of this file.

Likewise, core for the base-hw kernel and the new board can be built as
follows.

! $ make core KERNEL=hw BOARD=pine_a64lts
!    ...
!    MERGE    core-hw-pine_a64lts.lib.a
!  Program core/hw/core_hw_pine_a64lts

Similar to the build of bootstrap, we can find the result at the corresponding
subdirectory, here _core/hw/_. We find a single archive file named
_core-hw-pine_a64lts.a_ along with a stripped version of this file.

Next up, we are going to build a *system image* that contains both core and
bootstrap.
Now would be a good time to enable parallel builds. Edit the _etc/build.conf_
file by un-commenting the following line (removing the hash '#' character).

! #MAKE += -j4

One may also opt to write the BOARD and KERNEL arguments directly into the
_build.conf_ file as illustrated by the commented-out examples. This
spares the need to specify the arguments each time when issuing a build
command.

A system image contains bootstrap, core, and additional boot modules.
The first two puzzle pieces are already in place. But what about the boot modules?
In contrast to bootstrap and core, which are always the same for each
system scenario, the boot modules vary between system scenarios.
Genode system scenarios are defined in the form of run scripts.
The run script at _repos/base/run/log.run_ is a good starting point.
As defined by this particular run script, the system image for the "log"
system scenario is comprised of core, init, ld.lib.so, init, and test-log in
addition to a configuration. A system image (image.elf) for this scenario
would look like this:

[tikz img/base_hw_system_image]

Genode's run tool automates the process of assembling such Matryoshkas from
the various pieces. Let's give it a try:

! $ make run/log KERNEL=hw BOARD=pine_a64lts
! ...
! ... long sequence of compile steps
! ...
! genode build completed
! using 'ld-hw.lib.so' as 'ld.lib.so'
! core link address is 0xffffffc000000000
!
! Error: unknown image link address
!
!  File board/pine_a64lts/image_link_address not present in any repository.
!
! Makefile:329: recipe for target 'run/log' failed

This message should prompt us to have closer look at the run tool.

! $ cd genode
! $ grep -r "unknown image link address" tool
! tool/run/boot_dir/hw: puts stderr "\nError: unknown image link address\n"

The file _tool/run/boot_dir/hw_ is the part of the run tool that defines
the integration of a system image from its parts for the base-hw kernel.
It is worth skimming over the file to get a rough understanding of how
the system image is assembled from its ingredients. The error message
above comes from the function 'bootstrap_link_address' called during the
system-image integration step.

The link address is evaluated by the boot loader when loading the system image
as ELF binary. It defines the start of the text segment of the system image in
physical memory. As the physical memory layout differs between SoCs and
boards, we must provide a value that is suitable for the memory layout of the
Pine-A64-LTS board. From looking at Linux' _/proc/iomem_, we remember that the
system RAM of our board starts at 0x40000000.

As indicated by the error message above, the run tool expects to find
the link address in a file called _board/pine_a64lts/image_link_address_.
Let's create such a file with a sensible value. It is common practice
to leave some room at the very beginning of the memory, which is often
occupied by the boot loader. It is usually fine to link the system image to
64 KiB after the start of the physical memory.

! $ cd allwinner
! $ mkdir -p board/pine_a64lts
! $ echo 0x40010000 > board/pine_a64lts/image_link_address

With the link address defined, another attempt to build the system image for
the log scenario succeeds. The result can be found in the build directory's
_var/run/_ sub directory:

! $ find var/run
! var/run
! var/run/log.boot_modules.o
! var/run/log
! var/run/log/boot
! var/run/log/boot/image.elf
! var/run/log.core
! var/run/log.bootstrap
! var/run/log.config

The most interesting file is certainly _var/run/log/boot/image.elf_, which is
the final system image. To quickly validate the link address, let's check the
ELF entrypoint.

! $ readelf -a var/run/log/boot/image.elf | grep Entry
!   Entry point address:               0x40010000

The value looks familiar. While we are at it, the other files are also worth
inspecting.

:_var/run/log.boot_modules.o_: is an aggregate of all boot modules of the
  system scenario.

:_var/run/log.core_: is an ELF binary of core without the boot
  modules. The binary contains all debug information. This is handy for
  debugging the core component. For example, using this binary, the
  instruction pointer of a page fault within core can be related to the
  matching source code using 'objdump'.

:_var/run/log.bootstrap_: is an ELF binary of the bootstrap code without
  core and the boot modules. As for the core log.core binary, it is handy
  for debugging the bootstrap code.

:_var/run/log.config_: is the 'config' boot module passed to the initial
  init component. It corresponds to the snippet passed the 'install_config'
  function as found in the _log.run_ script.

By the way, one may prefer booting a uImage instead of an ELF image because a
uImage is compressed using gzip by default, which reduces the boot time. The
run tool supports that via the argument '--include image/uboot'. One can
either extend the RUN_OPT variable by adding a corresponding line to
_etc/build.conf_ or pass the option to the make command line:

! $ RUN_OPT='--include image/uboot' make run/log BOARD=pine_a64lts KERNEL=hw

After completing the build, the uImage file can be found at
_var/run/log/uImage_.

This is not magic. At this point, I recommend taking a look at the run tool's
snippets located at _tool/run/_. In particular, _tool/run/image/uboot_
contains the sequence of commands used for generating the uImage from the
ELF image.

Getting to grips using meaningful numbers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The faux system image that we just created contains information cowardly
copied from the imx8q_evk board, and which certainly mismatches the
pine_a64lts board. So let's revisit the files in our repository one by one and
look out for any numbers. Numbers are important. According to my experience,
hexadecimal numbers are especially important. Don't forget to squinch your
eyes when looking at them. Change them with caution.

! $ cd repos/allwinner
! $ find -type f
! ./lib/mk/spec/arm_v8/bootstrap-hw-pine_a64lts.mk
! ./lib/mk/spec/arm_v8/core-hw-pine_a64lts.mk
! ./board/pine_a64lts/image_link_address
! ./src/bootstrap/board/pine_a64lts/platform.cc
! ./src/bootstrap/board/pine_a64lts/board.h
! ./src/include/hw/spec/arm_64/pine_a64lts_board.h
! ./src/core/board/pine_a64lts/board.h

:_lib/mk/spec/arm_v8/bootstrap-hw-pine_a64lts.mk_:

The following line catches our attention:
! SRC_CC  += bootstrap/spec/arm/gicv3.cc

The i.MX8 SoC uses ARM's Generic Interrupt Controller version 3 (GICv3).
From booting Linux on the Pine-A64 board, we learned that the Allwinner SoC
uses the GIC version 2. Fortunately, the base-hw kernel supports both
versions. So we can change the line to:
! SRC_CC  += bootstrap/spec/arm/gicv2.cc

The NR_OF_CPUS value can stay unmodified because the Allwinner SoC has 4
cores.

:_lib/mk/spec/arm_v8/core-hw-pine_a64lts.mk_:

We merely also have to adjust the GIC version from 3 to 2.

:_src/bootstrap/board/pine_a64lts/platform.cc_:

The file contains a lot of i.MX8Q-specific initialization steps like
tweaking clocks and voltages. We can remove this code without looking back.
The body of the 'Bootstrap::Platform::Board' constructor can be reduced
to the mere initialization of the interrupt controller:
! {
!   ::Board::Pic pic { };
! }

The list of memory regions passed to the 'core_mmio' member can be
pruned to the single entry for the UART. The other entries that refer
to the IRQ controller should be removed because they refer to the wrong
version of the GIC anyway. We will supplement the proper regions for the
GICv2 later, once we turn our attention to interrupts.
! core_mmio(Memory_region { ::Board::UART_BASE, ::Board::UART_SIZE })

At this point, I am admittedly unsure about the 'wake_up_all_cpus' implementation,
in particular whether the opcode of the CPU_ON smc instruction would
match. I guess not. We will come to multi-processor support at a later
stage. So let's better remove the uncertainty by reducing
the implementation to
! void Board::Cpu::wake_up_all_cpus(void *) { }

:_src/bootstrap/board/pine_a64lts/board.h_:

We see several things that cry for adjustment.

* Updating the include guards
* Including the correct board definitions by replacing
  ! #include <hw/spec/arm_64/imx8q_evk_board.h>
  by
  ! #include <hw/spec/arm_64/pine_a64lts_board.h>
* Incorporating the GICv2 driver instead of the GICv3 driver by changing
  ! #include <hw/spec/arm/gicv3.h>
  to
  ! #include <hw/spec/arm/gicv2.h>
* Defining the C++ type 'Pic' such that it refers to the 'Hw::Gicv2' driver:
  ! using Pic = Hw::Gicv2;

:_src/include/hw/spec/arm_64/pine_a64lts_board.h_:

To our despair, the file is full of numbers.

* It includes the driver for the UART used for printing debug messages.
  Of course, the specified drivers/uart/imx.h driver won't work.
  While experimenting with the bare-metal serial output,
  we have learned that the Allwinner SoC uses a NS16550 UART controller. Let
  us pretend having a driver by changing the line to
  ! #include <drivers/uart/ns16550.h>

* The board-specific name space should reflect the name of our board:
  ! namespace Hw::Pine_a64lts_board {

* We want the C++ type 'Hw::Serial' to refer to our hypothetical NS16550
  driver.
  ! using Serial = Genode::Ns16550_uart;

* The RAM_BASE and RAM_SIZE values must match those we found from the
  look at Linux _/proc/iomem_.
  ! RAM_BASE = 0x40000000,
  ! RAM_SIZE = 0x7e000000,

* We already have found known-good values for UART_BASE and UART_SIZE
  during our bare-metal serial output experimentation. The UART_CLOCK
  value won't be needed in our case. So we define it as zero.
  ! UART_BASE  = 0x1c28000,
  ! UART_SIZE  = 0x1000,
  ! UART_CLOCK = 0,

* The IRQ_CONTROLLER_REDIST_BASE and SIZE are not used for the GICv2.
  So the values can be removed.

* The values for IRQ_CONTROLLER_DISTR_BASE and SIZE as well as
  VT_CPU_BASE and SIZE will become important once
  we will turn our attention to the interrupt controller. But this is not
  today. So we keep the existing numbers, keeping in mind that they won't
  work.

* When using the GICv2, we need to add the definition of
  IRQ_CONTROLLER_CPU_BASE and VT_CTRL_BASE.
  Until we use interrupts, we can pick an
  arbitrary number. To display good manners, let's leave the lowest 12 bits
  to zero, pretending that each device resource starts at a page boundary.
  ! IRQ_CONTROLLER_CPU_BASE     = 0xaaaaa000,
  ! IRQ_CONTROLLER_VT_CTRL_BASE = 0xbbbbb000,

: \clearpage

:_/src/core/board/pine_a64lts/board.h_:

The file contains mostly interrupt numbers. We will turn our attention to
interrupts later. Let's not touch them for now because we cannot validate
the values anyway at this point. Apart from these numbers, a few adjustments
must be made.

* Updating the include guard
* Including the board definitions from _pine_a64lts_board.h_
* Adjusting the GIC version of the included header from _gicv3.h_ to _gicv2.h_
* Importing the board-specific namespace 'Hw::Pine_a64lts_board'

To wrap up this step, let's check if we missed any leftover by grepping for
remaining occurrences of patterns like "imx" or "gicv3".
! $ grep -ri imx repos/allwinner

Now would also be a good time to revisit the file headers, updating the
information about the author, creation date, brief description, and copyright.
Should the code be considered to eventually become part of the Genode upstream
project at some point, it is sensible to leave the license disclaimer as is,
clarifying that the code is designated be a part of the Genode OS framework.

UART driver for bootstrap and core
----------------------------------

The next attempt to build the system image for the log scenario fails
predictably:

! $ make run/log KERNEL=hw BOARD=pine_a64lts
! ...
!     COMPILE  core_region_map.o
! In file included from /.../repos/allwinner/src/core/board/pine_a64lts/board.h:17,
!                  from /.../repos/base-hw/src/core/platform.h:37,
!                  from /.../repos/base-hw/src/core/core_region_map.cc:18:
! /.../repos/allwinner/src/include/hw/spec/arm_64/pine_a64lts_board.h:17:10:
!            fatal error: drivers/uart/ns16550.h: No such file or directory
!  #include <drivers/uart/ns16550.h>
!           ^~~~~~~~~~~~~~~~~~~~~~~~

We can find a number of blueprints for our new UART driver at
_repos/base/include/drivers/uart/_. By following the lines of the existing
drivers and combining our knowledge from the bare-metal serial experiments,
we can come up with the following little driver placed
at _allwinner/include/drivers/uart/ns16550.h_.

! #include <util/mmio.h>
!
! namespace Genode { class Ns16550_uart; }
!
!
! class Genode::Ns16550_uart : Mmio
! {
!   private:
!
!     struct Thr : Register<0x00, 32>
!     {
!       struct Data : Bitfield<0,8> { };
!     };
!
!     struct Lsr : Register<0x14, 32>
!     {
!       struct Thr_empty : Bitfield<5,1> { };
!     };
!
!   public:
!
!     Ns16550_uart(addr_t const base, uint32_t, uint32_t) : Mmio(base) { }
!
!     void put_char(char const c)
!     {
!       while (read<Lsr::Thr_empty>() == 0);
!
!       write<Thr::Data>(c);
!     }
! };

Like all drivers dedicatedly developed for Genode, it uses Genode's 'Register'
API to safely access bits of memory-mapped I/O registers. You can find the API
described in Section 8.18 "Utilities for user-level device drivers" in the
Genode-Foundations book.

Climbing the mountain step by step
----------------------------------

We are almost there. On our walk, we repeatedly try to build the system image,
look at the compiler and linker errors, fix them, and repeat.

! $ make run/log KERNEL=hw BOARD=pine_a64lts
!     ...
!     COMPILE  bootstrap/spec/arm/gicv2.o
! /.../repos/base-hw/src/bootstrap/spec/arm/gicv2.cc:
!                     In constructor 'Hw::Gicv2::Gicv2()':
! /.../repos/base-hw/src/bootstrap/spec/arm/gicv2.cc:23:28:
!                     error: 'NON_SECURE' is not a member of 'Board'
!   bool use_group_1 = Board::NON_SECURE &&
!                             ^~~~~~~~~~

The interrupt-controller driver apparently needs to distinguish the cases where
the kernel is running in the so-called "secure world" or "normal world" of
ARM TrustZone. If you want to learn more about schizophrenia as a feature
of ARM processors, let me point you to
[https://genode.org/documentation/articles/trustzone - our article on ARM TrustZone].
Admittedly, I'm not completely sure about which of both worlds are executing
our kernel. But it is probably safe to assume that the boot process switches
to the normal world before loading and starting our system image.
So we add the definition of NON_SECURE to
_allwinner/src/bootstrap/board/pine_a64lts/board.h_.

! namespace Board {
!   ...
!   static constexpr bool NON_SECURE = true;
! }

The next slope on our way up the hill:

! $ make run/log KERNEL=hw BOARD=pine_a64lts
!     ...
!     MERGE    bootstrap-hw-pine_a64lts.lib.a
! /.../genode-aarch64-ar: bootstrap/board/pine_a64lts/platform.o:
!                         No such file or directory
! /.../repos/base/mk/lib.mk:180: recipe for target
!                                'bootstrap-hw-pine_a64lts.lib.a' failed

We have to guide the build system to consider source files in the _allwinner_
repository, by adding the following line to
_lib/mk/spec/arm_v8/bootstrap-hw-pine_a64lts.mk_.

! vpath bootstrap/% $(REP_DIR)/src

: \clearpage

Next try. This time, we get a link error:

! /.../aarch64-none-elf/bin/ld: debug/core-hw-pine_a64lts.a(cpu.o):
!                               in function `Board::Pic::Pic()':
! /.../repos/base-hw/src/core/spec/arm/virtualization/gicv2.h:22:
!     undefined reference to `Board::Pic::Gich::Gich()'

It turns out that the virtualization-related parts of the GICv2 driver reside
in a distinct compilation unit located at
_base-hw/src/core/spec/arm/virtualization/gicv2.cc_, which is not yet included
in the build description for core. We have to add the following line to
_allwinner/lib/mk/spec/arm_v8/core-hw-pine_a64lts.mk_.

! SRC_CC += spec/arm/virtualization/gicv2.cc

With these minor obstacles addressed, we get a system image that should
largely be compatible with our board. The urge to try out the freshly
baked system image on the board is strong. Why not?

A first life sign of the kernel
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Testing the system image on the board comes down to the following few steps.

# Make sure to build the uImage using the 'image/uboot' RUN_OPT.

  ! $ RUN_OPT='--include image/uboot' make run/log BOARD=pine_a64lts KERNEL=hw

# Copy the uImage from _build/arm_v8a/var/run/log/uImage_ to the TFTP
  directory. In my case, that is _/var/lib/tftpboot/_.

# Boot the board and use U-Boot's 'bootp' and 'bootm' commands to load the
  uImage via TFTP and start it.

! => bootp 10.0.0.32:/var/lib/tftpboot/uImage
! BOOTP broadcast 1
! BOOTP broadcast 2
! BOOTP broadcast 3
! DHCP client bound to address 10.0.0.178 (1121 ms)
! Using ethernet@1c30000 device
! TFTP from server 10.0.0.32; our IP address is 10.0.0.178
! Filename '/var/lib/tftpboot/uImage'.
! Load address: 0x42000000
! Loading: #############################################################
!          2.7 MiB/s
! done
! => bootm
! ## Booting kernel from Legacy Image at 42000000 ...
!    Image Name:
!    Image Type:   AArch64 Linux Kernel Image (gzip compressed)
!    Data Size:    887610 Bytes = 866.8 KiB
!    Load Address: 40010000
!    Entry Point:  40010000
!    Verifying Checksum ... OK
!    Uncompressing Kernel Image
!
! Starting kernel ...
!
! Error: Assertion failed: id < _count && _cpus[id].constructed()
! Error:   File: /.../repos/base-hw/src/core/kernel/cpu.cc:205
! Error:   Function: Kernel::Cpu& Kernel::Cpu_pool::cpu(unsigned int)

The excitement is real! That's the first life sign of Genode's kernel!
We get three satisfactory results at once.
First, our custom 'Ns16550_uart' driver is working, as evidenced by the
beautifully formatted error messages. So we did not mess up any of the
important numbers there.
Second, in contrast to the archaic experiments with the bare-metal serial
output, which did not even use a stack, we can now enjoy the comfort
of Genode's C++ runtime. We don't feel like living in a cave any longer.