code:
Taking Linux out for a walk
===========================

In preparation of the porting of drivers from Linux to Genode, we have to
gather knowledge about the drivers' natural habitat. This article goes through
the steps of building a custom Linux system that is tailored to a driver of
our choice.

Most of Genode's device drivers are not written from scratch specifically for
Genode but ported from time-tested code bases such as the Linux kernel. In the
case of most ARM SoC drivers, the driver code is provided by the SoC vendors
and is often the only reference of how the hardware works. Given the
staggering complexity of the devices, their rapid product cycles, and the
diversity of employed IP cores, developing the drivers from scratch is out of
question.

Motivation
----------

In the past, we sometimes directly jumped into the driver-porting work guided
by mere intuition, transplanting chewable bites of Linux kernel code into
Genode components, and reanimating the code to life. In some cases like our
[https://genode.org/documentation/release-notes/10.08#Gallium3D_and_Intel_s_Graphics_Execution_Manager - our first port]
of the Intel GPU driver, we were lucky with bringing up a driver on Genode
without even testing the driver code on Linux beforehand. However, this
remained a rare exception. In most cases, especially when a predictable
success and development schedule is desired, we had to engage with the
building and integration of custom Linux systems as part of the driver-porting
work. Even though jumping thought the hoops of Linux is additional work, it
gives us three invaluable benefits.

First, it gives us *reassurance* how the driver works on the reference
hardware. We can set our expectations regarding the feature set, stability,
and performance. Once the driver is ported, we can reflect on the success by
the means of benchmarking the driver in both systems Linux and Genode.

Second, it allows us to *study* and *instrument* the driver's interaction with
the user-level interfaces and the hardware. For getting hold of the userland's
interaction with the driver, it is useful to exercize the driver interfaces -
think of 'ioctl' calls - by walking the beaten tracks.

Third and most importantly, it allows for *cross-correlating* the behavior
of the driver in its natural environment against its execution within an
isolated Genode component. We have to mimick Linux kernel APIs so that the
driver is happy, after all. The comparison of the driver running in both
environments is instructive for crafting the driver environment.

Picking a tangible goal
-----------------------

Among the various classes of peripherals, _network_ devices are an attractive
target for porting a first driver.
In contrast to the PIO device discussed in
Section [One Platform driver to rule them all],
network devices are _complicated enough_ to get a tangible benefit out of
the porting approach. Conversely, network drivers are _simple enough_ to not
get overwhelmed.
Furthermore, network connectivity is _easy to test_ in an automated way,
avoiding distractions from fiddly manual workflows. Finally, the _reward_ of
enabling networking support is quite substantial. A single driver opens up a
whole world of Genode system scenarios.

In the following, we will walk towards a minimally complex custom Linux
system by taking the following steps.

* Learning how to use U-Boot to start an _arbitrary_ Linux-based OS,
* Using a custom initrd based on Busybox,
* Booting a _custom-built_ kernel,
* Getting the _network device_ to work,
* Stripping down the _kernel configuration_,
* Making the findings _reproducible_

Bootstrapping Linux using U-Boot
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

For our development work flow, we'd like to retain the convenience of network
boot as explored in Sections [Getting acquainted with the target platform] and
[Bare-metal serial output].
Booting Linux differs from booting a single binary as we did so far. For
grokking those differences, it is nice to rely on known-to-work ingredients.

Remembering from Section [Getting acquainted with the target platform]
how well the image of the [https://www.armbian.com - Armbian]
distribution worked on the board,
we know where to look. There are various ways for accessing the content
of the image. When using Gnome, you may be able to mount the image by clicking
on it. In my case, I loop-mounted the image manually. First, looking in the
partition table to find out where the file system starts.

! $ fdisk -l Armbian_21.05....img
! Disk Armbian_21.05....img: 1,7 GiB, 1782579200 bytes, 3481600 sectors
! ...
! Armbian_21.05....img1  8192 3481599 3473408  1,7G 83 Linux

It starts at block '8192'. Given the usual block size of 512 and the help of a
multiplication device, we get to know the byte offset as an argument for
loop-mounting the file system.

! $ python -c "print 8192*512"
! 4194304
! $ sudo mount -oloop,offset=4194304 Armbian_21.05....img /mnt

In the file-system's _/boot/_ directory, the files of interest are:

:Image: is a symlink to the Linux kernel.
  In my case, it refers to _vmlinuz-5.10.34-sunxi64_, which is a file of
  21 MiB.

:config-5.10.34-sunxi64: is the kernel configuration, which will become
  handy once we build the kernel from source. Save it for later.

:dtb/allwinner/: is a directory of so-called device-tree files, which we
  will cover in a minute. For now, it suffices to know that each dtb file
  contains a hardware description of a specific board. Among the 48 dtb
  files present in the directory, I identified the one for the Pine-A64-LTS
  board via
  ! $ ls -1 /mnt/boot/dtb/allwinner/ | grep a64 | grep lts
  ! sun50i-a64-pine64-lts.dtb

:initrd.img-5.10.34-sunxi64: is the initial ram disk used for bootstrapping
  the userland.

For booting Linux, we have to supply the kernel (Image), the dtb file
(pine64.dtb) for our board, and the initrd via our TFTP directory and can use
the following U-Boot commands to bring the combination of the pieces to life.
The addresses are picked such that they do not overlap, using U-Boot's default
load address for the kernel.

# Load the kernel.
  ! => bootp 0x42000000 10.0.0.32:/var/lib/tftpboot/Image

# Load the initial RAM disk.
  ! => bootp 0x41000000 10.0.0.32:/var/lib/tftpboot/initrd

# Load the device-tree file
  ! => bootp 0x41f00000 10.0.0.32:/var/lib/tftpboot/pine64.dtb

# Modifying the device tree to supply kernel parameters.
  First, telling U-Boot's FDT tool where the location of our DTB data.
  ! => fdt addr 0x41f00000
  Making space for adding additional content.
  ! => fdt resize 0x1000
  Modifying the 'chosen' device-tree node that contains the kernel parameters
  such as the location of the initrd (start and end address)
  ! => fdt chosen 0x41000000 0x41996388
  or the kernel command line
  ! => fdt set /chosen bootargs "rdinit=/bin/sh"

# Start the kernel by specifying the kernel's address and DTB address.
  We can specify '-' as the second (initrd) argument because we already
  tell the kernel the location of the initrd via the 'chosen' DTB node.
  ! => booti 0x42000000 - 0x41f00000

It goes without saying that manually executing this sequence of
commands would be error-prone and unnerving. Fortunately, U-Boot allows
us to store a sequence of commands in an environment variable, like so:

! => setenv lx 'command; another command; ...'

The variable named 'lx' now contains the sequence of the  commands separated by
semicolons.
The value of 'lx' variable can be made persistent via the mechanism discussed
in Section [The U-Boot boot loader].
! => saveenv

The sequence of commands stored in the variable 'lx' can be executed via the
'run' command:
! => run lx

The machinery takes over...

! ethernet@1c30000 Waiting for PHY auto negotiation to complete........ done
! ...
! Using ethernet@1c30000 device
! TFTP from server 10.0.0.32; our IP address is 10.0.0.178
! Filename '/var/lib/tftpboot/Image'.
! Load address: 0x42000000
! Loading: #################################################################
!          #################################################################
!          ....
!          #####################################
!          1.6 MiB/s
! done
! Bytes transferred = 32020992 (1e89a00 hex)
! BOOTP broadcast 1
! DHCP client bound to address 10.0.0.178 (3 ms)
! Using ethernet@1c30000 device
! TFTP from server 10.0.0.32; our IP address is 10.0.0.178
! Filename '/var/lib/tftpboot/initrd'.
! Load address: 0x41000000
! Loading: #################################################################
!          ....
!          ##############
!          2.2 MiB/s
! done
! Bytes transferred = 10052488 (996388 hex)
! BOOTP broadcast 1
! DHCP client bound to address 10.0.0.178 (2 ms)
! Using ethernet@1c30000 device
! TFTP from server 10.0.0.32; our IP address is 10.0.0.178
! Filename '/var/lib/tftpboot/pine64.dtb'.
! Load address: 0x41f00000
! Loading: ##
!          1.2 MiB/s
! done
! Bytes transferred = 28418 (6f02 hex)
! ## Flattened Device Tree blob at 41f00000
!    Booting using the fdt blob at 0x41f00000
! EHCI failed to shut down host controller.
!    Loading Device Tree to 0000000049ff5000, end 0000000049ffffff ... OK
!
! Starting kernel ...
!
! [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
! [    0.000000] Linux version 5.10.34-sunxi64 ...
! ...
! ... a few hundred lines of boot messages ...
! ...
! [    6.593071] Freeing unused kernel memory: 5888K
! [   37.865818] vcc-1v2-hsic: disabling

Pressing enter...

! #

A root shell awaits our commands. Could we ask for more?

A custom initrd based on Busybox
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

With the clarity gained about the network boot process of Linux via U-Boot,
we can now stepwise replace each of the three ingredients.

First, we replace Armbian's initrd with a custom-built RAM disk based on
[https://www.busybox.net/ - Busybox]. This will give us a functional,
reproducible, and customizable userland to play with while being ten times
smaller than the initrd we scraped off the Armbian image. Since we ultimately
strive for a minimalistic Linux kernel with all batteries for our board
included and dynamic modules switched off, we don't need the initrd as a
carrier of kernel modules after all.

Presuming that a regular ARM tool chain and the cpio utilities are installed
(e.g., the Debian packages _gcc-aarch64-linux-gnu_ and _cpio_), the following
steps produce a custom initrd from scratch.

# Download and extract BusyBox

  ! $ wget https://busybox.net/downloads/busybox-1.29.3.tar.bz2
  !
  ! $ tar xjf busybox-1.29.3.tar.bz2
  ! $ mkdir build-busybox-aarch64
  ! $ cd busybox-1.29.3

# Configure BusyBox
  ! $ make O=../build-busybox-aarch64 defconfig
  ! $ make O=../build-busybox-aarch64 menuconfig

  Check the following setting in the menu:
  ! [*] Setting ->  Build static binary (no shared libs)

# Compile BusyBox with the installed cross-compile tool chain

  ! $ cd ../build-busybox-aarch64
  ! $ make CROSS_COMPILE=aarch64-linux-gnu- install -j6

# Useful setups for the ram disk

  ! $ cd <busbox-build-dir>/_install

  Create fstab by editing _etc/fstab_ as follows.

  ! none /proc  proc   defaults 0 0
  ! none /sys   sysfs  defaults 0 0

  Enable DNS resolution via a predefined name server.

  ! $ echo "nameserver 1.1.1.1" > etc/resolv.conf

  Create a run-level script using mdev by creating a _etc/init.d/rcS_
  file with the following content.

  ! #/bin/sh
  !
  ! mkdir -p /proc
  ! mkdir -p /sys
  ! mkdir -p /var/shm
  ! mkdir -p /var/run
  ! mkdir -p /var/tmp
  ! mkdir -p /tmp
  ! mkdir -p /home/tc
  ! mkdir -p /root
  ! mkdir -p /dev
  !
  ! /bin/mount -t tmpfs mdev /dev
  ! mkdir /dev/pts
  ! /bin/mount -t devpts devpts /dev/pts
  !
  ! echo "Mount everything ${1}"
  ! /bin/mount -a
  !
  ! echo /sbin/mdev > /proc/sys/kernel/hotplug
  ! /sbin/mdev -s
  ! echo "Finished"
  ! /bin/sh

# Create the initial ram disk using _cpio_

  ! $ find . | cpio -H newc -o | gzip > ../initrd

After replacing Armbian's initrd with our custom initrd in the TFTP directory,
the U-Boot command for tweaking the 'chosen' device-tree node must be slightly
adjusted to the different size.

Vendor kernel source
~~~~~~~~~~~~~~~~~~~~

For most SoCs, the chip vendor provides a vendor-blessed version of the
Linux kernel source somewhere on the internet. Those so-called vendor
kernels are usually a somewhat sour compromise. On the one hand, they contain the
know-how of the vendor regarding the device drivers. The vendor kernels are
usually tailored well to the hardware. On the other hand, the "tailoring" does
not always follow the written and unwritten rules of Linux kernel development,
standing in the way of cleanly integrating the vendor code into the upstream
kernel. Some vendors don't even try. Hence, after the vendor kernel for a
specific chip is publicly released, it must be assumed a dead branch of kernel
development, receiving no further love. In our situation - where we are after
the vendor's drivers - we have to take the SoC's vendor kernel as our
reference.

In the case of the Allwinner A64 SoC, the
[https://linux-sunxi.org/Pine64#BSP - vendor kernel] is based on Linux as old
as version 3.10. However, thanks to the tremendous efforts of the
[https://linux-sunxi.org - Sunxi] open-source community that works
independently from the SoC vendor, the A64 is fully supported by the upstream
Linux kernel by now.
The [https://linux-sunxi.org/Pine64#Linux_Kernel - instructions] for building
a kernel suitable for the A64 SoC come down to compiling the Linux kernel for
the AARCH64 architecture using its default configuration.

The bottom line is that we can pick a recent kernel from
[https://kernel.org] and go with it.

! $ wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.12.1.tar.xz
! ...
! $ tar xf linux-5.12.1.tar.xz

This will extract the kernel source to the _linux-5.12.1_ subdirectory.
In the following, we will refer to this directory as LX_DIR. Let's remember
the path in a environment variable. That's just for convenience.

! export LX_DIR=$PWD/linux-5.12.1

Building and booting a custom-built kernel
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Let's give the default kernel configuration a try. For hygienic reasons,
I prefer using a build directory separate from the source tree. Let's call
the build directory LX_BUILD_DIR pointing at some place that does not exist
yet.

! export LX_BUILD_DIR=$PWD/lx_build

When using the Linux build system targeting the AARCH64 architecture,
one always has to specify 'ARCH=arm64' as argument to make. Note that this
argument even changes the output of 'make help'. So we should apply it
consistently. Let's stuff it into an environment variable so that we don't
need to repeat it over and over again.

! export ARCH=arm64

Similarly to the 'ARCH' argument, we need to tell the kernel's build system
about the tool chain used for cross compiling the kernel. In our case,
we want to use Genode's tool chain, which is usually installed at
_/usr/local/genode/tool/<version>/bin/_ and prefixed with 'genode-aarch64-'.
The kernel's build system expects this information as CROSS_COMPILE argument
or environment variable.

! export CROSS_COMPILE=/usr/local/genode/tool/21.05/bin/genode-aarch64-

With these precautions taken, we can initialize the build directory
with the default configuration via

! make -C $LX_DIR O=$LX_BUILD_DIR defconfig

The kernel configuration can be found at the .config file inside the build
directory. To get an idea what a "default" configuration entails, the number
of enabled options is telling.

! $ grep =y $LX_BUILD_DIR/.config | wc -l
! 2482

I translate this number to _not processable by my mind_.

A further feel of defeat sets in when trying to compare the default
configuration with the _config-5.10.21-sunxi64_ found on the Armbian image.

! $ diff $LX_BUILD_DIR/.config config-5.10.21-sunxi64 | wc -l
! 10363

We seem to be out of luck if we think of correlating both configurations with
each other.

Without further ado, the kernel image can be built as follows.

! $ make -C $LX_BUILD_DIR Image -j8
!   ...
!
!   LD      vmlinux
!   SORTTAB vmlinux
!   SYSMAP  System.map
!   OBJCOPY arch/arm64/boot/Image

On my laptop, this takes about 20 minutes. As indicated by the build-system
messages, the resulting kernel 'Image' can
be found at the _arch/arm64/boot/_. It has a size of 32 MiB.

By the way, the kernel's build system offers a number of other useful targets
such as a compressed kernel image. It is worth taking a look at the options.

! $ make -C $LX_BUILD_DIR help

Here, we can learn that there exists a convenient target for creating DTB
files.

! $ make -C $LX_BUILD_DIR dtbs

So we can pick the dtb file matching our board from _arch/arm64/boot/dts/_,
in our case _allwinner/sun50i-a64-pine64-lts.dtb_ replacing the third
magical puzzle piece of the boot process by a variant created from source.

When trying out the fresh baked kernel on the board, we can see the kernel
booting up, showing the kernel log over the serial line, and presenting
us with the root shell. We have a solid ground to walk on!

Next up, let us turn our attention to networking. The first impulse is to
issue 'ifconfig' to see the presence of network devices. Well, the bad news
is that there are none.

We can look at this problem from several angles.

* Grep'ing the _.config_ file for patterns like ALLWINNER, SUNXI, NET, etc.
* Wandering through the menus of 'make menuconfig' on the hunt for cues.
* Comparing the boot logs of the Armbian kernel and the custom built kernel,
  looking out for network-related messages.

Of these points, the last one is probably the least futile. However, there
is an alternative route towards success and happiness: Let's have a closer
look at the device tree for our board.

Device-tree treasure trove
~~~~~~~~~~~~~~~~~~~~~~~~~~

The so-called
[https://github.com/devicetree-org/devicetree-specification/releases/download/v0.3/devicetree-specification-v0.3.pdf - device trees]
are semi-formal descriptions of a hardware platforms, which may consist of
several peripherals, buses, interrupt controllers, clocks, and so on.
In the context of Linux, it serves two purposes. First, it fills the gap of
dynamic device discovery on platforms where devices cannot
be probed in a reliable way at runtime. This applies for most ARM SoCs.
By looking at the device tree, the kernel learns which drivers are to use.
Second, the device tree parametrizes the individual drivers. This allows
drivers to stay clear from vendor-specific parameters hard-coded in the
source code. By taking parameters from the device tree, drivers can more
easily re-targeted to other SoC revisions.

For our aspiration to port drivers to Genode, device trees are a blessing.
Since they are curated by the SoC vendors, they contain a form of hardware
documentation that can often not be found anywhere else. Moreover, in
contrast to sketchy documentation - if available at all - the information
present in the device trees can be assumed to be correct because it plays a
role in driving the hardware. In a way, device trees look like a social
engineering trick to wrest hardware docs from vendors. To lift the
treasure, we have to look in the source tree at _arch/arm64/boot/dts/_.

! $ find $LX_DIR/arch/arm64/boot/dts

We find almost 800 files there. But the forest is well organized. It is
straight-forward to spot the vendor and narrow the search. In my case:

! $ find $LX_DIR/arch/arm64/boot/dts | grep allwinner
! ...
! $ find $LX_DIR/arch/arm64/boot/dts | grep allwinner | grep pine
! ...
! $ find $LX_DIR/arch/arm64/boot/dts | grep allwinner | grep pine | grep lts
! .../arch/arm64/boot/dts/allwinner/sun50i-a64-pine64-lts.dts

When looking into this file, we see that it uses the C preprocessor to
include a bunch of includes. To get the entire picture, we have to apply
the C-preprocessing step.

! $ cpp -I $LX_DIR/include -x assembler-with-cpp -P \
!       $LX_DIR/arch/arm64/boot/dts/allwinner/sun50i-a64-pine64-lts.dts \
!       > flat_pine64lts.dts

The '-x assembler-with-cpp' argument is needed to prevent the C preprocessor
from misinterpreting lines with a leading '#' as preprocessor directive.
The '-P' argument removes line markers from the output. It is useful to
save the result in a file (_flat_pine64lts.dts_), which we can consult
at any time later. For the Pine-A64-LTS board, the file comprises 1600 lines of
insights: the relationships of many important numbers to human-readable
terminology.

Enabling network support
~~~~~~~~~~~~~~~~~~~~~~~~

Refreshed from studying the device tree for our board, now is a good time to
come back to the topic of enabling the networking for our board. By skimming
over the flattened dts file, the following node featuring the term "ethernet"
catches our attention.

! emac: ethernet@1c30000 {
!  compatible = "allwinner,sun50i-a64-emac";
!  syscon = <&syscon>;
!  reg = <0x01c30000 0x10000>;
!  interrupts = <0 82 4>;
!  interrupt-names = "macirq";
!  resets = <&ccu 13>;
!  reset-names = "stmmaceth";
!  clocks = <&ccu 36>;
!  clock-names = "stmmaceth";
!  status = "disabled";
!  mdio: mdio {
!   compatible = "snps,dwmac-mdio";
!   #address-cells = <1>;
!   #size-cells = <0>;
!  };
! };

Let me draw your attention to the two lines defining a property called
'compatible'. Those properties denote the driver that should be used to talk
to this piece of hardware. It actually draws a direct connection to the source
code. To put the device-tree node in other words:

_To use ethernet on our board, the kernel configuration must include the_
_source code related to "allwinner,sun50i-a64-emac" and "snps,dwmac-mdio"_.

Let's start with "allwinner,sun50i-a64-emac".

! $ grep -r "allwinner,sun50i-a64-emac" $LX_DIR/drivers/net
! .../stmmac/dwmac-sun8i.c: { .compatible = "allwinner,sun50i-a64-emac",

Apparently, a sledgehammer is sometimes the perfect device for singling out a
needle from a haystack.
Let's not waste too much time with looking at the code. The only information
important to us is the name of the compilation unit 'dwmac-sun8i.c'.
As a matter of convention, the corresponding object file is usually present
in the makefile for the subsystem.

! $ grep "dwmac-sun8i.o" $LX_DIR/drivers/net/ethernet/stmicro/stmmac/Makefile
! obj-$(CONFIG_DWMAC_SUN8I)	+= dwmac-sun8i.o

What a revelation! We have just drawn the connection from a node found it
a device tree to a kernel-configuration option. When looking at the
kernel's '.config' file, it becomes clear that the kernel does not drive
the ethernet device with builtin drivers because the driver is configured
as a module.

! $ grep CONFIG_DWMAC_SUN8I $LX_BUILD_DIR/.config
! CONFIG_DWMAC_SUN8I=m

Often, kernel options have dependencies. To get a picture of the dependencies
of the 'CONFIG_DWMAC_SUN8I' driver, one can consult the accompanied _Kconfig_
files or show the 'Help' for the corresponding item in the menus of the
kernel's 'make menuconfig' interactive configuration tool.

! $ make -C $LX_BUILD_DIR menuconfig

Use search ('/') to search for the desired option (e.g., CONFIG_DWMAC_SUN8I),
learn about the location of the option in the menu hierarchy, and look out
for dependencies not marked with 'y'. In our concrete case, the two
options STMMAC_ETH and STMMAC_PLATFORM must be enabled to satisfy DWMAC_SUN8I.

To enable the options, one may use the interactive menu config tool. But
I prefer a non-interactive way that can be scripted. There exists a handy tool
at _scripts/config_ in the kernel source tree that allows us to enable and
disable options, like so:

! $ $LX_DIR/scripts/config --file $LX_BUILD_DIR/.config \
!        --enable STMMAC_ETH --enable STMMAC_PLATFORM --enable DWMAC_SUN8I

The tool merely sets or removes individual options. It must be followed
by an invocation of 'make olddefconfig' to resolve possible inconsistencies
and dependencies.

! $ make -C $LX_BUILD_DIR olddefconfig

By looking at the resulting '.config' we get the reassurance that all
dependencies are indeed met.

! $ grep CONFIG_DWMAC_SUN8I $LX_BUILD_DIR/.config
! CONFIG_DWMAC_SUN8I=y

The steps above may appear long-winded. But in contrast to hit-and-miss
juggling of kernel configurations, the steps form a deterministic and
explainable process.

With the driver for the "allwinner,sun50i-a64-emac" device-tree node
covered, we are left to repeat the process for the "snps,dwmac-mdio" node.
It turns out this node is covered by STMMAC_PLATFORM already.

As a convenient way to quickly test-drive our network-equipped Linux kernel,
the kernel can be instructed to issue a DHCP request as part of the boot
process. This can be enabled by specifying the argument 'ip=dhcp' to the
kernel command line. For reference, when using U-Boot's 'fdt' command,
the 'chosen' device-tree node can be adjusted as follows:

! => fdt set /chosen bootargs "rdinit=/bin/sh ip=dhcp"

Once all driver dependencies are resolved, 'ifconfig' reports the
Ethernet device with its IP address. At this point we know that network
packets were successfully transmitted in both directions.

! / # ifconfig
! ifconfig: /proc/net/dev: No such file or directory
! eth0      Link encap:Ethernet  HWaddr 02:BA:FE:7B:59:38
!           inet addr:10.0.0.178  Bcast:10.0.0.255  Mask:255.255.255.0
!           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
!           Interrupt:39
!
! lo        Link encap:Local Loopback
!           inet addr:127.0.0.1  Mask:255.0.0.0
!           UP LOOPBACK RUNNING  MTU:65536  Metric:1

Stripping down the kernel configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Even though the custom built kernel works in principle, the situation
it not satisfactory. First, the kernel image of 32 MiB carries a lot
of excess weight when considering that we merely want to take the network
driver for a joyride. Second, we learned that the default configuration
deliberately excludes features that we deem interesting.

To overcome this situation, we have to dive into the world of Linux kernel
configuration more than knee deep. But first, let's backup the known-to-work
'.config' file.

! cp $LX_BUILD_DIR/.config $LX_BUILD_DIR/config_works

In addition to the 'defconfig' build target that results in 2482 options
enabled in the '.config' file, the kernel's build system also features
a 'tinyconfig' target that leaves only 323 options enabled.
By using this configuration, the 'Image' build time goes down from 20
minutes to only 1.5 minutes with the image weighting only 1.6 MiB. That
is much more to my taste!

The only downside of the 'tiny' kernel is that it does not show a life
sign when booted on our board. But that is expected because the configuration
does not accommodate any specific SoC out of the box. In order to bring it to
life, all we need to do is to enable the right options, don't we?
The first goal should be to find the options needed to get the boot log over
the serial line. Once we accomplish that, we can turn our attention to
supporting our initrd. Once that works, we can revive the networking support.

At this point, we know that serial output works with the 'defconfig' kernel.
The 2482 options enabled for the defconfig have to contain the right ones.
Most obviously, we need to enable the platform support for our SoC
(ARCH_SUNXI).
We can let our intuition guide us for a bit. Which kernel options found in the
defconfig could possibly contribute to serial output of the kernel console?
Searching the _config_works_ file for terms like "SERIAL".
! $ grep =y $LX_BUILD_DIR/config_works | grep SERIAL
! ...
E.g., SERIAL_EARLYCON looks certainly useful to have.

The second strategy for finding the right options is the device-tree-driven
approach we used for the network device. Searching the DTS file for "serial"
leads us to a node referring to "snps,dw-apb-uart", which relates to
_8250_dw.c_ and _8250_early.c_, which in turn brings CONFIG_SERIAL_8250_DW
and CONFIG_SERIAL_8250_CONSOLE to our attention. Fast forward, the following
options can be directly inferred from the device tree:
SERIAL_8250, SERIAL_8250_16550A_VARIANTS, SERIAL_8250_DW, SERIAL_8250_CONSOLE.

[tikz img/lx_kernel_config]

Looking from above, looking from below, however we look, there is still a
gap of config options to fill. To uncover the missing options, we can apply
brute force, bisecting the configurations as illustrated by the following commands:

# Create a fresh tinyconfig
  ! $ make -C $LX_BUILD_DIR tinyconfig

# Enable options we already know we need for sure
  ! $ $LX_DIR/scripts/config --file $LX_BUILD_DIR/.config \
  !    --enable SERIAL_8250 ...

# Enable half of the candidates found in the backed up defconfig
  ! $ grep =y $LX_BUILD_DIR/config_works |\
  !    head -n 1200 |\
  !    sed "s/=y//" |\
  !    xargs -ixxx $LX_DIR/scripts/config \
  !                    --file $LX_BUILD_DIR/.config --enable xxx

  This command adjusts the _.config_ file by enabling the first 1200 config
  options we find enabled in the _config_works_ file, which are the upper half
  of the enabled options.

# Sanitize the _.config_ file
  ! $ make -C $LX_BUILD_DIR olddefconfig

# Build the kernel and copy the resulting image the TFTP directory
  ! $ make -C $LX_BUILD_DIR dtbs Image -j8 \
  !   && cp $LX_BUILD_DIR/arch/arm64/boot/Image /var/lib/tftpboot/

# Boot the board. If the kernel shows a life sign over serial, we know
  that all the needed options are covered by the first 1200 ones.
  Otherwise, we know that a missing option is somewhere beyond those 1200.
  So we can continue with the first step while cutting the search space
  in half for each iteration.

For example, in the case of the Pine-A64-LTS board, I found that the first
1200 options sufficed for the serial output. So I went for the first 600
options in the next iteration.

Since the search space is cut into half in each iteration, resolving the
mystery of missing kernel options comes down to about 10 iterations and a bit
of patience. Using this process, I uncovered the need for CONFIG_PRINTK,
CONFIG_BINFMT_ELF, or BLK_DEV_INITRD, which look obvious in hindsight but
are very unlikely to find by the means of grep, intuition, or device-tree
analysis.

By following this process, I eventually came up with a kernel has networking
and serial output enabled. The bisecting work is not taxing but rather
mechanic. It is nice knowing that it leads to predictable success. With about
500 options enabled and an image size of 4.2 MiB (uncompressed), the resulting
kernel is a workable basis for the upcoming porting and instrumentation work.

Making the findings reproducible
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

As with any Genode-related working topic, I'm trying to make the essence
of the above findings easily reproducible, for others and me. This way,
the next developer can pick up a topic where I left it.

To download the kernel using Genode's ports tool, we can start with the
following initial ports file placed at _allwinner/ports/a64_linux.port_.
The prefix 'a64' refers to the name of the Allwinner SoC, which expresses
our intent that the downloaded version of the Linux kernel is blessed for
the use of this particular SoC.

! LICENSE   := GPLv2
! VERSION   := 5.12.1
! DOWNLOADS := a64_linux.archive
!
! URL(a64_linux) := https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-$(VERSION).tar.xz
! SHA(a64_linux) := 123
! DIR(a64_linux) := src/linux

The SHA hash is not known at this point, just putting an arbitrary number 123
there. The accompanied _allwinner/ports/a64_linux.hash_ hash file can be
created with a made-up number.

! 456

With the port-description file and hash file in place, we can give the
'a64_linux' port a try.

! genode$ ./tool/ports/prepare_port a64_linux
! a64_linux  download https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.12.1.tar.xz
! Error: Hash sum check for a64_linux failed

Even though the hash-sum check predictably failed, the download of the archive
succeeded. It can be found in the _genode/contrib/_ directory in a
subdirectory named after the port file.

! genode$ ls -lh contrib/a64_linux-456.incomplete/linux-5.12.1.tar.xz
! ... 113M ... contrib/a64_linux-456.incomplete/linux-5.12.1.tar.xz

The correct SHA hash value is just one invocation of 'sha256sum' away:

! genode$ sha256sum contrib/a64_linux-456.incomplete/linux-5.12.1.tar.xz
! c0fc1cf...fe5f37  contrib/a64_linux-456.incomplete/linux-5.12.1.tar.xz

Now we can replace the dummy value 123 in the port description file with the
correct value and retry the 'prepare_port' call.

! genode$ ./tool/ports/prepare_port a64_linux
! a64_linux  extract linux-5.12.1.tar.xz (a64_linux)
! a64_linux  generate a64_linux.hash
! Error: allwinner/ports/a64_linux.port is out of date, expected 155f...

This time, the extraction step succeeded. However, the port tool rightfully
argues about the port hash, which is a hash over the port description file.
The hash ensures that port is consistent with the Genode source tree.
This can be conveniently updated using the
ports/update_hash tool.

! genode$ ./tool/ports/update_hash a64_linux
! generate a64_linux.hash

With the hash updated, the next attempt to 'prepare_port' succeeds:

! genode$ ./tool/ports/prepare_port a64_linux
! a64_linux  extract linux-5.12.1.tar.xz (a64_linux)
! a64_linux  generate a64_linux.hash
! genode$ ls contrib/a64_linux-155f8b01cd911f42f23178571d70a2220612b634/
! a64_linux.hash  linux-5.12.1.tar.xz  src

The _src/linux/_ subdirectory contains the source tree of the kernel.

! genode$ ls contrib/a64_linux-155f8b01cd911f42f23178571d70a2220612b634/src/linux/
! arch     CREDITS        fs       Kbuild   LICENSES     net      security  virt
! block    crypto         include  Kconfig  MAINTAINERS  README   sound
! certs    Documentation  init     kernel   Makefile     samples  tools
! COPYING  drivers        ipc      lib      mm           scripts  usr

Finally, to conserve the information about configuring and building a Linux
kernel tailored to our porting work, I added a Genode build target at
[https://github.com/genodelabs/genode-allwinner/blob/master/src/a64_linux/target.mk - allwinner/src/a64_linux/target.mk] /
[https://github.com/genodelabs/genode-allwinner/blob/master/src/a64_linux/target.inc - target.inc],
which applies the kernel configuration and builds the kernel image.
Thanks to this _target.mk_ file, the custom Linux kernel can be built
from within Genode's build directory via:

! build/arm_v8a$ make a64_linux


code:
source "../foundations/manual.gosh"

proc title_tikz {} { return {\input{img/platforms_title.tikz}} }

### IMAGE ###
proc process_image_latex {txtblock} {
	set img_info ""
	set img_size 80
	set img_angle "0"
	set img_star ""
	set img_relw "columnwidth"
	regexp {\[(image \w+.*)\]} [lindex $txtblock 0] dummy img_info
	if {$img_info == ""} return
	set img_name [lindex $img_info 1]
	regexp { (\d+)%} $img_info dummy img_size
	regexp { (\d+)°} $img_info dummy img_angle
	if {[regexp {full-span} $img_info dummy]} {
		set img_star "*"
		set img_relw "textwidth"
	}

	set img_cap ""
	foreach img_capline $txtblock {
		set txt [linetxt $img_capline]
		regsub {^\[.*\]} $txt "" txt
		regsub {^ *}     $txt "" txt
		append img_cap $txt " "
	}
	regsub { *$} $img_cap "" img_cap

	printline ""
	printline "\\begin{center}"
	printline "  \\epsfig{file=$img_name,angle=$img_angle,width=[expr $img_size.0/100]\\$img_relw}"
	printline "  \\captionof{figure}{[out_latex $img_cap]}"
	printline "  \\label{[label_latex $img_name]}"
	printline " \\end{center}\n"

}

repo_name: Fhiss/genode


code:
Networking
==========

Given the insights gained by driving a device using a tailored bare-bones
Linux system as discussed in the previous sections, we are ready to take the
next step, namely transplanting Linux driver code into self-sufficient Genode
components.
This section walks through the challenge of porting a network driver from
the Linux kernel to Genode. It thereby exemplifies Genode's device-driver
environment approach for the reuse of unmodified Linux kernel code, touches
crucial technicalities of the Linux kernel, and provides practical clues.

For reference, the result of the work described herein can be found at
the _genode-allwinner_ repository.

:Git repository of the Allwinner board support:

  [https://github.com/genodelabs/genode-allwinner]

Overview
--------

The activity of porting a Linux driver is too elaborate for one swift step.
To get a sense of measurable progress, it is useful and motivating to define
intermediate goals that can be wrapped up one by one.

# Creating a *minimal Linux kernel configuration* that accommodates barely
  more than the single driver we are interested in, similar to what
  we did in the Section [Taking Linux out for a walk].
  While pursuing this goal, one can solely keep the focus on the Linux kernel
  configuration.

# Getting a tangible feeling for the targeted device and the interplay
  with other devices. Looking at the *pruned device tree* as described
  in Section [Pruning device trees] is a
  good aid. Look at the cobweb of device nodes and try to make a mental
  picture out of it. E.g., in the case of the network driver, we have to
  consider the ethernet PHY, the actual network controller (emac), and spot
  the dependency from certain clocks and voltages as potential risks.
  To double-check the findings, it is recommended to test-drive Linux with the
  pruned device tree to see if it is still able to operate the device.

# Creating the initial source skeleton of the driver component and
  successfully compiling and linking Linux code into
  a *first executable ELF binary*.
  During this step, one can focus solely on the build system, symbols, and
  compilation units. One doesn't have to understand the code in order to
  link it. This step is described in Sections
  [Directory structure of the driver component] and
  [Identifying Linux source codes of interest].

# Creation of a *test scenario* (run script) and a convenient work flow to
  execute and quickly update the binary. At this step, which is covered
  by Section [Executable testbed], we are merely concerned with the
  relationships between components without looking inside them. For the work
  flow, it is satisfying to string together a few convenient shell commands
  to ease ones life.

# Complete the execution of *low-level Linux initcalls* until the first
  device resources are requested. Here, one has to get close to the Linux code
  but can ignore any hardware-specific concerns.

# Moving the work flow over to the *real hardware* leveraging the capabilities
  of Genode's run tool, and possibly tweaking it using custom plugins. The
  focus is on the use of shell commands and glueing them together using
  Tcl script snippets, possibly even
  [https://genodians.org/chelmuth/2019-03-13-powerplug - automating the powering]
  of the target hardware to make one's life better.

# Successively *completing all Linux initcalls* including those of the
  driver code. One has to iteratively boot the target hardware, look at the
  log output, compare the log with the Linux kernel log obtained
  beforehand. Whenever both logs diverge in non-plausible ways, investigate
  and instrument the code of both native Linux and Genode. In other words,
  taking a deep dive into the Linux kernel code, adding additional Linux
  subsystems, curating dummy functions, supplementing custom emulation code,
  and extending the platform-driver's configuration whenever encountering
  the driver's request for additional hardware resources.
  This step is addressed by Sections [Linux initcalls] and
  [The lx_emul building blocks].

# Once all initcalls are executed, *provoke a small operation* of the driver.
  For example, exercising the link detection of the network driver by
  plugging/unplugging the cable, comparing the resulting log output with
  native Linux. For the first time, the driver's actual functionality and
  its interplay with the physical world comes into focus.
  During the iterative debugging and learning, the tips and tricks given by
  Sections [Linux caveats] and [Debugging and development hints]
  may be of help.

# Get the *driver's core functionality* in the form of a self-sufficient
  program to work. In the network driver's case, this would be the
  determination of the device's MAC address as well as the transmission
  and reception of network packets. At this stage, the driver remains
  co-located with the test code in one program. In other words, the
  test program uses the Linux kernel's internal APIs directly. For the
  network driver, one can conveniently use Linux' builtin DHCP support
  as test program as described in below in Section
  [Using Linux' built-in DHCP support as networking test].

# Adding the *Genode service interface* to the driver, e.g., by using
  the building blocks of the 'genode_c_api'. To test this integration,
  the test scenario must be enhanced by a separate component that
  interacts with the driver component using a Genode session interface.
  Knowing that the Linux code and the device is operational,
  one can focus solely on the Genode integration at this stage.
  This step is covered in more detail in Section
  [Connecting the driver with a Genode session interface].

# Once the driver works reliably in a minimalistic setting, it is time
  to expose it to regular networking scenarios by *packaging* it into
  a form that is digestible by the arsenal of existing run scripts.
  The packaging step is covered in Section [Packaging the driver].

# With non-trivial work loads at hand, one can take a critical look at
  the driver's behavior, in particular at its performance,
  and *optimize* it if desired.

# Finally, one can wrap up the work by cleaning up the code, potentially
  consolidating parts shared with other drivers, reviewing the result, and
  documenting the component.

When broken up into these steps - each with a different focus and a tangible
intermediate result - the work can be conducted in manageable pieces and can
even passed-on between developers.

Directory structure of the driver component
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Being a network driver, it naturally should reside somewhere under
_src/drivers/nic/_. To avoid possible path ambiguities with network drivers
hosted in other Genode source repositories, placing the driver inside a
uniquely named subdirectory is a good practice. In our case - with the
driver being referred to as "EMAC" - we settle on the directory
_src/drivers/nic/emac/_.
This directory will host the following files, for which we can initially
create skeleton versions based on one of the already existing drivers. E.g.,
one may take the
[https://github.com/genodelabs/genode-allwinner/tree/master/src/drivers/nic/emac - emac driver]
as reference when porting a new network driver.

:target.mk: The build-description file of the driver

:main.cc: The main program, which initializes the Linux emulation
  ('Lx_kit::initialize') along with the Genode service frontend
  ('genode_uplink_init'), hosts the component's central I/O signal handler,
  defines the interplay between the execution of Linux and Genode code
  ('lx_emul_start_kernel' and 'Lx_kit::env().scheduler'), and supplies
  device-tree-binary information to the Linux code ('_dtb'). It is recommended
  to take a _main.cc_ of an existing driver as starting point.

:generated_dummies.c: Dummy implementations of symbols normally provided
  by the Linux kernel. As the name suggests, the content won't be manually
  maintained but generated. So we best start with an empty file.

:dummies.c: Whenever a generated dummy will be called, the execution will
  stop with a message along with a backtrace, which will prompt us to
  closely inspect the situation and decide whether the call of the dummy
  can be ignored by returning an appropriate return code or must be replaced
  by an actual implementation. In the former case, the dummy implementation
  must be moved from _generated_dummies.c_ to the manually curated
  _dummies.c_ file.

:lx_emul.c: This file contains the implementation of symbols that require more
  thought than merely being a dummy.

:lx_emul.h: This header is included by both _dummies.c_ and
  _generated_dummies.c_. It can thereby be used as a manually maintained
  supplement to the automatically generated _generated_dummies.c_.

:lx_user.c: This file contains the implementation of a representative of a
  Linux user task. It provides the symbol 'lx_user_init' that is called
  once the Linux kernel initialization is completed. As a skeleton, an
  empty implementation suffices.

  ! #include <lx_user/init.h>
  !
  ! void lx_user_init(void)
  ! {
  ! }

  At a later stage, _lx_user.c_ will be our hook for connecting the Linux
  kernel world with a Genode service interface.

:source.list: This file contains the selection of Linux source codes to be
  included in our driver. Each line refers to file specified relative to the
  root of the Linux kernel tree. Most of the development work revolves around
  the curation of this list. As a starting point, it is useful to take an
  existing source list as starting point, in particular the selection of
  _lib/_, _kernel/_, and _arch/_ files.

:src/include/lx_emul/initcall_order.h: In contrast to the files above, which
  reside local to the driver's directory, the _initcall_order.h_ header is
  used across the Linux drivers. It equips the Linux emulation with the
  information about the correct initcall sequence. Later, we will see how
  to generate this file automatically. Until then, the following empty
  skeleton will do.

  ! static const char * lx_emul_initcall_order[] = {
  !   "END_OF_INITCALL_ORDER_ARRAY_DUMMY_ENTRY"
  !};

Build magic
-----------

When reviewing the _target.mk_ file of the emac driver, it is obvious that
there must be more to the build rules than those few dull lines. The magic
happens in the lib-import file
[https://github.com/genodelabs/genode-allwinner/blob/master/lib/import/import-a64_lx_emul.mk - import-a64_lx_emul.mk].
This file is supplemented to the build process because the _target.mk_
specifies _a64_lx_emul_ as a library dependency.

! LIBS = base a64_lx_emul

The content of _import-a64_lx_emul.mk_ is worth studying.
It lists several building blocks of the _lx_kit_ and _lx_emul_ libraries
that are used across all Linux drivers ported for the A64 SoC,
it defines the compiler flags used for building Linux C code, and
it obtains the list of C source files from the driver's _sources.list_ file.
Furthermore, it evaluates the 'BOARDS' and 'DTS_EXTRACT' variables to
generate driver-specific device-tree binary files. Given this mechanism,
the _target.mk_ of a driver solely needs to declare the supported 'BOARDS'
and the driver-specific selection of device-tree nodes to produce a
ready-to-use dtb file, e.g.,
! BOARDS := pine_a64lts
! DTS_EXTRACT(pine_a64lts) := --select emac

Finally, the import file contains a number of tweaks and quirks such as
disabling certain warnings for individual compilation units or generating
build artifacts that are implicitly generated by the Linux build system
(crc32table.h).

Generated Linux headers
-----------------------

The Linux build system generates a number of header files when preparing a
build directory. When compiling Linux code outside the Linux build system - as
we do - those headers are badly missed. This is where the
[https://github.com/genodelabs/genode-allwinner/blob/master/lib/mk/spec/arm_v8/a64_linux_generated.mk - a64_linux_generated]
library comes in. This pseudo library has the sole purpose of creating a Linux
build directory with the generated headers we need. It takes the same Linux
[https://github.com/genodelabs/genode-allwinner/blob/master/src/a64_linux/target.inc - kernel configuration]
as used for the a64_linux target we discussed earlier in
Section [Taking Linux out for a Walk].

Identifying Linux source codes of interest
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

How to spot the files needed to drive our network device among the many
thousands of C files found in the Linux source tree?

Taking the device tree as our guide
-----------------------------------

In the remainder of this article, we refer an all-encompassing device-tree
source (DTS) file _flat_pine64lts.dts_ for our board. This file can be
extracted from the Linux kernel source via the C preprocessor as
described in Section [Device-tree treasure trove].

Given the _flat_pine64lts.dts_ file, let Genode's _dts/extract_ tool
(Section [Pruning device trees]) guide our attention:

!$ .../tool/dts/extract --select emac flat_pine64_lts.dts \
!                      | grep "compatible = "
!  compatible = "fixed-clock";
!  compatible = "fixed-clock";
!  compatible = "simple-bus";
!   compatible = "allwinner,sun50i-a64-system-control";
!    compatible = "mmio-sram";
!     compatible = "allwinner,sun50i-a64-sram-c";
!    compatible = "mmio-sram";
!     compatible = "allwinner,sun50i-a64-sram-c1",
!   compatible = "allwinner,sun50i-a64-ccu";
!   compatible = "allwinner,sun50i-a64-pinctrl";
!   compatible = "allwinner,sun50i-a64-emac";
!    compatible = "snps,dwmac-mdio";
!   compatible = "arm,gic-400";
!   compatible = "allwinner,sun50i-a64-rtc",
!   compatible = "allwinner,sun50i-a64-r-intc",
!   compatible = "allwinner,sun50i-a64-r-ccu";
!   compatible = "allwinner,sun50i-a64-r-pinctrl";
!   compatible = "allwinner,sun8i-a23-rsb";
!  compatible = "x-powers,axp803";
! compatible = "pine64,sopine-baseboard", "pine64,sopine",
!  compatible = "ethernet-phy-ieee802.3-c22";
! compatible = "pine64,pine64-lts", "allwinner,sun50i-r18",

Note that some 'compatible' attributes span multiple lines (the lines
ending with a comma). So it's probably best to manually inspect the
device-tree source to get the full information.

Recap that we already used the 'compatible' device-node attributes
in Section [Enabling network support]
to connect the dots between the device tree and kernel-configuration options.
Analogously, we can use those attribute values to look up the associated
source codes.

Let's take "snps,dwmac-mdio" as an pattern to grep Linux source tree:

! linux$ grep -rl "snps,dwmac-mdio" drivers
! drivers/net/ethernet/stmicro/stmmac/stmmac_platform.c

By looking at the Makefile where the driver code is located, we can
immediately spot the set of driver sources declared by looking at the listed
object files. This information is all we need to expand our _sources.list_
file.

! ...
! drivers/net/ethernet/stmicro/stmmac/stmmac_platform.c
! drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
! drivers/net/ethernet/stmicro/stmmac/stmmac_ethtool.c
! drivers/net/ethernet/stmicro/stmmac/stmmac_mdio.c
! drivers/net/ethernet/stmicro/stmmac/ring_mode.c
! drivers/net/ethernet/stmicro/stmmac/chain_mode.c
! drivers/net/ethernet/stmicro/stmmac/dwmac_lib.c
! drivers/net/ethernet/stmicro/stmmac/dwmac1000_core.c
! drivers/net/ethernet/stmicro/stmmac/dwmac1000_dma.c
! drivers/net/ethernet/stmicro/stmmac/dwmac100_core.c
! drivers/net/ethernet/stmicro/stmmac/dwmac100_dma.c
! drivers/net/ethernet/stmicro/stmmac/enh_desc.c
! drivers/net/ethernet/stmicro/stmmac/norm_desc.c
! drivers/net/ethernet/stmicro/stmmac/mmc_core.c
! drivers/net/ethernet/stmicro/stmmac/stmmac_hwtstamp.c
! drivers/net/ethernet/stmicro/stmmac/stmmac_ptp.c
! drivers/net/ethernet/stmicro/stmmac/dwmac4_descs.c
! drivers/net/ethernet/stmicro/stmmac/dwmac4_dma.c
! drivers/net/ethernet/stmicro/stmmac/dwmac4_lib.c
! drivers/net/ethernet/stmicro/stmmac/dwmac4_core.c
! drivers/net/ethernet/stmicro/stmmac/dwmac5.c
! drivers/net/ethernet/stmicro/stmmac/hwif.c
! drivers/net/ethernet/stmicro/stmmac/stmmac_tc.c
! drivers/net/ethernet/stmicro/stmmac/dwxgmac2_core.c
! drivers/net/ethernet/stmicro/stmmac/dwxgmac2_dma.c
! drivers/net/ethernet/stmicro/stmmac/dwxgmac2_descs.c

Taking the Linux build directory as our guide
---------------------------------------------

Alternatively to taking the device-tree as the starting point, the build
directory of our bare-bones Linux kernel contains instructive information,
specifically the object files that went into the kernel.

! build/arm_v8a$ find a64_linux/ -name "*.o"

Remember that we have previously slimmed down the Linux kernel configuration
as far as we could, keeping only the bare minimum needed for networking. So
the list of object files found in the Linux build directory serves as a
reasonably small superset of the compilation units that are of interest to us.
Any compilation unit not listed cannot be important.

By combining both perspectives, taking the 'compatible' device-nodes
attributes as cues while using the Linux kernel's object files as plausibility
check, our mental picture of the driver code and its dependencies becomes more
and more clear and our _sources.list_ file grows.

Build test
----------

With the _sources.list_ enriched with the list of Linux source codes we are
interested in, let's have the Genode build system chew a bit on our driver:

! build/arm_v8a$ make drivers/emac
!  ...
!  Program drivers/nic/emac/emac_nic_drv
!    COMPILE  arch/arm64/lib/memchr.o
!    COMPILE  arch/arm64/lib/memcmp.o
!    ...
!    COMPILE  drivers/net/ethernet/stmicro/stmmac/stmmac_main.o
!    COMPILE  drivers/net/ethernet/stmicro/stmmac/stmmac_platform.o
!    COMPILE  drivers/net/ethernet/stmicro/stmmac/stmmac_mdio.o
!    COMPILE  drivers/net/ethernet/stmicro/stmmac/stmmac_ptp.o
!    COMPILE  drivers/net/ethernet/stmicro/stmmac/stmmac_tc.o
!    COMPILE  dummies.o
!    COMPILE  main.o
!    LINK     emac_nic_drv
! drivers/net/ethernet/stmicro/stmmac/stmmac_main.o: in function `stmmac_cmdline_opt':
!                        stmmac_main.c:5379: undefined reference to `strsep'
! ...
! ... many more undefined references
! ...

We see the Linux source code being picked up and compiled! The build system
backs out not before the linking stage. During linking, however, the many
inter-dependencies of the driver code from the rest of the Linux kernel become
visible in the form of "undefined reference" errors.

Tying the loose ends to make the linker happy
---------------------------------------------

In principle, we could inspect and resolve each of those linking errors
manually. But given the volume of error messages, this would be tedious. In
this situation, Genode's _dde_linux/create_dummies_ tool comes to the rescue.

Let's remember the location of our driver's _generated_dummies.c_ file in an
environment variable named 'DUMMY_FILE', which will later be evaluated by the
_create_dummies_ tool:

!build/arm_v8a$ export DUMMY_FILE=/path/to/src/drivers/emac/generated_dummies.c

With the 'DUMMY_FILE' defined, we can instruct the tool to fill the file with
a dummy implementation for each of the unresolved references reported by the
linker:

!build/arm_v8a$ echo > $DUMMY_FILE ;\
!               ../../tool/dde_linux/create_dummies generate \
!                                    LINUX_KERNEL_DIR=a64_linux \
!                                    TARGET=drivers/nic/emac ;\
!               make drivers/nic/emac

In the command line above, we first wipe any prior content from the file,
then invoke the create_dummies tool, followed by another build of the driver
with the new version of generated file.
The _create_dummies_ tool is described in more detail at a dedicated
[https://genodians.org/skalk/2021-04-08-dde-linux-experiments-1#Generate_missing_implementations - article].

With the dummies generated, the linking of the executable binary succeeds!
This won't be the last time of issuing the command. In fact, each time after
adding or removing any content of the _sources.list_ file, it is best to
update _generated_dummies.c_ using the command above.

Executable testbed
~~~~~~~~~~~~~~~~~~

With a first executable binary built, it is time to give it a first run.
At this point, we are not yet concerned about accessing any actual hardware.
We merely want to obtain the first life sign of the component and see any
hint of Linux initialization code being executed.
The following run script named after the driver - in our case
_a64_emac_drv.run_ is an appropriate name - can serve as our initial test
scenario.

! build { core init timer drivers/platform drivers/nic/emac }
!
! create_boot_directory
!
! install_config {
!   <config>
!     <parent-provides>
!       <service name="LOG"/>
!       <service name="PD"/>
!       <service name="CPU"/>
!       <service name="ROM"/>
!       <service name="IO_MEM"/>
!       <service name="IRQ"/>
!     </parent-provides>
!
!     <default caps="100"/>
!
!     <start name="timer">
!       <resource name="RAM" quantum="1M"/>
!       <route> <any-service> <parent/> </any-service> </route>
!       <provides> <service name="Timer"/> </provides>
!     </start>
!
!     <start name="platform_drv">
!       <resource name="RAM" quantum="1M"/>
!       <provides> <service name="Platform"/> </provides>
!       <route> <any-service> <parent/> </any-service> </route>
!       <config devices_rom="config">
!         <policy label="emac_nic_drv -> ">
!         </policy>
!       </config>
!     </start>
!
!     <start name="emac_nic_drv">
!       <resource name="RAM" quantum="1M"/>
!       <route>
!         <service name="ROM"> <parent/> </service>
!         <service name="CPU"> <parent/> </service>
!         <service name="PD">  <parent/> </service>
!         <service name="LOG"> <parent/> </service>
!         <service name="Timer">    <child name="timer"/> </service>
!         <service name="Platform"> <child name="platform_drv"/> </service>
!       </route>
!       <config/>
!     </start>
!
!   </config>
! }
!
! build_boot_image { core ld.lib.so init timer platform_drv
!                    emac_nic_drv emac-pine_a64lts.dtb }
!
! run_genode_until forever

The 'emac_nic_drv' is connected to the 'platform_drv' but no device
resources are assigned to the corresponding policy yet. In fact, the
device hardware remains completely untouched, which allows us to
execute the run script for an arbitrary boards, in particular Qemu.
The use of Qemu instead of the targeted board at this early stage
is convenient.

The dtb file 'emac-pine_a64lts.dtb' as integrated into the boot
image is a side-product of building the emac driver. It is created according
to the definition of the 'BOARDS' and 'DTS_EXTRACT' variables in the driver's
target.mk file.

Linux initcalls
~~~~~~~~~~~~~~~

When executing the run script exemplified above, we are greeted with the
following log output.

! kernel initialized
! ROM modules:
!  ROM: [0000000040120000,0000000040120469) config
!  ROM: [0000000040009000,000000004000a000) core_log
!  ROM: [0000000040149000,000000004014a094) emac-pine_a64lts.dtb
!  ROM: [0000000040207000,000000004023cbe8) emac_nic_drv
!  ROM: [000000004023d000,00000000402841f8) init
!  ROM: [0000000040159000,0000000040206880) ld.lib.so
!  ROM: [0000000040121000,0000000040148d30) platform_drv
!  ROM: [0000000040007000,0000000040008000) platform_info
!  ROM: [000000004014b000,0000000040158710) timer
!
! Genode 21.05-10-g51f02a668d9
! 2010 MiB RAM and 64533 caps assigned to init
! [init -> emac_nic_drv] Error: Initcall __initcall_initialize_ptr_randomearly unknown in initcall database!
! [init -> emac_nic_drv] Error: Initcall __initcall_init_jiffies_clocksource1 unknown in initcall database!
! [init -> emac_nic_drv] Error: Initcall __initcall_stmmac_init6 unknown in initcall database!
! [init -> emac_nic_drv] Error: Initcall __initcall_sync_state_resume_initcall7 unknown in initcall database!
! [init -> emac_nic_drv] Error: Initcall __initcall_devlink_class_init2 unknown in initcall database!
! [init -> emac_nic_drv] Error: Function kmem_cache_init not implemented yet!
! [init -> emac_nic_drv] Backtrace follows:
! [init -> emac_nic_drv] 0x1024034
! [init -> emac_nic_drv] 0x1016898
! [init -> emac_nic_drv] 0x1022498
! [init -> emac_nic_drv] Will sleep forever...

The messages "Error: Initcall ... unknown in initcall database!" tell us that
the Linux code we incorporated into our component features initcalls that are
unknown to the 'lx_emul' execution environment. Hence, lx_emul won't know
the order, in which those calls must be executed. More information about
the initcall mechanism is available in a
[https://genodians.org/skalk/2021-04-08-dde-linux-experiments-1#Initcalls - dedicated article].

For us, it is important to know that the initcall order is supplemented to the
lx_emul library via the header file at _src/include/lx_emul/initcall_order.h_.
The content of this file depends on the Linux kernel configuration.
Fortunately, we won't need to maintain this header file by hand. Instead, the
handy _extract_initcall_order_ tool allows us to generate this file from the
_System.map_ of a built Linux kernel:

! build/arm_v8a$ ../../tool/dde_linux/extract_initcall_order extract \
!        LINUX_KERNEL_DIR=a64_linux
!        HEADER_FILE=/path/to/drivers/src/include/lx_emul/initcall_order.h

Note that the directory specified at 'LINUX_KERNEL_DIR' must contain a built
Linux kernel, specifically the _System.map_ file.

When re-running the run script with the updated _initcall_order.h_, the
initcall-related errors should disappear.

The lx_emul building blocks
~~~~~~~~~~~~~~~~~~~~~~~~~~~

A high-level overview of the anatomy of a DDE-Linux-based driver is provided
in the release documentation of
[https://genode.org/documentation/release-notes/21.08#Linux-device-driver_environment_re-imagined - Genode 21.08].
The lx_emul library provides three kinds of build blocks.

First, it provides
a custom C interface for low-level mechanisms of the runtime. The
corresponding functions are prefixed with 'lx_emul_'. The interface is
provided at _dde_linux/include/lx_emul/_.

Second, it provides alternative implementations of low-level Linux subsystems.
Those implementations reside at _dde_linux/src/lib/lx_emul/shadow/_.
For example, _shadow/mm/slub.c_ is an alternative to Linux'
_mm/slub.c_ that provides the same binary interface but implements it by the
means of the lx_emul mechanisms.

And third, it provides a few shadow headers at
_dde_linux/src/include/lx_emul/shadow/_ that strip away or tweak a few
unpleasant parts of the Linux-internal interfaces. In particular, it redirects
Linux' original initcall mechanism to the use of 'lx_emul_register_initcall'
and it hides low-level aspects of the memory model that are incompatible with
Genode. The latter is concerned with the conversion between virtual addresses,
'struct page' pointers, and DMA addresses.

Iterative crafting of the driver's runtime environment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The message "Error: Function ... not implemented yet!" (in the log output
above) followed by a backtrace is triggered by one of the dummy
implementations in 'generated_dummies.c'. It tells us that we need to replace
this particular dummy with either

* A dummy implementation in the manually curated 'dummies.c' with the
  call to 'lx_emul_trace_and_stop' replaced by a meaningful return value, or
* An actual implementation of the function in 'lx_emul.c', or
* Additional Linux source codes incorporated by extending the _sources.list_.

The decision must be taken on a case-by-case basis. To take the decision, it
is worthwhile to inspect the existing drivers. Drivers of the same kind
(network, framebuffer) tend to show similar patterns across SoCs.

Our job at this stage is the repeated execution of the run script while
resolving one unimplemented function in each iteration. Sometimes, this
process requires a deep dive into parts of the Linux kernel architecture in
order to asses the potential impact of the called dummy on the correct
functioning of the driver. Sometimes mere intuition may guide us. In any
case, the backtrace printed in the log output can be immensely helpful.
You may remember the *addr2line* utility mentioned in Section
[Option 2: One step of ground truth at a time].
It accepts an arbitrary sequence of numbers as standard input when started as
follows:

!build/arm_v8a$ /usr/local/genode/tool/current/bin/genode-aarch64-addr2line \
!                                          -e drivers/nic/emac/emac_nic_drv

To process the list of hexadecimal numbers appearing in the log, I use to copy
the numbers from the terminal using a rectangular selection (by pressing the
control key while selecting an area with the mouse) and pasting the content
into the 'addr2line' instance.

Moving to the target hardware
-----------------------------

At one point, we will ultimately reach a point where the driver tries to
obtain access to device resources.

![init -> emac_nic_drv] Error: memory-mapped I/O resource 0x1c00000
!                              (size=0x1000) unavailable

It's time to move the development from Qemu to the actual target hardware.

In order to grant the driver access to the requested resource, we first look
up the requested address in the _flat_pine64lts.dts_ file. In the example
above, the range belongs to a device called 'syscon'. With the information
found in the device node, we can enrich the platform driver's configuration
accordingly.

! <config>
!   <device name="syscon" type="allwinner,sun50i-a64-system-control">
!     <io_mem address="0x1c00000" size="0x1000"/>
!   </device>
!   <policy label="emac_nic_drv -> " info="yes">
!     <device name="syscon"/>
!   </policy>
! </config>

The change consist of two parts. First, a '<device>' is declared. Note that
the 'type' corresponds to the 'compatible' attribute of the DTS device node.
Second, the '<policy>' for the emac_nic_drv is changed to grant access of this
device to the driver. It is important to set the 'info' attribute to "yes",
which allows the driver to read the device meta information given in the
'<device>' node.

Besides memory-mapped I/O registers, device interrupts are the second type of
hardware resource of interest for device drivers. Sooner or later during the
driver initialization, we encounter a message like the following.

! [init -> emac_nic_drv] Error: irq 114 unavailable

The driver requests GIC interrupt 114. To find out what's behind this number,
the _flat_pine64lts.dts_ file is the right place for seeking the ground truth.
Note that interrupt numbers as found in DTS files correspond to GIC interrupts
numbers minus 32. So GIC interrupt 114 appears as number 114 - 32 = *82*. A
search in the DTS file for this number leads us to the matching device.

! emac: ethernet@1c30000 {
!   ...
!   reg = <0x01c30000 0x10000>;
!   ...
!   interrupts = <0 82 4>;
!   ...
! };

This information is all we need to craft a corresponding '<device>' node for
the platform-driver configuration.

! <device name="emac" type="allwinner,sun50i-a64-emac">
!   <io_mem address="0x1c30000" size="0x10000"/>;
!   <irq number="114"/>
! </device>

Linux caveats
~~~~~~~~~~~~~

In the past, we repeatedly encountered two kinds of trip-wires that one should
always keep in the back of the mind, namely Linux linker-script magic and
global variables.

A few kernel mechanisms depend of special support at the linker-script level,
most notably various flavours of initcall mechanisms, sometimes disguised as
a global table ('__clk_of_table', '__irqchip_of_table') magically created by
scattered table entries assigned to a special linker section. In contrast
to Linux, we cannot rely on linker-level mechanisms if we want to keep using
Genode's regular linker script. The lx_emul environment takes care of the
initcall flavors that we encountered so far using the pattern described
[https://genodians.org/skalk/2021-04-08-dde-linux-experiments-1#Initcalls - here].
But we know that there exist more categories of initcalls. In the event that a
certain initialization function is unexpectedly not called, it is worth
skimming the symbols of 'generated_dummies.o' for variables with 'table' in
their name. Another example for linker magic is the aliasing between the
'jiffies' and 'jiffies64' variables. Both variables must refer to the same
memory location (on little-endian architectures). This concrete issue is
covered by the _lib/import/import-a64_lx_emul.mk_ file.

The second trip wire is the presence of global variables that are specially
initialized by compilation units not featured in _sources.list_. In this case,
the _generated_dummies.c_ creates default-initialized variable instances,
which can break innocent library functions in subtle ways. For example,
_lib/hexdump.c_ contains the following global variable:

! const char hex_asc_upper[] = "0123456789ABCDEF";
! EXPORT_SYMBOL(hex_asc_upper);

This variable is implicitly used by _lib/vsprintf.c_ when printing "%d" format
strings. If default-initialized, a digit is wrongly rendered as null
(terminating the string) instead of the corresponding ASCII value, producing
all kinds of funny effects down the road. The global variable defined in
_lib/ctypes.c_ is equally important. If default-initialized, 'toupper' and
'strcasecmp' won't work as expected, breaking the program logic when used as
condition.

As a rule of thumb, when encountering erratic behavior, one should look out
for global variables in _generated_dummies.c_ and investigate their purpose.

Enabling Linux debug messages
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Several parts of the Linux kernel are garnished with debug messages that
reveal valuable insights of the kernel's behavior beyond the regular log
messages. The easiest way to obtain those messages for a given compilation
unit is adding the following line right at the beginning of the source file,
above the first '#include' directive:

! #define DEBUG

When booting Linux, one has to supply "debug" as kernel-command line argument.
When running the driver as Genode component, no further precaution is needed.

With respect to debug instrumentation, the following compilation units are
particularly fruitful:

:drivers/of/fdt.c:

  On ARM platforms, the kernel initialization is driven by the information
  of the supplied device tree. By enabling 'DEBUG' in this compilation unit,
  one becomes able to observe the processing of the device nodes found in the
  device-tree and how they are matched with the available drivers.

:drivers/base/dd.c:

  By enabling 'DEBUG' in this compilation unit, the probing of devices by
  the various drivers becomes visible. This is particular important for
  devices that depend on each other. Whenever a driver finds a precondition -
  such as the presence of another driver - not met, it backs out of the
  probing via Linux' defer mechanism ('EPROBE_DEFER'). Whenever the deferral
  of probing diverges between native Linux and the ported driver, one
  should investigate the root cause of the condition that led (or did not led)
  to an 'EPROBE_DEFER'.

Logging the execution of initcalls
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To unveil the execution sequence of initcalls and for relating messages
and backtraces printed in the log with the corresponding Linux code, two
little instrumentations are of great help.

:repos/dde_linux/src/lib/lx_emul/init.cc:

  By adding a message like the following to the 'lx_emul_register_initcall'
  function, we become able to relate the names of initcall functions with
  their corresponding addresses.

  ! Genode::log("lx_emul_register_initcall ", name, " call=", (void *)initcall);

  Since 'lx_emul_register_initcall' is called immediately at
  component construction time using the global ctors mechanism, this
  instrumentation gives us a complete list of initcalls. The names of those
  calls can easily be grep'ed in the Linux code to determine a suitable
  starting points for manual instrumentations.

:src/lib/lx_kit/init.cc:

  By changing the implementation of 'Lx_kit::Initcalls::execute_in_order'
  to print a log message in addition to 'entry->call()', we can know
  exactly when each initcall is executed.

  ! log("exec init call ", (void *)entry->call);

  The printed addresses correspond to those obtained in the first
  instrumentation. So when the kernel initialization gets stuck somewhere,
  one can look at the sequence of initcalls - and in particular the last
  initcall executed - that led to the situation.

Obtaining backtraces of blocked Linux tasks
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The Linux kernel code is not executed in a straight linear fashion but in the
form of many kernel threads that interact with each other using a variety
of mechanisms such as work queues. The lx_emul runtime implements a cooperative
task-execution model that folds all Linux kernel threads into a single flow
of control. To see what the Linux kernel threads are doing and in particular
the situation when they enter a blocking state, the following instrumentation
in _dde_linux/src/lib/lx_kit/task.cc_ is invaluable.

! #include <os/backtrace.h>
!
! void Task::block()
! {
!   log("Task::block: ", _name);
!   backtrace();
!   ...

The 'Task::block' method is called whenever a Linux kernel thread blocks. By
adding the two lines at the beginning of the method, we get hold of the
situation at each single task switch. It prints the name of the blocked kernel
thread along with the backtrace of the thread.

Another suitable point for an instrumentation is the 'Task::run' method. By
printing the '_name' after the '_setjmp' branch, one can obtain the sequence
of resumed (as opposed to blocked) kernel threads.

! void Task::run()
! {
!   if (_setjmp(_saved_env))
!     return;
!
!   log("Task::run: ", _name);
!   ...

De-referenced null pointers
~~~~~~~~~~~~~~~~~~~~~~~~~~~

The Linux kernel is anything but short of function pointers and callbacks. Hence,
sooner or later during the development, one may be faced with a de-referenced
null pointer like this:

! no RM attachment (READ pf_addr=0x18c pf_ip=0x1008fa0 from pager_object:
!                        pd='init -> emac_nic_drv' thread='ep')
! Warning: page fault, pager_object: pd='init -> emac_nic_drv' thread='ep'
!                                    ip=0x1008fa0 fault-addr=0x18c type=no-page

The very small page-fault address ('pf_addr') hints at a de-referenced null
pointer. The first impulse is looking up the instruction pointer 'pf_ip' in
the driver's binary using 'objdump'.

! build/arm_v8a$ /usr/local/genode/tool/current/bin/genode-aarch64-objdump \
!                            -lSd drivers/nic/emac/emac_nic_drv | less

In less, when searching for the instruction pointer value ('1008fa0'), one
can see the surroundings of the offending code.

! 1008f9c:       aa0003f3        mov     x19, x0
!.../src/linux/drivers/base/regmap/regmap.c:2720
!        if (!IS_ALIGNED(reg, map->reg_stride))
! 1008fa0:       b9418c00        ldr     w0, [x0, #396]

Could 'map' be a null pointer? If so, why? When looking into the code at the
displayed coordinates _regmap.c_ at line 2720, we encounter the function
'regmap_read' as a suitable point for instrumentation.

!#include <lx_emul.h>
!...
!int regmap_read(struct regmap *map, unsigned int reg, unsigned int *val)
!{
!  int ret;
!  printk("regmap_read map=%p\n", map);
!  if (!map)
!    lx_emul_trace_and_stop(__func__);
!
!  if (!IS_ALIGNED(reg, map->reg_stride))
!    return -EINVAL;

The 'printk' should validate our hypothesis that 'map' is indeed a null
pointer - just to double-check. The 'lx_emul_trace_and_stop' call gives us the
backtrace in this interesting situation. When running the code with these
instrumentations in place, the following output appears.

![init -> emac_nic_drv] regmap_read map=0
![init -> emac_nic_drv] Error: Function regmap_read not implemented yet!
![init -> emac_nic_drv] Backtrace follows:
![init -> emac_nic_drv] 0x1008fc0
![init -> emac_nic_drv] 0x1009044
![init -> emac_nic_drv] 0x100a9b8
![init -> emac_nic_drv] 0x10076c0
![init -> emac_nic_drv] 0x10060f8
![init -> emac_nic_drv] 0x1006938
![init -> emac_nic_drv] 0x10069a4
![init -> emac_nic_drv] 0x1001320
![init -> emac_nic_drv] 0x1001ac4
![init -> emac_nic_drv] 0x10074a0
![init -> emac_nic_drv] 0x1056f28
![init -> emac_nic_drv] 0x10481e8
![init -> emac_nic_drv] 0x10580f8

Thanks to the backtrace, we can track where the 'map' pointer comes from,
ultimately ending up at the call of 'syscon_regmap_lookup_by_phandle'.
In our case, this function was (wrongly) stubbed with a dummy function
returning NULL. As a way to double-check that the return value of this
function indeed corresponds to the de-referenced null pointer, one can tweak
the return value a little, returning a smallish magic number.

!struct regmap * syscon_regmap_lookup_by_phandle(struct device_node * np,
!                                                const char * property)
!{
!  return (struct regmap *)0x550;
!}

In the next run, we can observe that the page-fault address indeed changed
from 0x18c to 0x6dc.

! no RM attachment (READ pf_addr=0x6dc pf_ip=0x1008fc0 from pager_object:
!                   pd='init -> emac_nic_drv' thread='ep')
! Warning: page fault, pager_object: pd='init -> emac_nic_drv' thread='ep'
!                                    ip=0x1008fc0 fault-addr=0x6dc type=no-page

Apparently, we cannot simply shortcut the 'syscon_regmap_lookup_by_phandle'
function.

Ruling out potential cache-coherency issues
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Once the driver starts to interact with the device hardware, additional
uncertainties enter the picture. The most uncertain uncertainty is certainly
cache coherency. Nowadays, Linux drivers preferably use cached page mappings
for DMA buffers and manage the coherency between the device's perspective and
the CPU's perspective on those buffers via explicit cache-management (flush,
invalidate) operations. This cache management happens behind the surface of
functions like 'dma_map_page_attrs'.

To rule out the presence of cache coherency issues, we can force the driver to
use uncached mappings only by tweaking the allocators at
_dde_linux/src/include/lx_kit/env.h_. By changing the 'CACHED' argument of the
'memory' member to 'UNCACHED', all memory dynamically allocated by Linux
kernel code will be backed by uncached memory.

Should the driver work with this tweak, one can be pretty sure to have hit a
cache-coherency issue, likely missing the correct implementation of a
'dma_map' / 'dma_unmap' operation. Should the driver still does not work, the
problem lies somewhere else. Now would be the time to suspect cosmic rays.

Using Linux' built-in DHCP support as networking test
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Before equipping the driver with a Genode session interface, it is recommended
to first execute its core functionality as a standalone program. For a network
driver, the core functionality is the transmission and reception of network
packets.

The Linux kernel features builtin support for obtaining an IPv4 network
configuration at boot time via DHCP. The network-configuration protocol
involves the successful transmission and reception of multiple network
packets, and its completion is indicated by the IP address printed in the
kernel log. In other words, DHCP is the ideal first test workload for the
driver. It requires the following kernel configuration options:

! INET
! IP_PNP
! IP_PNP_DHCP

The implementation resides in _net/ipv4/ipconfig.c_, which must be added to
the _sources.list_ file of the driver.
When being part of a regular Linux kernel, this code evaluates the kernel
command line, namely the option "ip=dhcp". Since the lx_emul environment
has no notion of a kernel command line, we can manually force the code to
issue the DHCP request by modifying the implementation of the
'ip_auto_config' function by adding calls to 'skb_init' and
'ip_auto_config_setup' at the beginning of the function (right after the local
variable declarations).

!static int ip_auto_config_setup(char *addrs);
!
!static int __init ip_auto_config(void)
!{
!  ...
!  skb_init();
!  ip_auto_config_setup("dhcp");
!   ...

Capturing network traffic
~~~~~~~~~~~~~~~~~~~~~~~~~

There exist many ways to capture network traffic for observing the
interchange of DHCP protocol messages at the DHCP-server side. The tshark
tool is particularly nice. For capturing the traffic related to the MAC
address of my board, the following command line does an excellent job:

! tshark -i eno1 -t ad -Y 'eth.addr == 02:ba:fe:7b:59:38'

Using flood ping as a rudimentary stability check
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Once the driver has reached a seemingly operational state, having successfully
completed DHCP, it is a good time to put some stress on the driver. As a
litmus test, its interesting to see if a flood ping brings the driver to its
knees:

! sudo ping -f -c 1000 -s 1000 <ip-address>

Cross-correlation against the Linux kernel behavior
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Sharing one Linux kernel configuration among both our bare-bones Linux kernel
and the ported driver code allows for the detailed cross-correlation of the
driver behavior. This consistency is fostered by the
[https://github.com/genodelabs/genode-allwinner/blob/master/src/a64_linux/target.inc - src/a64_linux/target.inc]
file that is used by both the a64_linux target (configuring and building a
Linux kernel) and each driver component (via the a64_linux_generated library).
This way, any instrumentation of the Linux kernel code can be quickly tested
in both execution environments.

Connecting the driver with a Genode session interface
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Once we have validated that the driver is able to send and receive network
packets via the DHCP test, the time is ripe for integrating it into the Genode
environment. This integration comes down to two aspects. First, the test
scenario must be changed to move the network application into a component
separate from the driver, and second, the driver must interact with Genode's
uplink session interface.

The first part, the network application, can be accommodated by the NIC router
component. For reference, the following '<start>' node creates an instance
of the NIC router that issues a DHCP request once an uplink appears, and
prints the obtained IP address in the log.

! <start name="nic_router" caps="200">
!   <resource name="RAM" quantum="10M"/>
!   <provides>
!     <service name="Nic"/>
!     <service name="Uplink"/>
!   </provides>
!   <route>
!     <service name="Timer"> <child name="timer"/> </service>
!     <any-service> <parent/> </any-service>
!   </route>
!   <config verbose_domain_state="yes" dhcp_discover_timeout_sec="1">
!     <policy label_prefix="emac_nic_drv" domain="uplink"/>
!     <domain name="uplink"/>
!   </config>
! </start>

Of course, the driver must be able to reach the NIC router, which can be
achieved adding the following session route to the driver's '<start>' node.

! <start name="emac_nic_drv" caps="2000">
!   ...
!   <route>
!     ...
!     <service name="Uplink"> <child name="nic_router"/> </service>
!     ...
!   </route>
! </start>

The second part - bridging the gap between the Linux kernel code and Genode's
session interface - can best be addressed by the
[https://github.com/genodelabs/genode/blob/master/repos/os/include/genode_c_api/uplink.h - genode_c_api/uplink.h]
API and an implementation of the driver's
[https://github.com/genodelabs/genode-allwinner/blob/master/src/drivers/nic/emac/lx_user.c - lx_user.c],
which connects the 'genode_c_api' with the Linux netdevice interface.

Packaging the driver
~~~~~~~~~~~~~~~~~~~~

The final step is the packaging of the driver to make it available to a broad
range of Genode scenarios, in particular the run scripts based on the
'drivers_nic' subsystem such as _libports/run/fetchurl_lwip.run_.

The packaging is assisted by the _dde_linux/list_dependencies_ tool.
It determines the list of header dependencies for our driver by examining
dependency (.d) files. For reference, the _nic/emac/dep.list_ file is
generated via following command (the paths are abbreviated).

!build/arm_v8a$ ../../tool/dde_linux/list_dependencies \
!               TARGET_DIR=drivers/nic/emac \
!               LINUX_KERNEL_DIR=/path/to/linux/source/ \
!               SOURCE_LIST_FILE=.../allwinner/src/drivers/nic/emac/source.list \
!               DEP_LIST_FILE=.../allwinner/src/drivers/nic/emac/dep.list \
!               generate

As reference for the recipe files needed, the depot recipes at
[https://github.com/genodelabs/genode-allwinner/tree/master/recipes - allwinner/recipes/]
are helpful. Their roles are as follows:

:recipes/api/a64_linux/: This API recipe contains the parts of the Linux source tree
  that are relevant to build the drivers. It also features the parts of the
  Linux build system that are invoked to generate header files
  (for the a64_linux_generated library). Each DDE-Linux-based driver
  depends on this API archive. This recipe notably uses the information of the
  _dep.list_ and _source.list_ files.

:src/a64_emac_nic_drv/: This source archive contains the Genode parts of the
  network drivers. The Linux sources are taken from the api/a64_linux archive.

:recipes/pkg/drivers_nic-pine_a64lts/: This package aggregates all ingredients
  needed for a network-driver subsystem as expected by scenarios based on
  the convention of _drivers_nic_ packages, that is, run scripts using

  ! import_from_depot ... \
  !                   [depot_user]/pkg/[drivers_nic_pkg]

:recipes/raw/drivers_nic-pine_a64lts:

  The raw archive contains the init configuration of the driver subsystem.

While crafting those recipes, it is best to use a dummy depot user "x" so
that the intermediate results can easily be removed from the depot afterwards.
For reference, the following command extracts the archives from the source
tree and builds the binaries for the 'arm_v8a' architecture. The process
of developing the recipes comes down to repeatedly issuing this command
and extending the recipes until the binary archives are successfully built.

!genode$ ./tool/depot/create x/pkg/arm_v8a/drivers_nic-pine_a64lts \
!                            -j8 \
!                            FORCE=1 \
!                            UPDATE_VERSIONS=1