Linux for Embedded Devices - A deep dive

In the previous article, we saw what Linux is, what it is suitable for, and what is the basic structure from a kernel view. In the present article we discussed, deeper, what is the structure of that called userland.

In the land… of the users!

Normally, the application processors suitable to run Linux have an MMU (memory management unit) and (at least) two level of execution privileges:

Kernel mode: The most permissive mode that allows complete hardware access (in terms of memory access and instruction execution). The linux kernel is a big C program executing in this mode (all of the code… including dynamically loaded modules)
User mode: The most restrictive mode that disallows access to hardware and restricts the execution of instructions that can change the execution mode. The code running in this mode is known as “Userland code”.

The “userland” or “userspace” nomenclature, in opposition to “kernel land”, calling to “all things out of the kernel” refers to the set of code and data that are not executed as part of the kernel; this is: don’t have system privileges and they can't access the hardware except by making use of kernel services.

The exact manner to which userland is connected with kernel space is architecturally dependent but if the processor provides user mode execution, it also provides a special instruction or method to call kernel code in a controlled and secure way.

For example, in the old 80386, that did not have any special instruction to call the kernel, the operating system used a software interrupt for this (INT 0x80 in linux case, INT 0x2E in windows NT and similar). The i686 and later processors add the SYSCALL instruction for this purpose.

In any case, the transition from user space to kernel space is more expensive than normal instruction execution and should be minimized for a performant code.

For this reason, the major surface of the code is executed in userland by security reasons (if a userland code is malformed or writed by malicious intentions, the kernel space can catch the abnormal behavior and take actions like stop program or throws an error to other program)

Citizens of the userland

In this diagram, you can see a simplified schema with the common userland components:

C Library: Linux is made in C, and the C library is ubiquitous across all programs (even if the program is not written in C, it makes use of C library in the majority of cases). The C library is the most important library in the system because it defines a minimal set of functions and provides a compatibility layer for all dynamically linked programs.
Dynamic loader: a C program independently of any library that resolves dynamic library dependencies, loads it in memory and manages the sharing across all programs that use the libraries. A tiny linux system without dynamic load lacks this.
Various libraries that provide many functionality like network, input/output access, or graphics.
Programs that use these libraries and share data between them.
Programs that do not use any libraries and incorporate these code in a single binary. These programs are known as “statically linked programs” because all libraries are bundled in a single file.

Minimal linux system example

To illustrate the previous point, let's see an example of a minimal (but realistic) linux system:

In this example we can show any elements mentioned until this point:

A big C program as a Linux kernel with a devicetree attached to this containing system drivers and layout with some driver initialization options.
An in-RAM filesystem acting as a initramfs (maybe attached to the kernel with de devicetree in a single file) containing all userland elements:

A C library that contains basic code with functionality for others programs.
A various binaries providing specific functionalities like init (system initialization and program launching), remote shell access program and minimal system utilities like file copy, remove, list directory, required used by the scripts above this.
An init script containing all programs and their parameters to run (and the order to be executed).

The loader in this case is containing as a separated binary from C library but is part of the same program.
The utility marked as a busybox is really a single program that contains all required utilities like CP, LS, DF, MV, RM, even INIT or TELNETD.
All of this is started with the init process.

The way of boot process, making it all start

Before the kernel:

In order to get a functional Linux system, the hardware needs to load the kernel, the device tree and additionally find the root filesystem with the userland and put all in a working state.

The way that this happens varies from platform to platform but, in general, the process is beastly standardized.

The processor is power on: At this point, the CPU executes some first boot code contained internally in the chip (normally as a masked ROM, FLASH or other internal non-volatile storage). This piece of code is known as a first stage bootloader.

The first stage bootloader, initialize the processor and peripherals and find a piece of code into external devices like microSD card, eMMC, external flash or other devices (in old desktop PCs this piece is equiparable to the BIOS)

The first stage bootloader loads a piece of code from external storage (normally a fixed-size of code or platform-defined file in a well defined filesystem) and executes it. This part is known as a second stage bootloader.

The second stage bootloader will be implemented as a own write program or use a standard piece of code provided by free software projects like u-boot. In particular, u-boot provides a second stage bootloader infrastructure for many platforms and allows programmers to define a new platform in an easy and standard form.

This second stage bootloader prepares media devices and other peripherals to load the real bootloader (like the main u-boot). This is required in some platforms because the boot-rom only initializes some minimal and fixed set of hardware (like MMC interface or someone) leaving others peripheral like main memory uninitialized.

When all of the required peripherals are initialized, the second stage bootloader loads the real bootloader and executes it. This ends the initialization of the hardware (some secondary peripherals like USB or ethernet) and finds the kernel, the root filesystem and the devicetree.

Normally the real bootloader (also called third stage bootloader) provides some basic console to interact with the user via serial terminal or screen (in case of devices that have one) and, additionally, more advance script capabilities and fallback boot (if the bootloader not found kernel or some others parts, this may try to boot from network or secondary devices like usb or SD card)

If the bootloader found the kernel, the devicetree (or a kernel with bundled devicetree), try to load all in RAM, link the devicetree to kernel (passing the address of this in a special register of the processor) and execute the kernel.

After the kernel

At this point, the kernel is loaded in RAM and all of the hardware is ready to operate with it (at least the main memory and processor clock). Now, the kernel init find in devicetree the drivers of the hardware and try to load and initialize them. If some driver fails to load (or is not compiled into the kernel), ignore it and process with the next entry.

When all drivers are initialized, the kernel tries to mount the root filesystem. The root filesystem is the place to find the ‘/’ directory and all other directories, files and other filesystems are attached to it.

The root filesystem needs to be located in an accessible device (interface initialized with functional driver, by example, SDCard, eMMC or PCIe NVMe) and contained in a known filesystem format (format with driver installed in the kernel and accessible at the moment of the load and compatible with linux requirements)

The name and location of root filesystem is configured in devicetree or pass in the command line of the kernel (yes! the kernel accepts a command line from the bootloader) with rootfs=... parameter. By example, to boot a second partition in PCIe NVMe formatted in ext4 format the command line must be: rootfs=/dev/nvme0n2

If the root filesystem can not be mount, the kernel stall with a fatal message informing it and wait for reboot some time (normally 5 seconds or so on, but this behavior will be changed or disabled completely)

Due to these limitations (and the variability of the systems) many kernels attach a fallback root filesystem that is loaded in RAM by the bootloader beside the kernel and contain the minimum root filesystem to find the rest of the system. This is called initramfs and will be covered later.

When the root filesystem is located and mounted, the kernel is ready to find the init program. This program is the first userspace program and is the root of other programs. If the “init” parameter is not specified in the command line, the kernel tries several names and locations until one is found and can be loaded and executed. In order: /sbin/init, /etc/init, /bin/init, /bin/sh.

If all of them fail, the kernel stalls with a panic message informing it and waits for several seconds for reboot.

At this point, the kernel leaves the control to the init process. Some simple linux systems use only a custom init process as the unique program in the system but others require more flexibility and use more complex init systems. This will be covered in the next article!

See you later!!!!

‍

Martín Ribelotta

Embedded Linux Developer at Emtech S.A

‍

Any Comments or questions, please feel free to contact us: info@emtech.com.ar

‍