Advanced Programmable System Software Development

This Technical Brief describes advanced embedded software development techniques offered by the Missing Link Electronics “Soft” Hardware Platform. Because MLE adheres to defacto standards from the Open Source domain, has a powerful configuration management and integrated network connectivity, software and firmware development can be accelerated. This is explained by examples of MLE Linux kernel development and Linux device driver development.

Copyright © 2010 Missing Link Electronics, Inc. All rights reserved. Missing Link Electronics, the stylized Missing Link Electronics MLE logo are the service mark and/or trademark of Missing Link Electronics, Inc. All other product or service names and trademarks are the property of their respective owners.


Programmable Systems, as used in Next-Generation Embedded Systems and networked compute platforms for Industrial, Scientific and Medical (ISM) systems, require high degrees of optimization of the hardware and the software components. For designing such systems Missing Link Electronics provides platforms where a configurable microcontroller can be optimized towards the Open Source Software that runs on it and the particular application that the system must serve. Most functionality is implemented as an FPGA-based System-on-Chip which runs a full software stack comprising a boot-loader, a Linux operating system kernel and userspace application software. Because of the tight integration of hardware and software, the CPU instruction sets, the kernel device trees, the interrupt hierarchy, etc may be different for each implementation and as a result these components may not be interchangeable.

Therefore, MLE has dedicated special attention to ensure that components depending on each other can be associated and loaded / booted together. The concept is to use a so-called Unified Design Image (UDI) which combines an FPGA configuration bitfile, a boot-loader executable, a Linux kernel image, and possibly even userspace application software (see Figure 1). Now, during the system design phase it is important to try out many different system configurations - for functionality, performance and for robustness testing. To facilitate this, MLE has implemented a powerful and flexible multi-boot concept for Programmable System hardware and software which can hold multiple UDIs and offers failsafe system update functionality even for remote access [1].


Figure 1: Unified Design Images for MLE

The configuration management functionality is described in a triplet of MLE Technical Briefs: [2] goes into details of the implementation using CPLD devices and how to operate this functionality in general. [3] describes how to use this functionality from a hardware designers point of view. [4] (this paper) explains how to take advantage of this functionality for efficient firmware and software design.

In the following we will describe advanced software development techniques that can be used for the MLE “Soft” Hardware Platform. The MLE 1000 Series Rapid Prototyping System as one of the many instantiations of the MLE “Soft” Hardware Platform directly benefits from those techniques and will be used as an examplary setup throughout the rest of this document. We will refer to typical processes during development and how the MLE 1000 RPS with its network connectivity facilitates these processes. For more details on network connectivity, please have a look at Technical Brief [1].

Regarding the MLE 1000 RPS, software development includes writing and debugging userspace applications as well as Linux kernel development (sometimes referred to as firmware development). The most likely form of kernel development is the writing, testing and debugging of MLE Linux kernel modules. This means that the core of the kernel is extended via modular additions, like device drivers, rather than touched itself. Nevertheless we will first look at kernel core development on the MLE 1000 RPS.

Since the boot process of the MLE 1000 RPS has implications on the software development process, the following paragraphs will describe this boot process which starts at FPGA configuration and goes all the way to where an operating system kernel and all system services (daemons) have successfully been started. MLE adheres to the defacto Open Source standards. Therefore, most of this process is similar to any other GNU/Linux system. At each step we will discuss related software development issues.

For maximum efficiency we recommend a completely networked setup of the MLE 1000 RPS for software development. The setup and the MLE 1000 RPS’s network connectivity are described in the Technical Brief [1].

Figure 2 from that Technical Brief is repeated here for convenience: It shows an MLE 1000 RPS connected via LAN (or VPN) to one or more developer workstations and servers. The MLE 1000 RPS is the embedded target. The developer workstations operate as development host machines for embedded software development. The servers can be one or more machines providing networking services such as, DHCP, TFTP, NFS, etc. Of course, for smaller scale environments, all these services can also be provided by one single computer, acting as the developers workstation and server at the same time.


Figure 2: Typical Networked Setup

The descriptions in the following paragraphs refer to the setup of Figure 2. For the special task of MLE software development this setup can (and should) be enhanced with additional RS232 and USB connections. This way a developer is able to see debug messages from the early boot phase, where a network connection is not yet established.

Figure 3 shows the first part of the boot process.


Figure 3: MLE Boot Process, Part 1

At this point we skip the details of the electrical power up and FPGA configuration. It is sufficient to know, that the FPGA gets configured with an FPGA design file, which is stored in on-board Flash memory. Afterwards the FPGA resembles a System on Chip (SoC). The built-in CPU starts by executing some initial boot code to load a fully featured and network capable bootloader from on-board Flash memory.

The MLE bootloader is able to fetch an operating system kernel image from a remote server using DHCP and TFTP protocols. This procedure is very important for the software development flow: Assume a kernel developer has compiled a new kernel image and would like to test it. Most likely the compilation was done on a host machine by using a cross-compiler. The usual approach of copying the compiled executables to the target is a very tedious process. Often this involves physically transporting it in some way, like using a memory stick, or using a proprietary debug cable. Writing the image to some on-board Flash memory is another time-consuming variant.
With the MLE 1000 RPS and the MLE bootloader’s networking capability, a developer instead just needs to copy the new kernel image / executable into the appropriate directory on the TFTP server. The MLE bootloader will then fetch the image, decompress it if necessary, and finally transfer control to the new kernel’s entry point and thereby start it.

At this point the developer usually decides whether the new kernel is running as expected by looking at kernel messages written to the console. As with many other embedded targets, the default console of the MLE 1000 RPS is assigned to the RS232 interface. To access this interface via a network connection we recommend to use a developer workstation physically close to the location of the MLE 1000 RPS and to connect both using a so-called null modem cable. Via the developer workstation one now has remote access (via SSH, for example) and can access the console using a terminal emulation software such as minicom. Figure 4 shows the recommended modifications to the general setup shown in Figure 2.


Figure 4: Networked Setup Plus RS232 and USB Connections

Additionally, Figure 4 shows an optional USB connection between MLE 1000 RPS and a server. If the new kernel freezes the system, it is possible to control the MLE 1000 RPS directly via this USB connection which supports power cycling the system remotely, for example. For more information on this feature-rich USB connection and the related CPLD inside the MLE “Soft” Hardware Platform, please refer to the Technical Brief [2].

The MLE Linux behaves like any other modern GNU/Linux system: Kernel images loaded by the bootloader not only contain kernel code but also an Initial RAM Filesystem (initramfs) which provides a basic userspace application software environment. Such a combined Linux image is often called a uImage. In cases where interpreting kernel console messages may not be sufficient for debugging, i. e. deciding whether a new kernel is functioning correctly, this minimal environment can be used to interact with the system to test new kernel code from within userspace. Sometimes changing the initramfs, such as adding shell scripts, is sufficient for testing. In these situations mounting a Root Filesystem (rootfs) and starting the complete system, as described in the following paragraphs, can be skipped to reduce turn-around-time. Figure 5 visualizes this alternative quick-boot process, when using only the initramfs contained in the kernel’s uImage.


Figure 5: Quick-boot Process for Kernel Development

Changes to the initramfs are easy to be applied, because the initramfs is a part of the kernel image. Packaging new content on your host machine into the initramfs and copying the new kernel image plus initramfs to the TFTP server significantly accelerates the kernel development phase. Shell access to such a minimal system can be accomplished by integrating, for example, a TELNET server or simply by using the serial console.

After the kernel has been started and is running as expected, the second phase of the boot process is performed as shown in Figure 6.


Figure 6: MLE Boot Process, Part 2

We already mentioned the initramfs, which usually contains one or more scripts to be executed as the very first userspace process after the kernel is up. In the case of the MLE 1000 RPS and network booting, these scripts setup the network interface by querying a DHCP server. A DHCP server may also tell the location of the desired rootfs (IP address of the NFS server and the exported paths). The scripts mount the given rootfs via NFS and transfer control to a userspace startup process, typically called /sbin/init for the System V init process. While, of course, booting from a USB memory stick or a microSD card is also supported by MLE, using an NFS-mounted rootfs may be more efficient and reduce development turn-around-times, as it can be accessed remotely be the developer.

During the following startup sequence various Unix/Linux system services (so-called daemons) are started. Among them is an SSH server. Consequently, a developer may now login via SSH to get direct shell access to the MLE 1000 RPS.

At this point the process of kernel module development becomes Linux device driver development, and general userspace application development commences. Kernel modules and applications can be written and cross-compiled on the host workstation. Figure 7 shows the most important resources and services the MLE 1000 RPS offers for this type of standard Open Source development. It also visualizes the generic development flow.


Figure 7: Kernel Module and Application Software Development

Since the MLE Linux in the MLE 1000 RPS is capable to mount arbitrary NFS-exported files systems, a developer just needs to copy the target executables into such an NFS-exported directory to make them available for testing on the MLE 1000 RPS. Using an SSH connection to the MLE 1000 RPS one now can test, analyze, and debug new kernel modules or userspace application software. To support development and especially debugging, MLE Linux complies with standard Linux logging services like syslog and klogd. For example, dmesg can be called to display kernel messages, which helps when analyzing the behavior of kernel modules.

Last but not least the default MLE Linux includes a complete native C and C++ software development toolchain based on GNU GCC. Sometimes a native tolchain on the MLE 1000 RPS is a very welcome last resort when no other options for cross-compilation exist.

In this paper we have shown that the MLE “Soft” Hardware Platform closely follows defacto standards of Open Source firmware and software development. The advantage is that various well-known options exist during the development and debugging phase of the MLE Linux kernel, the device drivers and the application software. The ability to take proven shortcuts from the embedded software development domain can significantly reduce the turn-around-time.





[1] MLE TECHNICAL BRIEF 20100218. Network Connectivity of MLE 1000 Series, 2010.
[2] MLE TECHNICAL BRIEF 20100817. Advanced Programmable System Configuration
      Management, 2010.
[3] MLE TECHNICAL BRIEF 20100818. Advanced Programmable System Hardware De-
      sign, 2010.
[4] MLE T ECHNICAL B RIEF 20100819. Advanced Programmable System Software De-
      velopment, 2010.