Installing Xenomai 3

Installation steps

Xenomai follows a split source model, decoupling the kernel space support from the user-space libraries.

To this end, kernel and user-space Xenomai components are respectively available under the kernel/ and lib/ sub-trees. Other top-level directories, such as scripts/, testsuite/ and utils/, provide additional scripts and programs to be used on either the build host, or the runtime target.

The kernel/ sub-tree which implements the in-kernel support code is seen as a built-in extension of the Linux kernel. Therefore, the standard Linux kernel configuration process should be used to define the various settings for the Xenomai kernel components. All of the kernel code Xenomai currently introduces implements the Cobalt core (i.e. dual kernel configuration). As of today, the Mercury core needs no Xenomai-specific code in kernel space.

The lib/ sub-tree contains the various user-space libraries exported by the Xenomai framework to the applications. This tree is built separately from the kernel support. Libraries are built in order to support the selected core, either Cobalt or Mercury.

Installing the Cobalt core

Preparing the Cobalt kernel

Xenomai/cobalt provides a real-time extension kernel seamlessly integrated to Linux, therefore the first step is to build it as part of the target kernel. To this end, scripts/prepare-kernel.sh is a shell script which sets up the target kernel properly. The syntax is as follows:

$ scripts/prepare-kernel.sh [--linux=<linux-srctree>]
[--ipipe=<ipipe-patch>] [--arch=<target-arch>]

--linux
specifies the path of the target kernel source tree. Such kernel tree may be already configured or not, indifferently. This path defaults to $PWD.

--dovetail
specifies the path of the interrupt pipeline (aka Dovetail) patch to apply against the kernel tree. This parameter can be omitted if Dovetail has already been patched in, or the script shall suggest an appropriate one. The script will detect whether the interrupt pipeline code is already present into the kernel tree, and skip this operation if so.

--arch
tells the script about the target architecture. If unspecified, the build host architecture suggested as a reasonable default.

For instance, the following command would prepare the Linux tree located at /home/me/linux-3.10-ipipe in order to patch the Xenomai support in:

$ cd xenomai-3
$ scripts/prepare-kernel.sh --linux=/home/me/linux-3.10

Note: The script will infer the location of the Xenomai kernel code from its own location within the Xenomai source tree. For instance, if /home/me/xenomai-3/scripts/prepare-kernel.sh is executing, then the Xenomai kernel code available from /home/me/xenomai-3/kernel/cobalt will be patched in the target Linux kernel.

Configuring and compiling the Cobalt kernel

Once prepared, the target kernel can be configured as usual. All Xenomai configuration options are available from the “Xenomai” toplevel Kconfig menu.

There are several important kernel configuration options, documented in the Troubleshooting guide .

Once configured, the kernel can be compiled as usual.

If you want several different configs/builds at hand, you may reuse the same source by adding O=../build-<target> to each make invocation.

In order to cross-compile the Linux kernel, pass an ARCH and CROSS_COMPILE variable on make command line. See sections Building a Cobalt/arm kernel , Building a Cobalt/powerpc kernel , Building a Cobalt/x86 kernel , for examples.

Cobalt kernel parameters

The Cobalt kernel accepts the following set of parameters, which should be passed on the kernel command line by the boot loader.

Examples of building the Cobalt kernel

The examples in following sections use the following conventions:

$linux_tree
path to the target kernel sources

$xenomai_root
path to the Xenomai sources

Building a Cobalt/x86 kernel (32/64bit)

Building Xenomai/cobalt for x86 is almost the same for 32bit and 64bit platforms. You should note, however, that it is not possible to run Xenomai libraries compiled for x86_32 on a kernel compiled for x86_64, and conversely.

Assuming that you want to build natively for a x86_64 system (x86_32 cross-build options from x86_64 appear between brackets), you would typically run:

$ cd $linux_tree
$ $xenomai_root/scripts/prepare-kernel.sh --arch=x86 \
  --ipipe=ipipe-core-X.Y.Z-x86-NN.patch
$ make [ARCH=i386] xconfig/gconfig/menuconfig

configure the kernel (see also the recommended settings here ).

Enable Xenomai options, then build with:

$ make [ARCH=i386] bzImage modules

Now, let’s say that you really want to build Xenomai for a Pentium-based x86 32bit platform, using the native host toolchain; the typical steps would be as follows:

$ cd $linux_tree
$ $xenomai_root/scripts/prepare-kernel.sh --arch=i386 \
  --ipipe=ipipe-core-X.Y.Z-x86-NN.patch
$ make xconfig/gconfig/menuconfig

configure the kernel (see also the recommended settings here ).

Enable Xenomai options, then build with:

$ make bzImage modules

Similarly, for a 64bit platform, you would use:

$ cd $linux_tree
$ $xenomai_root/scripts/prepare-kernel.sh --arch=x86_64 \
  --ipipe=ipipe-core-X.Y.Z-x86-NN.patch
$ make xconfig/gconfig/menuconfig

configure the kernel (see also the recommended settings here ).

Enable Xenomai options, then build with:

$ make bzImage modules

The remaining examples illustrate how to cross-compile a Cobalt-enabled kernel for various architectures. Of course, you would have to install the proper cross-compilation toolchain for the target system first.

Building a Cobalt/powerpc kernel (32/64bit)

A typical cross-compilation setup, in order to build Xenomai for a ppc-6xx architecture running a 3.10.32 kernel. We use the DENX ELDK cross-compiler:

$ cd $linux_tree
$ $xenomai_root/scripts/prepare-kernel.sh --arch=powerpc \
  --ipipe=ipipe-core-3.10.32-powerpc-1.patch
$ make ARCH=powerpc CROSS_COMPILE=ppc_6xx- xconfig/gconfig/menuconfig

…select the kernel and Xenomai options, save the configuration

$ make ARCH=powerpc CROSS_COMPILE=powerpc-linux- uImage modules

…manually install the kernel image and modules to the proper location

Building Cobalt/arm kernel

Using codesourcery toolchain named arm-none-linux-gnueabi-gcc and compiling for a CSB637 board (AT91RM9200 based), a typical compilation will look like:

$ cd $linux_tree
$ $xenomai_root/scripts/prepare-kernel.sh --arch=arm \
  --ipipe=ipipe-core-X.Y.Z-x86-NN.patch
$ mkdir -p $build_root/linux
$ make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabi- O=$build_root/linux \
  csb637_defconfig
$ make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabi- O=$build_root/linux \
  bzImage modules

…manually install the kernel image, system map and modules to the proper location

Building Cobalt/arm64 kernel

Using Linaro toolchain with the prefix aarch64-linux-gnu- and compiling for the raspberry pi 3 board (cortex-a53), cross compiling is as follows:

$ cd $linux_tree
$ $xenomai_root/scripts/prepare-kernel.sh --arch=arm64 \
   --ipipe=ipipe-core-X.Y.Z-arm64-NN.patch
$ mkdir -p $build_root/linux
$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- O=$build_root/linux \
  defconfig
$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- O=$build_root/linux \
  Image dtbs modules

For arm64 there is one general defconfig in mainline, you’ll need to do any customize the defconfig as needed. Once the build is finished manually install the kernel image, devicetree and modules to the proper location.

Installing the Mercury core

For Mercury, you need no Xenomai-specific kernel support so far, beyond what your host Linux kernel already provides. Your kernel should at least provide high resolution timer support (CONFIG_HIGH_RES_TIMERS), and likely complete preemption (PREEMPT_RT) if your application requires short and bounded latencies.

Kernels with no real-time support can be used too, likely for basic debugging tasks, and/or running applications which do not have strict response time requirements.

Therefore, unlike with Cobalt, there is no additional steps for preparing and/or configuring the kernel for Mercury.

Installing the Xenomai libraries and tools

Prerequisites

Generic requirements (both cores)

• GCC must have support for legacy atomic builtins (__sync form).

• GCC should have a (sane/working) support for TLS preferably, although this is not mandatory if building with --disable-tls.

• If you plan to enable the user-space registry support (i.e. --enable-registry), then CONFIG_FUSE_FS must be enabled in the target kernel running the real-time applications. In addition, the FUSE development libraries must be available from the toolchain.

• If you plan to build from the sources available from the Xenomai GIT tree (git.xenomai.org), the autoconf (>= 2.62), automake and libtool packages must be available on your build system.

Cobalt-specific requirements

• The kernel version must be 5.10 or better.

• The Dovetail interrupt pipeline must be available for your target kernel.

• A timestamp counter (TSC) is required from running on a x86_32 hardware. TSC-emulation using a PIT register is not available.

Mercury-specific requirement

• There is no particular requirement for Mercury setups, although using a NPTL-based glibc or uClibc is recommended.

Configuring

If building the source obtained from the Xenomai GIT tree (git.xenomai.org), the configure script and Makefiles must be generated in the Xenomai source tree. The recommended way is to run the automatic reconfiguration script shipped, from the top of the source tree:

$ ./scripts/bootstrap

When run, the generated configure script prepares for building the libraries and programs, for both the Cobalt and Mercury cores. The core-specific code which may be needed internally is automatically and transparently selected at compilation-time by the build process.

The options listed below can be passed to this script.

Generic configuration options (both cores)

When this option is enabled, the system creates a file hierachy at <user>/<session>/<pid> under the registry root path, where you can access the internal state of the active real-time objects. The session label is obtained from the –session runtime switch. If no session name is specified, anon@<pid> will be used. E.g. looking at the properties of a VxWorks task could be done as follows:

If not specified in the configuration switch, the registry root path will be /var/run/xenomai.

$ cat /var/run/xenomai/root/anon@12656/12656/vxworks/tasks/windTask
name       = windTask
errno      = 0
status     = ready
priority   = 70
lock_depth = 0

You may override the default root of the registry hierarchy either statically at build time by passing the desired root path to the –enable-registry configuration switch, or dynamically by using the --registry-root runtime option passed to the application.

When running over Xenomai/cobalt, the /proc/xenomai interface is also available for inspecting the core system state.

–enable-lores-clock
Enables support for low resolution clocks. By default, libraries are built with no support for tick-based timing. If you need such support (e.g. for pSOS ™ or VxWorks ™ APIs), then you can turn it on using this option.

The POSIX API does not support tick-based timing. Alchemy may use it optionally.

–enable-clock-monotonic-raw
The Xenomai libraries requires a monotonic clock to be available from the underlying POSIX interface. When CLOCK_MONOTONIC_RAW is available on your system, you may want to pass this switch, otherwise CLOCK_MONOTONIC will be used by default.

The Cobalt core implements CLOCK_MONOTONIC_RAW, so this switch is turned on by default when building with --with-core=cobalt. On the contrary, this option is turned off by default when building for the Mercury core, since we don’t know in advance whether this feature does exist on the target kernel.

–enable-tls
Xenomai can use GCC’s thread local storage extension (TLS) to speed up the retrieval of the per-thread information it uses internally. This switch enables TLS, use the converse --disable-tls to prevent this.

Due to GCC bugs regarding this feature with some release,architecture combinations, whether TLS is turned on by default is a per-architecture decision. Currently, this feature is enabled for x86 and powerpc by default, other architectures will require --enable-tls to be passed to the configure script explicitly.

Unless --enable-dlopen-libs is present, the initial-exec TLS model is selected.

When TLS is disabled, POSIX’s thread-specific data management services are used internally (i.e. pthread_set/getspecific()).

–enable-dlopen-libs
This switch allows programs to load Xenomai-based libraries dynamically, using the dlopen(3) routine. Enabling dynamic loading introduces some overhead in TLS accesses when enabled (see --enable-tls), which might be noticeable depending on the architecture.

To support dynamic loading when --enable-tls is turned on, the global-dynamic TLS model is automatically selected.

Dynamic loading of Xenomai-based libraries is disabled by default.

–enable-async-cancel
Enables fully asynchronous cancellation of Xenomai threads created by the real-time APIs, making provision to protect the Xenomai implementation code accordingly.

When disabled, Xenomai assumes that threads may exit due to cancellation requests only when they reach cancellation points (like system calls). Asynchronous cancellation is disabled by default.

Fully asynchronous cancellation can easily lead to resource leakage, silent corruption, safety issues and all sorts of rampant bugs. The only reason to turn this feature on would be aimed at cancelling threads which run significantly long, syscall-less busy loops with no explicit exit condition, which should probably be revisited anyway.

–enable-smp
Turns on SMP support for Xenomai libraries.

SMP support must be enabled in Xenomai libraries when the client applications are running over a SMP-capable kernel.

–disable-sanity
Turns off the sanity checks performed at application startup by the Xenomai libraries. This option sets a default, which can later be overriden using the –[no-]sanity options passed to a Copperplate-based Xenomai application. Sanity checks are enabled by default when configuring.

–enable-fortify
Enables _FORTIFY_SOURCE when building the Xenomai code unless –enable-debug=full is also given on the command line, in which case –enable-fortify is silently ignored.

–disable-valgrind-client
Turns off the Valgrind client support, forcing CONFIG_XENO_VALGRIND_API off in the Xenomai configuration header.

–enable-doc-build
Causes the inline Xenomai documentation based on the Doxygen markup language to be produced as PDF and HTML documents. Additional documentation like manpages based on the Asciidoc markup language is produced too.

Cobalt-specific configuration options

Mercury-specific configuration options

[1]

Cross-compilation

In order to cross-compile the Xenomai libraries and programs, you will need to pass a --host and --build option to the configure script. The --host option allow to select the architecture for which the libraries and programs are built. The --build option allows to choose the architecture on which the compilation tools are run, i.e. the system running the configure script.

Since cross-compiling requires specific tools, such tools are generally prefixed with the host architecture name; for example, a compiler for the PowerPC architecture may be named powerpc-linux-gcc.

When passing --host=powerpc-linux to configure, it will automatically use powerpc-linux- as a prefix to all compilation tools names and infer the host architecture name from this prefix. If configure is unable to infer the architecture name from the cross-compilation tools prefix, you will have to manually pass the name of all compilation tools using at least the CC and LD, variables on configure command line.

The easiest way to build a GNU cross-compiler might involve using crosstool-ng, available here .

If you want to avoid to build your own cross compiler, you might if find easier to use the ELDK. It includes the GNU cross development tools, such as the compilers, binutils, gdb, etc., and a number of pre-built target tools and libraries required on the target system. See here for further details.

Some other pre-built toolchains:

• Mentor Sourcery CodeBench Lite Edition, available here ;

• Linaro toolchain (for the ARM architecture), available here .

Examples of building the Xenomai libraries and tools

The examples in following sections use the following conventions:

$xenomai_root
path to the Xenomai sources

$build_root
path to a clean build directory

$staging_dir
path to a directory that will hold the installed file temporarily before they are moved to their final location; when used in a cross-compilation setup, it is usually a NFS mount point from the target’s root directory to the local build host, as a consequence of which running make{nbsp}DESTDIR=$staging_dir{nbsp}install on the host immediately updates the target system with the installed programs and libraries.

In the examples below, make sure to add --enable-smp to the configure script options if building for a SMP-enabled kernel.

Building the x86 libraries (32/64bit)

Assuming that you want to build the Mercury libraries natively for a x86_64/SMP system, enabling shared multi-processing support. You would typically run:

$ mkdir $build_root && cd $build_root
$ $xenomai_root/configure --with-core=mercury --enable-smp --enable-pshared
$ make install

Conversely, cross-building the Cobalt libraries from x86_64 with the same feature set, for running on x86_32 could be:

$ mkdir $build_root && cd $build_root
$ $xenomai_root/configure --with-core=cobalt --enable-smp --enable-pshared \
  --host=i686-linux CFLAGS="-m32 -O2" LDFLAGS="-m32"
$ make install

After installing the build tree (i.e. using “make install”), the installation root should be populated with the librairies, programs and header files you can use to build Xenomai-based real-time applications. This directory path defaults to /usr/xenomai.

The remaining examples illustrate how to cross-compile Xenomai for various architectures. Of course, you would have to install the proper cross-compilation toolchain for the target system first.

Building the PPC32 libraries

A typical cross-compilation setup, in order to build the Cobalt libraries for a ppc-6xx architecture. In that example, we want the debug symbols to be generated for the executable, with no runtime overhead though. We use the DENX ELDK cross-compiler:

$ cd $build_root
$ $xenomai_root/configure --host=powerpc-linux --with-core=cobalt \
  --enable-debug=symbols
$ make DESTDIR=$staging_dir install

Building the PPC64 libraries

Same process than for a 32bit PowerPC target, using a crosstool-built toolchain for ppc64/SMP.

$ cd $build_root
$ $xenomai_root/configure --host=powerpc64-unknown-linux-gnu \
  --with-core=cobalt --enable-smp
$ make DESTDIR=$staging_dir install

Building the ARM libraries

Using codesourcery toolchain named arm-none-linux-gnueabi-gcc and compiling for a CSB637 board (AT91RM9200 based), a typical cross-compilation from a x86_32 desktop would look like:

$ mkdir $build_root/xenomai && cd $build_root/xenomai
$ $xenomai_root/configure CFLAGS="-march=armv4t" LDFLAGS="-march=armv4t" \
  --build=i686-pc-linux-gnu --host=arm-none-linux-gnueabi- --with-core=cobalt
$ make DESTDIR=$staging_dir install

Unlike previous releases, Xenomai no longer passes any arm architecture specific flags, or FPU flags to gcc, so, users are expected to pass them using the CFLAGS and LDFLAGS variables as demonstrated above, where the AT91RM9200 is based on the ARM920T core, implementing the armv4 architecture. The following table summarizes the CFLAGS and options which were automatically passed in previous revisions and which now need to be explicitely passed to configure, for the supported SOCs:

ARM configure options and compilation flags

It is possible to build for an older architecture version (v6 instead of v7, or v4 instead of v5), if your toolchain does not support the target architecture, the only restriction being that if SMP is enabled, the architecture should not be less than v6.

Building the ARM64 libraries

ARM64 is only supported from the git repos on the next branch.

Using the Linaro toolchain with the prefix aarch64-linux-gnu- for the Raspberry Pi 3 board (cortex-a53), cross compililation from a x86_64 host would be as follows:

mkdir $build_root/xenomai && cd $build_root/xenomai
../xenomai/configure CFLAGS="-mtune=cortex-a53" LDFLAGS="-mtune=cortex-a53" \
--build=i686-pc-linux-gnu --host=aarch64-linux-gnu --with-core=cobalt \
--enable-smp CC=aarch64-linux-gnu-gcc LD=aarch64-linux-gnu-ld
$ make DESTDIR=$staging_dir install

Passing a value for the -mcpu flag will help generate optimized code for a specific cpu type but it’s not necessary.

Testing the installation

Booting the Cobalt kernel

In order to test the Xenomai installation over Cobalt, you should first try to boot the patched kernel. Check the kernel boot log for messages like these:

    $ dmesg | grep -i xenomai
    I-pipe: head domain Xenomai registered.
    [Xenomai] Cobalt vX.Y.Z enabled

If the kernel fails booting, or the log messages indicates an error status instead, see the Troubleshooting guide .

Testing the real-time system (both cores)

First, run the latency test alone:

    $ /usr/xenomai/bin/latency

The latency test should display a message every second with minimum, maximum and average latency values. If this test displays an error message, hangs, or displays unexpected values, see the Troubleshooting guide .

If the latency test does not show inacceptable latency figures, you should try to run the xeno-test test next, in order to assess the worst-case latency of your system. Try:

    $ xeno-test --help

xeno-test is simple script running a series of unit tests of Xenomai 3 features, then the latency test under a user-specified load. The aim of this script is to allow Xenomai users to run reproducible latency measurements, under a load which would approximate the load of their system in production.

The dohell script is a companion of xeno-test, derived from snippets posted on the Linux Kernel Mailing List, which allows to generate a synthetic load (network load, disk I/O load, cpu load, and Linux syscalls coverage with LTP).

For network load, a server must be listening on another machine, for connections on a TCP port. The default port used for this is the “discard” port available in the inetd daemon.

The xeno-test command line used by the Xenomai developers to measure latency enables all the possible dohell sources of load:

xeno-test -l "dohell -s <serverip> -l <path/to/ltp> -m <mntpoint>" -g <filename>

On ARM machines running with with FCSE enabled in “guaranteed” mode, it does not make much sense to run the LTP testsuite, the limitation to 95 processes would cause several key tests to fail, so, we use dohell +-b+ option to periodically spawn the +hackbench+ test with 80 threads (to not exceed the 95 processes limit). The command-line becomes:

xeno-test -l "dohell -s <serverip> -b <path/to/hackbench> 7200" -g <filename>

Calibrating the Cobalt core timer

The accuracy of the Cobalt timing services depends on proper calibration of its core timer. Sound factory-default calibration values are defined for each platform Xenomai supports, but it is recommended to calibrate the core timer specifically for the target system.

See the documentation about the autotune(1) utility.