When configuring the Linux kernel, some options should be avoided.
CONFIG_CPU_FREQ
This allows the CPU frequency to be modulated with workload, but many
CPUs change the TSC counting frequency also, which makes it useless for
accurate timing when the CPU clock can change. Also some CPUs can take
several milliseconds to ramp up to full speed.
CONFIG_CPU_IDLE
Allows the CPU to enter deep sleep states, increasing the time it takes
to get out of these sleep states, hence the latency of an idle system.
Also, on some CPU, entering these deep sleep states causes the timers
used by Xenomai to stop functioning.
CONFIG_KGDB
This option should not be enabled, except with x86.
CONFIG_CONTEXT_TRACKING_FORCE
This option which appeared in kernel 3.8 is forced off by the legacy
I-pipe support for interrupt pipelining since 3.14 onward, as it is
incompatible with interrupt pipelining, and has no upside for regular
users. However, you have to manually disable it for older kernels when
present. Common effects observed with this feature enabled include
RCU-related kernel warnings during real-time activities, and
pathologically high latencies. Current pipelines like Dovetail have no
issue with this option though.
This means that the kernel crashes before the console is enabled. You
should enable the CONFIG_EARLY_PRINTK option. For some architectures
(blackfin, x86, arm), enabling this option also requires passing the
earlyprintk parameter on the kernel command line. See
Documentation/kernel-parameters.txt for possible values.
For the ARM architecture, you have to enable CONFIG_DEBUG_KERNEL and
CONFIG_DEBUG_LL in order to be able to enable CONFIG_EARLY_PRINTK.
Please make sure to check the Kernel configuration section first.
If nothing seems wrong there, try capturing the OOPS information using a
serial console or netconsole, then post it to the xenomai mailing
list
, along with the kernel
configuration file (aka .config) matching the kernel build.
Your distribution may be configured to pass the quiet option on the
kernel command line. In this case, the kernel does not print all the log
messages, however, they are still available using the dmesg command.
The most probable reason is that no hardware timer chip is available for Xenomai timing operations.
Check that you did not enable some of the conflicting options listed in the Kernel configuration section.
With AMD x86_64 CPUs
You will most likely also see the following message:
I-pipe: cannot use LAPIC as a tick device
I-pipe: disable C1E power state in your BIOS
The interrupt pipeline outputs this message if C1E option is enabled in
the BIOS. To fix this issue, disable C1E support in the BIOS. In some
Award BIOS this option is located in the Advanced BIOS Features→ menu
(AMD C1E Support).
Disabling the AMD K8 Cool&Quiet feature in the BIOS will NOT solve
this problem.
With other CPU architectures
The interrupt pipeline implementation may lack a registration for a
hardware timer available to Xenomai timing operations (e.g. a call to
ipipe_timer_register()).
You may have an issue with System Management Interrupts on your x86 platform. You may want to look at this document .
This x86-specific issue might still happen on legacy hardware with no MSI support. See this document .
See this entry .
On the ppc64 platform, check whether CONFIG_PPC_64K_PAGES is defined
in your kernel configuration. If so, then you likely need to raise all
Xenomai parameters defining the size of internal heaps, such as
CONFIG_XENO_OPT_SYS_HEAPSZ, CONFIG_XENO_OPT_GLOBAL_SEM_HEAPSZ and
CONFIG_XENO_OPT_SEM_HEAPSZ, so that (size / 64k) > 2. The default
values for these parameters are currently based on the assumption that
PAGE_SIZE = 4k.
The Xenomai 2.x nucleus issues this warning if the kernel
configuration enables the local APIC support (CONFIG_X86_LOCAL_APIC),
but the processor status gathered at boot time by the kernel says that
no local APIC support is available. There are two options for fixing
this issue:
either your CPU really has no local APIC hardware, in which case you need to rebuild a kernel with LAPIC support disabled.
or it does have a local APIC but the kernel boot parameters did not specify to activate it using the lapic option. The latter is required since 2.6.9-rc4 for boxes which APIC hardware is disabled by default by the BIOS. You may want to look at the file Documentation/kernel-parameters.txt from the Linux source tree, for more information about this parameter.
or,
This message may happen when starting a Xenomai 2.x or 3.x application respectively. On the x86 architecture, the configure script option mentioned allows Xenomai to use the vsyscall mechanism for issuing system calls, based on the most efficient method determined by the kernel for the current system. This mechanism is only available from NPTL-enabled glibc releases.
Turn off this feature for other libc flavours.
As mentioned in the message, the target kernel is lacking Cobalt support. See this document for detailed information about installing Cobalt.
Another symptom of the previous issue, i.e. the Cobalt core is not enabled in the target kernel.
This is the result of an attempt to run a Xenomai application as an
unprivileged user, which fails because invoking Cobalt services requires
CAP_SYS_NICE. However, you may allow a specific group of users to
access Cobalt services, by following the instructions on this
page
.
A variant of the previous issue, specific to the Mercury core though: your application needs the CAP_SYS_NICE and CAP_IPC_LOCK capabilities to be granted access to the system services the Xenomai libraries need, such as mlockall(2) (see capabilities(7)). Running the application with root privileges is a way to gain those capabilities.
Same as below:
The ABI concerned by this message is the system call binary interface
between the Xenomai libraries and the real-time kernel services it
invokes (e.g. libcobalt and the Cobalt kernel with Xenomai 3.x). This
ABI may evolve over time, only between major Xenomai releases or testing
candidate releases (i.e. -rc series) though. When this happens, the ABI
level required by the application linked against Xenomai libraries may
not match the ABI exposed by the Xenomai co-kernel implementation on the
target machine, which is the situation this message reports.
To fix this issue, just make sure to rebuild both the Xenomai kernel support and the user-space binaries for your target system. If however you did install the appropriate Xenomai binaries on your target system, chances are that stale files from a previous Xenomai installation still exist on your system, causing the mismatch.
Each major Xenomai release (e.g. 2.1.x, 2.2.x … 2.6.x, 3.0.x …) defines such kernel/user ABI, which remains stable across minor update releases (e.g. 2.6.0 → 2.6.4). This guarantee makes partial updates possible with production systems (i.e. kernel and/or user support). For instance, any application built over the Xenomai 2.6.0 binaries can run over a Xenomai 2.6.4 kernel support, and conversely.
Debian-based distributions (notably Ubuntu) may ship with pre-installed Xenomai libraries. Make sure that these files don’t get in the way if you plan to install a more recent Xenomai kernel support.
Although the program in question may be present, this message may happen
on ARM platforms when a mismatch exists between the kernel and user
library configurations with respect to EABI support. Typically, if user
libraries are compiled with a toolchain generating OABI code, the result
won’t run over a kernel not enabling the CONFIG_OABI_COMPAT option.
Conversely, the product of a compilation with an EABI toolchain won’t
run on a kernel not enabling the CONFIG_AEABI option.
When a Xenomai application starts, the set of core features it requires is compared to the feature set the kernel provides. This message denotes a mismatch between both sets, which can be solved by fixing the kernel and/or user build configuration. Further details are available from this page .
On SMP-capable architectures, both kernel and user-space components (i.e. Xenomai libraries) must be compiled with the same setting with respect to SMP support.
SMP support in the kernel is controlled via the CONFIG_SMP option. The
--enable-smp configuration switch enables this feature for the Xenomai
libraries (conversely, --disable-smp disables it).
Using Xenomai libraries built for a single-processor configuration (i.e.
--disable-smp) over a SMP kernel (i.e. CONFIG_SMP=y) is NOT
valid. On the other hand, using Xenomai libraries built with SMP support
enabled over a single-processor kernel is fine.
Possible reasons for this error are:
you booted a kernel without Xenomai or I-pipe support, a kernel with I-pipe and Xenomai support should have a /proc/ipipe/version and /proc/xenomai/version files;
the kernel you booted does not have the CONFIG_XENO_SKIN_* option
enabled for the skin you use, or CONFIG_XENO_OPT_PERVASIVE is
disabled;
Xenomai failed to start, check the kernel log;
you are trying to run Xenomai user-space support compiled for x86_32 on an x86_64 kernel.
Your user-space application unexpectedly commits a lot of virtual
memory, as reported by “top” or /proc/<pid>/maps. Sometimes
OOM situations may even appear during runtime on systems with limited
memory.
The reason is that Xenomai threads are underlaid by regular POSIX
threads, for which a large default amount of stack space memory is
commonly reserved by the POSIX threading library (8MiB per thread by the
glibc). Therefore, the kernel will commit as much as
8MiB * nr_threads bytes to RAM space for the application, as a
side-effect of calling the mlockall() service to lock the process
memory, as Xenomai requires.
This behaviour can be controlled in two ways:
via the stacksize parameter passed to the various thread creation
routines, or pthread_attr_setstacksize() directly when using the
POSIX API.
by setting a lower user-limit for the initial stack allocation from the application’s parent shell which all threads from the child process inherit, as illustrated below:
ulimit -s <initial-size-in-kbytes>
Possible reasons may be:
Stack space overflow issue now biting some real-time kernel thread?
Spurious delay/timeout values computed by the application (specifically: too short).
A case of freeze is a system call called in a loop which fails without its return value being properly checked.
On x86, whenever the nucleus watchdog does not trigger, you may want to try disabling CONFIG_X86_UP_IOAPIC while keeping CONFIG_X86_UP_APIC, and arm the kernel NMI watchdog on the LAPIC (nmi_watchdog=2). You may be lucky and have a backtrace after the freeze. Maybe enabling all the nucleus debug options would catch something too.
The first test to run to see if Xenomai is running correctly on your platform is the latency test. The following sections describe the usual reasons for this test not to run correctly.
You have launched latency -t 1 or latency -t 2 which both require
the kernel to have been configured with the
CONFIG_XENO_DRIVERS_TIMERBENCH option.
The most common reason for this issues is a too short period passed with
the -p option, try increasing the period. If you enable the watchdog
(option CONFIG_XENO_OPT_WATCHDOG, in your kernel configuration), you
should see the watchdog triggered (period too short?)
message.
The built-in Xenomai watchdog has stopped the latency test because it
was using all the CPU in pure real-time mode (aka primary mode). This
is likely due to a too short period. Run the latency test again,
passing a longer period using the -p option this time.
The latency test runs, but you are seeing high latencies.
make sure that you carefully followed the Kernel configuration section .
if running on a Raspberry Pi SBC, make sure you don’t hit a firmware issue, see https://github.com/raspberrypi/firmware/issues/497 .
if running on a x86 platform, make sure that you do not have an issue with SMIs, see the section about SMIs .
if running on a x86 platform with a legacy USB switch available from the BIOS configuration, try disabling it.
if you do not have this option at BIOS configuration level, it does
not necessarily mean that there is no support for it, thus no
potential for high latencies; this support might just be forcibly
enabled at boot time. To solve this, in case your machine has some
USB controller hardware, make sure to enable the corresponding host
controller driver support in your kernel configuration. For
instance, UHCI-compliant hardware needs CONFIG_USB_UHCI_HCD. As
part of its init chores, the driver should reset the host controller
properly, kicking out the BIOS off the concerned hardware, and
deactivate the USB legacy mode if set in the same move.
if you observe high latencies while running X-window, try disabling
hardware acceleration in the X-window server file. With recent
versions of X-window, try using the fbdev driver. Install it
(Debian package named xserver-xorg-video-fbdev for instance), then
modifiy the Device section to use this driver in
/etc/X11/xorg.conf, as in:
Section "Device"
Identifier "Card0"
Driver "fbdev"
EndSection
With olders versions of X-window, keep the existing driver, but add the
following line to the Device section:
Option "NoAccel"
The switchtest test creates many kernel threads, an operation which consumes memory taken from internal pools managed by the Xenomai real-time core.
Xenomai 2.x and 3.x series require CONFIG_XENO_OPT_SYS_HEAPSZ to be
large enough in the kernel configuration settings, to cope with the
allocation requests.
Xenomai 2.x may also require to increase the
CONFIG_XENO_OPT_SYS_STACKPOOLSZ setting.