Professional Documents
Culture Documents
Paul N. Leroux
Technology Analyst
QNX Software Systems Ltd.
paull@qnx.com
Abstract
Designers of embedded systems have become increasingly interested in the Linux operating
system, largely because of its open source model. But, as it turns out, the standard Linux kernel
can’t deliver the hard realtime capabilities such as predictable response times and microsecond
latencies — that a large number of embedded systems demand. Several products have emerged to
fill the Linux realtime gap, with mixed success. For instance, some vendors have taken a dual-
kernel approach that provides a fragile runtime environment for realtime tasks, while forcing
developers to write new drivers and system services — even when equivalent services already
exist for Linux. Meanwhile, others have proposed a “pure” Linux solution that, to be effective,
would require a rewrite of the Linux driver and virtual file system frameworks. In this paper, we
look at an alternate approach — using a POSIX-based RTOS designed specifically for embedded
systems — that not only allows Linux developers to keep their programming model, but also
maintains the key advantages of Linux’s open source model. As an added benefit, this approach
allows embedded developers to enjoy OS services unavailable with either standard Linux or
realtime Linux extensions. To illustrate the viability of this approach, the paper ends with
examples of various Linux applications already ported QNX Neutrino, the POSIX-based RTOS
from QNX Software Systems.
Real Time or Real Linux? A Realistic Alternative
Filling the Realtime Gap however, has emerged as the “standard.” In fact,
some approaches even deviate from the standard
For the embedded systems designer, Linux poses a
Linux/POSIX programming model.
dilemma. On the one hand, Linux lets the designer
leverage a large pool of developers, a rich legacy
of source code, and industry-standard POSIX APIs. Real Time Implemented
At the same time, the standard Linux kernel can’t Outside of Linux
deliver the “hard” realtime capabilities such as
For instance, most vendors provide “realtime Linux”
guaranteed response times and microsecond
— note the quotation marks — by running Linux as
latencies — that many embedded devices require.
a task on top of a realtime kernel (see Figure 1). Any
The reasons for this are rooted in Linux’s general- tasks that require deterministic scheduling also run in
purpose architecture. Take process scheduling, for this preemptible kernel, but at a higher priority than
instance. Rather than use a preemptive priority- Linux. These tasks can thus preempt Linux whenever
based scheduler, as an RTOS would, Linux they need to execute and will yield the CPU to Linux
implements a “fairness” policy so that every process only when their work is done.
gets a reasonable opportunity to execute. As a result,
high-priority, time-critical processes can’t always
gain immediate access to the CPU. In fact, the OS Dual-Kernel Model
will sometimes interrupt a high-priority process to
give a lower-priority process a share of CPU time.
User Applications
Also, the standard Linux kernel isn’t preemptible.
IPC
A high-priority process can never preempt a kernel
call, but must instead wait for the entire call to Linux Kernel
complete — even if the call was invoked by the Realtime Tasks
lowest-priority process in the system. This makes it
Realtime Kernel
difficult, if not impossible, to create a design where
critical events are consistently handled within short Hardware
(and predictable) timeframes.
Mind you, it’s wrong to think that this scheduling Figure 1 — In the dual-kernel model, Linux runs as the
model constitutes a deficiency in Linux. The model lowest-priority task in a separate realtime kernel.
have to be addressed within the Linux driver and Compatible to the core
virtual file system (VFS) models. This would How well does QNX Neutrino support the Linux
effectively require a rewrite of those frameworks — programming model? The answer lies, to a great
and of all device drivers and file systems employing degree, in the POSIX APIs adopted by Linux.
them. Without these modifications, realtime tasks Though rooted in Unix practice, these APIs were
could experience unpredictable delays when defined by the POSIX working groups purely in
blocked on a service. As a further problem, most terms of “interface,” not “implementation.” Simply
existing Linux drivers aren’t preemptible. Thus, to put, the APIs aren’t tied to any OS or OS architec-
ensure predictability, programmers would also have ture. As a result, an RTOS can support the same
to insert preemption points into every driver in the POSIX APIs as Linux (along with various subtleties
system — something that few Linux developers of Linux compatibility) without adopting Linux’s
have experience doing. non-deterministic kernel.
In any case, it’s unclear which realtime modifica- The QNX Neutrino RTOS supplies proof for this
tions, if any, will eventually be integrated into the implementation-independent approach: It offers
standard Linux kernel. After all, most Linux-based POSIX-compliant APIs, but implemented on a
systems rely on the kernel’s current approach, which realtime, microkernel architecture (see Figure 2).
sacrifices predictability to achieve higher overall Importantly, QNX doesn’t implement POSIX as an
throughput. A more deterministic kernel — with its add-on layer. Rather, the very core of the QNX
attendant reduction in average response times — Neutrino RTOS — the QNX microkernel — was
may simply be out of step with what most Linux designed from the beginning to support POSIX
developers want. realtime facilities, including threads. POSIX, and
hence Linux, compatibility runs deep.
The Best of Both Worlds Embedded Linux: a new definition
® ®
The QNX Neutrino represents an entirely different This issue of API compatibility is taking on surpris-
and altogether more viable approach to the problems ing importance as OEMs attempt to use Linux in
we’ve been discussing. Instead of trying to make
developers squeeze predictable response times out
of Linux, QNX Neutrino offers a proven realtime QNX Microkernel Model
operating environment that:
HA File Device
• allows Linux developers to keep their existing Manager System Driver
APIs and programming model
Message Passing
• addresses the shortcomings of realtime Linux
Microkernel
extensions through a much tougher runtime
POSIX TCP/IP Java
model, greater design options, and a unified App Manager VM
environment for realtime and non-realtime
applications
Figure 2 — In QNX Neutrino, the microkernel contains
• maintains key benefits associated with Linux’s only the most fundamental OS services. All other
open source model, such as easier trouble- services are provided through optional, memory-
shooting and OS customization — in fact, protected processes that can be stopped and started
dynamically. To achieve this modularity, QNX Neutrino
QNX Neutrino’s architecture offers distinct
uses message passing as the fundamental means of
advantages on both counts IPC for the entire system.
their embedded products. Until recently Linux was, helped save the Unix world from fragmentation. In
more than anything else, a unified community of fact, the ELC specification will be based on existing
developers creating software for a relatively narrow POSIX standards, such as the POSIX 1003.1, which
range of environments mostly web servers and the QNX Neutrino RTOS supports today. As a result,
workstations based on x86 hardware. This common the QNX Neutrino RTOS will inherently support
focus meant that Linux developers could count on a embedded Linux applications, while simultaneously
variety of benefits: binary compatibility, a consistent providing all of the benefits of a true RTOS designed
OS kernel, a pool of readily adaptable source code, from the ground up for embedded systems (more on
and so on. these benefits later).
debugged with standard, source-level tools familiar issuing calls such as open(), read(), write(), or lseek()
to every Linux developer. on the pathname. For instance, the QNX serial-port
Better yet, QNX Neutrino allows applications to driver typically registers the pathname /dev/ser1 to
access all drivers and OS services via a single, represent the first serial port. Any application that
consistent form of IPC: synchronous message needs to access that port simply issues an open() on
passing. This approach offers several advantages. /dev/ser1.
For instance, because QNX message passing is So where does the message passing come in? Well,
synchronous, it automatically coordinates the from the application’s perspective, the open() looks
execution of communicating programs, thereby and behaves like a standard POSIX call; there’s
eliminating the need to handcode — and debug — nothing special about it. But, underneath the hood,
complex synchronization services in every process the QNX Neutrino C library converts the call into an
(see Figure 2). Moreover, message passing inherently io_open message and forwards the message to the
simplifies the task of partitioning a complex system serial driver. If the application subsequently wished to
into well-defined building blocks that can be write a character to the serial port, a similar sequence
developed, tested, and maintained individually. would occur: the client would issue a write(), the C
Troubleshooting becomes much easier as a result. library would construct an io_write message, and the
Does this mean, however, that developers must use message would be sent to the driver.
proprietary message-passing functions? Not at all. There’s an interesting and highly beneficial side-
QNX messaging is implemented so seamlessly that effect here: the exact same message can be sent
applications and OS services don’t have to invoke from any client application to any service, provided
special functions to exchange messages — the the service supports that particular function. For
messages can be sent and received transparently, instance, to write a character to a serial port or to put
using standard POSIX calls. Now, that may sound a a character into a disk file, an application would issue
bit like magic, so let’s look at how it works. the same write() function in either case; the only
In QNX Neutrino, any service-providing program difference would be where the message was sent.
(e.g. a driver) can “advertise” its services to other In other words, the application is cleanly decoupled
programs by registering a pathname in the pathname from the services it relies on. This decoupling
space. Programs can then access those services by simplifies development since all interactions between
applications and system services can be implemented
MsgReply() or SEND
using a simple, POSIX-based programming model. It
MsgError() blocked also eases migration to new designs since applications
MsgReceive()
don't have to include hardware- or protocol-specific
MsgSend() code.
MsgSend()
RECEIVE
READY Two more (short) points about QNX message
blocked
MsgReceive() passing before we move on. First, developers can
MsgReply() or
REPLY access the QNX messaging framework directly, using
MsgError() blocked
three simple calls: MsgSend(), MsgReceive(), and
This thread
Other thread MsgReply(). Second, message passing isn’t the only
form of IPC that QNX Neutrino supports. Developers
Figure 3 — QNX message passing simplifies the
can, in fact, work with several standard forms of IPC,
synchronization of processes and threads. For instance,
the act of sending a message automatically causes the including signals, POSIX message queues, shared
sending thread to be blocked and the receiving thread memory, pipes, and FIFOs.
to be scheduled for execution.
Since QNX Neutrino is a POSIX operating system, it’s generally a simple exercise to port any open source
program — including any Linux program — that has been written to the POSIX API.
The following table lists a sample of the open-source applications that have already been ported to QNX
Neutrino. For a comprehensive listing, visit http://www.qnx.com/developer/download.
* Only available as part of the free QNX Momentics NC edition, which can be downloaded from http://www.qnx.com/nc.
Once those packages are installed, run the configure script. Once that’s done, run make. Assuming
there were no errors in the build, type make install. Your Linux port is now complete.
When problems do arise, they tend to fall into the following categories:
• The configure script failed because it couldn’t guess what OS you were using — If this
happens, it means that the program hasn’t updated its configure scripts since QNX Neutrino v6
was added to the GNU build system. Copying config.sub and config.guess from
/usr/share/automake-1.6/ over the same files in the program’s source tree should solve
the problem.
• The required libraries aren’t available — Many common libraries have already been ported to
QNX Neutrino, and can be downloaded from the QNX website or from a community package
repository. If you still can’t find the library you need, you’ll need to download its source code
and port it yourself. Generally, porting libraries is as easy as porting POSIX applications. (And
once you’re done, you might want to submit the port to the QNX website or a community
repository so that others can use it!)
• When porting a library, the configure script doesn’t think it can create shared libraries —
Again, this is caused by out-of-date configure scripts. If you’re happy with static libraries you
can ignore this problem, but if you really want shared libraries, you can edit the configure script.
There is usually a case $host_os in block that sets up various parameters for building
shared libraries in the configure script. (Tip: Search for “soname_spec” to find the block.)
Once you’ve found it, you can add the following to the block:
# QNX ELF
*qnx* | *nto*)
version_type=qnx
need_lib_prefix=no
need_version=no
library_names_spec='${libname}${release}.so$versuffix
${libname}${release}.so$major $libname.so'
soname_spec='${libname}${release}.so$major'
shlibpath_var=LD_LIBRARY_PATH
shlibpath_overrides_runpath=no
dynamic_linker='QNX ldqnx.so'
;;
You’ll also want to copy the ltmain.sh from /usr/share/libtool over the ltmain.sh in
the library source code. This should update the configure script for building shared libraries under
QNX Neutrino v6.
• When linking the program, socket-related symbols were missing — This happens when the
configure script incorrectly assumes that socket-based symbols are in the C library. You could edit
the configure script to fix it, but usually it’s just easier to add -lsocket to the LIBS= lines in
the makefiles.
If an open source program doesn’t use the GNU build system, it can still be pretty easy to port, once you
tweak the makefiles and remove or add a header file or two from the source files.
Quake III provides a good example of how easy it is to port Linux source to QNX Neutrino. The entire
port was completed in a few days, with most of the time spent tweaking the makefiles, changing the sound
code to work with the QNX implementation of ALSA, and converting the X code over to the Photon
microGUI (the native QNX windowing system). Ninety-five per cent of the Quake code just worked.
We began with a QNX Neutrino host for our development platform and with a QNX Neutrino board
support package for our multi-processor PowerPC target. To obtain an OSPF router daemon, we elected
to port the Linux-based Zebra router protocol suite.
We used the following command to generate the QNX-specific, PowerPC-targeted header and makefiles
for Zebra:
The –host parameter is, in fact, unnecessary because the included config.guess script properly
determines that QNX Neutrino is running self-hosted on an Intel architecture PC.
The –target parameter instructs the configure script to generate QNX-target, big-endian, PowerPC
makefiles.
The –prefix parameter instructs the configure script to generate makefiles that will use /opt/zebra
as the installation directory for the installable products. QNX Neutrino’s default installation location for
third-party software is rooted at /opt. Use discretion and consider QNX Neutrino’s package installer if
you want to install into alternate locations.
Results
With the Zebra configuration and build, we encountered two configuration issues immediately:
• The configure operation failed to locate socket APIs and set the communication method between
the route daemons and QNX Neutrino to be ioctl-based.
• Sources including zebra.h failed to compile due to a missing fcntl.h header file.
The first issue arose because Linux incorporates the Berkeley Socket Library into the GNU C library,
glibc, whereas QNX Neutrino implements it within libsocket.a. Correcting this was easy: we simply
added a test for a QNX Neutrino host to the configure file and added –lsocket to the linker options
with the line LIBS=-lsocket $LIBS. Once we reconfigured and recompiled the code, the problem
was solved.
We encountered the second issue because Linux places the file control header into /usr/include/sys
whereas QNX Neutrino places it into /usr/include. To correct this, we added an OS-specific QNX
define to config.h.in (the template for config.h) and added a conditional compilation flag to
zebra.h to include <fcntl.h> rather than <sys/fcntl.h> when the host OS is QNX Neutrino.
Reconfiguring and compiling demonstrated that this solved the header file issue. Here’s the code snippet:
#ifdef __QNXNTO__
#include <fcntl.h>
#else
#include <sys/fcntl.h>
#endif
With these minor issues resolved, we compiled and installed the Zebra suite without errors. To run the
OSPF software router, we added appropriate TCP/IP service entries to /etc/services for the
networking daemons and configured QNX Neutrino’s telnetd. We then had our OSPF router running
on SMP PowerPC hardware.
Conclusion
Members of the QNX user community have been able to easily port many open-source applications
to the QNX Neutrino RTOS, including Samba (CIFS) servers, NTP time servers, CORBA managers,
SNMP managers, packet filters, commercial protocol suites, X applications, distributed processing
systems such as PVM, shells such as csh/tcsh, scripting languages such as Perl, Python, and TCL,
Quake, and so on.
Packaged versions of those applications are available from the QNX user community. For instance, the
http://www.qnxzone.com site provides references to many ported applications.