RTOS, Latencies, Linux and SW libraries when choosing an OS solution for an embedded product with higher requirements for features

Read time: 20+ minutes

Introduction

For quite some time we have the world of RTOS solutions developing itself boldly. New distributions of Linux come (and sometimes they are discontinued), the kernel evolves slowly and securely, but many times RTOS systems, that theoretically may be better for your device, come and go without much noise as well.

Let’s imagine the following scenario:

We are a company developing an embedded device that needs a GUI, graphics and support for higher level HW interfaces (e.g. HDMI, WAN, Bluetooth, USB, 7 inch display). And higher CPU load. And we need to choose an OS. BUT – it must be an RTOS.

This article has the purpose of giving you basic info and what to consider when making the choice in terms of characteristics and features. The other purpose is to explain why Linux is one of the best options, and also to point shortly what other RTOS solutions are available around.

******* ******* ******* ******* ******* ******* ******

The basics

RTOS

RTOS is an abbreviation that means Real Time Operating System. Thus there is no such thing as the RTOS OS. Just to illustrate – here are some real RTOS systems: BeRTOS, eCos, FreeRTOS, MicroC/OS-II and MicroC/OS-III, Neutrino, VxWorks, Zephyr, uOS. This small list includes both free and proprietary examples. And many of those have ports for a lot of architectures.

One more important thing about RTOS – the term Real Time can refer to tasks from few hundred microseconds up to hours or days. Thus Real Time doesn’t mean speed, resources, or anything else but the following: succeeding to perform a given task in a predefined period of time. This is normally verified by measuring on a test bench thousands of times this performance, including with changing environment conditions and in multiple worst case scenarios. And when the task is always performed within the specified time frame – we have a Real Time OS.

Linux

The pure Linux itself has a lot of nice features – actually it possess so much features that no other OS on the market can give you – free, open source, with immense opportunity for customization and trimming, with existing hundreds variants that may serve from quite small special purpose devices to the largest servers, with libraries that can support hundreds of types of HW, optimized resource managers (for memory, servers, GPUs), able to work on any type of multi-processor machines, with multiple type of specific libraries for scientific calculations and thousand other options. In addition it has vast long-term detailed documentation, completely opened source, full changes and history documentation (along with the reasons for them), thousands of books, forums, etc.

There were two problems in the past: The monolithic kernel and the lack of preemption and determinism.

About the monolithic kernel – when the HW was weak it was hardly possible to fit a whole PC OS into an embedded device. For the last 10+ years the HW increased its capabilities immensely. And this problem was solved.

About the second very specific disadvantage in the world of  RTOS and Embedded devices: Initially Linux was a general purpose OS, without task preemption, and thus non-deterministic. This means – if some task takes too long to release the CPU or System resources, other devices may stop working until they get access to them.  Due to this – some special distributions with additional hacks and extensions were started in the past (since around 2002), which eliminated this initial characteristic. These hacks and the initial versions of the code to support some RTOS features like preemption, priority inversion and few others related e.g. to IRQs (Interrupt Request routines) were developed initially by Ingo Molnar, Thomas Gleixner, and Steven Rostedt. Their ideas attracted contributions from many more developers over the years. Those changes started being adopted slowly from the Linux Kernel Developers group (since around 2004) and step by step we have now a state that allows Linux to work fully as a Real Time OS.

For those that understand the numbers – one specific experiment was made in 2016 and announced by Jan Altenberg of Linutronix (at Embedded Linux Conference in Portland, Oregon): a specific test system “OSADL Latency Box” was used to trigger events with frequency 12000 times in a second (12 KHz) for 12 hours. In addition the test setup was made with 100% CPU load. What was measured each time was the reaction of the system and the delay between a stimulus from the test box and the result of the Linux target processing. There is more about why the test was made, but important was that the results were quite stable – the Linux system with correctly set preemption and Real Time settings was responding with average time ~40 microseconds and worst time ~95 microseconds. With 100% CPU load achieved via running of several tens of applications simultaneously. This means that we have predictable deterministic syste

Open Source

Regarding the open source – one of the main pluses of the open source SW is that you can use parts of it to develop a device with already prepared libraries. You can pay a developer to widen some of them for the common benefit and provide that part of the work to everyone, and then use the bundle of this library and some other libraries (e.g. proprietary of your company or bought from external companies) to create your product. It is a complete win-win scenario for all (considering that you will not release your product openly, and that is the regular business scenario). And this is how the open source SW contributes immensely to the community and companies all over the world. Once a library is created – it can be used from everyone. It can be later taken from you, trimmed and you can put additional features. At the moment that I make improvement – all users benefit (e.g. I find a bug). At the moment that someone else makes some attribution – I have the option to directly adopt the new features and fixes.

Second pros – there are really thousands of libraries. And you can use most of them.

The details on Open Source and how to use it in any commercial product based on licensing scheme you can read here.

Third pros – the history and all the information for the SW is most of the time available thanks to bug trackers.

Fourth pros – let’s say that you need one specific feature – finding a freelancer, who takes part in the development of this library to add sth. specially for you is possible. Whereas commercial companies rarely add sth. specially for you, they will ask for bigger payment, and if they agree at all – you most probably will get it slower. I have seen it in some projects.

Cons – theoretically you don’t get certified product. But – when all those developers decided to make an RTOS – they have most of the time included features to test the performance and qualities of your end product, i.e. you will be able to check the parameters. And whatever proprietary system you get – you should never miss this step. Simply because using a proprietary OS would never guarantee that the final product you created will comply with standards (and all possible user case scenarios

Proprietary OS

Regarding the proprietary solutions (i.e. paid OS): If we pay for them – what do we get more? Theoretically we get some support package (depending many times on how much money we would pay), and we get standardized, tested, measured system, that is already proved for its qualities in multiple commercial devices. Not that open source products are not used in commercial products. Of course they are used. And many free and open source Real Time OSes are also measured, and can be tested. Some can be standardized actually (or at least the products they are used in, not a subject of this article).

But as already mentioned – you still have to make the small changes for you mostly by yourself (cause nobody cares for small things), support is limited and many times internal information on why and what was created may be concealed, and at the end of the day – you still have to test your end product thoroughly.

Despite this – may times some of the proprietary OSes are really good, result of many years of development and do possess really good features

Embedded device

Nowadays this term is so wide and diverse, that it covers any self-sustained device from the smallest micro range till big PC-like platforms that run full-scale linux or windows inside them. So – consider this when reading further. And consider that in this study I care about the bigger systems, and not e.g. smallest scale sensors, or one-chip micro devices running on battery for the next 3 years with most of the time sleeping.

Again – we look for an OS for a big device, with graphics, libraries, drivers, etc. And it must be an RTOS.

******* ******* ******* ******* ******* ******* ******

Measuring few basic parameters of an RTOS regarding speed and timing

Latency and Jitter – the basics to define deterministic systems

Latency is the speed of reaction of our system to a given event. Example – if I have a device that shall perform an action (e.g. video enabled device with a screen that plays a movie clip) after the pressing of a button – the Latency would be the time from pressing the button till the movie is really started on the screen. Latencies can be measured for any type of triggers and resulting actions. In the world of OS we may measure the latency between the interrupt from the button until the task X (which shall perform the action) really starts operating. Also we may split this time in two – the OS latency until the task is started, and our task initialize time from first call to the task until the action actually starts being visible for the user.

There is one important point about the Latencies – any dedicated custom microcontroller/microCPU platform with FIQ (Fast interrupt Request) HW is still one of the fastest RTOS techniques that can reach latencies under few microseconds. This means – for some specific tasks it is impractical to consider regular Linux on a regular platform to be a solution. This however is not our scenario. And again to show how Linux developers provided solution even for this – when the experiment was made (the one I mentioned few paragraphs above) – Jan Altenberg of Linutronix made customizations and configured the core to use FIQ on the target (an ARM based CPU that had this HW option). When he made the tests – the worst Latency (or reaction time) measured was ~40 microseconds. With best reaction times down to ~5 microseconds.

So – buckle up and hold yourself tight – Linux is already RTOS, actually since several years. And yes – there are hundreds of industrial and consumer products that provide already Linux RTOS solutions on the market. In billions of different devices.

***** ***** ***** ***** *****

Jitter (as defined by Rafael Aroca and Glauco Caurin in [1]) is indirect information obtained from several latency measures, consisting of a random variation between each latency value. In a RTOS, the jitter impact could be notorious, as it is analyzed by Proctor when trying to control step motors. For example, the pulses duration controls the motor rotation, but the jitter induce the torque to vary, causing step losses in the motor [Proctor and Shackleford 2001]. To compute jitter, the time difference between two consecutive interrupt latency measures is calculated. Finally, the greatest encountered difference is selected as the worst jitter of this system.

***** ***** ***** ***** *****

Few images to present graphically what jitter is in different occasions:

1. Waveform timing variations (Neil Roberts -July 14, 2003, link)

Now here is what happens when we have a waveform controlled by clock – the jitter causes distortions. If those are within acceptable rate it is OK. If those are not – we have to change our design.

Effects of Jitter

When e.g. digital audio is played back – its speed is controlled by a clock.  In a perfect world the period of each clock cycle would be the same, but in the real world they always vary.  This variance from an ideal clock is called jitter.  The variance causes distortion in the reproduced waveform. Jitter is a measure of the extent to which execution timing fails to meet deterministic expectations. Most real-time applications require timing behavior to consistently execute within a small window of acceptable jitter, as shown in the illustration from National Instruments:

***** ***** ***** ***** *****

One final example relevant to our tests: Let’s imagine that we measure for thousands of times the delay of an event in microseconds (us) – and we then build a graphic of the different delays values against the number of occurrences (in this case presented as a histogram):

In this particular case the jitter means that we cannot guarantee 100% that we will finish our task before 4000 us. The graphic is taken from servers Disk latencies measurements. Why there are two peaks and what it means – feel free to google more on bimodal jitter. I will be satisfied to point out that there is a potential problem and it should be investigated – as normally the peak should be one.

***** ***** ***** ***** *****

When do we measure those characteristics?

In the world of RTOS – at the moment that we add our tasks to the system, our code, actuators and functionality – we should measure any important delays in as much as possible combinations of external stimulus and different system load. After the latency measurements – this one will define the maximum speed of operation, and after checking for it we may either reduce the speed under the limits for it to occur, or we may eliminate a problem in our design.

If we neglect it and it happens in combination also with resonance frequencies of events occurrences – this can practically stall some function, or even block completely the program flow of our device.

Example 1: I have seen this e.g. while developing video SW for Nokia mobile phones. The result of lack of synchronization between CMOS image sensor readout and the internal video capture state machine was e.g. complete stalling of the displayed video and no data in the saved file.

Example 2: While developing the Alert Manager for a family of clusters (car instrument panels) – when we had jitter in the GUI high level machine – this led to complete disruption of the displayed content, freezes, etc. The problem of course is if we not check it in worst case scenarios and this occurs at customer side.

Final notes on why we may have real trouble with this in a high level CPU nowadays – there are internal caches, HW pieces that differ between processors, CPUs that allow run-time instructions optimizations and many other specifics. The result is that even if we are able to make at least some partial static analysis of our system for the reaction to an IRQ – in full scale Linux / Windows (and practically any other system with high level OS with separated kernel and user space, and both HW and SW optimizations) – the main way to test for the jitter is with real time tests via Black Box approach. That’s why airplane, medical and automotive devices are tested for hundreds and thousands of hours with dedicated devices.

For small scale RTOSes on a regular MCUs without special HW optimizations – the jitter can be very well analyzed also statically (though this is never enough and any standardization requires HW tests). But practically they pass also the hundreds and thousands hours of testing.

Worst Case Response Time

This measurement is as well performed mostly after all our parts of a specific function are added to the system. Now that we have the full system – we shall develop a test, which will put all the given resources to be used at maximum level, and check again the Latency from trigger A, until action B is performed. In other words – the Worst Case response time is the biggest Latency that may occur for a given measurement. Actually Worst Case response time is understood many times after all jitter / latencies tests.

******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* *******

******* ******* ******* ******* ******* ******* *******

Characteristics, resources, and libraries based on HW and SW features

One of the major points to consider for the choice of OS is the required resources. Here is a simple comparison – regular PC Linux distributions require between 0,6 and 2 GB minimum hard drive space, at least several hundred megabytes of RAM, at least several hundred MHz of CPU frequency. Optimized RTOS variants of Linux may reduce those requirements in factors of 10 to e.g. 50 times less.

In comparison for MicroC/OS-II, as written on their website: The μC/OS kernels allow for unlimited tasks and kernel objects. The kernels’ memory footprint can be scaled down to contain only the features required for your application, typically 6–24 KBytes of code space and 1 KByte of data space.

You can see the difference – the first may go up to GBytes and hundreds of MHz, whereas the second goes to Kbytes and is applicable for microcontrollers running as slowly as several KHz.

So: How do we evaluate for the upper parameters and what may influence the choice of an RTOS?

General OS characteristics and resources requirements

Any RTOS can be measured for Latencies and further for Jitter and other product related specifics. But here is what to consider further when choosing one for your device

1. Speed of task switching

The bigger the system, the more resources are required. Of course if we need MMU, dynamic threads and other features – we should have more resources in terms of CPU, RAM, EEPORM. By default the high level operating systems have bigger overhead, but those are supported by modern developments in CPU architectures like highly evolved MMUs. As it is really complicated to statically analyze the task / context switching in a big modern CPU – normally this should be checked via real tests. Example for modern Linux testing can be found here. On the other hand – e.g. the ARMv8 Cortex A53, used in the RaspberryPi 3B already has L1 and L2 caches, an MMU with TLB and other optimizations.

Just to have the reference – an old school minimalistic RTOS written in C, which is optimized for speed, may take less than several tens of instructions. Even if we have no optimizations, and so some instructions require several clock cycles – at speed e.g. 600MHz – 200 clock cycles are 0.33 microseconds (the numbers are totally generalized!). If the MCU is pipelined, and the OS is optimized – these times may go down up to several tens of nanoseconds.

This is why modern CPUs and OSes have struggled hard to provide faster and faster HW and SW for optimizing the context / task / thread switching.

I was searching for more information, but people do the benchmarking without posting the results on the web, so it was hard to find one. However Quora is giving an example (back in 2012!, and speeds are going up in the last 7 years!): https://www.quora.com/How-long-does-a-context-switch-take. What is cited there is several hundreds of nanoseconds on Pentium 4 CPU.

Final point on this matter – your design should in general avoid creating hundreds of threads. If you need so many threads – then it should be some really special system! And about benchmarking – Linux provides quite a lot of tools in its perf module

2. Required services

What services do we need – interrupts handling special wrapper, dynamic loading of drivers, HW HotPlug support (classical example – for a USB stick

3. Memory footprint

These are HDD, SDCARD, eMMC, EEPROM – once we know the main features – we go to check how much memory is required from the different options for us. Are those scalable easily? What is the minimum and maximum required memory

4. RAM memory footprint

Depending on the features we should check the required RAM. Usually dynamic tasks, dynamically loaded drivers and more services require more RAM. And we need to know what is the minimum required by the different OSes and maximum after we add our tasks

5. Available libraries

Critically important – even if we choose a great OS – at the moment that we need e.g. MP4 video decoding, but there is no port of libraries for it for this OS – this may lead to further delays and additional price for development, and sometimes for special proprietary libraries

6. Data storage at power off

Necessity and ability to store data at power off, retrieving it at power on – This feature is normally not obligatory OS related, but let’s say – Linux will be able to store and later retrieve files and data from and to any type of storage

7. Support

If we choose a proprietary OS – what support we get, at what price, is it a package of certain amount of hours, is there common support, will we have dedicated contacts? If we pay and don’t know these details – it would be really stupid money spending

8. Аvailable documentation

This should be checked in advance. Especially for proprietary SW

9. Available forums

We care about the possibility to search for lessons learned – most of the Open Source products have such. And it is important for us if we chose to use proprietary SW whether there is a community. Example – in the past the company KEIL (owned since 2005 by ARM) (specialized in commercial embedded tools like compilers, IDEs, OS ports and others) – had a forum, that I used several times many years ago. And the forum helped me find most of the answers I needed

******* ******* ******* ******* ******* ******* ******

Features of RTOS to consider when choosing depending on HW and SW needs

As explained in the beginning – Embedded devices can be from the smallest to the biggest scale. As a result we need some way to compare the different OSes based on what they deliver as functionality by default. So here are the basic points on the different features that we may use further as a base for our comparison

Hardware related

Here we care whether the given OS has already support and drivers for the given pieces of HW. We can always port drivers. But if we have them in advance – the time-to-market will be reduced significantly.

• RS232/485, UART, I2C, SPI, GPIO, RTC – as those features are really basic nowadays there is no serious OS to consider here that will not support them
• CAN, LIN – basic communication for automotive products
• MOST, Flexray – again mostly for automotive, what is interesting that those are high speed high data volume networks
• LVDS, MIPI DSI – for accessing common type LCD displays
• LAN
• WAN / WiFi (Low Power WAN, LoRa WAN – Specific Long Transmission Distance WAN)
• BLE (Bluetooth Low Energy)
• 6LoWPAN – IPv6 over Low-Power Wireless Personal Area Networks
• USB Host – ability to connect e.g. USB sticks to our device
• USB Device – ability to connect our device to a PC
• USB 3.0 – drivers
• HDMI – drivers
• SD CARD reader ability – despite that this is quite common – it is a specific feature
• Firewire – drivers
• PCMCIA – drivers
• Audio – capability in general, plus some drivers if possible

• TV tuner – despite that this is seldom, it may be important for the classic case like a Set-Top box

• Webcam support – for some devices – e.g. high class real estate security system, or for a specific communication device
• Think of any other HW piece that might be necessary for your device

Software related

• Command Line Interface – critical if we need runtime internal OS access
• File System – critical for any bigger system
• Runtime Analysis – important if we care seriously about performance. After all we search for an RTOS solution. This however is relatively easy to implement and shouldn’t be defining factor. I have written logging systems and have checked performance with GPIOs and other methods. On the other hand – if you need full scale analysis of used resources (RAM, CPU, memory, etc.), and not simply timings and latency checks – you will need this. Finally – most RTOSes provide at least some measurement of the basic runtime parameters.
• Support for Libraries – the most classical example is Linux – there is no other open source platform with this amount of libraries, drivers, modules, ports for any HW, etc.
• GUI – do we need it? If our device has a display GUI may be helpful. On the other hand we may want to build or use our own GUI. For Linux there are tens of architectures with hundreds of versions from the simples to the most sophisticated GUIs
• TLS/SSL – if we need remote encrypted access – this would be really important. Again available by default in Linux
• POSIX API – if we have libraries or modules based on POSIX – this will be critically required. Available by default in Linux.
• NVIDIA drivers – some of NVIDIA GPU cores have explicit port of the drivers for systems different of Windows, OS X (MAC) and Linux – if we choose different OS and we need / have some NVIDIA HW – this might be quite important
• OSEK (Offene Systeme und deren Schnittstellen für die Elektronik in Kraftfahrzeugen; English: “Open Systems and their Interfaces for the Electronics in Motor Vehicles“) – i.e. a standard applicable for Automotive devices
• AI capability – now this one is tricky. Artificial Intelligence means mostly that we will have to implement e.g. neural networks or other high intelligence algorithms. This usually implies higher complexity of the OS. And again for Linux there are already a lot of solutions.
• Think of any other important feature, that derives from the specifics of your or other products

******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* *******

******* ******* ******* ******* ******* ******* *******

Classification of Embedded devices based on features with few examples

It is practically impossible to even imagine listing all possible device classes. But we can try with some examples to have basics on classification. Please consider that these are just few theoretical examples and one would definitely need to consider a lot more when designing similar devices for real!

I emphasize again – Please consider that these are just few theoretical examples that I made up in half an hour – just to present potential scenarios. One would definitely need to consider a lot more when designing such devices for real!

Now that you have been through the devices – we have seen how many details have to be considered when making the choice. Finally – each OS change requires more or less some (and sometimes even serious) adaptions of the architecture. Even more – if we decide to work with Linux – it makes sense if we use the features it provides. As everybody who ever read at least few lines on Linux – almost everything in it may be accessed via file interface.

At the moment that we switch to minimalistic custom RTOS that has different than the Linux file concept – the code can hardly be ported for few days.

Of course if we comply to the POSIX API, and write our Linux code POSIX compatible, and switch to RTOS that supports POSIX – we will have no trouble. The same applies in the other direction.

On the other hand – if we develop a device for an year already, with market / SOP date coming close, and with several next versions of the device in our planning – most of the time we will stick to the architecture we have chosen in the beginning, and search for the best way to expand, improve and widen its capabilities. Switching to other architecture means corresponding price in time and resources, thus rarely done. That’s why we have to choose carefully. The next paragraph is on potential regular RTOS solutions, if you want to go to the potential Linux solutions – skip to point 4.2

******* ******* ******* ******* ******* ******* ******

List of live and widely adopted RTOSes and few general points on OS choice and change

Following is a table based on Wikipedia’s page: https://en.wikipedia.org/wiki/Comparison_of_real-time_operating_systems. The original Wiki page contains 190 RTOSes, almost no Linux based among them! I didn’t even imagine that there is such diversity! I knew or have heard of 20% of those. In addition I care about OSes that give me big amount of scalability. So this is not my choice for my device, but if you need a classical RTOS – this is your list – only 41 left for you to have fun with.

What I did was serious effort to remove all obsolete, stopped, and not supported OSes. This means I have checked and read all their home pages, and checked a lot more than the “one paragraph advertising resume” on the home page. In addition I have removed those dedicated only to specific architectures, and several ultra-small OSes. It turned out, that most of those OSes doesn’t provide SW / driver packages that can easily extend the supported-by-default HW extensions. Or at least they didn’t state it clearly. As those are considered RTOSes, most of them have no graphics support at all.

The result is a list of commercial and open source OSes that are live, widely used and are good choice mostly for minimalistic, small or specific embedded devices.

Of course whenever you read this article and choose some OSes to check in detail – please check the supported boards, CPU architectures and so on. Most probably some of them will have widened the supported list and rarely would have abandoned support for some older boards / HW.

The table below is available to download from this link.

******* ******* ******* ******* ******* ******* ******

The Linux solution

This was literally a struggle. Not that 10 years of different HW platforms, projects, OSes and whatever were not :D. But the fun, as well as appetite, comes with the advance and the getting closer to the final solution.

A month ago I started working with Arch Linux to test some basics in relation to a potential project. But it has quite a footprint – OK, I will create bigger device, I need the drivers, the CPU will be powerful enough, etc., and still the pure Arch Linux image is not small. Version 2018.11.01 from November this year, with Kernel 4.18.16 is 586 MB. Initially I have chosen Arch as it is one of the older and quite stable Linux distributions, with great community and support, dedicated to deliver small pure Linux without GUI, always updated with the latest Kernel version, practically great for development. In addition – it is said about it – if you want to go hard and force yourself to learn – this is the distribution for you. Second addition – it is so stable that a lot other groups of developers use it as a base to create their distributions. Marvelous. Awesome! :).

However  if I decide to put it in an embedded device, and I add a GUI and a bunch of libraries I need – it looks like I’m going at least for 1 GB of Flash memory / EEPROM / eMMC. This is far too much. I want to reduce the cost, and I don’t want to have all possible features that I can dream of. After all I strive for scalability, configuration, and as an engineer – always to put in my devices exactly and only what I need. Not to mention that removing the unnecessary things many times helps to speed, sometimes removes unexpected problems, reduces even the cost in merge-update procedures (cause you don’t need to care for things that you don’t use and loose whatever resources on them), etc.

So I thought – there should be some even smaller distributions. And I searched. I have checked every possible minimalistic or lightweight Linux distribution, and it turned out – all of them provide already a GUI, and they are all PC oriented.

The smaller OSes, most of which are listed above in the RTOSes table, have not even bare resemblance to Linux. On top of that – after reading for hours on microkernels and OSes based on them – it turned out those provide always almost no services. Instead they have only high performance and smallest footprint. And once you add enough high level libraries – this would result in reduced speed and performance. Because as each high level service will be in “User Space” – the total overhead will lead to less performance compared to a regular monolithic Kernel that provides all the features incorporated in advance.

The Linux Kernel itself is the best combination of great performance, maximum optimization, constant improvements, the biggest list of services, and ported for maximum number of platforms. A bit more on the Linux Kernel design is available in this article from 2017.

The struggle seemed at dead end for me at one point, but again – the one who strives will find what he needs. There was a solution, and here it is:

Any Linux can be trimmed! Arch Linux as well. This means – we can practically remove all the unnecessary SW. So if I don’t need e.g. package X – I can simply uninstall it.

Now comes the question – how to make my own Linux, only with the features that I need? And the answer is: there are several ways to create your own distribution, and others to write this into a custom image. One would ask – isn’t this really a big pain / problem / overhead? I would say No. The reasons:

• If I have my own team and they have to study some proprietary OS – I have to pay for this OS, and they have to invest the time to study
• If I choose minimalistic OS – my team still needs to study some specific OS, and develop all the upper drivers, layers, etc.
• If I chose to create my own Linux distro – my team will have to learn, but any SW that will be created later will be the most portable ever.
• Finally: if I compare the number of OSE, PikeOS, Y@sos or Zephyr OS experienced developers, to the number of Linux developers on the market – Linux somehow beats them all. And as I want some bigger device, and not small specific devices that require explicitly minimalistic RTOS – yes, that’s my solution.

******* ******* ******* ******* ******* ******* ******

Arch Linux vs Debian

First I need to add a bit more info on Debian as it is the only real competitor to Arch for minimalistic, pure, proven distribution, with community and already available packages. It turned out there is a minimalistic Debian called GRML – and it has a small version that is normally around 300 MB. But it is ported only for i686+ and AMD64. The pure minimal Debian 9.6.0-arm64-netinst.iso is only 203 MB. But it later adds more packages via internet.

Debian provides a lot more packages (over 50,000) with drivers by default. And it has as well a dedicated tool chain for creating your own custom distribution. Debian provides one of the most stable versions of Linux OS in the world. It has unstable, testing and stable branches, and before going to stable one release keeps in testing for several months. So it is rock solid. Finally Debian is one of the oldest basic distros – it started in 1993. And it is ported for a lot more architectures than Arch by default.

On the other hand Arch Linux is 10 years younger, has a lot better Wiki archive, provides quite less packages by default (a bit over 10,000), but gives you the ability to build yourself over 40,000 more. Its popularity grew so much, that it is now one of the main root distros, feeding many other derivatives.

Just to mention about other main distributions: Ubuntu – Debian based, Mint – Ubuntu based, – i.e. Debian. The not Debian main distros (apart from Arch): Slackware, RedHat, Enoch and Android. The last one is not Linux, but it is Linux based. One of the greatest ways to understand why Linux is powerful is just by looking the distributions trees.

Finally more info on the comparison of both: here and here

******* ******* ******* ******* ******* ******* ******

How to make minimalistic Linux custom trimmed for your device and the final Linux choice

Is it actually a problem, that our chosen distribution Arch is not ported for all the possible CPU architectures around? Because we are going to build our own system. And I plan to do it from scratch. Let’s see.

After searching and reading a lot – there are several ways of building your own distribution that are convenient for our use case as follows:

• LFS – stands for Linux From Scratch – this is a project that provides you with step-by-step instructions for building your own custom Linux system, entirely from source code. It provides complete flexibility, has a lot of tools to help you add packages later, can give you really minimalistic image, and is practically a book that is posted online. The project exists for 19 years, i.e. it is quite stable, used, proved, with periodic updates and a great Wiki. Actually their web page is the start page of the Wiki. It is focused on cross-compiling, i.e. you can build for many architectures (apart from x86, i686 and ARM it supports by default as Itanium, SPARC, MIPS, and Alpha.) However – one note – LFS has the purpose of creating a full scale Linux distribution. This means it is not targeted on embedded devices.

• Custom Debian with some of the following tools (again not targeted for embedded devices, thus only mentioning): DebianCustomCD or Respian.
• Via Buildroot – as written on their Homepage: a simple, efficient and easy-to-use tool to generate embedded Linux systems through cross-compilation. This system provides you with detailed instructions on how to make a full image with Root File System. It is really a simple tool compared to Yocto. One of the drawbacks of the simplicity is that if one package is updated – to change it on the target you have to rebuild the complete image. On top of everything – it is not so easy to build several variants for different boards and with modifications. But the simplicity of the scripts of Buildroot leads to total ~1000 lines of code.
• Via Yocto – this is the best option for our case. Yocto is an open source system for cross-compilation, which is targeted to embedded devices. It also produces an image with Root File System, but provides a lot more flexibility, ability to build modified variants of the targets, support for hundreds of BSPs (Board Support Packages). Compared to Buildroot it has a lot more library packages by default (including some that are rarely useful like the old Qt3). You can even build a complete compiler tool chain for your target. The only drawback is that it is not simple – the scripts that support it are total over 60000 lines of code, including also Python and a custom internal language for some options. Despite the complexity – Yocto is community and industry supported, with active development, with a lot of manuals and HowTos, so – it is worth the learning. And it is our best option.

Finally – a bit more details on the comparison of Yocto and Buildroot can be found in this article from 2016, based on a common session from two developers working for the platforms. The resume is quite good and is from the Embedded Linux Conference in San Diego, CA. And despite that it is now almost two years old – it gives so good overview of how and why each system was made, that it is still valid.

******* ******* ******* ******* ******* ******* ******

I hope this overview was interesting to read and gave you all the basics on how to choose an OS, and why this choice is critically important. Of course each of the points allows quite deeper explanation, but the purpose of this article was not to teach basics in Embedded SW Engineering. So if you want to know more and get the details on any of the points – google them. In my experience with a bit of searching, and reading 10-20 results from any search engine – one can find the right two or three articles that will give him enough information to understand what he needs. And another piece of advice – search for newest articles, better posted in the last year, and always triple check and cross-compare the information. The mere advertisement of this or that solution, without the solid back information and deeper understanding, is not a well-informed choice.

In conclusion I would say – don’t underestimate any of the points mentioned here. Otherwise you risk your endeavor, no matter if you are a developer, architect, project manager or entrepreneur. Read the details, check, digest and cross-compare information in advance. If you do all the steps – later you will benefit from making the 99% solid-grounded choice. And usually this means the best results on the market later.

********

Resources:

********

Open, free and limited source licenses in the world of Software development – By me, December 2018: https://triplehelix-consulting.com/open-free-and-limited-source-licenses-in-the-world-of-software-development/

Inside Real Time Linux: 2016, at Embedded Linux Conference in Portland, Oregon, by Jan Altenberg of Linutronix: https://www.linux.com/news/event/elce/2017/2/inside-real-time-linux

Waveform timing variations (Neil Roberts -July 14, 2003)  https://www.edn.com/design/communications-design/4139019/Phase-noise-and-jitter–a-primer-for-digital-designers

How long does a context switch take? – Oct 11, 2012, answer to a question in Quora:  https://www.quora.com/How-long-does-a-context-switch-take

********

Wikipedia – Comparison of real-time operating systems

https://en.wikipedia.org/wiki/Comparison_of_real-time_operating_systems

********

Perf – one of the Linux modules for performance evaluation: https://elinux.org/Perf

Is Linux kernel design outdated? May 8, 2017: https://www.infoworld.com/article/3195147/linux/is-linux-kernel-design-outdated.html

GNU/Linux Distribution Timeline: https://upload.wikimedia.org/wikipedia/commons/1/1b/Linux_Distribution_Timeline.svg

Arch Linux VS Debian, December 2017, by David Morelo : https://linuxhint.com/arch-linux-vs-debian/

Arch compared to other distributions – in particular with Debian: https://wiki.archlinux.org/index.php/arch_compared_to_other_distributions#Debian

********

Linux From Scratch : http://www.linuxfromscratch.org/

Buildroot : https://buildroot.org/

Yocto Project : https://www.yoctoproject.org/

Deciding between Buildroot & Yocto, By Nathan Willis, April 6, 2016 :  https://lwn.net/Articles/682540/