Everything about updating Linux devices

The Year of the Linux Desktop may get postponed every year, but Linux has come to dominate pretty much every other market segment - from servers, industrial machines and mobile phones (via Android), to a host of embedded devices.

In the modern connected world, it is more important than ever to make sure all devices are kept up-to-date with all the latest security and feature updates. In this article, we’ll dive into the possibilities of managing system updates on various flavors of Linux.

But first, let’s see how it’s different compared to smaller embedded devices.

What is “firmware”?

Broadly speaking, there are two main types of architecture that can be used for embedded (IoT) devices:

• A microcontroller unit (MCU) that typically integrates a small processor and at most a couple megabytes of flash memory and RAM in a single package. An MCU typically runs either “bare metal” applications or a real-time operating system (RTOS) such as FreeRTOS or Zephyr. Software running on these kinds of devices is largely monolithic, with everything from the OS to drivers and applications compiled in a single binary running in a single memory address space. There is typically no file system, at least not for the firmware, though one can be used for settings or interacting with external media.
• A central processing unit (CPU) that is connected to external RAM and mass storage. On embedded platforms, these are often integrated with additional peripherals, forming a System on a Chip (SoC). CPUs usually run full-fledged operating systems, typically Linux, that support processes, separate address spaces, memory protection, and the file system.

Of course, you can also run an RTOS on a large SoC or even a desktop computer, and there are variants of Linux which fit on some MCUs.

For those from the MCU/RTOS world, the “firmware” will be that single executable that you build when linking your RTOS, drivers, libraries and application code. It usually lives in a specific range of flash memory addresses, with the rest often reserved for the bootloader, swap space for firmware updates, and non-volatile settings storage. There are a couple of possible approaches to firmware updates, but the general idea is to replace one binary with a different version of it. Those binaries are just blobs of data, and because there is a single binary being built, versioning is pretty simple - when you build firmware, you give it a version number and that’s it. You can’t upgrade the RTOS kernel without rebuilding the entire firmware, so all dependency management and version control has to happen at compile time, on developer machines.

Users of desktop computers will know that larger OSes are vastly different. In fact, when talking about full-fledged computers, the operating system is rarely referred to as “firmware” at all. The term tends to be reserved for BIOS or UEFI software that lives on your motherboard and is responsible for early initialization of the hardware. But it is not very useful as an operating environment in and of itself.

So let’s see how all this maps onto various types of Linux systems. To get a baseline, we’ll start with the traditional desktop and server distributions.

Working with the “big guys”

A typical desktop Linux distribution - think Ubuntu, Fedora, Arch, or even Raspberry Pi OS - is, in essence, a collection of packages. You may be familiar with *.deb files and the apt tool on Ubuntu or Debian, or *.rpm and dnf on Fedora. apt and dnf are examples of package managers, which install, uninstall and update packages while also ensuring that they play nicely together - for example, by enforcing dependencies.

Packages exist for common software like the Firefox browser or LibreOffice, as well as core system components like the Linux kernel, the C library or the GRUB bootloader. Those core component packages will often include additional scripts responsible for updating the boot partition or reconfiguring the bootloader to use a different kernel version by default. However, the main point remains that the installed system is a collection of packages, each versioned separately. Many packages will declare dependencies on each other, e.g. a web browser may require the C library in a version of at least 2.17, but earlier than 3.0. The job of a package manager is to ensure that they aren’t broken.

Security and feature updates will be delivered by pushing newer package versions into a central repository. Distribution maintainers will make sure that all packages currently in the repository work with each other, so that users shouldn’t have dependency issues. Some distributions, like Arch, are considered “rolling release” - packages are released as software is updated upstream, so both minor security updates and major feature upgrades are delivered the same way.

However, in more traditional distributions like Ubuntu, there will be separate repositories for each of the OS releases - most mainstream distributions are released twice a year. Core packages are usually not updated to new major versions within an OS release. However, an in-place upgrade to a newer release is relatively similar - the OS is reconfigured to use the repositories of a newer release and an otherwise mostly normal system upgrade is performed - possibly with some additional scripting to take care of any major architectural changes, like when various distributions have changed the init system from sysvinit to Upstart and then to systemd.

This approach is very flexible, as it allows distribution maintainers to push updates to each system component separately, keeping the update downloads small. However, it also means that each individual installation is unique, and update issues may not be easily reproducible. The user is also generally expected to be present during the update process, to address any issues that may arise. For example, if the configuration files have been modified in a way that the package manager can’t handle, making it unsuitable for automated remote updates. Not only that, but there is very little resilience to errors - an unsuccessful update is likely to leave the system in an inconsistent and possibly unbootable state. Traditional distributions also usually don’t enforce the system integrity in any way - you may install things that aren’t packages (e.g. compile software yourself), for which the dependencies won’t be tracked in any way - subsequent system upgrades may easily break that software or even cause conflicts.

Modern distributions will also support containerized application packages like Snap or Flatpak, which avoid using the distribution’s native core packages as much as possible. Instead, they run applications in their own sandboxes with dedicated core components - different versions of which can be run side by side. I’ll leave these issues aside for the time being, though.

The immutable fix

Commercial operating systems have long alleviated these problems by introducing mechanisms that prevent key system files from being modified. You might have heard about the System File Checker on Windows or System Integrity Protection on macOS. If you try to mess with files in the C:\Windows\System32 or /System directories, the OS will prevent you from causing damage, even if you’re running with administrative privileges.

Not only that - on macOS Catalina and later, the base system installation lies on a separate, read-only volume, and any modifications made by the user are overlaid on top of it but logically separated. Windows does not have such a feature on the file system level - but it keeps track of what a “clean” system is supposed to look like, so you can use the “Reset This PC” feature to functionally restore it, even without any installation media or downloads.

Since the mid-2010s, various Linux distributions have attempted to replicate those principles – often called “immutable” or “atomic” distributions – in the world of free software. The basic idea is to have most of the file system read-only, and encourage the user to install software in a containerized manner. However, the details depend heavily on a specific distribution. Here are a couple of examples:

• Fedora Silverblue started as Project Atomic in 2014 and has been an official variant of Fedora since late 2018. Along with its other immutable siblings like Fedora IoT and CoreOS, it is arguably the “original” immutable distribution for the desktop. However we may qualify it, it relies heavily on OSTree, a piece of software that treats the entire root filesystem in a manner similar to a Git repository. Updates to the base distribution are released by the devs pushing a new “commit” to the OS repository. provides a way to select the “commit” at boot time, which means you can just choose the previously working version at boot time if anything goes wrong. You’re encouraged to install third-party software via containers such as Flatpak, but for low-level stuff like drivers, the rpm-ostree tool can be used to layer arbitrary packages as “commits” on top of the upstream base system. Those changes will be replayed when updating the system - and if anything goes wrong, you can always switch to the original state at boot time.
• Ubuntu Core, first released as a variant of Ubuntu 15.04 “Vivid Vervet”, is the reference distribution for the Snap packaging system. Snap containers are essentially read-only SquashFS images that are mounted on-demand - and Ubuntu Core, through an elaborate bootloader configuration, boots directly into the “core” snap and only allows running other snap packages. If you need anything else, you’ll need to containerize it through something like Docker or even just chroot. Snap allows multiple versions of the same package to be installed side-by-side, so if there are dependencies between packages, each of them sees its own virtualized view of the file system, with only the right dependencies present. Updates are provided by including newer versions of the packages in the Snap Store, and you can boot into an older version of the “core” snap if anything goes wrong. Ubuntu Core is an IoT-focused distribution, but a desktop variant based on the same principles is slated for release in 2024.
• Vanilla OS is a rather new distribution based on Ubuntu 22.10, which actually focuses more on atomicity and transactionality than on immutability. On install, it creates two copies of the root filesystem - “A” and “B”, which are mounted read only. Updates and other maintenance tasks happen much like on a traditional Ubuntu system, but not written directly to the running filesystem. Instead, a piece of software called ABRoot creates a record of changes against the current state, and the “opposite” (e.g. B if A is currently running) partition is updated to reflect the new state if the transaction was successful, to be used after a reboot. That way, the last known working state is always available as a fallback. Arbitrary transactions are supported like this, so software can be installed through traditional packages, even if containerized technologies are preferred.
• SteamOS on the Steam Deck takes the “A/B” concept even further - the partitions are kept in sync with the OS releases on the block level, i.e. the on-disk state will always be bit-for-bit identical to some released version. On update, the “opposite” partition is synchronized with a remote image using RAUC and desync - even if the user has modified the partition in any way, this sync process will restore the pristine state. Third-party software is supported primarily via containerization and user-local installation. The user may re-enable read-write access to the root filesystem, but such changes will be discarded during the next system update. Overlays such as systemd-sysext may be used to keep changes permanent, though dependencies may break at any time.
• Finally, since version 11, Android has been providing a “Virtual A/B” mechanism that improves upon the A/B images concept by minimizing the storage requirements. On the logical level, the procedure works as described above for the Steam Deck, but the “A” and “B” partitions are virtual - a special driver remaps physical blocks of the flash memory in such a way that unchanged blocks are not duplicated, and updated blocks may be compressed.

All of these approaches have various tradeoffs between flexibility, reliability, and storage efficiency. However, each is designed to be directly accessed and maintained by the end users. In the world of embedded systems, reliability and predictability often take priority, leading to automated, remote updates which are executed without the user interacting with the system directly. On the other hand, flexibility is usually not as important, as the software is generally not installed or uninstalled as part of normal usage.

Understanding updates in embedded Linux

There is no single handbook for every application. Some embedded systems may opt for traditional package-based distributions if a particular application requires such flexibility. Others may opt for filesystem-level immutability, using solutions such as OSTree.

However, the most common update paradigm seems to be block-level updates of immutable partitions. This comes in two main variants:

• Symmetric (A/B) images - as described above for Vanilla OS and SteamOS, two functionally equivalent copies of the OS are installed on the storage medium, and any update re-flashes the one that’s not currently running. The roles are swapped after a successful upgrade, but the previous version remains a “last known good” state, to be used when a rollback is required in case of failure.

• Asymmetric (main + recovery) images - the system essentially dual-boots between a “main” OS that provides all the normal functionality of the device, and a specialized “recovery” partition that can be used for repair, and also for updates. The update process will involve rebooting into the recovery image, which will re-flash the “main” partition, either performing the download itself, or utilizing an image downloaded before the reboot. In case anything goes wrong, the download and flash procedure can be repeated when the recovery image is booted again.

Note that the partition layout may be more complicated than just two “A/B” or “main+recovery” partitions. Actual layouts usually also include a partition for non-volatile data such as configuration files or user storage, and an additional partition for the bootloader. Separate from the main system image may also be found the Linux kernel, peripheral firmware and additional applications. In the case of the A/B paradigm, those may be either duplicated alongside each of the A/B root filesystems or shared between both.

The root filesystem partition itself may or may not be mounted read-only on a running system. Sometimes it’s easier to mount it read-write to allow software that e.g. writes system logs and other volatile data onto disk to run unmodified. However, in such a case it is expected that all such data will be lost after an upgrade, so actually non-volatile data need to reside on a separate read-write partition, and the software directed to use it - either via compile-time configurations, symlinks on the root partition, or mounted as an overlay.

As already mentioned, the A/B paradigm is also the basis of system upgrades on ChromeOS, Android and SteamOS, among others. On the other hand, the “main+recovery” paradigm should be familiar to laptop users - a recovery partition that allows the OS to be reinstalled from the Internet has been standard on Apple computers since Mac OS X Lion (2011). Windows Recovery Environment, though somewhat more limited, has been available since Windows Vista (2006), while Windows laptop vendors were known to introduce custom recovery partitions years prior to that.

The A/B paradigm should also feel familiar to those experienced in solutions based on MCUs and RTOSes. However, there are some differences in the upgrade flow. On RTOS-based devices, the bulk of the upgrade logic is contained in the bootloader such as MCUboot or X-CUBE-SBSFU. The typical flow involves writing the raw update image onto the inactive partition, and the bootloader handles the rest after a reboot which may involve swapping the two partitions around, verifying checksums and cryptographic signatures, or even decrypting an encrypted image. On Linux-based devices, all these tasks are typically performed by the updater software before the reboot.

The simplest form of such “updater software” is just performing a bit-for-bit dd copy from the update image onto the system partition, and perhaps switching a flag in the bootloader to boot from the previously inactive partition. However, there are a number of additional features that are desirable for a production solution, including:

• Integrity checks to make sure that the image being flashed is not corrupted and will be bootable
• Cryptographic signature checks to make sure that the image comes from a trusted vendor and has not been tampered with
• Encrypted image support to make it more difficult to analyze or reverse engineer
• Delta image or chunked download support so that it is not necessary to download the entire image from scratch with every upgrade
• Scripting support so that e.g. configuration files and other non-volatile storage can be updated or migrated between upgrades, bootloader can be reconfigured, and so forth
• Integrated OTA functionality so that the device can check whether a new update image is available and apply it automatically.

Additionally, where tamper-resistance is important, the Linux kernel provides an additional layer of security - the dm-verity module. This allows the creation of a hash tree, which allows detection of any data modification on the protected block device.

dm-verity is enabled by default on Android and ChromeOS devices, and can also be used on any embedded device. When it is combined with other hardening techniques such as secure boot and kernel module signing, a device that will only ever run verified and signed file system images can be created.

To support all those features, a specialized class of software for managing software updates on embedded Linux systems emerged.

In this section you will learn about some of the readily available open-source software for managing updates on embedded Linux systems.

SWUpdate

According to its Github page, SWUpdate is “a Linux Update agent with the goal to provide an efficient and safe way to update an embedded Linux system in the field”. The project itself is licensed under GPLv2, but can also be controlled via a library that is released under LGPLv2.1. The first release dates back to July 2014, and the project has been being actively developed ever since, with a new stable version typically released twice a year.

It is designed for block-level upgrades for immutable filesystems. Its explicit design goal is that “SWUpdate should be configured to fit into a project, not vice versa”, so it can be made to run on pretty much any Linux system, in many paradigms including both “A/B” and “main+recovery”. The SWUpdate project itself does not impose any design choices upon the firmware vendor, so you still need to decide, for example, how to handle non-volatile storage.

Most integrations are based on either Yocto or, less often, Buildroot. The swupdate package is available in upstream Buildroot repository, but all configuration of both the system (partitioning etc.) and the updater has to be provided manually. Likewise, the basic integration with Yocto is available as a meta-swupdate layer; simple configurations for Raspberry Pi and Beaglebone devices are available as meta-swupdate-boards, but these are only intended as examples, not as production-ready configurations.

SWUpdate is modular to an extent, and contains the following parts:

• SWUpdate core - can take an update file and apply it; the file can be either available locally or, depending on compile-time configuration, downloaded on the fly using libcurl; the core updater is command-line-based, but can also be used as a library
• Web Server / Mongoose - serves a simple local web page that can be used to upload an update file by pointing a web browser at the embedded device’s IP address
• Suricatta - polls a compatible OTA update server for new update packages and applies them automatically.

SWUpdate defines a uniform update file format, which typically uses the *.swu extension. Under the hood, it is a cpio archive that may contain any number of archives, as well as a configuration file that describes how they should be applied. Depending on the SWUpdate’s compile-time configuration, the configuration file may support arbitrary scripting in Lua or Unix shell, as well as signing and/or encrypting of the packages, in which case proper signature or encryption will be required for successful upgrade. This file format is designed to be “streamed” linearly – that is, applied on the fly as it is being downloaded, without the need for any intermediate storage.

SWUpdate also supports “delta” images in two forms - the images can be rdiff images, utilizing the same algorithm as rsync, to be applied over a known previous state. Alternatively, the *.swu file may only contain the metadata while the actual data is synchronized over the network using a zchunk format.

SWUpdate is widely used in embedded Linux ecosystems, especially those based on Yocto.

RAUC

RAUC (Robust Auto-Update Controller) is another solution that, in the words of its maintainers, “controls the update process on embedded Linux systems”. It is a newer project, with the first release dated February 2017, and is updated multiple times a year. It is licensed as LGPLv2.1.

RAUC is developed by Pengutronix and integrated with their own embedded Linux build system, PTXdist. Official integrations are also readily available for Yocto (meta-rauc), Buildroot, Debian, Ubuntu, Arch Linux and NixOS. The Yocto integration is additionally served by example configurations for a number of popular development platforms, provided through the meta-rauc-community repository.

Much like SWUpdate, RAUC supports both “A/B” and “main+recovery” paradigms and multiple partitions. Its update file format, *.raucb (RAUC bundle) is a SquashFS image that can additionally contain its signature, verity hash tree, and may also be encrypted. Within that bundle, there is a manifest file that describes other components, and any number of images in a variety of formats - either bit-for-bit filesystem images or directory archives.

Update bundles can be local files, or downloaded on-demand from an HTTP(S) server. However, unlike SWUpdate, on-demand downloads are handled through an integrated network block device driver and HTTP Range requests. RAUC relies on the ability to mount the SquashFS bundle, so linear streaming is not possible without caching the entire bundle locally.

Delta upgrades can be handled through either a built-in “adaptive updates” mechanism, or through casync or desync. In all three cases, download optimization is performed locally and the update bundle needs to be available in its entirety on the update server.

RAUC can be controlled either from the command line or via D-Bus. It is only concerned with applying the updates, so there is no integrated web server or automatic update checks. In general, it is less flexible, but also more lightweight than SWUpdate.

RAUC is used to update the Steam Deck and Deutsche Bahn ICE train information terminals.

Mender

Mender comes up often in the context of OTA updates on embedded Linux devices. Developed by Northern.tech, it is both an open-source project and a commercial product, designed with simplicity in mind. While SWUpdate or RAUC provide extensive configuration options to fit any possible project, Mender focuses on the typical use case of updating systems based on either Yocto or Debian, using OTA firmware and software management through Mender’s own server software.

Both Mender client and server are open source, licensed under Apache License 2.0. However, Northern.tech heavily promotes its own hosted instances of the Mender server, which offer additional commercial features. The client software is also tightly coupled with the server tech, although the client program also supports performing standalone updates from local files or arbitrary URLs.

Mender artifacts are signed but do not support encryption. They can contain full partition images (designed for the A/B paradigm), delta images, or “Update Modules”, which support arbitrary scripting and may be used for application-level updates. Full partition updates are streamed over the network, so the entire file need not be saved in intermediate storage.

Mender may seem like a very compelling option thanks to its simplicity and low barrier of entry. However, the tight coupling between the client and server sides, combined with not being based on open standards, means vendor lock-in may be an issue. It is also much less flexible and configurable than either SWUpdate or RAUC.

How does AVSystem fit into all of this?

At AVSystem, we believe that OMA SpecWorks’ LightweightM2M is the right solution not only for remote management and monitoring of IoT devices, but also for managing the OTA firmware update process. Our Coiote IoT Device Management Platform offers a comprehensive solution for managing firmware update campaigns on both single devices and entire fleets.

Our LwM2M client library Anjay has been most popular on systems based on MCUs and RTOSes, but it is highly portable and can work on embedded Linux systems as well. Anjay implements a Firmware Update state machine compliant with the open standard OMA LwM2M protocol, which allows the user to implement the callbacks to any backend they want. This may very well be SWUpdate, RAUC, Mender’s standalone updater, or any other update solution for embedded Linux.

As you can read in the tutorial, Anjay handles the download itself and, through the aforementioned callbacks, passes the firmware image as a linear stream of bytes to the application code. For this reason, this implementation will work best with update agent software that enables linear streams to be applied on the fly, like SWUpdate or Mender.

The table below shows a comparison of how the callbacks can be implemented for different target platforms:

There are additional possibilities for managing updates in the LwM2M ecosystem. Multi-component Firmware Update has recently been introduced in Anjay and Coiote, which may be useful for updating non-monolithic systems, for example when some applications have a different release cycle to the OS itself. On the other hand, the LwM2M Software Management specification may be well suited for systems based on more traditional packages (*.deb, *.rpm, etc.) or containers (e.g. Docker).

Conclusion

The open-source nature of Linux means that it is endlessly adaptable and can fit almost any possible application. It also means that there are a multitude of possible strategies for updating the system. It’s important to understand the advantages and weaknesses of different solutions, and choose one that is appropriate for your particular application. Open-source firmware management software and readily available OTA update server solutions can help bring that solution to life, while staying true to the best practices used in the embedded Linux ecosystem.