Secure OTA Firmware Updates: From Risk Assessment to Fleet Rollout
Facundo Zingarelli
Wednesday, November 11, 2026
Key points
A secure update mechanism follows from the product's risk assessment. Authenticity and integrity (a signature checked on the device) are the baseline; anti-rollback, robustness against failed updates, image confidentiality and the installation policy are decisions, each with a hardware and process cost.
Existing stacks cover most of it: MCUboot on microcontrollers; Mender, RAUC and SWUpdate on embedded Linux. Each leaves different layers to the integrator, such as boot-time verification or anti-rollback.
Flash for a second image, OTP fuses or a secure element, an updatable bootloader and the ability to change signing keys must be decided before the hardware is frozen. Signing keys, reproducible builds and staged fleet rollouts then have to keep working for the whole support period.
Secure over-the-air updates are a requirement for connected products, both to fix vulnerabilities over a product's life and to meet regulations such as the Cyber Resilience Act. This article treats the update mechanism not as a feature added at the end of development but as a design that follows from a product's risk assessment.
It runs from why secure updates matter, through turning a risk assessment into an update design, to mapping those properties onto existing stacks such as MCUboot, Mender, RAUC and SWUpdate, the signing keys behind every update, and the rollout that brings each release to a fleet, closing with the decisions that cut across all of them.
1. Why secure updates
Vulnerabilities in a connected product are usually found after it ships, in its own code or in the libraries and stacks it integrates. A firmware update mechanism is how they are fixed in the field; without one, the options are a recall, a site visit, or accepting the risk. Because the mechanism can install arbitrary code, it is also part of the attack surface, and its design has to be covered by the risk assessment like any other interface.
This article covers the design of that mechanism on microcontrollers and embedded Linux: the properties it needs, how the risk assessment sets them, and how common open-source implementations such as MCUboot, RAUC, SWUpdate and Mender provide them, whether updates arrive over the air (OTA) or through a local interface.
Regulatory triggers
Most frameworks that apply to embedded products require an update capability. They describe the outcome and leave the mechanism to the manufacturer:
EU Cyber Resilience Act (CRA): vulnerabilities must be addressable through security updates, distributed securely and without delay during the support period, which is at least five years unless the product is expected to be in use for less. Our article on the Cyber Resilience Act for embedded teams covers these obligations.
Radio Equipment Directive (RED), Article 3(3)(d), (e) and (f): applicable since 1 August 2025 to internet-connected radio equipment, with EN 18031 as the harmonised standard. EN 18031-1 groups the update requirements under the secure update mechanism (SUM): whether a mechanism is required, that it installs only genuine software, and whether updates are automated.
IEC 62443-4-2: for industrial components, EDR 3.10 (support for updates), with the enhancement RE(1) for update authenticity and integrity.
UNECE R156: for vehicles, a Software Update Management System (SUMS) is a condition of type approval. ISO 24089 describes how to implement it.
US FD&C Act, Section 524B: for medical devices that include software and can connect to the internet ("cyber devices"), required in premarket submissions since 2023. Manufacturers must have a plan to monitor and address postmarket vulnerabilities, and provide updates and patches on a regular cycle for known unacceptable vulnerabilities and promptly for critical ones.
Outside medical devices, US requirements are sector-specific or voluntary, such as the NIST IR 8259A device capability baseline, which federal procurement builds on, and the FCC's voluntary Cyber Trust Mark label.
A typical mistake is designing the update path against the checklist of one framework. Products sold into several markets often fall under more than one, and each assessment then reopens the design. We recommend designing from the risk assessment and documenting how the result covers each applicable framework.
None of these regulations names a bootloader, a partition layout or a signature algorithm. They require that a product can be updated securely for as long as it is supported, and leave the engineering to the manufacturer. That engineering is the subject of this article: which properties an update mechanism needs, how the risk assessment decides which of them a product requires, and how to implement them on microcontrollers and embedded Linux while avoiding the failure modes that are hard to correct after shipping, such as insufficient flash for a second image, units left unbootable by an interrupted update, or signing keys that cannot be rotated.
2. From risk assessment to update design
The properties commonly discussed for a secure update mechanism are authenticity, integrity, anti-rollback, robustness against failed updates, and confidentiality of the image. Authenticity and integrity apply to practically every connected product. The others depend on the product, its environment and its threat model, and the risk assessment is where they are decided and justified.
This section covers which properties are a baseline, which are design decisions, and what each decision costs in hardware and process.
Authenticity and integrity
An interface that writes firmware without verifying it lets anyone with access to it run their own code on the device. A connected product therefore verifies, before installing or running an image, that it comes from the manufacturer and was not modified. EN 18031-1 (SUM-2) and IEC 62443-4-2 EDR 3.10 RE(1) address this explicitly; under the CRA, it falls under the integrity requirement in Annex I, Part I, point (2)(f).
The usual approach is an asymmetric signature, for example ECDSA P-256 or Ed25519, over the image or a manifest. The private key stays with the manufacturer, and the device holds the public key in the bootloader, OTP memory or a secure element. The risk assessment decides how deep this goes: where that key is anchored, whether verification also runs at every boot, and who can use the signing key.
A typical mistake is relying on TLS alone. TLS protects the download, not the image. An image modified on the build or update server arrives over TLS without an error, and local paths such as USB, UART or BLE often have no TLS at all. We recommend signing every image at build time and verifying it on the device, on every path that writes firmware, including recovery and factory modes.
From our projects. On an OpenWrt-based high-power cellular router, we ported the firmware off a customised fork onto the upstream release, keeping the custom logic in separate packages so security fixes keep flowing, and integrated secure boot so the device verifies the software it runs.
Decisions the risk assessment drives
Beyond authenticity and integrity, the remaining properties address specific risks. Each risk has several valid answers: accepting it, or one of several measures that differ in the protection they give and what they cost. Which one is appropriate depends on how the risk is rated in the risk assessment, by impact, likelihood and the number of units affected, weighed against the cost of the measure.
The table lists the options for each risk, from accepting it to the strongest measures, with their main cost in brackets. It does not indicate which one to choose.
Risk | Options, from accepting the risk to the strongest measure |
|---|---|
Modified firmware is run after a direct write to flash, for example through a debug port or by replacing an external flash chip | • Accept the risk |
An older, validly signed image with known vulnerabilities is installed | • Accept the risk |
The contents of the image are disclosed, exposing proprietary code or helping an attacker analyse it | • Accept the risk, with plaintext images |
A unit is left unusable by an interrupted or faulty update | • Accept the risk, with recovery on site or replacement |
Units in the field stay vulnerable because updates are not installed | • Manual updates by the user or a service technician (depends on someone acting) |
The signing key is compromised, or the signature algorithm weakens, during the product's lifetime | • Accept the risk, with a single fixed key |
As an example, consider a connected thermostat whose worst-case local compromise exposes the temperature history of one home. Modifying its firmware through the debug port requires opening a unit inside the user's home and affects only that unit, so that risk can reasonably be accepted, or addressed by locking the debug ports. A remote attack through the update path affects the whole fleet, which is why authenticated updates stay justified for the same product. An industrial controller or a payment terminal goes through the same rows and, with a higher impact rating, usually ends up on stronger options.
Bandwidth and energy are a separate input. They decide between full images and delta updates, and the choice of transport, such as cellular, BLE, LoRaWAN or a wired connection.
Several of these options are hard to add after the hardware is fixed. Flash size and the availability of OTP fuses, RPMB or a secure element are component-selection decisions. A product that ships without room for a second image is limited to update schemes that fit in the remaining space.
A typical mistake is taking a reference configuration from a vendor SDK or a tutorial as is, either with every option enabled or with the minimum needed to boot. Neither is based on the product's risks. We recommend recording, for each row, the rated risk, the option chosen and the reason, including when the risk is accepted, so that the decision can be reviewed when the product or its environment changes.
From our projects. On a cellular anti-theft bike tracker, we set the installation policy from the device's state, updating the modem and BLE module without user involvement at three priority levels: routine updates only while charging, critical ones immediately on activated devices, and mandatory ones immediately on every device, including units not yet activated. External flash held the latest firmware for both modules, so either could be rolled back.
Dependency on secure boot
Verification in the update client protects the update path. It does not protect against an attacker who can write to flash directly, for example through an open debug port or by replacing an external flash chip. The device then runs whatever is in flash, unless the software is checked before it runs. That check is what secure boot means in its broadest sense: the device verifies the authenticity and integrity of the software it boots. How it does that has no single technical answer. It is a design decision with its own trade-offs, and which one a product needs follows from the first row of the table above.
On microcontrollers, a single bootloader such as MCUboot often handles both installation and boot-time verification. On embedded Linux, the two are usually split between the update client, such as RAUC, SWUpdate or Mender, and the boot chain, for example ROM code, U-Boot, the kernel and root filesystem verification. Secure boot will be covered in a separate article.
A typical mistake is verifying images only at install time, in the update client, while the bootloader runs any image it finds, on a product whose risk assessment rated direct flash writes as a risk to mitigate. We recommend designing the update mechanism and the boot chain together, with consistent assumptions about what an attacker can reach and which keys each stage trusts.
From our projects. On an automotive Android system built on the NXP i.MX8M Plus, we integrated the update mechanism with the boot chain: signed A/B OTA over HTTPS through the AOSP Update Engine, secure boot anchored in NXP AHAB (signed SPL, U-Boot and kernel), Android Verified Boot 2.0 for the Android partitions, and key storage protected by ARM TrustZone.
3. Mapping the properties to an implementation
The previous section derived, from the risk assessment, which properties a product's update mechanism needs. Implementing them means deciding, for each property, which layer provides it: the server or tool that builds and distributes images, the update client on the device that downloads and installs them, or the bootloader and boot chain that decide what runs. It also means deciding where that layer comes from: an existing stack, an integration of existing components, or the manufacturer's own code. Few teams write every layer themselves, so in practice the design starts from an existing stack, and what that stack provides decides what is left to build.
The subsections below take the properties in turn. Each starts with the usual mechanism and the layer that has to own it, then names stacks that provide it, and ends with what is left when the chosen stack does not. A table at the end summarises how four common open-source stacks cover each property: MCUboot on microcontrollers, and Mender, RAUC and SWUpdate on embedded Linux.
Authenticity and integrity
Every other property assumes that the device runs only images from the manufacturer. Establishing that means authenticating each image, and the common mechanism is a digital signature. The signature is usually asymmetric: it is created at build time with a private key that stays with the manufacturer and verified on the device with the matching public key, so the device holds only a public key and never a secret that could forge an image. A symmetric scheme, where the same key signs and verifies, is possible but places that forging key on every unit, which is why asymmetric signing is the usual choice. What the check covers depends on the layer that performs it.
A check in the bootloader runs before the image is put in place, and can be repeated at every boot, which also catches an image written to flash directly, at the cost of boot time. A bootloader-layer stack such as MCUboot works this way on microcontrollers, for example in Zephyr RTOS projects. A check in the update client runs at install time and protects the update path only: once the image is installed, the boot chain runs whatever is on the partition. Embedded Linux update clients such as Mender, RAUC and SWUpdate verify at install time, each with its own signed format, and leave verification at boot, for example a signed FIT image in U-Boot and dm-verity for the root filesystem, to the boot chain.
When the stack verifies only at install time and the risk assessment called for mitigating direct flash writes, the missing check has to be added in the boot chain. That is usually an integration of existing boot components rather than new code, and will be covered in a separate article.
A typical mistake on Linux is treating the update client's signature check as boot-time verification. We recommend deciding explicitly, from the risk assessment, whether the boot chain also has to verify what it boots.
Robustness against failed updates
A verified image can still fail: the installation can be interrupted, or the new software can boot and not work. The usual mechanism keeps two copies of the software, boots the new one on trial and falls back to the previous one unless the new one is confirmed. It needs two layers to cooperate: the bootloader, which counts boot attempts and selects the copy, and the running software, which confirms the update once it has checked itself.
On microcontrollers, the bootloader can own both copies. MCUboot, with one of its swap strategies, exchanges the two image slots, boots the new image as a test and restores the previous one at the next reset if the application has not confirmed it. The strategy is a build-time choice, and the overwrite-only strategy is simpler to implement, but because it keeps no second copy it cannot revert a failed update. On embedded Linux, the work is usually split: the update client writes to the inactive root filesystem partition and marks it for trial, and the bootloader, for example U-Boot, GRUB or barebox, counts boot attempts and falls back. Mender and RAUC work this way, and SWUpdate supports the same double-copy layout, although it is also commonly used with a single copy and a rescue system.
Where the stack lacks a layer, the gap is usually on the bootloader side: a vendor bootloader with no second slot or no attempt counter. Integrating an existing bootloader is generally cheaper than writing the trial and fallback logic, which has to survive power loss at every step and be tested on the product.
The confirmation step is where the product's health check goes, and the quality of that check decides how many failures the fallback catches. A typical mistake is confirming the image as soon as the application starts. We recommend confirming only after the functions the product needs, such as connectivity to the update server, have been checked.
From our projects. On a marine 5G router running OpenWrt, we built a fail-safe update mechanism using Mender: each new image was verified before installation, and a failed update rolled back automatically to a known-good copy in external flash.
Anti-rollback
A signature proves where an image comes from, not that it is current. An older release with known vulnerabilities is still validly signed, so preventing its installation needs a separate check, against a version or counter that only moves forward. The layer that holds that value decides what the check protects against.
A version check in the update client is the cheapest option. It stops a downgrade through the update path, but anything that writes flash directly bypasses it; SWUpdate, for example, offers it as a minimum-version setting. A security counter checked by the bootloader and stored in protected non-volatile memory also covers direct writes. MCUboot compares a counter in the image with one held by the platform, and leaves reading and updating that counter to a platform interface the integrator implements. A hardware monotonic counter, for example in OTP fuses, eMMC RPMB or a TPM, is the strongest form, and is usually integrated by the manufacturer rather than provided by the update stack.
Not every stack provides a check. Mender, for example, matches an artifact's dependencies by exact value and uses its version field only for inventory reporting, so it does not refuse an older but validly signed release; a version check can be added by the integrator in a state script or a custom Update Module, with the same limit as any check in the update client.
A value that only moves forward also means that a withdrawn release cannot be reinstalled, so reverting a faulty release across a fleet is planned as a new release with a higher version, as a later section describes.
Confidentiality
Signatures and counters control what runs on the device, not who can read the image. Disclosure of the image is nearly always a risk; what the risk assessment decides is whether to mitigate it, and how far. Where it is mitigated, the image is encrypted, and the layer that decrypts it decides where the protection ends. TLS between client and server protects the image in transit only; it stays readable on the update server and in the device's storage.
Encryption at rest is removed either by the bootloader, as the image is put in place, or by the update client during installation. MCUboot takes the first approach: the image stays encrypted in the secondary slot, with its AES key wrapped for the device, although not with every upgrade strategy. RAUC and SWUpdate take the second: RAUC encrypts a bundle for one or more recipient public keys, and SWUpdate decrypts images with a symmetric AES key held on the device. Mender protects the image in transit with TLS; encrypting the stored image is not part of its documented features and is left to the integrator, for example through a custom Update Module.
The key that decrypts the image is on the device, so the protection is only as strong as that key's storage. One key for the whole fleet means one extracted key exposes every image; keys per device or per batch need provisioning at that granularity. When the stack has no decryption, it is usually added in the update client, as an installation step that decrypts before writing.
A typical mistake is encrypting every image with one key built into every unit and treating that as protection of the code, when a single extracted unit exposes every release. We recommend choosing the key granularity from the rating of the disclosure risk, and accepting the risk explicitly when that rating does not justify the cost.
Installation policy
Once an image can be verified and installed safely, the remaining question is when it is installed and who decides. Policy belongs to whichever layer sees the fleet. On a single device, the application decides when to download, when to reset and whether to ask the user; across a fleet, a server decides which devices receive which release, and when.
Some stacks include that server. Mender combines a deployment server with its client: deployments target device groups, and the device side can pause an update before installing or rebooting, which fits products that need user approval or maintenance windows. Others provide the device side and connect to a separate server, commonly Eclipse hawkBit, through a client such as SWUpdate's built-in suricatta mode or the separate rauc-hawkbit-updater for RAUC. A bootloader such as MCUboot has no policy of its own; on microcontrollers the application decides when an update starts, for example through a management protocol such as MCUmgr.
Where the stack has no server, the options are running an open-source server, using a hosted service, or building the minimum the product needs. How releases are rolled out across a fleet is covered in a later section.
A typical mistake is choosing a device-side stack without deciding where the policy will live, and finding late in the project that the product needs fleet management. We recommend deciding the server layer together with the client, from how the product is operated: by an end user, by an operator, or not at all.
How four stacks cover the properties
The table summarises the subsections above for four common open-source stacks. "Built-in" means the stack provides the property, with a short note on how; "integrator-supplied" means the stack leaves it to the manufacturer, who integrates or writes it.
Property | MCUboot | Mender | RAUC | SWUpdate |
|---|---|---|---|---|
Authenticity and integrity | Built-in: bootloader signature check (RSA, ECDSA P-256, Ed25519), optionally at every boot | Built-in: signed Artifact, checked by the client at install (ECDSA P-256, RSA); boot-time check integrator-supplied | Built-in: CMS-signed bundle, checked at install; boot-time check integrator-supplied | Built-in: signed sw-description (RSA, CMS or GPG), checked at install; boot-time check integrator-supplied |
Robustness against failed updates | Built-in: slot swap, test boot and revert (swap strategies only) | Built-in: A/B root filesystem, U-Boot or GRUB boot count, commit-step state scripts | Built-in: A/B slots, bootloader attempt counter, mark-good after boot | Built-in: double copy, success signalled to the bootloader; also single copy with rescue system |
Anti-rollback | Built-in: hardware security counter (platform implements storage); version check with overwrite-only | Integrator-supplied: state script or Update Module | Built-in (limited): installs only a higher semantic version in polling mode; no general downgrade block | Built-in: minimum-version check in the client |
Confidentiality | Built-in: AES image encryption, key wrapped with ECIES, RSA-OAEP or AES-KW; not with direct-XIP | Integrator-supplied: TLS in transit only | Built-in: AES-256 encrypted bundle, per recipient key | Built-in: AES-CBC image encryption, symmetric key on the device |
Installation policy | Integrator-supplied: application or MCUmgr | Built-in: deployment server, device groups, pause before install or reboot | Integrator-supplied: separate server, for example hawkBit via rauc-hawkbit-updater | Built-in: hawkBit client (suricatta); server separate |
When the stack does not provide a layer
Each subsection above ended with the same question: what is left when the chosen stack does not provide the layer that owns a property. Often it does not: the SDK has no update client, the bootloader has no second slot or rollback counter, or the bootloader comes from the silicon vendor and cannot itself be updated in the field. The missing layer then has to come from somewhere, and each source has its own cost, from the least to the most code to own:
Adopt a framework that integrates the missing layer. The least code to own. The product then depends on the framework's support for the hardware, its licence and its roadmap.
Integrate existing components. For example, porting MCUboot to a vendor SDK, or combining RAUC or SWUpdate with an existing boot setup and a separate server. Each component is maintained upstream, but the integration between them is the manufacturer's own code.
Write the missing layer. Full control, and full ownership: the code is security-critical and has to be maintained, tested and covered by vulnerability handling for the whole support period. Power loss during installation, fallback and signature verification all have to be tested on the product.
A typical mistake is a self-written bootloader that checks a CRC instead of a signature, or that switches between two images without a confirmation step, so an image that boots but does not work is never reverted. Another is a bootloader that cannot be updated, which means a vulnerability in it cannot be fixed in the field.
We recommend treating the update stack as a criterion when the SDK, framework and hardware are selected, not after. The cost of any self-maintained layer should be estimated over the full support period, not only for the first implementation. The general case for weighing a framework by its ecosystem is covered in Why the framework and its ecosystem matter, in our Cyber Resilience Act article.
What the update stack does not decide
Whichever stack is used and whatever is built around it, every property above rests on authenticating each image, and the update stack does not manage the keys behind it. Where those keys are stored, who can use them, how they are rotated and how long they must remain valid is the subject of the next section.
4. Signing keys and long-term support
Every feature in the previous sections rests on one check: the device accepts an image only if it is signed with the manufacturer's key. That makes the private key the most valuable secret in the update mechanism. If it is stolen, an attacker can sign firmware that every device in the fleet will install. If it is lost, a device that was not built to switch keys cannot be updated securely again.
This section covers where signing keys are kept, how they are replaced when they have to be, and what has to stay available for as long as the product is supported. The options for replacing a key, such as spare key slots or a root key that signs key updates, were listed in the second section and, on most products, have to be built into the device before it ships. This section is about operating them.
Where signing keys live and who can use them
Whoever can use the private key can sign firmware the fleet will install, so where the key is kept decides who that is. How well it has to be protected is rated in the risk assessment like any other risk, and the options range from no extra cost to dedicated hardware. From the least to the most complex:
Key stored as a secret in the CI system. The key is a protected variable, available only to release jobs on protected branches or tags. It costs nothing and keeps the key out of the repository and off developers' machines. It does not protect the key from anyone who can change the release pipeline or administer the CI system, since the key exists as a file while the job runs.
Cloud key management service (KMS). The key is generated in the service and cannot be exported; the pipeline sends a hash and receives a signature. The cost is low, typically a small monthly fee per key and per signing operation, and every use is logged. Someone with access to the pipeline can still have an image signed, but cannot take the key away.
Hardware security module (HSM), on site or as a cloud HSM, accessed through an interface such as PKCS#11. The same model as a KMS, with dedicated hardware and certifications such as FIPS 140-3, at a considerably higher cost. It is usually justified by the impact rating, or required by a customer or a sector standard.
Offline root key with online signing keys. The root key is kept offline and used only to sign or revoke the signing keys that the pipeline uses, which can be stored with any of the options above. A compromised signing key can then be replaced without touching the root of trust on the device. The cost is mainly process: a documented procedure for each use of the root key.
Independently of where the key is stored, two practices cost little. Production images are signed only by the release pipeline, from a tagged release and after an approval step. And separate keys for development and production builds, and per product line, limit what a single key can install: a development image cannot be installed on a production unit, and a key leaked from one product does not affect the others.
A typical mistake is a single key file shared across products and test builds, kept in the repository or copied to developers' laptops. Anyone with access to it, and anyone who later gets access to a backup or a disk image, can sign production firmware. We recommend starting at least with a key held only by the CI system and used only by release jobs, and moving to a KMS or HSM when the impact rating or a customer requirement justifies it.
Key rotation and revocation in practice
Even a well-protected key may have to be replaced during the product's lifetime: it can leak, the account or service that holds it can be lost, or its algorithm can weaken. A device can switch to a new key only if it was built for it, with either a second public key already on the device or a root key that can authorise a new signing key. Without either, replacing the key means replacing the bootloader, which on most products is not possible in the field.
With that in place, a planned rotation follows a fixed order:
Release an update, signed with the current key, that makes the device trust the new key.
Wait until the fleet has installed it, and track the units that have not.
Sign new releases with the new key.
Revoke the old key on the devices, so that images signed with it are no longer accepted.
After a compromise, the same steps apply, but the attacker also holds the old key until step 4. The time this takes across the fleet is the exposure window, and it is set by how fast updates are installed, which is a reason in itself for automatic installation.
Units that are offline during a rotation, such as stock in a warehouse or devices stored by customers, come back still trusting only the old key. If the old key has been revoked on the server side or destroyed, they cannot be updated. A bridge release signed with the old key, which installs the new key, has to stay available for them, or the old key has to be kept, securely, until they have been reached.
A typical mistake is finding out that rotation is impossible only after a key is compromised. We recommend rehearsing a rotation on test units before the first production release, including a unit that has been offline for the whole process.
Planning for the support period
Storage and rotation protect the key only as long as they keep working, and a connected product stays in the field for years after the team that set them up has moved on. How many years is increasingly not only the manufacturer's decision. Connected devices that stopped receiving updates, or never received them, have been behind large incidents, such as the Mirai botnet in 2016, built from cameras and routers that shipped with default credentials. Regulators have responded by requiring manufacturers to keep supporting their products for a minimum time: in the EU, the CRA sets the support period to at least five years, unless the product is expected to be in use for less. Many industrial products are used well beyond that in any case. Over that period, three things have to keep working:
The keys and the signing infrastructure. The CI or KMS account, its access rights and its backups have to outlive staff changes, supplier changes and company restructuring. A key that only one person can access is a single point of failure.
The signature algorithm. The RSA and elliptic-curve signatures used today would be broken by a sufficiently large quantum computer. Because nobody can say when one will exist, NIST, in the final IR 8547 (October 2026), deprecates these algorithms after 2030 and disallows them after 2035. A product expected in the field beyond the early 2030s therefore needs post-quantum signatures, or the ability to switch to them, in its design.
The build. A security fix for a five-year-old release requires rebuilding it: the same toolchain, SDK, dependencies and signing setup. If the build cannot be reproduced, the fix has to be ported to a newer release, which may not fit the older hardware.
A typical mistake is treating key management as a one-time setup done when the first image is signed. We recommend documenting the key management process, the rotation procedure and the build environment as part of the product's technical documentation, and reviewing them at each major release.
With the keys and the build kept available, the manufacturer can produce a signed release at any point in the support period. How that release reaches the fleet, and what happens when it fails on some units, is the subject of the next section.
5. Rolling out to a fleet
The previous section ensured that the manufacturer can produce a signed release at any point in the support period. A valid signature, however, says nothing about whether the release works. An image that passed every test in the lab is installed on thousands of units that differ in hardware revision, configuration and network, and a defect that only shows on some of them is still signed. Pushed to the whole fleet at once, it fails everywhere at once, and the robust installation from the previous sections only helps where the failure is caught by the health check.
This section covers how a release reaches the fleet: keeping security fixes small enough to roll out quickly, rolling out in stages while watching the results, handling units that fail or are offline, and what the process leaves behind as a record. The device side, installation, fallback and anti-rollback, was covered in the third section and is assumed here. Rolling out is the job of the server or whatever process replaces it.
Separating security and feature releases
A rollout is only as fast as the confidence in the release allows. A security fix shipped together with new features carries their risk as well: it needs the same testing, it is rolled out with the same caution, and if a feature fails, the fix is withdrawn with it. The exposure window from the previous section is then set by the slowest feature in the release, not by the fix.
The usual answer is to maintain a release branch per supported version, where security fixes are applied on their own and released as small patch versions, while features go into the next minor or major release. A patch release changes little, so it can be tested and rolled out quickly: a small change is a small risk.
The cost is maintaining several branches in parallel for the support period, which is the build reproducibility from the previous section applied to more than one version. Products with a short support period or a small fleet often keep a single branch and accept slower security releases.
A typical mistake is fixing a vulnerability only on the main branch and shipping it with the next feature release, months later. We recommend deciding at the first release how many versions are supported in parallel, and how a security fix reaches each of them.
Staged rollouts and monitoring
A defect that escapes testing is found by the first units that install the release. If those are the whole fleet, the defect is found everywhere at the same time, and for a defect that the device-side health check does not cover, which depending on what that check tests might be a bug that drains the battery, breaks a sensor or prevents the next update, nothing reverts it automatically. A staged rollout limits how many units are affected before someone, or something, notices.
The options differ in how much of this is automated. From the least to the most costly:
Manual stages. The release goes first to internal test units, then to a small group of field units, for example a friendly customer or the manufacturer's own installations, and only then to the rest. It needs nothing beyond the ability to target a group of devices, and protects against defects that show within the observation time on the first groups. It depends on someone checking the results before each step.
Phased rollout on the server. The deployment is split into phases, each a share of the target devices, with a delay between them, for example 5 %, then 15 %, then the rest after two days. Mender offers this in its Enterprise plan; a device installs when it next polls the server and its phase is active. It removes the manual steps, but the rollout continues by itself unless someone stops it.
Automatic stop on failure. A script or service reads the deployment results and the devices' health data, and aborts the deployment when failures exceed a threshold. Mender, for example, reports the status of each device in a deployment and exposes an abort operation through its management API, so the stop can be built on top of it; Mender documents no automatic abort-on-failure threshold. It protects against a rollout that continues unattended, at the cost of defining what "failure" means and maintaining the integration.
Each option relies on knowing how the release behaves. The deployment status alone, success or failure per device, only reports what the device-side health check detects. Defects outside it show up in health data that the product reports anyway or has to start reporting: reboots and crash counts, connectivity, battery, and whether the device still checks in for updates. The last is the most important, since a release that breaks the update client cannot be fixed by the next one.
A typical mistake is a first phase so small or so short that it cannot show the defects it is meant to catch, for example a handful of units observed for an hour when the failure appears after a day of operation. We recommend sizing the first phase and its duration from the product's failure modes, and including units of every hardware variant in it.
From our projects. For a portable Linux media projector, we built the OTA system using Mender around a staged deployment pipeline: each release moved through development, then test, then production device groups, and we validated fixes on a large device group over several days before mass production.
Targeting a heterogeneous fleet
A fleet is rarely uniform. Over its life, a product accumulates hardware revisions, components replaced because of supply problems, regional variants with different radios, and units that are several releases behind. An image built for one revision can fail on another, and a release tested from the previous version may not install from a much older one. Rolling out to "all devices" ignores these differences, so the rollout has to know which unit is which.
The first protection is in the image itself: a Mender Artifact declares the device types it is compatible with, and the server does not deploy it to other devices; MCUboot images can carry a dependency on the version of another image. That prevents installing an image built for a different hardware. Beyond that, devices are grouped by the attributes the risk assessment and the test plan care about, typically hardware revision, current firmware version, region or customer. Mender supports static groups, a fixed list of devices, in all plans, and dynamic groups, a filter on the attributes the device reports, in its Enterprise plan.
Groups also decide the order of the rollout: a release can go first to the hardware variant it was tested on most, and last to the oldest units or the ones that are hardest to reach if something goes wrong.
A typical mistake is a device that does not report its hardware revision, so units with a replaced component cannot be told apart once a release fails on them. We recommend having each device report, from its first release, the attributes needed to target it: hardware revision, bootloader version and installed firmware version at least.
Bandwidth, scheduling and offline devices
Releasing an update to the whole fleet at the same moment also loads the infrastructure at that moment: the server, the content delivery network, and for cellular devices, the data plan, which is often billed per megabyte. And at any given time, part of the fleet is offline: switched off, out of coverage, or in stock. A rollout has to reach them later, without the operator tracking each one.
Three measures address this, from the cheapest:
Spreading the load. Devices that poll the server at an interval, and phases, already spread the downloads over time. This costs nothing beyond configuration.
Retrying and resuming. A download interrupted by a lost connection resumes rather than restarting, and a device that failed can try again. Mender offers automatic retries of failed deployments, counted per device, in its Professional and Enterprise plans. Retries hide transient failures, so they have to be counted separately from devices that fail repeatedly.
Delta updates. Only the difference between the installed and the new image is sent, which reduces data use and install time. A delta applies only to the exact version it was generated from, so a fleet on several versions needs several deltas, or a full image as fallback. Mender provides delta updates for root filesystems in its Enterprise plan, and partly in Professional; the Enterprise server can generate the deltas itself and falls back to the full Artifact where none is available.
Offline devices are the reason a deployment cannot simply end. In Mender, a deployment to a static group finishes once every device in it has reported success or failure, so it stays open until the last offline unit comes back; a deployment to a dynamic group stays open and also picks up devices that join the group later. Either way, a unit that returns after months receives the release then, together with the key rotation from the previous section if it missed one.
A typical mistake is sizing the data plan and the server for the steady state and discovering the cost of a full-image release over cellular after the first one. We recommend estimating the cost of a full-fleet release, in data and in time, before choosing between full images and deltas.
From our projects. On the same automotive Android system, we produced both full and incremental update packages, signed and delivered over HTTPS, so a unit one release behind could take the smaller package.
Handling failed updates in the field
Even with stages and monitoring, some releases fail on some units. On each device, the fallback from the third section reverts what the health check catches. Above that, the operator has two actions: stopping the deployment, and bringing the units that already installed the release back to a working version.
Stopping is the simpler one. Aborting a deployment in Mender stops it for devices that have not yet installed it; units that already did keep the new version.
Bringing those units back is where fleet-level rollback meets the anti-rollback from the previous sections. With anti-rollback, a device refuses an older image by design, so the previous release cannot simply be redeployed. The fix is to roll forward: rebuild the previous, working code as a new release with a higher version and, on MCUboot, a security counter no lower than the one already installed, then deploy it like any other release. Without anti-rollback, redeploying the previous release works, which is also why the risk assessment has to rate the risk of an attacker doing the same. Either way, units that fail before they can check in again, for example after a defect in the update client, need a local recovery path or a site visit.
A typical mistake is incrementing the security counter in every release. Each increment makes all earlier releases permanently uninstallable, including the last known good one, and uses up one of the limited increments of a hardware counter. We recommend incrementing it only when a release fixes a vulnerability that older releases must not be able to reintroduce, and rehearsing a roll-forward on test units before the first production release.
From our projects. For a fleet of marine routers, we ran the cloud side on a Mender server in Kubernetes on AWS: the firmware pipeline built and tested each release, deployed it to Mender and rolled it out to internal devices first, and field devices rolled back automatically when an update failed.
What the update process leaves as evidence
When a vulnerability is reported, the first questions are which units run the affected version, and when the fix reached them. Answering them needs records that the update process produces anyway, if they are kept: which release went to which devices, when each device installed it or failed, and why. The same records show a customer, an auditor or a market surveillance authority that updates are actually delivered, not only available.
Mender keeps a deployment history per device and per deployment, with the status of each device and the log a device uploads when its update fails; audit logs of user actions are available in its Enterprise plan. Without such a server, the equivalent is a log of what the device reports after each update, kept for the support period.
A typical mistake is keeping deployment data only as long as the server's default retention, so the history of a release is gone when it is needed years later. We recommend defining which update records are kept, and for how long, as part of the vulnerability handling process.
From our projects. In the same projector's system, we built a fleet view showing each device's status, location and installed software: the record needed to answer which units run which release.
Staged rollouts, targeting, retries and records each prevent a specific failure, and each is worth checking before the first production release.
6. Conclusion
The previous section brought a signed release from the manufacturer to the units in the field. Taken together, the sections show that the update mechanism is not a feature added at the end of development: its shape follows from the risk assessment. Authenticity and integrity are the baseline, and every other property is a decision, set by how each risk is rated and what the measure costs. The article followed those decisions from the properties a product needs, through how existing stacks such as MCUboot and Mender provide them, to the keys behind the signature and the rollout that brings each release to the fleet.
A few points cut across the sections rather than belonging to any one of them. To an attacker, the update client, the boot chain, the signing keys and the rollout are a single chain entered at its weakest stage, so they are best designed together, against one statement of what an attacker can reach and which keys each stage trusts. Most of the design can change in a later release, but the decisions fixed when the hardware is chosen or the first boot stage is written cannot: flash for a second or recovery image, the fuses or secure element that anti-rollback and key anchoring need, a bootloader that can be updated or taught a new key, and headroom to change the signature algorithm. Each has to be settled explicitly in the risk assessment, including the choice to accept a risk, before the hardware is frozen. What the design leaves behind is not a set of features but a record of those decisions, kept for the support period: it is what lets the design be reviewed when the product or its threats change, and what shows an auditor that each protection, or its absence, was deliberate. Finally, the keys and the reproducible build have to outlive the team that set them up, owned by a role rather than a person and exercised rather than only documented, and the whole mechanism has to be re-tested on production hardware, since the sample keys and unlocked debug access of an evaluation board are not what ships.
Closing
The first section started from vulnerabilities that are found after a product ships, in its own code or in the libraries and stacks it integrates. Every decision in this article serves one purpose: being able to fix them in the field, securely, for as long as the product is supported.
FAQ
Is TLS enough to secure OTA firmware updates?
No. TLS protects the download, not the image. An image modified on the build or update server arrives over TLS without an error, and local paths such as USB, UART or BLE often have no TLS at all. Sign every image at build time and verify the signature on the device, on every path that writes firmware, including recovery and factory modes.
What is the difference between MCUboot and Mender, RAUC or SWUpdate?
MCUboot is a bootloader for microcontrollers: it verifies the signature, can repeat the check at every boot, swaps image slots and reverts an unconfirmed update. Mender, RAUC and SWUpdate are update clients for embedded Linux: they verify a signed update at install time and write it to an inactive partition, while boot-time verification is left to the boot chain. Mender also includes a deployment server; RAUC and SWUpdate usually connect to one such as Eclipse hawkBit.
Do we need A/B partitions for safe updates?
Not necessarily. An update that leaves a unit unusable ranks high in impact on almost any product, so the risk assessment drives the complexity of the recovery mechanism rather than whether to have one. Where local recovery is acceptable, a lighter mechanism such as serial recovery mode or a recovery image can be enough. Where the unit has to recover on its own, two image slots or A/B partitions with a confirmation step and automatic fallback are the strongest option, but they roughly double the flash used by the application or root filesystem. The choice has to be made before the hardware is fixed.
How do you roll back a bad release when anti-rollback is enabled?
You roll forward. With anti-rollback, a device refuses an older image by design, so the last working code is rebuilt as a new release with a higher version and, on MCUboot, a security counter no lower than the one installed, then deployed like any other release. Increment the security counter only for releases that fix a vulnerability older releases must not reintroduce.
Do firmware signing keys need post-quantum algorithms?
If the product is expected to stay in the field beyond the early 2030s, plan for it. NIST IR 8547 deprecates today's RSA and elliptic-curve signatures after 2030 and disallows them after 2035. Either use post-quantum signatures such as LMS or ML-DSA from the start, or design the bootloader and update format so the signature algorithm can be changed during the product's lifetime.
Designing the update mechanism for a connected product?
iomico works on these update mechanism decisions day to day, across microcontroller and embedded Linux products, from the risk assessment through to signing keys and fleet rollout. If you are facing them on a product of your own and would like help, or a review of an existing design, get in touch.
Sources
MCUboot documentation — Design, imgtool, Encrypted images, Building and using MCUboot with Zephyr
Mender documentation — Artifact, Sign and verify, Operating system updates overview, Integration checklist, Security, Device-side API, Device group, Deployment, Delta update, Abort deployment API, Polling intervals
Mender plans and pricing — https://mender.io/plans/pricing
RAUC documentation — Basics, Reference, Integration, Advanced
rauc-hawkbit-updater — https://rauc-hawkbit-updater.readthedocs.io/en/latest/
SWUpdate documentation — Signed images, Overview, Command line options, Encrypted images, Asymmetric encrypted images, Suricatta
U-Boot — FIT signatures
dm-verity — https://docs.kernel.org/admin-guide/device-mapper/verity.html
Cyber Resilience Act — Regulation (EU) 2024/2847, Article 13
NIST IR 8547 (final, October 2026) — Transition to Post-Quantum Cryptography Standards
CISA Alert TA16-288A — Heightened DDoS Threat Posed by Mirai and Other Botnets
Radio Equipment Directive (RED) — Directive 2014/53/EU, Delegated Regulation (EU) 2022/30, Implementing Decision (EU) 2025/138 (EN 18031 references)
IEC 62443-4-2 — Technical security requirements for IACS components
UN Regulation No. 156 and ISO 24089 — UN R156 (software update and SUMS), ISO 24089:2023
US FD&C Act, Section 524B — FDA, Cybersecurity in Medical Devices
NIST IR 8259A — IoT Device Cybersecurity Capability Core Baseline
FCC U.S. Cyber Trust Mark — fcc.gov/cybertrustmark
