The device shipped on schedule. It passed the security review: firmware was signed, secure boot was enabled, and an SBOM sat in the repository next to the release tag.

Weeks later a CVE landed against a compression library that the product bundled somewhere in its dependency tree. The compliance team asked a simple question — which field units contain the vulnerable version, and can we show the shipped binary matches the SBOM we filed? Nobody could answer it quickly.

Two engineers checked out the release tag and rebuilt it. They got two different binaries. The signing key, it turned out, had lived on a shared CI server that several contractors could reach. And the SBOM had been generated from the source manifest, not from the image that was actually flashed at the factory.

None of that was a broken cipher. The signatures verified and secure boot worked. What had never been engineered was the layer where the firmware is built, signed, and provisioned — and that is where the supply chain was weak.

Wrong Assumption

Why It Fails

Non-reproducible builds break provenance. If two clean checkouts of the same commit produce different binaries, there is no way to show the released image came from the reviewed source. Timestamps, absolute build paths, unpinned toolchain versions, and network-fetched dependencies all leak nondeterminism into the output. Reproducible containerized builds with pinned toolchains are the precondition for provenance — without them an SBOM describes an image nobody can reconstruct.

The SBOM describes source, not the shipped image. Many SBOMs are generated from a package manifest or a source scan, so they list what the build was supposed to include. The shipped image can diverge: statically linked transitive dependencies, vendored code copied into the tree, or a build flag that pulls a different library version. In SPDX or CycloneDX form the document looks complete, but it is answering a different question than the auditor is asking.

Signing keys leak on the build server or the line. A private signing key in a CI secret readable by every job, or handed to a contract manufacturer without hardware protection, is the highest-value target in the chain. Once it leaks, downstream controls collapse — an attacker signs firmware the device will happily accept. Holding the signing key in an HSM, so it is never exportable, is what keeps the chain of trust from having an unprotected core.

Vulnerability handling has no traceability. CRA requires manufacturers to handle vulnerabilities across the support period. In practice that means being able to answer, fast, which product releases — and ideally which field units — contain a newly disclosed CVE. Unit-level traceability is not spelled out in the law, but without at least a CVE-to-component-to-release link that answer becomes manual archaeology measured in days, not the hours a coordinated disclosure allows.

Secure boot without a real root of trust. Secure boot is only as strong as where its verification key is anchored. A public key or its hash belongs in immutable or one-time-programmable storage — boot ROM, eFuse, or OTP; TPMs and secure elements protect device identity and key material, and TrustZone can isolate the verification logic, depending on the SoC. Anchoring that first key in immutable hardware is what makes the first link resistant to software-level modification.

In practice these rarely arrive one at a time. A CVE disclosure exposes the traceability gap; the traceability gap forces a rebuild; the rebuild is not reproducible; the non-reproducible build cannot be tied to the filed SBOM. Stacked together, they turn a routine patch into a compliance scramble.

Hidden System Complexity

source repo → open-source & vendor dependencies → toolchain / compiler → CI build → SBOM generation → firmware signing (key custody) → factory provisioning / flashing → secure boot (device) → OTA update → field fleet → CVE disclosure → impact analysis → patch → rebuild & sign → staged OTA rollout → rollout monitoring

A problem seen at the field level usually originates several stages up. A CVE against a transitive dependency is invisible until someone maps it back through the build to a component version, then forward to the firmware releases and the units that carry it. Break any link — an SBOM that omits transitive dependencies, a build that cannot be reproduced, a provisioning step with no record of which key signed which batch — and the whole trace goes cold.

The manufacturing edge adds a second axis. Provisioning, key injection, and firmware flashing happen on a line the product team often does not control directly. That is the secure device provisioning and factory key injection layer: every unit leaves the line with the right identity, the right firmware, and a record of both.

Failure Patterns

Scenario 1. A connected industrial controller passes its security review with signed firmware and secure boot. During CRA readiness prep, the team cannot reproduce the released binary from the tagged commit — the CI image pulled an unpinned cross-compiler that has since moved a minor version. The filed SBOM no longer maps to any binary the team can rebuild, so the provenance claim is hard to defend before a single line of firmware is inspected.

Scenario 2. A medical-adjacent IoT gateway bundles a TLS library statically. A high-severity CVE is published against that library. The SBOM, generated from the top-level package manifest, never listed it as a transitive dependency, so automated CVE scanning reports the product as unaffected. The exposure is real; the paperwork says otherwise, and it surfaces only when a customer's own scanner flags the shipped binary.

Scenario 3. An OEM outsources flashing to a contract manufacturer and ships the production signing key as a file so the line can sign units. Months later, firmware signed with that key but never released by the OEM appears on grey-market devices. The devices accept it because secure boot is doing exactly what it was told: trust anything this key signs. The weak point was key handling on the line.

Embedded Security Engineering

Embedded supply-chain weaknesses — non-reproducible builds, source-only SBOMs, exposed signing keys, missing CVE traceability — are process and provenance problems, not cryptography problems. Closing them takes reproducible build pipelines, build-bound SBOMs, hardware-backed key custody, and a vulnerability-handling trail wired into CI/CD, not another penetration test on the finished device. Promwad engineers secure boot and OTA flows, SBOM-enabled release pipelines, controlled embedded build environments, device identity, and secure provisioning for connected products, aligned with CRA-readiness practices.

Explore Embedded Security Engineering →

Embedded Security Platforms and Technology Ecosystem

Solution Approach

Step 1: Make the build reproducible and bind the SBOM to it. Pin the exact compiler, SDK, and dependency versions in a containerized CI image, strip build-path and timestamp nondeterminism, and generate the SBOM during that controlled build from the resolved dependency graph and package metadata — then validate it against the final firmware image. A build that produces different binaries on different machines has no provenance, and a version-controlled, reproducible toolchain is where that gap closes before any SBOM tooling is bolted on top.

Step 2: Keep signing keys in a server-side HSM and provision identity at the line. Hold the private signing key in an HSM where it cannot be exported, so firmware is signed without the raw key ever touching a build server or CI secret. At the factory, provisioning injects the public verification key and a per-device identity into each unit's secure element, and logs every batch against the firmware version and key used. On outsourced lines this secure-element provisioning discipline is a requirement, not a contractual footnote.

Step 3: Wire CVE-to-fleet traceability into the pipeline. Store each SBOM against its firmware release and each release against the units provisioned with it, then run continuous dependency scanning against CVE feeds so a new disclosure resolves to an affected-component, affected-release, and (where records allow) affected-unit list. This does not add a CRA requirement; it makes CRA vulnerability handling workable and cuts response time from days to hours.

A CVE that takes days to map to a fleet is a traceability gap before it is anything else. The pipeline sets how fast you can answer "who is affected"; if the SBOM, the release, and the provisioning record are not linked, nothing downstream shortens that answer.

Real Trade-Offs

Reproducible builds remove provenance ambiguity but demand a locked, containerized toolchain and disciplined dependency pinning — real setup cost, and friction every time a dependency has to move. On regulated products that cost is recovered at the first audit and the first CVE response.

A hardware secure element or TPM anchors the root of trust and protects keys, but adds BOM cost, provisioning complexity, and a firmware integration path. For high-volume consumer devices a well-managed OTP-fused key may be enough; for industrial and networking products the secure element usually earns its place.

A build-bound SBOM validated against the shipped image is far more useful than a source-manifest SBOM, but it has to be regenerated and re-validated on every build, which only works once the build is reproducible. Bolting SBOM tooling onto a non-reproducible pipeline produces a document that looks compliant and proves little.

Where the product also carries a functional-safety case — IEC 61508 or, for vehicles, ASPICE and ISO 26262 — the supply-chain provenance trail and the safety assurance case can reuse the same configuration-management and release-traceability backbone, but the safety and cybersecurity artifacts remain distinct evidence with distinct goals and should not be treated as one deliverable.

Qualifying Symptoms

• Two clean checkouts of the same commit produce different firmware binaries.
• The SBOM comes from a source manifest, so transitive or statically linked dependencies are missing from it.
• A new CVE takes days of manual work to map to affected products and units.
• The production signing key lives on a build server, in a CI secret, or as a file handed to a contract manufacturer.
• Secure boot is enabled, but the verification key is anchored in rewritable storage rather than OTP, a TPM, or a secure element.
• There is no record tying a firmware release to the specific units flashed with it on the line.
• Dependency scanning reports a component as absent that the shipped binary actually contains.

At this point the work is build-pipeline and provisioning engineering, not another cryptographic review of the finished device. In practice: a reproducible build, a build-bound SBOM validated against the image, hardware-backed key custody, and a CVE-to-fleet traceability trail.

For products on embedded Linux or a custom kernel, the Linux and kernel engineering layer is often where provenance is won or lost — the build system, the BSP, and the dependency graph all originate there. Where the update path is the exposure, a chain-of-trust update pipeline closes the loop from signing to install. And if none of this is wired into how software is verified, QA and test automation will confirm the firmware works without ever confirming it can be traced.

This class of problem shows up most in connected industrial and networking products and IoT gateways where teams built solid on-device cryptography early but treated the build pipeline and provisioning line as integration details — and then hit a CRA deadline or a coordinated CVE disclosure that demanded provenance they had never produced. Automotive and medical teams meet the same engineering problem under their own sector regimes.

Review Your Embedded Supply Chain Security