From Handwritten Makefiles to Hermetic Builds: How Embedded Firmware Pipelines Are Evolving

From Handwritten Makefiles to Hermetic Builds: How Embedded Firmware Pipelines Are Evolving

 

The Makefile was the first serious build system for embedded firmware, and for many teams it remains the current one. A Makefile that invokes arm-none-eabi-gcc, links against a linker script, and produces a .elf binary is an entirely adequate build tool for a single product with one developer. It becomes inadequate when multiple developers work on the same codebase across different host operating systems, when CI servers produce binaries that differ from local developer builds, when a product ships and must be rebuilt five years later to produce an identical binary for a field service audit, or when a security patch must be applied to a production firmware and the team needs confidence that only the patch changed the output.

Each of these requirements exposes the same fundamental limitation: a Makefile describes compilation rules but does not describe the environment that executes those rules. The compiler version, the system headers, the shared library versions available on the build host, the environment variables set by the developer's shell profile — all of these influence the build output and are all outside the Makefile's scope. Two developers running the same Makefile on machines configured differently produce different binaries. A CI server configured differently from the developers' machines produces a binary that passes CI but fails on hardware for reasons that take days to diagnose.

The progression of embedded build systems from Makefiles through CMake, Yocto/Buildroot, and toward Nix/Guix-based hermetic pipelines is a progression of increasing control over the build environment, with each generation solving the problem the previous one could not.

The Makefile Era and Its Limitations

The handwritten Makefile encodes build rules as shell commands. Its dependency tracking is based on file modification timestamps: if the source file is newer than the object file, recompile. This model is simple and transparent, and for decades it was sufficient because the implicit assumption it carries — that the build environment on the developer's machine is consistent — was approximately true when teams were small and machines were manually configured.

The problems that accumulate with Makefiles in commercial embedded development are predictable:

  • Implicit host dependencies: the Makefile invokes gcc or arm-none-eabi-gcc, but does not specify which version. A developer with GCC 12.2 and a CI runner with GCC 13.1 produce different optimization results, different debug symbols, and sometimes different behavior — all from the same source.
  • Missing clean targets: incremental builds based on timestamp comparison fail silently when build artifacts from a previous build configuration contaminate the current one. The famous "works on my machine" failure.
  • No environment pinning: system headers, library paths, and even locale settings affect build output. A developer whose PATH resolves a different Python interpreter than the CI runner produces different code generation results for tools that generate source code.
  • Cross-compilation fragility: embedded cross-compilation requires precisely coordinated toolchain components — compiler, assembler, linker, sysroot, C library headers. Makefiles typically hardcode paths to these components, binding the build to a specific machine layout that must be manually replicated on every new developer machine.

The CMake build system, which displaced many hand-written Makefiles in the embedded C++ world, addressed the rule specification problem: CMake provides a platform-independent build description that generates Makefiles, Ninja build files, or IDE project files for different host environments. West, the meta-build tool used by Zephyr, adds workspace management — tracking the versions of multiple repositories that together constitute a complete build, expressed in a west.yml manifest file. Both are significant improvements over raw Makefiles for build organization and portability, but neither solves the environment problem: CMake and West describe what to build, not the environment in which to build it. The build output still depends on the toolchain and system libraries available on the host.

Yocto and Buildroot — Full-Stack Reproducibility for Embedded Linux

For teams building embedded Linux firmware, Yocto and Buildroot represent the first generation of build systems that attempt to control the entire build environment rather than just the build rules. Both build the complete software stack — bootloader, kernel, root filesystem, application layer — from source, pulling in precisely specified component versions and building everything with a cross-compilation toolchain that is itself built as part of the process.

Yocto's reproducibility story is quantifiable: the Yocto Project documentation cites 99.8 percent binary reproducibility for core-image-minimal. This means that building the same Yocto configuration twice produces binaries where 99.8 percent of the bytes are identical between builds, with the remaining 0.2 percent attributable to timestamps or other non-determinism being actively tracked and reduced. For embedded Linux products, this level of reproducibility means that the firmware binary that ships to production can be recreated from the same source and configuration years later — a requirement for field service and for demonstrating the absence of unauthorized modifications.

The layer model that Yocto uses — where each layer is a set of BitBake recipes, classes, and configuration files that can be version-pinned and composed — provides a structured mechanism for managing the complexity of a full embedded Linux stack. A layer for the BSP, a layer for the communication middleware, a layer for the application software, each versioned independently and composed by the distro configuration. The Yocto Scarthgap LTS release, designated in 2024, commits to four years of maintenance — a support lifecycle explicitly designed for commercial embedded products with long qualification timelines.

Buildroot occupies a different point in the tradeoff space: simpler and faster than Yocto, appropriate for smaller systems with less complexity, but with less flexibility for customization at scale. Buildroot's build is faster primarily because it does not implement the package management, layer composition, and extensible SDK infrastructure that Yocto does. For a product that fits within Buildroot's capabilities, it produces a cleaner build system with less conceptual overhead; for a product that needs the full flexibility of Yocto's recipe system, Buildroot hits limitations that require workarounds.

The following table summarizes the key differences:

Build system

Environment control

Reproducibility

Complexity

Best fit

Makefile

None — depends on host

Non-reproducible

Low

Single developer, simple product

CMake + West

Partial — Kconfig, pinned repos

Host-dependent

Medium

RTOS firmware, multi-repo projects

Buildroot

Full source build

High

Medium

Small embedded Linux, fast iteration

Yocto / OE

Full source build + layer model

Very high (99.8%)

High

Complex embedded Linux, commercial products

Nix / Guix

Hermetic, content-addressed

Cryptographic

Very high

Long-lifetime products, supply chain security

What Reproducibility Actually Means — and Why It Is Not Solved

The distinction between "high reproducibility" and "hermetic reproducibility" matters for specific embedded use cases, and understanding it requires clarity about what can undermine the reproducibility that Yocto or Buildroot nominally provide.

Yocto achieves high reproducibility by being very specific about which versions of components are built. But the build environment itself — the host Linux kernel, the host libc, the host Python version used to run BitBake, the host glibc version used by the native tools that Yocto builds — still influences the build output. When a Yocto build produces different results on Ubuntu 22.04 versus Ubuntu 24.04, the layer configuration is identical but the host environment differs. Yocto addresses this partially through its buildtools tarball — a set of host native tools built separately that the build scripts activate before running BitBake, reducing but not eliminating host environment sensitivity.

Hermetic reproducibility — the property that the same input hash always produces the same output hash, regardless of the host operating system, regardless of when the build runs, regardless of the developer's system configuration — requires a fundamentally different approach to build environment management. This is what Nix and Guix provide.

A Nix derivation describes a build in terms of its complete input closure: the sources, the build tools, the compiler, the linker, the dependencies — all identified by cryptographic hash. The build executes in a sandbox: a Linux namespace with a separate root filesystem containing only the declared dependencies, no access to the host filesystem outside the Nix store, no network access by default, and a fixed timezone and locale. Two derivations with the same input hashes, run on any machine with any host OS, produce identical output hashes. The content-addressed Nix store at /nix/store ensures that the same derivation hash always resolves to the same build artifacts.

For embedded firmware, the practical implication is this: a Nix flake that describes the complete cross-compilation environment — pinned compiler toolchain, pinned libraries, pinned build tools — produces the same firmware binary on every developer machine, on every CI runner, and will produce the same binary five years from now as long as the input sources are available. The reUpNix research project demonstrated Nix-based reproducible embedded Linux stacks, reducing NixOS's basic installation size by up to 86 percent for constrained embedded devices and making system updates failure-atomic.

Nix for Embedded Development in Practice

The practical starting point for using Nix in embedded firmware development is the Nix development shell (nix develop or nix-shell). A shell.nix or flake.nix file declares the cross-compilation toolchain, host build tools, and any code generation dependencies as a complete, reproducible environment. A developer who enters nix develop in the project directory gets exactly the declared toolchain without installing anything system-wide, without modifying PATH manually, and identically to every other developer on the team and every CI runner.

For a bare-metal ARM Cortex-M firmware project, the Nix shell environment declaration pinning the entire arm-none-eabi toolchain looks approximately like:

nix
{
description = "STM32 firmware build environment";
inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-25.05";
outputs = { self, nixpkgs }: {
devShells.x86_64-linux.default = nixpkgs.legacyPackages.x86_64-linux.mkShell {
packages = with nixpkgs.legacyPackages.x86_64-linux; [
gcc-arm-embedded # pinned ARM toolchain
cmake ninja # build system
openocd # debugging
python3 # code generation scripts
];
};
};
}

The nixpkgs.url pins the entire Nixpkgs repository to a specific commit, which pins every package in the environment to a specific version. Updating any dependency is an explicit act with a visible diff in the flake.lock file — not an implicit consequence of a developer upgrading their system packages.

For embedded Linux targets, Nix can manage the cross-compilation environment for Yocto itself — ensuring that the host tools BitBake uses are hermetically specified — or replace Yocto entirely for products whose scope fits within what Nix-native cross-compilation supports. The reUpNix approach uses NixOS as the embedded Linux distribution rather than a Yocto-generated image, with Nix managing the entire system closure including the kernel and init system. This provides the strongest reproducibility guarantee but requires accepting that the embedded system is NixOS-based, which has implications for BSP support (NixOS's hardware support is broad but not as comprehensive as Yocto's for specialized industrial hardware) and for system image size.

One documented challenge for Nix in long-lifetime embedded products is source availability: Guix has documented that approximately 3.6 percent of package sources from 2022 are already missing from their original locations, and roughly 8 percent of sources from five years prior are unreachable. The Guix project's integration with Software Heritage — an archive of 94 percent of Guix package sources as of 2024 — is the practical answer to this. For commercial embedded products, the implication is that reproducibility over a 15-25 year product lifetime requires either maintaining a private source mirror or relying on a dedicated archival infrastructure.

GNU Guix — Stronger Guarantees, Different Tradeoffs

GNU Guix implements the same content-addressed, hermetic build model as Nix but with design decisions that give it distinct characteristics relevant to embedded use. Guix uses Scheme (specifically GNU Guile) as its configuration and package definition language rather than Nix's purpose-built functional language, which means Guix package definitions are programs in a general-purpose language with all the tooling that implies. For teams comfortable with Scheme or Lisp, this is a significant productivity advantage; for teams without that background, it is a steeper entry point than Nix's more common adoption.

Guix's full-source bootstrap is the most technically significant distinction from Nix for supply chain security. Guix has achieved the ability to build its entire software stack from a 357-byte seed program, bootstrapping GCC from first principles through a sequence of intermediate compilers. This directly addresses the attack vector that Ken Thompson described in his 1984 "Reflections on Trusting Trust" paper: if the compiler used to build the compiler was itself compromised, the compromise propagates to everything built with it, invisibly and undetectably. Guix's bootstrappable build chain provides a cryptographic argument that the toolchain it produces has not been silently modified — a property that is relevant for high-security embedded products in defense, critical infrastructure, and medical applications.

Guix's grafting mechanism allows security updates to a shared library to be applied without rebuilding every package that depends on it — a practical feature for products where the rebuild-all-on-dependency-update model of Nix creates operational friction during security patching.

 

docker

 


Docker and Container-Based Environments — the Pragmatic Middle Ground

Between the full hermeticity of Nix/Guix and the toolchain-dependent fragility of hand-maintained Makefiles, Docker-based build environments represent the pragmatic choice that most commercial embedded teams are currently adopting. A Docker image that contains the specific cross-compilation toolchain, build tools, and code generators provides environment reproducibility at the container level: the build environment is the container image, any developer who pulls the container gets the same environment, and CI runs in the same container.

Docker containers are not hermetically reproducible in the Nix sense — the container image is itself a product of its own build process whose inputs may not be fully pinned — but they are reproducible enough for most commercial embedded development workflows. The team pins the container image by digest (the SHA256 hash of the full image), ensuring that CI and developer builds use the exact same environment. The container image is built from a Dockerfile that specifies exact package versions. The image is stored in a private registry with retention policies that ensure availability for the product's lifetime.

The limitation that Docker containers do not address is source provenance: Docker containers standardize the build environment but do not prevent the build from pulling sources dynamically at build time, introducing network dependencies that undermine reproducibility. A container-based build that runs git clone or pip install as part of the build is not reproducible; a container-based build that uses only pre-fetched sources available inside the container or in a private mirror approaches the reproducibility level of a Yocto build with proper source mirroring.

The practical recommendation for teams currently using hand-maintained Makefiles or CMake with implicitly managed toolchains is to move to Docker-based environments as the immediate improvement, with Yocto for embedded Linux products and Nix for hermetic cross-compilation environments as the next step for teams with the most demanding reproducibility and supply chain security requirements.

Quick Overview

Embedded build system evolution follows a progression from hand-written Makefiles that implicitly depend on the host environment, through CMake and West that describe rules but not environment, through Yocto and Buildroot that build full software stacks with high reproducibility (Yocto achieves 99.8 percent binary reproducibility for core-image-minimal), toward Nix and Guix that provide hermetic, content-addressed builds where identical input hashes always produce identical output hashes regardless of host system. Docker container environments represent the practical intermediate step for teams moving away from host-dependent builds. EU Cyber Resilience Act requirements for SBOM generation and software supply chain transparency create regulatory pressure that reinforces the engineering case for reproducible build infrastructure. The reUpNix research demonstrates that the Nix model is adaptable to constrained embedded targets, reducing NixOS installation size by up to 86 percent for embedded Linux deployments.

Key Applications

Medical device firmware with 20+ year product lifetimes where reproducing a specific binary for field service audit requires recreating the exact build environment from years prior, critical infrastructure and defense embedded systems where supply chain integrity requires demonstrating that the toolchain was not compromised, industrial embedded Linux products requiring Yocto LTS four-year support cycles with SBOM generation for CRA compliance, automotive ECU development where binary reproducibility across development teams and supplier boundaries is a contractual requirement, and any embedded team where CI-produced binaries differ from developer-produced binaries due to host environment inconsistency.

Benefits

Hermetic builds eliminate "works on my machine" failures by ensuring every developer and every CI runner produces identical binaries from the same source. Content-addressed artifact caching in Nix means unchanged components are never rebuilt — a Nix build that changes one recipe reuses all cached artifacts for unchanged components, producing only the minimal necessary rebuilds. Source-pinned, version-locked build environments enable exact reproduction of any past firmware build without maintaining a manually configured legacy machine. SBOM generation as a build artifact — which Yocto supports natively and Nix/Guix support through derivation closure introspection — satisfies CRA component inventory requirements automatically.

Challenges

The Nix expression language has a steep learning curve for teams without functional programming background; Guix's Scheme is familiar to Lispers but unknown to most embedded C developers. Nix's Nixpkgs coverage of specialized embedded toolchains — vendor-specific SDKs, proprietary cross-compilers, hardware-specific code generators — is less comprehensive than Yocto's BSP ecosystem, requiring custom derivations for some platforms. Long-term source availability remains an unsolved problem: approximately 3.6 percent of Guix package sources from 2022 are already missing from their original locations, requiring private source mirrors for true long-term reproducibility. Migrating an existing Yocto-based product to Nix involves substantial rework with uncertain upside for products already achieving acceptable reproducibility.

Outlook

Yocto's Scarthgap LTS four-year support cycle, SPDX 3 SBOM generation, and ongoing reproducibility improvement work position it as the stable commercial baseline for embedded Linux products through at least 2028. Nix's adoption in high-security embedded contexts — demonstrated by its use in the MITRE eCTF embedded security competition for reproducible firmware environments and by Cyberus Technology's NixOS-based medical device and robotics products — is growing steadily as supply chain security requirements increase. The convergence of regulatory pressure (CRA SBOM requirements) with the technical properties of reproducible builds is making the investment in hermetic build infrastructure justifiable for a broader set of commercial embedded products.

Related Terms

reproducible build, hermetic build, content-addressed store, Nix derivation, Nix flake, flake.lock, nixpkgs, Guix, Guile Scheme, bootstrappable build, Software Heritage, reUpNix, Yocto, BitBake, OpenEmbedded, Buildroot, CMake, West, Makefile, cross-compilation, arm-none-eabi, toolchain pinning, build sandbox, Docker, container image digest, SBOM, SPDX, source mirroring, supply chain security, build reproducibility, binary reproducibility, functional package management, buildtools tarball, layer model, Scarthgap LTS, BitBake recipe, Ninja, Ken Thompson attack, trust chain

 

Contact us

 

 

Our Case Studies

 

FAQ

What makes a build hermetic and why does it matter for embedded firmware?

 

A hermetic build executes with access only to explicitly declared, cryptographically identified inputs — no access to the host filesystem, no implicit network downloads, no implicit dependency on host system packages, environment variables, or timestamps outside the build's control. The same declared inputs always produce the same output. It matters for embedded firmware because a non-hermetic build can produce different binaries from the same source depending on the developer's system configuration, the CI runner's OS version, or the date of the build — making it impossible to demonstrate that two builds of the same source are identical, and making it impossible to reproduce a specific firmware binary years after its initial build.
 

How does Nix achieve reproducibility where Yocto and CMake do not?

 

Nix identifies every build artifact by the cryptographic hash of its complete input closure: the source files, the build tools, the compiler, and all dependencies, each themselves identified by hash. Builds execute in a sandbox with access only to the declared inputs and no access to the host filesystem or network. Two builds with identical input hashes produce identical output hashes on any machine with any host OS. Yocto achieves high reproducibility by specifying component versions precisely, but the build still executes in the host environment; host OS differences can produce different results. CMake describes build rules but does not control the environment — it assumes the toolchain and libraries available on the host are consistent across developers and CI.
 

What is the reUpNix approach and what does it demonstrate for embedded Linux?

 

The reUpNix project published at SIGPLAN/SIGBED 2023 demonstrated NixOS-based reproducible embedded Linux stacks. It addressed NixOS's limitations for embedded deployment by reducing the basic installation size by up to 86 percent, making system updates failure-atomic through content-addressed store semantics, and integrating third-party OCI container images with up to 24 percent less on-disk space through file-level deduplication. It demonstrates that the Nix model — which was primarily designed for server and desktop Linux — can be adapted for resource-constrained embedded targets, providing stronger reproducibility and update safety guarantees than Yocto or Buildroot while maintaining a manageable system image size.
 

Why does Guix's full-source bootstrap matter for high-security embedded products?

 

Guix's full-source bootstrap builds its entire toolchain from a 357-byte verifiable seed program, avoiding dependence on any pre-compiled compiler binary. This addresses the supply chain attack described by Ken Thompson in 1984: a compromised compiler can inject malicious code into everything it compiles, including future versions of itself, invisibly and persistently. Any build pipeline that starts from a pre-compiled compiler binary cannot rule out this attack without trusting the binary's provenance. Guix's bootstrappable build chain provides a mathematical argument that the toolchain is clean — relevant for defense, critical infrastructure, and medical embedded products where supply chain integrity is a security requirement.