Install EVE-OS over the Network with PXE

Introduction

With PXE, an edge device installs EVE-OS (Edge Virtualization Engine) over the network instead of from a USB drive. This is useful for testing EVE-OS versions, where devices are reinstalled regularly, and for deploying EVE-OS to many edge devices. This article describes how to build a PXE server with Docker Compose, prepare the edge device, and take it from installation to onboarding in Edge Infrastructure Services. It covers an HTTP setup and a TFTP-only setup for EVE-OS 13.4.0 and greater on 64-bit x86 (amd64) edge devices with UEFI.

Overview

The PXE server needs:

Three services: DHCP (Kea), TFTP (tftp-hpa), and HTTP (NGINX), running as Docker containers.
Five files from the EVE-OS netboot archive, copied to the folders of the TFTP server and the web server.
One change: a single line added to ipxe.efi.cfg.

The edge device requests the files in this order:

 Edge device
   │
   ├─ PXE    DHCP ──► IP address, next-server, boot file ipxe.efi
   │         TFTP ──► ipxe.efi
   │
   ├─ iPXE   DHCP ──► IP address, next-server, boot file ipxe.efi
   │         TFTP ──► ipxe.efi.cfg
   │         HTTP ──► EFI/BOOT/BOOTX64.EFI
   │
   ├─ GRUB   HTTP ──► EFI/BOOT/grub.cfg
   │         HTTP ──► installer.iso
   ▼
 Installer runs

GRUB reads its configuration file from the same location that delivered the GRUB binary. Because ipxe.efi.cfg points at the web server, GRUB retrieves both EFI/BOOT/grub.cfg and installer.iso over HTTP. Place grub.cfg next to BOOTX64.EFI in the web root, not in the TFTP root.

All installation settings come from installer.iso. Installation settings in ipxe.efi.cfg have no effect.

The installation starts in one of two ways:

From the network: the PXE client in the device firmware loads ipxe.efi from the PXE server.
From the hard disk: the device firmware starts ipxe.efi directly from the hard disk, and the PXE block is skipped.

From iPXE onward, both ways are identical and lead to the same installation result.

Prerequisites

A PXE server host running Linux, with a static IP address on the imaging network. The containers use network_mode: host, which gives them direct access to the host network. DHCP and TFTP require this. Docker Desktop on macOS and Windows does not provide host networking in the same way, so it is not suitable for the PXE server.
Docker Engine 23.0 or greater on the PXE server host. That release and later include Docker Compose v2, which provides the docker compose command used in this article. The older docker-compose command with a hyphen is Compose v1 and is not supported.
Network access from the PXE server host to Docker Hub, to download the EVE-OS netboot archive and the container images.
The EVE-OS version to install, 13.4.0 or greater.
An edge device with 64-bit x86 (amd64) architecture and UEFI firmware, able to boot from the network or from a hard disk that you can write to.

Values Used in the Examples

All examples in this article use the following values. Replace them with the values of your network.

Value	Example	Used in
IP address of the PXE server	`192.168.104.2`	`ipxe.efi.cfg`, `kea-dhcp4.conf`
Network interface of the PXE server	`ens19`	`kea-dhcp4.conf`
Subnet	`192.168.104.0/24`	`kea-dhcp4.conf`
Address pool for edge devices	`192.168.104.100 - 192.168.104.200`	`kea-dhcp4.conf`
Default gateway	`192.168.104.1`	`kea-dhcp4.conf`
DNS server	`8.8.8.8`	`kea-dhcp4.conf`
EVE-OS version	`17.0.0-lts-kvm-amd64`	Download command

Decisions Before the Setup

Boot order: testing or fleet deployment

Testing and lab use: network boot first

In the BIOS/UEFI setup of the edge device, set the boot order to network boot (PXE) first and hard disk second.

 Every boot
   │
   ├─ PXE server answers ──────────► Reinstall EVE-OS
   │
   └─ PXE server does not answer ──► Boot EVE-OS from hard disk

To install a different EVE-OS version, replace the files on the PXE server and restart the edge device.

Every edge device that starts from the network (PXE) while the PXE server answers is reinstalled. Its hard disk is overwritten.

Fleet deployment: iPXE on the hard disk, BIOS/UEFI locked

Use this approach when you deploy EVE-OS to many edge devices and each device must be installed one time only. With this approach:

Installation happens one time only. The edge device starts from the hard disk. iPXE on the hard disk performs the network installation, and EVE-OS replaces iPXE. After installation, nothing on the device can start a network installation again.
No work on the device after installation. The BIOS/UEFI is set and locked before installation, so nobody has to return to the device to change the boot order.
Measured boot stays valid. UEFI measures the list of boot options into TPM register PCR-1, and EVE-OS uses PCR-1 to seal the vault key. Because the BIOS/UEFI settings do not change after installation, the recorded value stays the same and the vault continues to unseal.

Write ipxe.efi to the hard disk of the edge device.
In the BIOS/UEFI setup of the edge device, set the boot order to hard disk first.
Lock the BIOS/UEFI setup, for example with an administrator password.

 First start                    Every later start
 ┌──────────────────┐           ┌──────────────────┐
 │ Hard disk:       │           │ Hard disk:       │
 │ - ipxe.efi       │           │ - EVE-OS         │
 └────────┬─────────┘           └────────┬─────────┘
          ▼                              ▼
 iPXE installs EVE-OS           EVE-OS starts from
 and replaces itself            the hard disk

Changing the boot order after the edge node is onboarded changes the PCR-1 value. EVE-OS is then unable to unseal the vault key, and the edge node logs a policy check failure naming PCR index 1. Attaching any device that carries a bootable partition, most often a USB drive, has the same effect.

Network: single network or staged

Single network: installation and onboarding on one network

  ┌──────────────────┐              ┌──────────────────┐
  │    PXE server    │              │  Edge Infra-     │
  │ DHCP, TFTP, HTTP │              │  structure Svcs  │
  └────────┬─────────┘              └────────┬─────────┘
           │                         ┌───────┴────────┐
           │                         │ Router / WAN   │
           │                         └───────┬────────┘
           │      ┌──────────────┐           │
           └──────┤    Switch    ├───────────┘
                  └──────┬───────┘
                         │
                 ┌───────┴────────┐
                 │  Edge device   │
                 └────────────────┘

In this case the DHCP server (Kea) must provide a default gateway and DNS servers. Refer to the “DHCP: Kea configuration” section.

Staged: separate imaging and onboarding networks

The edge device installs on an isolated imaging network and onboards on a separate onboarding network.

  ┌──────────────────┐
  │    PXE server    │
  │ DHCP, TFTP, HTTP │
  └────────┬─────────┘
           │
    ┌──────┴───────┐
    │    Switch    │
    └──────┬───────┘
           │
  ┌────────┴─────────┐
  │   Edge device    │
  └──────────────────┘

The onboarding network must provide a DHCP server with a default gateway and DNS servers, and a route to your Edge Infrastructure Services cluster.

HTTP Setup

Services: Docker Compose with Kea, tftp-hpa, and NGINX

Create the following directory structure.

 pxe/
 ├── docker-compose.yml
 ├── artifacts/                  extracted netboot archive
 ├── kea/
 │   ├── etc/kea-dhcp4.conf      Kea configuration
 │   ├── lib/                    Kea lease database
 │   └── log/                    Kea logs
 ├── nginx/
 │   ├── conf.d/custom.conf      NGINX configuration
 │   ├── www/                    files delivered over HTTP
 │   └── log/                    NGINX logs
 └── tftp/
     ├── tftpboot/               files delivered over TFTP
     └── log/                    TFTP logs

Create docker-compose.yml in the pxe folder.

services:
  kea:
    image: docker.cloudsmith.io/isc/docker/kea-dhcp4
    network_mode: host
    volumes:
      - ./kea/etc:/etc/kea
      - ./kea/lib:/var/lib/kea
      - ./kea/log:/var/log/kea
    restart: unless-stopped

  tftp:
    image: alpine:3.22
    network_mode: host
    command: >
      sh -c "apk add --no-cache tftp-hpa &&
        { syslogd -n -O /var/log/tftp/tftpd.log & } &&
        exec in.tftpd -L -vvv --secure --user root /tftpboot"
    volumes:
      - ./tftp/tftpboot:/tftpboot:ro
      - ./tftp/log:/var/log/tftp
    restart: unless-stopped

  nginx:
    image: nginx:alpine
    network_mode: host
    command: ["nginx-debug", "-g", "daemon off;"]
    volumes:
      - ./nginx/conf.d:/etc/nginx/conf.d:ro
      - ./nginx/www:/usr/share/nginx/html:ro
      - ./nginx/log:/var/log/nginx
    restart: unless-stopped

The file deliberately has no version key. Compose v2 treats that key as obsolete and prints a warning if it is present.

The tftp container installs tftp-hpa from the Alpine package repository at every start. The PXE server host needs internet access whenever this container starts.To avoid that, build a container image with the package already installed.

FROM alpine:3.22
RUN apk add --no-cache tftp-hpa
CMD ["sh", "-c", "syslogd -O /var/log/tftp/tftpd.log && exec in.tftpd -L -vvv --secure --user root /tftpboot"]

Build it, for example with docker build -t local/tftp:3.22 ., then reference it in docker-compose.yml.

  tftp:
    image: local/tftp:3.22
    network_mode: host
    volumes:
      - ./tftp/tftpboot:/tftpboot:ro
      - ./tftp/log:/var/log/tftp
    restart: unless-stopped

Create nginx/conf.d/custom.conf.

server {
    listen 80;
    root /usr/share/nginx/html;
    autoindex on;
}

Files: netboot archive, placement, and changes

Change to the artifacts folder, then download and extract the netboot archive.

cd pxe/artifacts
docker run --rm lfedge/eve:17.0.0-lts-kvm-amd64 installer_net | tar -xf -

Optional: to use a customized installer.iso, refer to Get a Custom EVE-OS Image.
Copy the files to the following folders.

 Archive content           Copy to
 ──────────────────────    ──────────────────────────────────────────
 ./ipxe.efi           ───► ./tftp/tftpboot/ipxe.efi
 ./ipxe.efi.cfg       ───► ./tftp/tftpboot/ipxe.efi.cfg (one line added)
 ./EFI/BOOT/BOOTX64.EFI ─► ./nginx/www/EFI/BOOT/BOOTX64.EFI
 ./EFI/BOOT/grub.cfg  ───► ./nginx/www/EFI/BOOT/grub.cfg
 ./installer.iso      ───► ./nginx/www/installer.iso

grub.cfg belongs in the web root next to BOOTX64.EFI, because GRUB reads its configuration file from the same location that delivered the GRUB binary. A copy in the TFTP root is never read. If you use a customized installer.iso, copy it to ./nginx/www/installer.iso instead.

Add one line to ./tftp/tftpboot/ipxe.efi.cfg. Find the following commented line near the top of the file.

# set url https://foo.bar/

Add a set url line beneath it that points to your web server. The trailing slash is required, because iPXE appends EFI/BOOT/BOOTX64.EFI directly to this value.

set url http://192.168.104.2/

Leave the rest of the file unchanged. The set console, set eve_args, and set installer_args lines have no effect in EVE-OS 13.4.0 and greater, but the iseq ${buildarch} lines that follow them select the bootloader for your architecture and must stay in place. EFI/BOOT/grub.cfg needs no changes. GRUB was chainloaded over HTTP, so it already reads installer.iso over HTTP from the web root.

DHCP: Kea configuration

Create kea/etc/kea-dhcp4.conf.

{
  "Dhcp4": {
    "interfaces-config": {
      "interfaces": ["ens19"]
    },
    "lease-database": {
      "type": "memfile",
      "persist": true,
      "name": "/var/lib/kea/kea-leases4.csv"
    },
    "subnet4": [
      {
        "id": 1,
        "subnet": "192.168.104.0/24",
        "pools": [
          { "pool": "192.168.104.100 - 192.168.104.200" }
        ],
        "next-server": "192.168.104.2",
        "boot-file-name": "ipxe.efi",
        "option-data": [
          { "name": "routers", "data": "192.168.104.1" },
          { "name": "domain-name-servers", "data": "8.8.8.8" }
        ]
      }
    ],
    "loggers": [
      {
        "name": "kea-dhcp4",
        "severity": "DEBUG",
        "debuglevel": 99,
        "output_options": [
          { "output": "/var/log/kea/kea-dhcp4.log", "flush": true }
        ]
      }
    ]
  }
}

Setting	Effect
`next-server`, `boot-file-name`	The edge device starts the network installation.
`routers`	The edge device has a default gateway to Edge Infrastructure Services.
`domain-name-servers`	The edge device can resolve the hostname of your Edge Infrastructure Services cluster.

Serve ipxe.efi as the boot file name on every DHCP request, including the request that iPXE itself sends after it starts. iPXE appends .cfg to this name to build the path to its own configuration file. To install only specific edge devices, remove next-server and boot-file-name from the subnet and add a reservation per edge device:

"reservations": [
  {
    "hw-address": "bc:24:11:52:78:98",
    "ip-address": "192.168.104.71",
    "next-server": "192.168.104.2",
    "boot-file-name": "ipxe.efi"
  }
]

Start and install

In the pxe folder, start the containers.

docker compose up -d
docker compose ps

Start the edge device. The console shows output similar to the following:

>>Start PXE over IPv4.
  Station IP address is 192.168.104.71
  Server IP address is 192.168.104.2
  NBP filename is ipxe.efi
 Downloading NBP file...
  NBP file downloaded successfully.

iPXE 1.0.0+ -- Open Source Network Boot Firmware -- https://ipxe.org
Configuring (net0 bc:24:11:52:78:98)...... ok
tftp://192.168.104.2/ipxe.efi.cfg... ok
http://192.168.104.2/EFI/BOOT/BOOTX64.EFI... ok
Downloading installer. This may take some time. Please wait patiently.

*Boot 17.0.0-lts-kvm-amd64-installer

On EVE-OS 13.4, 14.5, 16.0, and 17.0 LTS releases, including 17.0.0-lts, the edge device downloads installer.iso twice. This roughly doubles the installation time and the network traffic. The installation still succeeds. The issue is fixed in EVE-OS 17.4.0.

Troubleshooting

Symptom	Cause
The boot stops after the `BOOTX64.EFI` line.	GRUB cannot find `EFI/BOOT/grub.cfg`. Confirm the file is in `./nginx/www/EFI/BOOT/`, not only in the TFTP root.
The boot stops after `ipxe.efi.cfg` loads.	The `set url` line is missing or has no trailing slash.
The device never requests a boot file.	`next-server` or `boot-file-name` is missing from the Kea configuration, or the reservation does not match the device MAC address.

For more diagnostics, refer to Troubleshoot EVE-OS Installation.

Onboarding

Single network: after the installation, EVE-OS starts from the hard disk and connects through the router to Edge Infrastructure Services.
Staged: after the installation, switch off the edge device, connect it to the onboarding network, and switch it on. EVE-OS starts from the hard disk and connects to Edge Infrastructure Services.

To register the edge node, refer to Onboard an Edge Node.

TFTP-Only Setup

The TFTP-only setup uses the HTTP setup without NGINX. Every file is delivered over TFTP, and no file needs to be edited.

ZEDEDA does not recommend this setup. GRUB’s TFTP transport cannot seek within a file, so every backward seek restarts the transfer from the first byte. For an installer ISO image of approximately 500 MB, that means several complete transfers before the kernel starts. GRUB’s HTTP transport requests only the byte ranges it reads. Use the HTTP setup unless you cannot run a web server on the imaging network.

Remove the nginx service from docker-compose.yml.

services:
  kea:
    image: docker.cloudsmith.io/isc/docker/kea-dhcp4
    network_mode: host
    volumes:
      - ./kea/etc:/etc/kea
      - ./kea/lib:/var/lib/kea
      - ./kea/log:/var/log/kea
    restart: unless-stopped

  tftp:
    image: alpine:3.22
    network_mode: host
    command: >
      sh -c "apk add --no-cache tftp-hpa &&
        { syslogd -n -O /var/log/tftp/tftpd.log & } &&
        exec in.tftpd -L -vvv --secure --user root /tftpboot"
    volumes:
      - ./tftp/tftpboot:/tftpboot:ro
      - ./tftp/log:/var/log/tftp
    restart: unless-stopped

The tftp container installs tftp-hpa at every start, as described in the “Services” section. The Dockerfile workaround described there applies here too.

Copy all files unchanged to ./tftp/tftpboot/.

 Archive content           Copy to
 ──────────────────────    ─────────────────────────────────────
 ./ipxe.efi           ───► ./tftp/tftpboot/ipxe.efi
 ./ipxe.efi.cfg       ───► ./tftp/tftpboot/ipxe.efi.cfg
 ./EFI/BOOT/BOOTX64.EFI ─► ./tftp/tftpboot/EFI/BOOT/BOOTX64.EFI
 ./EFI/BOOT/grub.cfg  ───► ./tftp/tftpboot/EFI/BOOT/grub.cfg
 ./installer.iso      ───► ./tftp/tftpboot/installer.iso

No set url line is needed. With that value empty, iPXE resolves EFI/BOOT/BOOTX64.EFI relative to the TFTP server it loaded its configuration file from, and GRUB then reads grub.cfg and installer.iso from the same TFTP root. If you use a customized installer.iso, copy it to ./tftp/tftpboot/installer.iso instead.

Start Here

Infrastructure Services Overview

Core infrastructure

Build

Applications and delivery

Edge Kubernetes

Operations

Monitor

Administration and Security

Troubleshooting

Partnerships

Install EVE-OS over the Network with PXE

Introduction

Overview

Prerequisites

Values Used in the Examples

Decisions Before the Setup

Boot order: testing or fleet deployment

Testing and lab use: network boot first

Fleet deployment: iPXE on the hard disk, BIOS/UEFI locked

Network: single network or staged

Single network: installation and onboarding on one network

Staged: separate imaging and onboarding networks

HTTP Setup

Services: Docker Compose with Kea, tftp-hpa, and NGINX

Files: netboot archive, placement, and changes

DHCP: Kea configuration

Start and install

Troubleshooting

Onboarding

TFTP-Only Setup

Next Steps

Onboard an Edge Node

Get a Custom EVE-OS Image

Troubleshoot EVE-OS Installation

Get EVE-OS

​Introduction

​Overview

​Prerequisites

​Values Used in the Examples

​Decisions Before the Setup

​Boot order: testing or fleet deployment

​Testing and lab use: network boot first

​Fleet deployment: iPXE on the hard disk, BIOS/UEFI locked

​Network: single network or staged

​Single network: installation and onboarding on one network

​Staged: separate imaging and onboarding networks

​HTTP Setup

​Services: Docker Compose with Kea, tftp-hpa, and NGINX

​Files: netboot archive, placement, and changes

​DHCP: Kea configuration

​Start and install

​Troubleshooting

​Onboarding

​TFTP-Only Setup

​Next Steps

Onboard an Edge Node

Get a Custom EVE-OS Image

Troubleshoot EVE-OS Installation

Get EVE-OS

Introduction

Overview

Prerequisites

Values Used in the Examples

Decisions Before the Setup

Boot order: testing or fleet deployment

Testing and lab use: network boot first

Fleet deployment: iPXE on the hard disk, BIOS/UEFI locked

Network: single network or staged

Single network: installation and onboarding on one network

Staged: separate imaging and onboarding networks

HTTP Setup

Services: Docker Compose with Kea, tftp-hpa, and NGINX

Files: netboot archive, placement, and changes

DHCP: Kea configuration

Start and install

Troubleshooting

Onboarding

TFTP-Only Setup

Next Steps