Infrastructure Error Resolutions

Use this section for issue-driven workflows that do not belong to a single object page. For API error codes, see Errors.

403: Server Failed to Authenticate Request

Problem

Your node returns a 403 error while updating EVE-OS or downloading an edge app image.

Error messages

During EVE-OS upgrade, seen in the status tab under the EVE-OS Images block:

RESPONSE Status: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.

During edge app image download, seen in the status tab:

Syncer Download Status of image name: develop-20230712.2-ubuntu-2001.packer, location: /persist/vault/downloader/pending/3f13e0116cba0df1cecf2b47a91af7d5c93cf77a0198843e8384a971a83801b3.c92788c1374f6072cefdb9fa8a59d1adfa4235dc27b863c8e6789822171a98a4 - error could not get properties for blob: -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, /pillar/vendor/github.com/Azure/azure-storage-blob-go/azblob/zc_storage_error.go:42

===== RESPONSE ERROR (ServiceCode=AuthenticationFailed) =====

Description=403 Forbidden, Details: (none)

Cause

The node’s system clock is not synchronized with the Network Time Protocol (NTP) server. When the timestamp in the request header differs from the server’s time by more than 15 minutes, the server interprets the request as a potential replay attack and rejects authentication.

Solution

Ensure your node is synchronized with the NTP server. A clock difference greater than 15 minutes causes authentication to fail against the image datastore.

Maintenance Mode

A node enters maintenance mode when EVE-OS detects a condition that prevents it from operating safely. Applications stop running, but the node remains reachable through the EVE-OS Local UI. The run state ribbon on the node overview tab shows Maintenance Mode. To identify the specific reason, check the notification bell on the node detail page.

Low disk space

Available storage on the node has reached zero. EVE-OS reserves a minimum amount of disk space for the operating system and log files. When the reserve is exhausted, EVE-OS suspends volume creation and enters maintenance mode to prevent data corruption. Free disk space by deleting unused application images, edge containers, or volumes from the node. Maintenance mode clears automatically when storage becomes available. No reboot is required.

Vault locked up

The encrypted vault (EVE-OS’s protected storage partition for application data and cryptographic keys) did not unlock within the expected time after boot. The most common cause is a TPM (Trusted Platform Module) PCR value mismatch: the node’s measured boot state has changed and no longer matches the stored attestation policy. Typical triggers include an EVE-OS upgrade, a BIOS or firmware update, a change to secure boot settings, or replacement of hardware components. Resolution depends on the cause:

If a recent EVE-OS upgrade or firmware update triggered the mismatch, update the node’s PCR template to reflect the new PCR values. Maintenance mode clears automatically after attestation succeeds. See Flexible PCR Policy and Manage PCR Templates.
If the vault key was never backed up to Edge Infrastructure Services (for example, on a node that failed during initial onboarding), re-flash and re-onboard the device.
If TPM hardware failure is suspected, see the TPM encryption failure section in this article.

Node certs refused

Edge Infrastructure Services rejected the node’s signing certificate. This occurs when the device’s certificate changes after a factory reset or a re-installation of EVE-OS that regenerates the /persist partition. Edge Infrastructure Services treats a certificate change as a security violation and puts the node in maintenance mode.

The maintenance mode reason might appear as Unknown in the current UI. Look for a notification containing text similar to Failed to send EdgeNodeCerts: SenderStatusNotFound to confirm this is the cause.

To resolve, delete the device record from Edge Infrastructure Services and re-onboard the node. Deleting the device record removes any device-specific configuration attached to it.

In the sidebar, select Infrastructure.
Locate the node and open its detail page.
Select Delete to remove the device record.
Re-onboard the device using its onboarding key. See Onboard Nodes for instructions.

TPM encryption failure

The node’s TPM hardware failed its periodic self-check. EVE-OS tests the TPM’s encrypt and decrypt functions hourly. If the test fails, EVE-OS enters maintenance mode to prevent data loss from an unreliable encryption subsystem.

The maintenance mode reason might appear as Unknown in the current UI. Look for a notification containing text similar to TPM manager reported TPM error: failed to encrypt key using TPM to confirm this is the cause.

The TPM check retries hourly. If the failure is transient, maintenance mode clears automatically when the check passes. If the failure persists, the TPM hardware has likely failed and the device requires repair or replacement. Contact ZEDEDA Technical Support.

Attestation Status Shows “Not Done”

Problem

The Attestation field in the node’s Security panel shows Not Done even though attestation is configured. The UI shows Not Done for any attestation state other than a successful pass, including nodes that have never completed attestation and nodes whose attestation is actively failing. To distinguish between the two: a node that previously attested successfully has a date in the Last Attested At field. A blank value means the node has never successfully attested.

Cause

The most common cause of attestation failure on a previously healthy node is a change in measured boot state. A PCR (Platform Configuration Register) value no longer matches the stored attestation policy. Typical triggers are an EVE-OS upgrade, a BIOS or firmware update, a change to secure boot or USB boot settings, or replacement of hardware components. Less common causes, such as a network interruption causing a nonce mismatch or an integrity token mismatch between attestation cycles, resolve automatically on the next retry and do not require admin action.

Solution

On the node detail page, go to the Status tab and find the Attestation section.
Select Show Reported PCR Values to view the current PCR values reported by the device.
Compare the reported values against the PCR template for this node’s hardware model in Marketplace > Hardware Models > your model > Attestation.
If the values have changed (for example, after an EVE-OS upgrade), update the PCR template to match the new reported values. EVE-OS retries attestation automatically after the template is updated.

See Manage PCR Templates and Remote Attestation for detailed instructions. If attestation continues to fail after updating the PCR template, contact ZEDEDA Technical Support.

Node Reboot Reasons

The Last Reboot Reason field on the node overview tab records why the node last restarted. The field shows a one-line label with no additional context. Normal reasons (First Boot After Install, User Requested, and EVE-OS Update) indicate expected behavior and require no action. The following sections describe the unexpected reboot reasons that require investigation.

EVE-OS update fallback

The node attempted an EVE-OS upgrade but could not reach Edge Infrastructure Services within the fallback window (default: 5 minutes) after booting the new version. EVE-OS automatically reverted to the previous stable version and rebooted. After the fallback reboot, the node runs its previous EVE-OS version. If network connectivity is restored, the node reconnects to Edge Infrastructure Services normally. Confirm the active version on the node overview tab. To retry the upgrade, verify that the node has stable Edge Infrastructure Services connectivity, then reinitiate the update. See Update EVE-OS on Nodes. The fallback window is configurable via the timer.update.fallback.no.network node configuration setting (default: 300 seconds, minimum: 60 seconds). If your deployment has intentional connectivity gaps during updates, increase this value before initiating upgrades.

Disconnect

The node was unable to reach Edge Infrastructure Services for longer than the disconnect timeout (default: 7 days) and rebooted to attempt recovery. After rebooting, the node reconnects to Edge Infrastructure Services automatically if network connectivity is available. Check that the node’s management interface has connectivity and that Edge Infrastructure Services is reachable. For nodes that are expected to operate offline for extended periods, configure the timer.reboot.no.network setting to a longer value. See Enable Offline Mode.

Out of memory

The Linux kernel’s out-of-memory (OOM) killer terminated a process on the node due to memory exhaustion. EVE-OS detected the terminated process and rebooted to restore normal operation. Review the memory allocation of edge apps deployed to this node. If apps consistently consume more memory than is available, reduce their memory allocation, remove apps from the node, or migrate workloads to a node with more RAM. If the OOM event occurs without heavy app workloads, verify that the node meets the minimum memory requirements for EVE-OS and its deployed apps combined.

Network Configuration Verification Failed

Problem

The notification bell on the node detail page shows a message in the following format:

Not using the latest network config (submitted at <timestamp>), instead using older network config (submitted at <timestamp>) due to error: <error detail>

This notification means EVE-OS could not verify the current network adapter configuration and fell back to an older configuration. The <error detail> at the end of the message describes why verification failed.

Cause

EVE-OS maintains an ordered list of network configurations pushed from Edge Infrastructure Services. When the most recent configuration fails connectivity tests, EVE-OS automatically falls back to the previous configuration and records the reason. The <error detail> in the notification identifies the underlying problem. Three common patterns:

No IP address obtained. The message contains text such as "no suitable IP address available" or "waiting for interface IP address(es) and/or DNS server(s)". The management interface did not receive an IP address. Check the physical connection, switch port, VLAN configuration, DHCP server availability, or static IP settings.
IP address obtained but Edge Infrastructure Services unreachable. The interface has an IP and DNS server but cannot connect to Edge Infrastructure Services. A firewall rule, proxy setting, or routing issue is blocking outbound connectivity. Check firewall rules and proxy configuration on the network.
Edge Infrastructure Services responded with an error. The interface has connectivity but Edge Infrastructure Services returned a certificate or server error. This is commonly caused by a TLS-inspecting proxy intercepting the connection. Check proxy settings and any TLS inspection policies on the network.

Solution

On the node detail page, select the Adapters tab.
Review the management port configuration (IP settings, VLAN, proxy, and DNS).
Correct the configuration and select Save.

EVE-OS picks up the new configuration immediately and retests connectivity. If verification succeeds, the notification clears. If the node cannot reach Edge Infrastructure Services at all and you need to restore connectivity without modifying the configuration, enable the last-resort network configuration. See Enable Last Resort.

Insufficient Memory for App Deployment

Problem

The notification bell on an app instance detail page shows one of the following messages. When no memory is expected to free up:

Remaining memory bytes <N> app instance needs <N>

When other app instances are shutting down and memory is expected to become available:

App instance needs <N> bytes but only have <N>; waiting for one or more halting app instances to free up <N> bytes

The app instance does not start.

Cause

The node does not have enough free memory to launch the app instance. The first message indicates that no other app instances are shutting down. The second message indicates that EVE-OS detected halting app instances and is waiting for those instances to release memory before retrying.

Solution

If the second message is visible, wait for the halting app instances to finish shutting down. EVE-OS retries automatically and the app starts when enough memory is available. If the first message is visible:

Check memory usage on the Overview tab of the node detail page.
Stop one or more running app instances on the same node to free memory.
EVE-OS retries the failed app instance automatically when enough memory becomes available.

To prevent recurrence, reduce the memory allocation of the failing app instance before redeploying it, or deploy it to a node with more available RAM.

Insufficient Disk Space for Volume Creation

Problem

The notification bell on a volume detail page or app instance detail page shows:

Remaining disk space <N> volume needs <N>

The volume is not created and any app instance that depends on it does not start.

Cause

The node does not have enough free disk space to create the volume at the requested size. This is distinct from the Maintenance Mode: Low Disk Space scenario, which occurs when the entire storage reserve is exhausted and the node enters maintenance mode. A volume creation failure can occur while the node is otherwise healthy and running other apps normally.

Solution

Free disk space on the node by deleting unused volumes or application images. EVE-OS retries the volume creation automatically when sufficient space becomes available. No manual restart of the volume or app instance is required. See Manage Volumes for instructions on deleting volumes.

403: Unauthorized Deployment

Problem

Edge Infrastructure Services is not permitting your node to deploy your application.

Affected versions

EVE-OS (Edge Virtualization Engine) versions 10.4.0 through 10.4.3.

Error messages

In the logs, initial /attest calls fail:

devicename:my_edge_node msg:[ATTEST] Error All attempts to connect to zedcloud.zededa.net/api/v2/edgedevice/id/f4dd387d-5856-42cb-bd02-e90048711f62/attest failed: [send via eth0: interface eth0: no suitable IP address available]

In the status tab of the edge app instance:

Found error in content tree CONTENT_TREE_NAME attached to volume DATASTORE_NAME: Syncer Download Status of image name: superset:latest, location: - error error getting manifest: GET https:// /oauth2/token?scope=repository%3Asuperset%3Apull&service=zededaacrwsvywdevedgeimages.azurecr.io: UNAUTHORIZED: authentication required, visit https://aka.ms/acr/authorization for more information.

Cause

EVE-OS fails to transmit the required certificates. This happens when a network problem occurs during onboarding and causes the first attest calls to fail. EVE-OS does not attempt to retransmit certificates after the first failure.

Solution

Upgrade to an LTS version greater than 10.4.3. See Update EVE-OS on Nodes.

400: Authentication Information Format

Problem

Your node displays a 400 error in the status tab while downloading an EVE-OS image during an upgrade. The download fails because EVE-OS cannot authenticate with the image datastore.

Affected versions

EVE-OS (Edge Virtualization Engine) versions earlier than 9.4.15.

Error messages

Seen in the node’s status tab:

Description=400 Authentication information is not given in the correct format. Check the value of Authorization header., Details: (none) HEAD https://zededacentral.blob.core.windows.net/diskimages/10.4.8-lts-kvm-amd64?timeout=61 Authorization: REDACTED

The full blob storage error shows InvalidAuthenticationInfo as the Azure service code:

===== RESPONSE ERROR (ServiceCode=InvalidAuthenticationInfo) =====
Description=400 Bad Request, Details: (none)
RESPONSE Status: 400 Bad Request
X-Ms-Error-Code: [InvalidAuthenticationInfo]

Cause

EVE-OS fails to retrieve the Edge Infrastructure Services certificate over a weak internet connection during onboarding or a previous boot. Without the certificate, EVE-OS cannot generate a valid authentication header for the image datastore, causing the 400 response when it attempts to download the upgrade image.

Workaround

Power cycle the node several times. Each reboot gives EVE-OS another opportunity to retrieve the Edge Infrastructure Services certificate. The 400 error stops after the certificate is successfully fetched. If the 400 error persists after multiple power cycles, contact ZEDEDA Technical Support.

Solution

Upgrade to EVE-OS 9.4.15 or greater. See Update EVE-OS on Nodes.

Device Stuck in Booting State

Problem

Your node becomes stuck displaying the status Booting after an EVE-OS upgrade. The node appears unresponsive, but its applications continue running.

Affected versions

EVE-OS versions 8.5.x and 9.3.0, and all versions earlier than 8.12.0.

Error messages

The node takes a long time to fetch the Edge Infrastructure Services certificate:

error getCertsFromController failed: All attempts to connect to zedcloud.pmwtus.zededa.net/api/v2/edgedevice/certs failed: [send via eth0: interface eth0: no suitable IP address available send via eth1: interface eth1: no suitable IP address available]

The node cannot fetch its configuration due to a certificate update mismatch:

error removeAndVerifyAuthContainer: local server cert hash(32) does not match in authen (32)
error RemoveAndVerifyAuthContainer verify auth error removeAndVerifyAuthContainer: local server cert hash 32bytes does not match in authen, V2 server true

The issue resolves after the new Edge Infrastructure Services certificate is fetched:

info Controller cert delete
info Controller cert create
info BaseOs status create 9.4.4-lts-kvm-amd64

Cause

The node cannot obtain the Edge Infrastructure Services certificate because of a lack of internet connectivity on the management interface. As a result, EVE-OS skips certificate fetching during boot. Certificate retrieval takes longer on older EVE-OS versions and depends on connection quality.

Workaround

Force the certificate to roll out by power cycling the node until it successfully receives the certificate. This might require several power cycles.

Solution

Upgrade to an LTS version. See Update EVE-OS on Nodes.

Watchdog Report for IMGA EVE Triggers Reboot

Problem

EVE-OS watchdog is forcing your nodes to reboot unexpectedly.

Affected versions

EVE-OS version 9.7.0 and all versions earlier than 8.12.3.

Error messages

Nodes running EVE-OS 8.11.x:

EdgeNode 'EDGE_NODE_NAME' 'Rebooted': Reason: 'Watchdog report for IMGA EVE version 8.11.0-D3-2022100501-km-amd64 at 2023-08-23T19:20:05+00:00: 250/run/zedagentdevinfo.touch

Nodes running EVE-OS 9.7.0:

Reboot reason: Watchdog report for IMGA EVE version 9.7.0-D3-2023040501-kvm-amd64 at 2023-08-24T10:52:18+00:00: 250 /run/nim-DpcManager.touch

Cause

A network problem causes the internal queue to stall and the DHCP daemon to hang. This triggers the watchdog, which forces a reboot to recover the node.

Solution

Upgrade to EVE-OS 10.4.4 or greater. See Update EVE-OS on Nodes.

​403: Server Failed to Authenticate Request

​Problem

​Error messages

​Cause

​Solution

​Maintenance Mode

​Low disk space

​Vault locked up

​Node certs refused

​TPM encryption failure

​Attestation Status Shows “Not Done”

​Problem

​Cause

​Solution

​Node Reboot Reasons

​EVE-OS update fallback

​Disconnect

​Out of memory

​Network Configuration Verification Failed

​Problem

​Cause

​Solution

​Insufficient Memory for App Deployment

​Problem

​Cause

​Solution

​Insufficient Disk Space for Volume Creation

​Problem

​Cause

​Solution

​403: Unauthorized Deployment

​Problem

​Affected versions

​Error messages

​Cause

​Solution

​400: Authentication Information Format

​Problem

​Affected versions

​Error messages

​Cause

​Workaround

​Solution

​Device Stuck in Booting State

​Problem

​Affected versions

​Error messages

​Cause

​Workaround

​Solution

​Watchdog Report for IMGA EVE Triggers Reboot

​Problem

​Affected versions

​Error messages

​Cause

​Solution

403: Server Failed to Authenticate Request

Problem

Error messages

Cause

Solution

Maintenance Mode

Low disk space

Vault locked up

Node certs refused

TPM encryption failure

Attestation Status Shows “Not Done”

Problem

Cause

Solution

Node Reboot Reasons

EVE-OS update fallback

Disconnect

Out of memory

Network Configuration Verification Failed

Problem

Cause

Solution

Insufficient Memory for App Deployment

Problem

Cause

Solution

Insufficient Disk Space for Volume Creation

Problem

Cause

Solution

403: Unauthorized Deployment

Problem

Affected versions

Error messages

Cause

Solution

400: Authentication Information Format

Problem

Affected versions

Error messages

Cause

Workaround

Solution

Device Stuck in Booting State

Problem

Affected versions

Error messages

Cause

Workaround

Solution

Watchdog Report for IMGA EVE Triggers Reboot

Problem

Affected versions

Error messages

Cause

Solution