Skip to main content
When a node in a cluster becomes unavailable, Edge Infrastructure Services automatically manages application placement to minimize downtime. The behavior differs depending on whether the outage is planned or unexpected.

Planned Maintenance

Planned maintenance covers three user-initiated operations on a cluster node: reboots, shutdowns, and EVE-OS upgrades initiated from the console or ZCLI. The cluster handling for all three is identical: the node is drained of workloads before the operation continues. The drain process stops all application instances on the node and allows them to be scheduled on another node in the cluster. There can be a delay in this process if a clustered volume instance is degraded. For example, if one or more replicas need to be rebuilt following a recent unplanned outage on a different node. In that case, the drain blocks the current node until the volume instance has been restored to full health by reading data off the remaining healthy replicas. The time required for the drain and any rebuild varies based on cluster node disk I/O performance, inter-node network performance, and the amount of data to be replicated. If no volume rebuild is needed, application instances fail over within a few minutes of the maintenance request.

Reboots and shutdowns

The drain process begins immediately. Application instances can expect approximately 1–2 minutes of downtime. Actual downtime varies depending on hardware and the health of the cluster volume instance for each application.

EVE-OS updates

The drain process begins after the EVE-OS update image has been downloaded to the node locally, verified successfully, and activated from the console. Application instance downtime follows existing EVE-OS expectations for systems not part of a cluster.

Application Failback

Failback is the process by which applications return to their original node after it comes back online. The cluster identifies all application instances not running on their originally preferred node and reschedules them back to that node. Failback begins automatically after a cluster node completes its boot process and rejoins the cluster.

Unexpected Outages

For unexpected node outages (hardware failures, power loss, network disconnections) failover is intentionally delayed to avoid unnecessary disruption from brief, intermittent connectivity issues. The cluster first detects the node as unreachable after a minimum of 1 minute. This status may take an additional minute or more to appear in the console due to EVE-OS status reporting frequency. If the node remains unreachable for more than 5 minutes, the cluster begins the failover process for all application instances running on that node. The 5-minute threshold prevents application restarts during brief inter-node network interruptions.

Next Steps

Manage clusters

Cluster lifecycle operations including node replacement.

Clusters Overview

Background on cluster types and automatic application migration.

Create clusters

Cluster creation prerequisites and setup steps.