Skip to main content
Use the ZEDEDA CLI (ZCLI) to manage node clusters from the command line. This article covers zcli edge-node-cluster commands for creating a cluster, checking its status, replacing a node, recovering a two-node high-availability (HA) cluster, upgrading EVE-OS (Edge Virtualization Engine) across the cluster, and deleting the cluster, with syntax and examples for each operation. A node cluster groups nodes so that application instances and their volumes fail over between them. A standard cluster uses three nodes. A two-node HA cluster uses two nodes plus a witness that supplies the third vote the cluster needs to stay available. See Two-node HA clusters. If you want to manage ZEDEDA Edge Kubernetes Service clusters instead, see Manage ZEDEDA Edge Kubernetes Service Clusters.

​
Before You Start

​
Create a Node Cluster

​
Create a node cluster

Create a node cluster by naming it, assigning it to a project, and listing the member nodes. Each --nodes entry pairs a node name with the network interface that node uses for cluster traffic, separated by a colon. Repeat --nodes once for each member node.
The syntax also lists --title, --description, and --tags. The current ZCLI release accepts these flags but does not send them to Edge Infrastructure Services, so they have no effect. The cluster title is set to the cluster name.
Example:

​
Two-node HA clusters

A two-node HA cluster runs on exactly two nodes. The cluster state is stored in etcd, a database that needs votes from a majority of its members to keep quorum. Two nodes alone cannot keep a majority if one fails, so a two-node HA cluster adds a witness: a lightweight member that runs only etcd and supplies the third vote. Edge Infrastructure Services places the witness automatically at an address at the top of the cluster prefix and runs it on the cluster’s seed node. The show output identifies the seed node in the Role column. See View node cluster configuration. To create a two-node HA cluster, pass --ha-type=2node-ha with exactly two --nodes entries. ZCLI rejects the command if you pass a different number of nodes. Example:

​
View a Node Cluster

​
View node cluster configuration

Run show without a name to list every node cluster in your enterprise. Pass a name or UUID to view the configuration of one cluster.
Example:
The cluster configuration includes the following high-availability details:
  • HA Type shows none or 2node-ha. The cluster list shows the same value in the HAType column.
  • Witness IP shows the witness address for a two-node HA cluster.
  • Quorum Recovery Generation appears after a two-node HA cluster has been recovered with 2node-ha-reset. See Recover a Two-Node HA Cluster.
  • The node table includes a Role column. The seed node shows seed, or seed, witness in a two-node HA cluster, because the witness runs on the seed node.
Add --upgrade to view the EVE-OS upgrade progress for each member node instead of the cluster configuration. See View upgrade status.

​
View node cluster status

View the runtime status of a node cluster, including the health of its members. Add --raw to print the unprocessed status that the cluster reports.
Example:
For a two-node HA cluster, the status also includes the following fields:
  • HA State shows the high-availability state of the cluster, for example DEGRADED after a recovery with 2node-ha-reset. This field does not appear for other clusters.
  • Witness IP shows the witness address.
  • Witness State shows the state the witness reports. The value is NOT REPORTED until the witness reports a state, for example right after you enable two-node HA or recover the cluster.
  • Witness Error shows an error message when the witness reports one.
For an explanation of the status values, see Node Cluster Status and Health.

​
Update a Node Cluster

​
Replace a node in a cluster

Adding a node to a cluster or removing a node from a cluster is not supported, except when you convert a single-node cluster to a two-node HA cluster or add a replacement node to a recovered two-node HA cluster. See Convert a single-node cluster to two-node HA. Otherwise, you can only replace one node with another. To replace a node, run update and pass the complete list of member nodes with the replacement node in place of the node you are removing. The --nodes flag is required, so include every node that should remain in the cluster.
Example:
The syntax also lists --title, --description, and --tags. The current ZCLI release accepts these flags but does not send them to Edge Infrastructure Services, so they have no effect. An update keeps the existing title, description, and tags of the cluster.

​
Convert a single-node cluster to two-node HA

To turn a single-node cluster into a two-node HA cluster, run update with --ha-type=2node-ha and pass the existing node and the second node. Example:
The following rules apply after a cluster is two-node HA:
  • The cluster keeps the two-node HA scheme whether or not you pass --ha-type on later updates.
  • The cluster cannot grow beyond two nodes. ZCLI rejects an update with any number of --nodes entries other than two, or one while the cluster is DEGRADED after a recovery.
  • You cannot change the scheme back with --ha-type=none. To change the scheme, delete the cluster and create it again.

​
Override the Kubernetes version or configuration on a node

The Kubernetes version and configuration overrides are EVE-OS configuration properties that you set on each node, not on the cluster. Use zcli edge-node update with the --config flag.
  • k3s.version sets the Kubernetes (K3s) version on the node. The value is a complete K3s version string, for example v1.34.3+k3s1. Supported with EVE-OS 16.7.0 and greater.
  • k3s.config.override adds options to the K3s configuration on the node. The value is a base64-encoded YAML file. Supported with EVE-OS 16.5.0 and greater.
Apply either override only during a maintenance window with the node offline. Kubernetes terminates on the node immediately, downloads the new version or configuration, and restarts. Example:

​
Recover a Two-Node HA Cluster

If the seed node of a two-node HA cluster fails and cannot come back, the cluster loses quorum. Use 2node-ha-reset to recover the cluster on the remaining node. The command does the following:
  • Permanently removes the current seed node from the cluster.
  • Promotes the remaining node to seed and resets its etcd database to a single member.
  • Moves every application instance to the remaining node.
  • Moves the witness to the remaining node, where it switches to the other of the two addresses reserved for it at the top of the cluster prefix.
The cluster stays two-node HA but is DEGRADED, with one physical node, until you add a replacement node.
This command is destructive and cannot be undone. Power off the failed seed node before you run it. If the failed node comes back on the network while it still holds its old cluster state, it can disrupt the recovered cluster.

​
Reset the quorum of a two-node HA cluster

Run 2node-ha-reset with the exact cluster name. ZCLI does not match partial names for this command, and it rejects clusters that are not two-node HA.
Example:
Without --force, ZCLI prints a warning that names the node it will remove and the node it will promote, and then asks you to confirm with Y or N. You can run the command again on a cluster that has already been reset. Edge Infrastructure Services repeats the recovery on the remaining node without resetting it a second time, and no node is removed.

​
Add a replacement node

After the reset, run update with the remaining node and the replacement node to return the cluster to two nodes. Because the cluster is already two-node HA, --ha-type is optional. Example:
To confirm that the cluster has recovered, check HA State and Witness State with zcli edge-node-cluster status. See View node cluster status.

​
Upgrade EVE-OS on a Cluster

Upgrading EVE-OS on a node cluster is available only through ZCLI. The console no longer offers a cluster-level upgrade: the Release Manager does not list clusters, and the cluster detail page has no Upgrade EVE-OS action. Always upgrade a cluster with zcli edge-node-cluster upgrade rather than running zcli edge-node eveimage-update on each member. The cluster-level upgrade takes down one node at a time, so the remaining members keep serving application instances while each node drains, reboots, and rejoins the cluster.

​
Upgrade a node cluster

Start a rolling EVE-OS upgrade across every node in the cluster. The --image value is the name of the EVE-OS image to install. Cluster nodes require the EVE-k flavor of EVE-OS.
Example:
An upgrade might take hours to complete. Before each node reboots, the cluster rebuilds any replicated volume that would be left with a single replica, and node operations are blocked while a clustered volume has no replica redundancy. For details, see App Instance Failover in Clusters.

​
View upgrade status

Check the progress of a cluster upgrade with show --upgrade. You must pass a cluster name or UUID.
Example:
The response is a table with one row per member node and the columns NodeID, UpgradeableEveOS, Status, CreatedAt, and UpdatedAt. Rerun the command until every node reports a completed status.

​
Delete a Node Cluster

​
Delete a node cluster

Delete a node cluster you no longer need. When you delete a cluster, every member node reboots and reconfigures itself as a standalone node.
Example:

​
Next Steps

Create Node Clusters

Requirements and GUI steps for creating a node cluster.

Manage Node Clusters

GUI-based cluster management.

Node Cluster Status and Health

Interpret cluster and node status values.

Overview

All ZCLI command procedures, organized by resource type.