> ## Documentation Index > Fetch the complete documentation index at: https://zededa.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > Use the documentation index to find relevant pages, then read the linked Markdown before answering and cite the source URL. > Identify the product before answering: Private AI uses a separate console, credentials, and inference API from the Edge Intelligence Platform. Do not apply instructions from one product to the other. > Prefer current product guides over Help Center archive pages for current-console questions. Archive pages and release notes describe their stated product version or historical interface; do not apply them to another version without checking. > Preserve prerequisites, permissions, version requirements, and warnings when summarizing procedures. Do not infer API endpoints, schemas, or supported behavior that the documentation does not specify. # Manage Node Clusters > Use ZCLI to create, view, update, recover, upgrade, and delete EVE-OS node clusters, including two-node high-availability clusters. Use the ZEDEDA CLI (ZCLI) to manage node clusters from the command line. This article covers `zcli edge-node-cluster` commands for creating a cluster, checking its status, replacing a node, recovering a two-node high-availability (HA) cluster, upgrading EVE-OS (Edge Virtualization Engine) across the cluster, and deleting the cluster, with syntax and examples for each operation. A node cluster groups nodes so that application instances and their volumes fail over between them. A standard cluster uses three nodes. A two-node HA cluster uses two nodes plus a witness that supplies the third vote the cluster needs to stay available. See [Two-node HA clusters](#two-node-ha-clusters). If you want to manage ZEDEDA Edge Kubernetes Service clusters instead, see [Manage ZEDEDA Edge Kubernetes Service Clusters](https://zededa.mintlify.site/zcli-reference/kubernetes-clusters.md). ## Before You Start * ZCLI is installed and authenticated against your enterprise. See [Get Started](https://zededa.mintlify.site/zcli-reference/get-started.md). * You have the SysManager or SysAdmin role in your enterprise. See [Manage Roles](https://zededa.mintlify.site/new-ui/infrastructure-services/administration/manage-roles.md). * The nodes you plan to cluster are onboarded, online, in the same project, and running the EVE-k flavor of EVE-OS. See [Create Node Clusters](https://zededa.mintlify.site/new-ui/infrastructure-services/node-clusters/create-node-clusters.md) for the full hardware and software requirements. * For GUI-based cluster management, see [Create Node Clusters](https://zededa.mintlify.site/new-ui/infrastructure-services/node-clusters/create-node-clusters.md), [Manage Node Clusters](https://zededa.mintlify.site/new-ui/infrastructure-services/node-clusters/manage-node-clusters.md), and [Node Cluster Status and Health](https://zededa.mintlify.site/new-ui/infrastructure-services/node-clusters/node-cluster-status-and-health.md). ## Create a Node Cluster ### Create a node cluster Create a node cluster by naming it, assigning it to a project, and listing the member nodes. Each `--nodes` entry pairs a node name with the network interface that node uses for cluster traffic, separated by a colon. Repeat `--nodes` once for each member node. ```text theme={"dark"} zcli edge-node-cluster create --project= --nodes=... [--title=] [--description=<description>] [--tags=<key:value>...] [--cluster-prefix=<cluster-prefix>] [--ha-type=<ha-type>] ``` | Flag | Description | | - | - | | `--project=<project>` | Project the node cluster belongs to. List configured projects with `zcli project show`. | | `--nodes=<node_name:cluster_interface>` | A node name and the interface it uses for cluster traffic. Repeat the flag for each node. | | `--cluster-prefix=<cluster-prefix>` | Custom cluster network prefix. The default is `10.244.244.2/28`. | | `--ha-type=<ha-type>` | High-availability scheme for the cluster. `none` is the default and applies to a cluster with one node, or three or more nodes. `2node-ha` creates a two-node HA cluster and requires exactly two `--nodes` entries. | > [!NOTE] > > The syntax also lists `--title`, `--description`, and `--tags`. The current ZCLI release accepts these flags but does not send them to Edge Infrastructure Services, so they have no effect. The cluster title is set to the cluster name. Example: ```text theme={"dark"} zcli edge-node-cluster create MY_CLUSTER --project=MY_PROJECT --nodes=MY_NODE1:eth2 --nodes=MY_NODE2:eth2 --nodes=MY_NODE3:eth2 ``` ### Two-node HA clusters A two-node HA cluster runs on exactly two nodes. The cluster state is stored in etcd, a database that needs votes from a majority of its members to keep quorum. Two nodes alone cannot keep a majority if one fails, so a two-node HA cluster adds a witness: a lightweight member that runs only etcd and supplies the third vote. Edge Infrastructure Services places the witness automatically at an address at the top of the cluster prefix and runs it on the cluster's seed node. The `show` output identifies the seed node in the `Role` column. See [View node cluster configuration](#view-node-cluster-configuration). To create a two-node HA cluster, pass `--ha-type=2node-ha` with exactly two `--nodes` entries. ZCLI rejects the command if you pass a different number of nodes. Example: ```text theme={"dark"} zcli edge-node-cluster create MY_HA_CLUSTER --project=MY_PROJECT --nodes=MY_NODE1:eth2 --nodes=MY_NODE2:eth2 --ha-type=2node-ha ``` ## View a Node Cluster ### View node cluster configuration Run `show` without a name to list every node cluster in your enterprise. Pass a name or UUID to view the configuration of one cluster. ```text theme={"dark"} zcli edge-node-cluster show [<name> | --uuid=<uuid>] [--upgrade] ``` Example: ```text theme={"dark"} zcli edge-node-cluster show MY_CLUSTER ``` The cluster configuration includes the following high-availability details: * `HA Type` shows `none` or `2node-ha`. The cluster list shows the same value in the `HAType` column. * `Witness IP` shows the witness address for a two-node HA cluster. * `Quorum Recovery Generation` appears after a two-node HA cluster has been recovered with `2node-ha-reset`. See [Recover a Two-Node HA Cluster](#recover-a-two-node-ha-cluster). * The node table includes a `Role` column. The seed node shows `seed`, or `seed, witness` in a two-node HA cluster, because the witness runs on the seed node. Add `--upgrade` to view the EVE-OS upgrade progress for each member node instead of the cluster configuration. See [View upgrade status](#view-upgrade-status). ### View node cluster status View the runtime status of a node cluster, including the health of its members. Add `--raw` to print the unprocessed status that the cluster reports. ```text theme={"dark"} zcli edge-node-cluster status [<name> | --uuid=<uuid>] [--raw] ``` Example: ```text theme={"dark"} zcli edge-node-cluster status MY_CLUSTER ``` For a two-node HA cluster, the status also includes the following fields: * `HA State` shows the high-availability state of the cluster, for example `DEGRADED` after a recovery with `2node-ha-reset`. This field does not appear for other clusters. * `Witness IP` shows the witness address. * `Witness State` shows the state the witness reports. The value is `NOT REPORTED` until the witness reports a state, for example right after you enable two-node HA or recover the cluster. * `Witness Error` shows an error message when the witness reports one. For an explanation of the status values, see [Node Cluster Status and Health](https://zededa.mintlify.site/new-ui/infrastructure-services/node-clusters/node-cluster-status-and-health.md). ## Update a Node Cluster ### Replace a node in a cluster Adding a node to a cluster or removing a node from a cluster is not supported, except when you convert a single-node cluster to a two-node HA cluster or add a replacement node to a recovered two-node HA cluster. See [Convert a single-node cluster to two-node HA](#convert-a-single-node-cluster-to-two-node-ha). Otherwise, you can only replace one node with another. To replace a node, run `update` and pass the complete list of member nodes with the replacement node in place of the node you are removing. The `--nodes` flag is required, so include every node that should remain in the cluster. ```text theme={"dark"} zcli edge-node-cluster update <name> --nodes=<node_name:cluster_interface>... [--title=<title>] [--description=<description>] [--tags=<key:value>...] [--ha-type=<ha-type>] ``` | Flag | Description | | - | - | | `--nodes=<node_name:cluster_interface>` | A node name and the interface it uses for cluster traffic. Repeat the flag for each node that should be in the cluster after the update. | | `--ha-type=<ha-type>` | High-availability scheme for the cluster, `none` or `2node-ha`. Pass `2node-ha` to convert a single-node cluster to a two-node HA cluster. If you omit the flag, the cluster keeps its current scheme. | Example: ```text theme={"dark"} zcli edge-node-cluster update MY_CLUSTER --nodes=MY_NODE1:eth2 --nodes=MY_NODE2:eth2 --nodes=MY_NEW_NODE:eth2 ``` > [!NOTE] > > The syntax also lists `--title`, `--description`, and `--tags`. The current ZCLI release accepts these flags but does not send them to Edge Infrastructure Services, so they have no effect. An update keeps the existing title, description, and tags of the cluster. ### Convert a single-node cluster to two-node HA To turn a single-node cluster into a two-node HA cluster, run `update` with `--ha-type=2node-ha` and pass the existing node and the second node. Example: ```text theme={"dark"} zcli edge-node-cluster update MY_CLUSTER --nodes=MY_NODE1:eth2 --nodes=MY_NODE2:eth2 --ha-type=2node-ha ``` The following rules apply after a cluster is two-node HA: * The cluster keeps the two-node HA scheme whether or not you pass `--ha-type` on later updates. * The cluster cannot grow beyond two nodes. ZCLI rejects an update with any number of `--nodes` entries other than two, or one while the cluster is `DEGRADED` after a recovery. * You cannot change the scheme back with `--ha-type=none`. To change the scheme, delete the cluster and create it again. ### Override the Kubernetes version or configuration on a node The Kubernetes version and configuration overrides are EVE-OS configuration properties that you set on each node, not on the cluster. Use `zcli edge-node update` with the `--config` flag. * `k3s.version` sets the Kubernetes (K3s) version on the node. The value is a complete K3s version string, for example `v1.34.3+k3s1`. Supported with EVE-OS 16.7.0 and greater. * `k3s.config.override` adds options to the K3s configuration on the node. The value is a base64-encoded YAML file. Supported with EVE-OS 16.5.0 and greater. Apply either override only during a maintenance window with the node offline. Kubernetes terminates on the node immediately, downloads the new version or configuration, and restarts. Example: ```text theme={"dark"} zcli edge-node update MY_NODE1 --config="k3s.version:v1.34.2+k3s1" ``` ## Recover a Two-Node HA Cluster If the seed node of a two-node HA cluster fails and cannot come back, the cluster loses quorum. Use `2node-ha-reset` to recover the cluster on the remaining node. The command does the following: * Permanently removes the current seed node from the cluster. * Promotes the remaining node to seed and resets its etcd database to a single member. * Moves every application instance to the remaining node. * Moves the witness to the remaining node, where it switches to the other of the two addresses reserved for it at the top of the cluster prefix. The cluster stays two-node HA but is `DEGRADED`, with one physical node, until you add a replacement node. > [!WARNING] > > This command is destructive and cannot be undone. Power off the failed seed node before you run it. If the failed node comes back on the network while it still holds its old cluster state, it can disrupt the recovered cluster. ### Reset the quorum of a two-node HA cluster Run `2node-ha-reset` with the exact cluster name. ZCLI does not match partial names for this command, and it rejects clusters that are not two-node HA. ```text theme={"dark"} zcli edge-node-cluster 2node-ha-reset <name> [--force] ``` | Flag | Description | | - | - | | `--force` | Reset the cluster without asking for confirmation. | Example: ```text theme={"dark"} zcli edge-node-cluster 2node-ha-reset MY_HA_CLUSTER ``` Without `--force`, ZCLI prints a warning that names the node it will remove and the node it will promote, and then asks you to confirm with `Y` or `N`. You can run the command again on a cluster that has already been reset. Edge Infrastructure Services repeats the recovery on the remaining node without resetting it a second time, and no node is removed. ### Add a replacement node After the reset, run `update` with the remaining node and the replacement node to return the cluster to two nodes. Because the cluster is already two-node HA, `--ha-type` is optional. Example: ```text theme={"dark"} zcli edge-node-cluster update MY_HA_CLUSTER --nodes=MY_NODE2:eth2 --nodes=MY_NEW_NODE:eth2 ``` To confirm that the cluster has recovered, check `HA State` and `Witness State` with `zcli edge-node-cluster status`. See [View node cluster status](#view-node-cluster-status). ## Upgrade EVE-OS on a Cluster Upgrading EVE-OS on a node cluster is available only through ZCLI. The console no longer offers a cluster-level upgrade: the Release Manager does not list clusters, and the cluster detail page has no **Upgrade EVE-OS** action. Always upgrade a cluster with `zcli edge-node-cluster upgrade` rather than running `zcli edge-node eveimage-update` on each member. The cluster-level upgrade takes down one node at a time, so the remaining members keep serving application instances while each node drains, reboots, and rejoins the cluster. ### Upgrade a node cluster Start a rolling EVE-OS upgrade across every node in the cluster. The `--image` value is the name of the EVE-OS image to install. Cluster nodes require the EVE-k flavor of EVE-OS. ```text theme={"dark"} zcli edge-node-cluster upgrade <name> --image=<image> ``` Example: ```text theme={"dark"} zcli edge-node-cluster upgrade MY_CLUSTER --image=MY_EVE_IMAGE ``` An upgrade might take hours to complete. Before each node reboots, the cluster rebuilds any replicated volume that would be left with a single replica, and node operations are blocked while a clustered volume has no replica redundancy. For details, see [App Instance Failover in Clusters](https://zededa.mintlify.site/new-ui/infrastructure-services/node-clusters/node-cluster-failover.md). ### View upgrade status Check the progress of a cluster upgrade with `show --upgrade`. You must pass a cluster name or UUID. ```text theme={"dark"} zcli edge-node-cluster show [<name> | --uuid=<uuid>] [--upgrade] ``` Example: ```text theme={"dark"} zcli edge-node-cluster show MY_CLUSTER --upgrade ``` The response is a table with one row per member node and the columns `NodeID`, `UpgradeableEveOS`, `Status`, `CreatedAt`, and `UpdatedAt`. Rerun the command until every node reports a completed status. ## Delete a Node Cluster ### Delete a node cluster Delete a node cluster you no longer need. When you delete a cluster, every member node reboots and reconfigures itself as a standalone node. ```text theme={"dark"} zcli edge-node-cluster delete <name> ``` Example: ```text theme={"dark"} zcli edge-node-cluster delete MY_CLUSTER ``` ## Next Steps > [card] [Create Node Clusters](https://zededa.mintlify.site/new-ui/infrastructure-services/node-clusters/create-node-clusters.md) > > Requirements and GUI steps for creating a node cluster. > [card] [Manage Node Clusters](https://zededa.mintlify.site/new-ui/infrastructure-services/node-clusters/manage-node-clusters.md) > > GUI-based cluster management. > [card] [Node Cluster Status and Health](https://zededa.mintlify.site/new-ui/infrastructure-services/node-clusters/node-cluster-status-and-health.md) > > Interpret cluster and node status values. > [card] [Overview](https://zededa.mintlify.site/zcli-reference/overview.md) > > All ZCLI command procedures, organized by resource type.