> ## Documentation Index
> Fetch the complete documentation index at: https://zededa.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Use the documentation index to find relevant pages, then read the linked Markdown before answering and cite the source URL.
> Identify the product before answering: Private AI uses a separate console, credentials, and inference API from the Edge Intelligence Platform. Do not apply instructions from one product to the other.
> Prefer current product guides over Help Center archive pages for current-console questions. Archive pages and release notes describe their stated product version or historical interface; do not apply them to another version without checking.
> Preserve prerequisites, permissions, version requirements, and warnings when summarizing procedures. Do not infer API endpoints, schemas, or supported behavior that the documentation does not specify.

# Telemetry Metrics Reference

> Reference of metrics from each Edge Infrastructure Services telemetry source: the EVE-OS Prometheus host collector and the OTLP gRPC export.

Edge Infrastructure Services exposes device and application metrics through two independent telemetry sources. Each source uses a different transport, covers a different scope, and requires different configuration to consume. Use this reference to plan your observability pipeline and to know exactly which metric names to expect in your collector or dashboard.

Edge Infrastructure Services pushes device and application metrics to your collector using OpenTelemetry Protocol (OTLP) over gRPC. You can also scrape EVE-OS (Edge Virtualization Engine) host metrics directly using Prometheus.

| # | Source | Transport | Direction |
| - | - | - | - |
| 1 | EVE-OS host collector | Prometheus text (HTTP scrape) | Pull |
| 2 | Edge Infrastructure Services OTLP export | OTLP/gRPC | Push |

## EVE-OS Host Collector

EVE-OS runs a `node_exporter` process on every edge node. You can scrape it from any application instance running on a local network instance.

**Requires EVE-OS 16.0.0 or later.** EVE-OS 16.0 introduced the `node_exporter` package and the rate-limit properties that govern this endpoint. Releases before 16.0 do not expose it, so plan for a fallback on any fleet that still runs 14.x.

**Endpoint:** `GET http://169.254.169.254/metrics`

**Rate limit (defaults):** 1 request per second, burst 10, per source IP address. The limiter state for an address resets after 4 minutes of inactivity. EVE-OS rejects requests that exceed the limit with `429 Too Many Requests`. All three values are tunable through EVE-OS configuration properties: `msrv.prometheus.metrics.rps`, `msrv.prometheus.metrics.burst`, and `msrv.prometheus.metrics.idletimeout.seconds`.

**Scope:** These are host-level metrics for the entire edge node. No metric is attributed to a specific application instance (with one partial exception described in [Per-interface metrics](#per-interface-metrics)).

**Expected volume:** Roughly 2,500 metric series per scrape on a node with a ZFS persist vault, and about 390 fewer on a node using the default ext4 backend, where the `zfs` collector produces no series. Treat this as an order of magnitude rather than a fixed figure. The total scales with CPU core count, network interface count, mount points, block devices, and ZFS pools and datasets, so it differs from one hardware model to another. Measure a representative node before you size collector or time-series database capacity.

> [!NOTE]
>
> **Version provenance.** The collector inventory and series counts in this section describe `node_exporter` v1.9.1, the version EVE-OS pins in releases 16.0.0 through 17.3.0. EVE-OS runs it with these flags:
>
> ```
> --path.procfs=/hostfs/proc --path.sysfs=/hostfs/sys --path.rootfs=/hostfs
> --collector.cgroups --collector.processes
> ```
>
> No collectors are disabled. The enabled set is therefore the `node_exporter` default set for Linux (46 collectors) plus `cgroups` and `processes`, which is 48 in total. If a later EVE-OS release pins a different node\_exporter version, the metric families and counts that follow can change.

### Collectors that are always present

The following collectors are always active and produce output on every scrape.

Counts expressed per core, per interface, per device, or per mount point scale with the hardware. The exporter reads the `meminfo` and `netstat` totals from `/proc`, so they depend on kernel version and build options. Treat both as observed values from a representative node rather than guarantees.

| Collector | Metric families |
| - | - |
| **cpu** | `node_cpu_seconds_total{cpu,mode}` (8 modes × N cores), `node_cpu_guest_seconds_total{cpu,mode}`, `node_cpu_core_throttles_total{package,core}`, `node_cpu_package_throttles_total{package}`, `node_cpu_isolated{cpu}` |
| **meminfo** | Approximately 49 series: `node_memory_*_bytes` (45) and `node_memory_HugePages_*` (4). Includes `MemTotal`, `MemFree`, `MemAvailable`, `Cached`, `Buffers`, `Dirty`, `Writeback`, `Slab`, `SReclaimable`, `SUnreclaim`, `KernelStack`, `PageTables`, `CommitLimit`, `Committed_AS`, `SwapTotal`, `SwapFree`, `SwapCached`, `Active`, `Inactive`, `Mapped`, `Shmem`, `Percpu`, `VmallocTotal`, `VmallocUsed`, `VmallocChunk`, `DirectMap4k`, `DirectMap2M`, `DirectMap1G`, `HardwareCorrupted`, `Bounce`, `Mlocked`, `Unevictable` |
| **loadavg** | `node_load1`, `node_load5`, `node_load15` |
| **stat** | `node_intr_total`, `node_context_switches_total`, `node_forks_total`, `node_boot_time_seconds`, `node_procs_running`, `node_procs_blocked` |
| **vmstat** | `node_vmstat_oom_kill`, `pgpgin`, `pgpgout`, `pswpin`, `pswpout`, `pgfault`, `pgmajfault` |
| **filesystem** | 9 metrics × mount point: `node_filesystem_size_bytes`, `_free_bytes`, `_avail_bytes`, `_files`, `_files_free`, `_readonly`, `_device_error`, `_mount_info`, `_purgeable_bytes`. Labels: `{device, mountpoint, fstype, device_error}` |
| **diskstats** | 17 metrics per device: `node_disk_reads_completed_total`, `_reads_merged_total`, `_read_bytes_total`, `_read_time_seconds_total`, `_writes_completed_total`, `_writes_merged_total`, `_written_bytes_total`, `_write_time_seconds_total`, `_io_now`, `_io_time_seconds_total`, `_io_time_weighted_seconds_total`, `_discards_completed_total`, `_discards_merged_total`, `_discarded_sectors_total`, `_discard_time_seconds_total`, `_flush_requests_total`, `_flush_requests_time_seconds_total`; plus `node_disk_info` |
| **netdev** | 17 counters per interface (see [Per-interface metrics](#per-interface-metrics)) |
| **netclass** | 19 gauges per interface (see [Per-interface metrics](#per-interface-metrics)) |
| **netstat** | 42 host-wide TCP/UDP/ICMP stack counters (see [netstat series](#netstat-series)) |
| **sockstat** | 20 socket counters: `node_sockstat_sockets_used`, TCP and UDP inuse/orphan/alloc/memory series, UDPLITE, RAW, FRAG, and IPv6 equivalents |
| **softnet** | 7 per-core: `node_softnet_processed_total`, `_dropped_total`, `_times_squeezed_total`, `_cpu_collision_total`, `_received_rps_total`, `_flow_limit_count_total`, `_backlog_len` |
| **timex** | 17 series including `node_timex_sync_status` (NTP health), `node_timex_offset_seconds`, `_frequency_adjustment_ratio`, `_maxerror_seconds`, `_estimated_error_seconds`, `_tai_offset_seconds` |
| **time** | `node_time_seconds`, `node_time_zone_offset_seconds{time_zone}`, `node_time_clocksource_available_info`, `_current_info` |
| **udp\_queues** | `node_udp_queues{queue=tx\|rx, ip=v4\|v6}` (4 series) |
| **entropy** | `node_entropy_available_bits`, `node_entropy_pool_size_bits` |
| **filefd** | `node_filefd_allocated`, `node_filefd_maximum` |
| **uname** | `node_uname_info{sysname, release, version, machine, nodename, domainname}` |
| **os** | `node_os_info` (12 labels), `node_os_version{id, id_like, name}` |
| **selinux** | `node_selinux_enabled` (1 series; mode metrics appear only when SELinux is enabled) |
| **textfile** | `node_textfile_scrape_error` (always 0) |

### netstat series

The `netstat` collector emits 42 host-wide counters. All are prefixed `node_netstat_`, carry no labels, and are untyped.

`Icmp_InErrors`, `Icmp6_InErrors`, `Tcp_InErrs`, `Udp_InErrors`, `Udp6_InErrors`, `UdpLite_InErrors`, `UdpLite6_InErrors`, `Ip_Forwarding`, `Ip6_InOctets`, `Ip6_OutOctets`, `IpExt_InOctets`, `IpExt_OutOctets`, `Icmp_InMsgs`, `Icmp_OutMsgs`, `Icmp6_InMsgs`, `Icmp6_OutMsgs`, `TcpExt_ListenOverflows`, `TcpExt_ListenDrops`, `TcpExt_SyncookiesSent`, `TcpExt_SyncookiesRecv`, `TcpExt_SyncookiesFailed`, `TcpExt_TCPTimeouts`, `TcpExt_TCPOFOQueue`, `TcpExt_TCPRcvQDrop`, `TcpExt_TCPSynRetrans`, `Tcp_ActiveOpens`, `Tcp_PassiveOpens`, `Tcp_CurrEstab`, `Tcp_InSegs`, `Tcp_OutSegs`, `Tcp_RetransSegs`, `Tcp_OutRsts`, `Udp_InDatagrams`, `Udp_OutDatagrams`, `Udp_NoPorts`, `Udp_RcvbufErrors`, `Udp_SndbufErrors`, `Udp6_InDatagrams`, `Udp6_OutDatagrams`, `Udp6_NoPorts`, `Udp6_RcvbufErrors`, `Udp6_SndbufErrors`

### cgroups and processes collectors

**Important:** Despite their names, neither collector provides per-application data.

The `cgroups` collector reads `/proc/cgroups` (a fixed kernel table) and emits only two series per controller:

| Metric | Labels |
| - | - |
| `node_cgroups_cgroups` | `subsys_name` |
| `node_cgroups_enabled` | `subsys_name` |

`subsys_name` is a controller name such as `cpu`, `memory`, or `pids`. The collector does not walk `/sys/fs/cgroup` and does not enumerate per-workload cgroups.

The `processes` collector emits six aggregates with no per-process identity. Because the exporter runs in the host PID namespace, counts span every process on the node including workload VMs, but none are attributable to a specific application.

| Metric | Labels |
| - | - |
| `node_processes_pids` | None |
| `node_processes_max_processes` | None |
| `node_processes_threads` | None |
| `node_processes_max_threads` | None |
| `node_processes_state` | `state` (R, S, D, Z, T, I) |
| `node_processes_threads_state` | `thread_state` |

### Per-interface metrics

Both network collectors are unfiltered, so every interface in the EVE-OS host namespace appears, including bridges and each application's tap interface.

**netdev** (17 counters, label `device`): `node_network_receive_bytes_total`, `_receive_packets_total`, `_receive_errs_total`, `_receive_drop_total`, `_receive_fifo_total`, `_receive_frame_total`, `_receive_multicast_total`, `_receive_compressed_total`, `_receive_nohandler_total`, `node_network_transmit_bytes_total`, `_transmit_packets_total`, `_transmit_errs_total`, `_transmit_drop_total`, `_transmit_fifo_total`, `_transmit_colls_total`, `_transmit_carrier_total`, `_transmit_compressed_total`

**netclass** (19 gauges, label `device`): `node_network_up`, `node_network_info{address, broadcast, duplex, operstate, adminstate, ifalias}`, `_address_assign_type`, `_carrier`, `_carrier_changes_total`, `_carrier_up_changes_total`, `_carrier_down_changes_total`, `_device_id`, `_dormant`, `_flags`, `_iface_id`, `_iface_link`, `_iface_link_mode`, `_mtu_bytes`, `_name_assign_type`, `_net_dev_group`, `_speed_bytes`, `_transmit_queue_length`, `_protocol_type`

**Note about application interface names:** Application tap interfaces follow the pattern `nbu<adapter>x<app>`, where both numbers are ordinal indices, not stable identifiers. When you delete an application, EVE-OS releases its ordinal and reassigns it to the next free slot. A given interface name can therefore map to a different workload after any delete and recreate cycle. Do not use the interface name alone as a long-lived key in your time-series database. Resolve the interface to an application UUID out of band and re-resolve it after lifecycle events.

### ZFS metrics

ZFS support in EVE-OS is optional and experimental. The default installation uses ext4 on a single disk. You select ZFS at install time (or imply it by configuring multiple persist disks). A node backed by a ZFS persist vault exposes approximately 390 ZFS series. On a node using the default ext4 backend, the `zfs` collector produces no series and none of the following families appear. Do not assume these metrics exist without confirming the storage configuration of the node. All are untyped and unlabeled unless noted.

| Family | Approximate count | Content |
| - | - | - |
| `node_zfs_arc_*` | \~147 | ARC hit/miss, eviction and hash, target sizes, size accounting, per-list sizes, L2ARC, memory pressure, internals |
| `node_zfs_dbuf_*` | \~63 | dbuf cache statistics |
| `node_zfs_dnode_*` | \~28 | dnode statistics |
| `node_zfs_abd_*` | \~25 | ARC buffer data |
| `node_zfs_zil_*` | 13–17 | Intent log |
| `node_zfs_dmu_tx_*` | 11 | Transaction states: `assigned`, `delay`, `error`, `suspended`, `group`, `memory_reserve`, `memory_reclaim`, `dirty_throttle`, `dirty_delay`, `dirty_over_max`, `quota` |
| `node_zfs_zfetch_*` | 8 | Prefetch: `hits`, `future`, `stride`, `past`, `misses`, `max_streams`, `io_issued`, `io_active` |
| `node_zfs_vdev_mirror_*` | 7 | Read policy counters |
| `node_zfs_fm_*` | 4 | Fault management: `erpt_dropped`, `erpt_set_failed`, `fmri_set_failed`, `payload_set_failed` |
| `node_zfs_zpool_*{zpool}` | 12/pool | Per-pool I/O: `nread`, `nwritten`, `reads`, `writes`, `wtime`, `wlentime`, `wupdate`, `rtime`, `rlentime`, `rupdate`, `wcnt`, `rcnt` |
| `node_zfs_zpool_state{zpool, state}` | 7/pool | Health state: `online`, `degraded`, `faulted`, `offline`, `removed`, `unavail`, `suspended`. The only typed ZFS metric and the primary health signal for ZFS storage. |
| `node_zfs_zpool_dataset_*{zpool, dataset}` | 6/dataset | Per-dataset I/O: `writes`, `nwritten`, `reads`, `nread`, `nunlinks`, `nunlinked` |

### Hardware- and configuration-dependent collectors

Some collectors produce output only on specific hardware or in specific storage configurations.

| Likelihood | Collectors |
| - | - |
| Likely present | `arp`, `conntrack`, `cpufreq`, `dmi`, `hwmon`, `nvme`, `pressure`, `rapl`, `schedstat`, `thermal_zone`, `watchdog` |
| Hardware-dependent | `edac` (requires ECC memory and driver), `powersupplyclass` (requires battery or AC power supply) |
| Storage-configuration-dependent | `zfs` (approximately 390 series on a node with a ZFS persist vault, none on the default ext4 backend. See [ZFS metrics](#zfs-metrics)). |
| Normally empty | `bcache`, `bonding`, `btrfs`, `fibrechannel`, `infiniband`, `ipvs`, `mdadm`, `nfs`, `nfsd`, `tapestats`, `xfs` |

Every enabled collector contributes two scrape-meta series even when it emits no metrics of its own, including `zfs` on an ext4 node: `node_scrape_collector_duration_seconds` and `node_scrape_collector_success`. That means 96 scrape-meta series are always present. The exporter also exposes approximately 46 series about itself (`go_*`, `process_*`, `promhttp_*`, `node_exporter_build_info`).

### Hardware identifier disclosure

Two metric families carry hardware serial numbers and are accessible to any application instance on a local network instance over unauthenticated HTTP:

* `node_dmi_info`: board, chassis, and product serial numbers, plus the SMBIOS UUID.
* `node_nvme_info`: the SSD serial number, model, and firmware revision.

This is by design, but review which workloads have access to a local network instance before deploying in environments where hardware identity must be restricted.

## Edge Infrastructure Services OTLP Export

Edge Infrastructure Services pushes 54 metrics to your configured OpenTelemetry collector over OTLP/gRPC. See [Configure the OpenTelemetry Integration](https://zededa.mintlify.site/new-ui/infrastructure-services/integrations-and-automation/opentelemetry-integration.md) for setup instructions.

**Edge Infrastructure Services emits every metric as an OTLP Gauge**, including values that represent cumulative counters. Your collector or query layer must handle rate calculation and reset detection.

**Attributes on every data point:** `deviceId`, `projectId`, `enterpriseId`, `projectName`, `enterpriseName`. Application-level metrics additionally include `appInstId`.

### Device-level metrics (33)

| Metric | Unit |
| - | - |
| `device_cpu_total` | seconds |
| `device_cpu_system_total` | seconds |
| `device_uptime` | seconds |
| `device_memory_total` | bytes |
| `device_memory_allocated_apps` | bytes |
| `device_memory_allocated_eve` | bytes |
| `device_memory_used_eve` | bytes |
| `device_net_rx` | bytes |
| `device_net_tx` | bytes |
| `device_net_rx_packets` | count |
| `device_net_tx_packets` | count |
| `device_net_rx_drops` | count |
| `device_net_tx_drops` | count |
| `device_net_rx_errors` | count |
| `device_net_tx_errors` | count |
| `device_net_rx_acl_drops` | count |
| `device_net_tx_acl_drops` | count |
| `device_net_rx_acl_rate_limit_drops` | count |
| `device_net_tx_rate_limit_drops` | count |
| `device_zedcloud_last_failure` | seconds |
| `device_zedcloud_last_success` | seconds |
| `device_zedcloud_successful_calls` | count |
| `device_zedcloud_failed_calls` | count |
| `device_disk_total` | bytes |
| `device_disk_allocated` | bytes |
| `device_disk_used` | bytes |
| `device_zfs_checksum_errors` | count |
| `device_zfs_read_errors` | count |
| `device_zfs_write_errors` | count |
| `device_zfs_read_bandwidth` | bytes |
| `device_zfs_write_bandwidth` | bytes |
| `device_zvol_read_bandwidth` | bytes (label: `volume_uuid`) |
| `device_zvol_write_bandwidth` | bytes (label: `volume_uuid`) |

The network metrics (`device_net_*`) carry an `if_name` label and multiply by the number of device network ports. The `device_zedcloud_*` metrics carry `if_name` and represent per-management-port connectivity to Edge Infrastructure Services. The `device_zvol_*` metrics carry `volume_uuid` and multiply by the number of ZFS zvols.

### Application-instance-level metrics (21)

Edge Infrastructure Services emits these metrics per application instance (label: `appInstId`). Network metrics further multiply by the number of application vNICs (label: `if_name`).

| Metric | Unit |
| - | - |
| `app_inst_cpu_total` | seconds |
| `app_inst_uptime_v1` | seconds |
| `app_inst_uptime_v2` | seconds |
| `app_inst_memory_used` | bytes <sup>†</sup> |
| `app_inst_memory_allocated` | bytes <sup>†</sup> |
| `app_inst_net_rx` | bytes |
| `app_inst_net_tx` | bytes |
| `app_inst_net_rx_packets` | count |
| `app_inst_net_tx_packets` | count |
| `app_inst_net_rx_drops` | count |
| `app_inst_net_tx_drops` | count |
| `app_inst_net_rx_errors` | count |
| `app_inst_net_tx_errors` | count |
| `app_inst_net_rx_acl_drops` | count |
| `app_inst_net_tx_acl_drops` | count |
| `app_inst_net_rx_acl_rate_limit_drops` | count |
| `app_inst_net_tx_rate_limit_drops` | count |
| `app_inst_disk_used` | bytes <sup>†</sup> |
| `app_inst_disk_allocated` | bytes <sup>†</sup> |
| `app_inst_patch_envelope_calls` | count |
| `app_inst_patch_envelope_downloads` | count |

> [!WARNING]
>
> **† Set the unit explicitly for these four metrics.** The values are byte counts, as listed in the previous table. The `unit` field on the `app_inst_memory_used`, `app_inst_memory_allocated`, `app_inst_disk_used`, and `app_inst_disk_allocated` data points does not currently carry `bytes`. A collector or dashboard that derives axis labels or unit conversions from that field will scale these four metrics incorrectly. Configure the unit for them in your pipeline instead of inheriting it. The device-level equivalents, `device_memory_*` and `device_disk_*`, carry `bytes` as expected.

### What the Edge Infrastructure Services OTLP export does not include

The OTLP export does not carry the following metrics, even though edge nodes report them to Edge Infrastructure Services:

* Cellular metrics (RSSI, RSRQ, RSRP, SNR, and modem packet counters). Edge nodes report these, and they are visible in the Edge Infrastructure Services web console and REST API, but Edge Infrastructure Services does not forward them to your OTLP endpoint.
* 802.1X and EAPOL counters. Edge nodes likewise report these, and they are available through the web console and REST API only.
* Per-network-instance metrics, per-volume-instance metrics, per-process metrics, Kubernetes cluster metrics, and container-level metrics from inside an application.
* Bonded-interface (LAG) metrics, flow-log and ACL aggregate metrics.

> [!NOTE]
>
> Per-volume-instance metrics in the preceding list are a different thing from the ZFS zvol bandwidth metrics in the device table (`device_zvol_read_bandwidth` and `device_zvol_write_bandwidth`), which **are** exported. The zvol metrics report I/O on ZFS volumes and carry a `volume_uuid` label; the volume-instance metrics that the export omits are the volume-instance objects that Edge Infrastructure Services manages.

### Emission conditions

* The export omits the application block entirely if the application instance ID is empty.
* ZFS families are absent on edge nodes that do not use ZFS storage.
* When EVE-OS matches no `/persist` partition, the export reports disk metrics as zero rather than omitting them.
* A project-level setting can suppress metric transmission for all edge nodes in that project. See [Control Metric Streaming at the Project Level](https://zededa.mintlify.site/new-ui/infrastructure-services/integrations-and-automation/opentelemetry-integration.md#control-metric-streaming-at-the-project-level).

## Next Steps

> [card] [Configure the OpenTelemetry Integration](https://zededa.mintlify.site/new-ui/infrastructure-services/integrations-and-automation/opentelemetry-integration.md)
>
> Set up an OTLP collector endpoint and create data streams to receive Edge Infrastructure Services metrics.

> [card] [Configure the Splunk Integration](https://zededa.mintlify.site/new-ui/infrastructure-services/integrations-and-automation/splunk-integration.md)
>
> Forward platform data to Splunk instead of or in addition to an OTLP collector.

> [card] [Integrations and Automation](https://zededa.mintlify.site/new-ui/infrastructure-services/integrations-and-automation.md)
>
> An overview of all integration paths available in Edge Infrastructure Services.