Skip to main content

​
Introduction

The on-device benchmarking feature validates ML model performance by running inference workloads on real edge hardware. Each benchmark uses the Helm chart configured in the target device pool, which deploys to ZEDEDA Edge Kubernetes Service clusters associated with that device pool. After creating a benchmark, use the benchmark run detail page to monitor progress and validate that the model meets your latency and throughput requirements before deployment. This article covers the steps that follow benchmark creation. For device pool setup and benchmark creation steps, see Benchmark Models and Manage Device Pools.

​
Open a Benchmark Run

  1. In the bottom navigation bar, select Inference.
  2. In the sidebar, select Inference > Deploy > Evaluation.
  3. Select the Benchmarks tab.
  4. Select the benchmark name to open the detail page.

​
Benchmark Status Flow

The benchmark detail page displays the current status of the run. The status progresses through the following stages: When the status is Queued, Pending, Scheduled, Deploying, or Running, the page polls for updates every 10 seconds and displays an animated refresh indicator. A blue banner reads: “Benchmark is currently running. Results will appear as they become available.” If the benchmark encountered an error, a red banner displays the error message.

​
Benchmark Detail Page

The detail page has three tabs: Overview, Performance, and Results.

​
Overview tab

The Overview tab is available at all stages of the benchmark run. It contains the following sections. Status ribbon The status ribbon at the top of the page shows:
  • Current status.
  • Progress percentage.
  • Number of results received versus results expected.
  • Elapsed duration.
  • Number of devices and models included in the run.
Evaluation details The Evaluation Details card shows the benchmark configuration as entered when the benchmark was created. Progress and configuration This section displays the run parameters: Models under test Lists the model names and versions included in the benchmark. Target devices and clusters Lists the device pools and ZEDEDA Edge Kubernetes Service clusters running the benchmark workload. Device and version status table Displays live per-device progress. Each row represents one device/model combination and shows a stage badge indicating its current state.

​
Performance tab

The Performance tab becomes available after the benchmark completes or partially completes. It displays latency and throughput charts organized by model version and device type, letting you visually compare configurations.

​
Results tab

The Results tab shows result data tables with the following metrics per model/device combination: Latency (milliseconds) Throughput
  • Frames per second (fps) for computer vision models.
  • Tokens per second (tps) for generative AI models.
Resource utilization
  • CPU usage (%).
  • Memory (MB).
  • GPU memory (MB).
Reliability
  • Error rate.
Power and thermal
  • Average and peak power (watts).
  • Temperature readings (°C).

​
Interpret Results

Use the following guidance to evaluate whether a model/device combination is suitable for production deployment.
  • Compare P99 latency against your application’s maximum tolerable response time. P99 latency represents the worst-case experience for nearly all users.
  • Throughput (fps or tps) should meet the expected peak concurrent request load for your use case.
  • An error rate above 0% might indicate that the model or hardware combination needs investigation before deployment.
  • GPU memory usage should stay below device capacity to avoid out-of-memory failures under load.
  • Use the device comparison view to select the right hardware profile for production deployment.

​
Cancel or Delete a Benchmark Run

Cancel a running benchmark Select Cancel from the actions menu on the benchmark detail page or the Benchmarks list. Partial results collected before cancellation are preserved and visible on the Results tab. Delete a benchmark run Select Delete from the actions menu to permanently remove the benchmark run and all associated results. This action cannot be undone. You cannot delete a run in the Pending, Queued, Scheduled, Deploying, or Running status; cancel it first.

​
Troubleshoot a Failed Benchmark

If a benchmark shows a Failed status or a red error banner, work through the following checks in order.
  1. Note the error message in the red banner at the top of the detail page.
  2. Open Infrastructure > Fleet > Kubernetes and verify that the ZEDEDA Edge Kubernetes Service clusters listed in the Device Pool are healthy and reachable.
  3. Confirm that the model version is available in your model library and that the provider connection is working.
  4. Try re-creating the benchmark with a shorter duration or smaller batch size to isolate the issue.
For additional troubleshooting steps, see Inference Troubleshooting.

​
In This Section