Benchmarks

Benchmarks measures how a model or route behaves on the selected Nexus machine. Use a quick check for everyday comparisons and the Cluster tab for a sustained distributed-inference test.

Benchmarks measures speed and responsiveness. It does not judge answer quality.

Run a quick check

1. Open Benchmarks > Quick Check.
2. Choose a model, preset, route, or other compatible target.
3. Choose a short, long, or context-heavy task.
4. Set Total replies and At once.
5. Select Run.

Nexus sends one warm-up request before the measured replies. The warm-up prepares a cold local model but is not included in the result.

At once is concurrency: 1 runs replies one by one; a higher value measures a busy period. Start with 1, then compare with a realistic concurrent load.

Running a cloud target can incur provider charges. A benchmark also creates real inference load, so avoid running it during latency-sensitive work.

Read the results

The result separates:

• First response: the typical wait before output starts.
• Reply speed: tokens per second after output starts.
• Completed replies: successful replies compared with the requested total.
• Concurrency and workload: the test conditions needed to reproduce the result.

An incomplete result includes only replies that finished. If timing was unavailable for some or all replies, response counts can still be valid while reply speed is blank.

Use History to reopen recent results, compare two compatible runs, repeat a setup, or export CSV. Quick-check history is stored in this browser for the selected machine and keeps the most recent results. Export anything you need to retain or share.

Compare fairly

Change one variable at a time:

• Keep the workload and concurrency fixed when comparing models.
• Keep the model, context, and hardware fixed when comparing runtime settings.
• Run once after a restart to see cold behavior and again to see warm behavior.
• Compare local results on an otherwise idle machine.

A faster reply can still be less useful, and a model with a slower first response can produce tokens quickly after it starts. Review both measurements.

Run cluster tests

When Cluster is available, the Cluster tab runs saved speed and capacity tests against an active formation.

• A speed test measures first-token delay and writing speed for the current model and cluster layout.
• A capacity test increases simultaneous replies and checks them against the minimum speed and maximum first-token delay you choose.

Run cluster tests while the formation is otherwise idle. They can take several minutes and create sustained load. If you change the model, context, participating Macs, distribution mode, or Thunderbolt fabric, run the test again.

The resulting capacity recommendation is evidence for the current setup, not a permanent limit. Applying a new MLX simultaneous-reply limit can restart the runtime and stop active local replies.

See Cluster for the operating workflow.

Troubleshoot a failed test

• Another benchmark is running: Wait for it to finish on this machine.
• Target disappeared: Refresh, then choose a current model or route.
• No speed sample: Confirm the target streams timing data; response duration may still be available.
• Local model failed to start: Open Models or Activity and resolve the model job first.
• Cluster test is unavailable: Confirm Cluster is entitled, formed, and running on the selected machine.
• Permission refused: Select a machine you can operate or ask an administrator for access.