Smart Balancer

Pro Enterprise

Turn scattered inference capacity into one dependable endpoint. Smart Balancer distributes requests across compatible models and machines, keeps throughput steady, and steers around destinations that are busy, draining, or unavailable.

Your applications do not need to know the fleet topology. They call one pool address:

@pool/<slug>

What changes for your applications

Without a balancer, each client must be tied to a specific model endpoint and handle failures itself. With Smart Balancer:

• Applications use one stable model name even as capacity changes.
• Healthy destinations can share concurrent work.
• Busy or unavailable destinations stop receiving new assignments.
• Operators can drain capacity for maintenance without abruptly interrupting current requests.
• Request history explains where work went and why.

Smart Balancer improves the reliability of capacity you already have. It does not merge model weights: every destination in a pool must independently serve a compatible model. Use Cluster when several Macs must cooperate to run one model.

The balancing workspace

Smart Balancer brings together:

• Pools: stable model addresses backed by multiple compatible destinations
• Nodes: Nexus machines or authenticated compatible endpoints that provide capacity
• Deployments: model availability across managed Nexus machines
• Live: active work, waiting work, throughput, and destination health
• History: bounded decision traces for destination selection and failover

Understand pools, targets, and nodes

• A pool is the stable @pool/<slug> model name used by applications.
• A target is one model destination inside that pool.
• A node is another Nexus machine that makes its models available to the controller.
• The controller is the Nexus machine that receives the application request, manages the queue, and selects the target.

One pool can include local runtimes, models on other Nexus machines, and compatible provider endpoints. Each target keeps its own model, capacity, health, and credentials.

Prepare capacity

Before creating a pool, make sure you have:

1. At least one reachable Nexus machine or compatible inference endpoint.
2. A compatible model available on every destination you want to include.
3. A client token or node credential scoped only to the access that connection requires.

Connect multiple Nexus machines through Fleet first. Prefer a trusted LAN or private VPN, and verify every advertised address before adding it as capacity.

Create a pool

1. Open Smart Balancer.
2. Add or select the nodes that will provide capacity.
3. Give the pool a recognizable name and stable slug.
4. Choose a compatibility mode.
5. Add one target for each model destination.
6. Choose a balancing strategy.
7. Review target priority and capacity weight.
8. Validate the pool and run a dry run.
9. Create the pool and use its @pool/<slug> value as the application's model name.

Applications can call the pool directly, or a Smart Route can select it as a target.

Choose a compatibility mode

Compatibility mode defines what the pool promises when it moves a request between targets.

Exact replicas

Use Exact replicas when every target serves the same model. This is the safest choice for distributing load across copies of one local model or one consistent deployment.

Nexus may ask you to confirm models that report the same identity but cannot prove their underlying builds are identical. Confirm only when you control the copies and know they are interchangeable.

Equivalent generation

Use Equivalent generation when targets may serve different models that you consider acceptable for the same generation workload.

Equivalent models can produce different answers, latency, token usage, context behavior, and safety characteristics. Use this mode deliberately and verify that every target supports the request types applications will send. It is not a substitute for Smart Routes when different kinds of work should intentionally use different models.

Choose a balancing strategy

Smart is the recommended default. Other strategies are useful when you need a simpler or more deterministic policy.

Unavailable, incompatible, draining, or unhealthy targets are skipped regardless of strategy.

Configure priority and capacity

Priority tiers

Priority controls which group of targets Nexus prefers:

• Priority 1 is the primary tier.
• Targets with the same priority share traffic.
• Higher-numbered tiers act as failover when lower-numbered targets are not eligible or do not have room.

Use equal priorities for active-active capacity. Use higher numbers for standby machines or more expensive provider endpoints that should be reached only when primary capacity cannot serve the request.

Capacity weight

Capacity weight controls both how many requests a target should serve simultaneously and its relative share inside one priority tier. A weight-4 target receives roughly twice the opportunity of a comparable weight-2 target.

Match capacity to the target's real parallelism:

• Keep the detected value when it reflects the runtime accurately.
• Lower it for a workstation that has other duties, long-context workloads, or a machine also participating in Cluster.
• Set hosted endpoints according to the provider capacity available to your account.
• Avoid setting it above a local runtime's parallelism. Extra work will wait at that runtime instead of in the pool, where Nexus can make failover decisions.

Capacity is a scheduling limit, not a promise that every workload will sustain that many fast responses. Watch live throughput and adjust after representative traffic.

Queueing, affinity, and retries

When every eligible target is occupied, Smart Balancer can hold requests in a bounded pool queue. Waiting work is admitted fairly across client tokens so one busy application does not consume another application's entire pending allowance.

The displayed waiting count is operational guidance, not a guaranteed queue position. Client cancellation, target recovery, request limits, and timeouts can all change admission order.

For conversations, Nexus can keep related turns on the same target to preserve model warmth and reusable prompt state. Send a consistent X-Msty-Nexus-Session-Id, prompt_cache_key, or supported user value when the client can provide one. Affinity yields when the previous target is materially busier or unavailable.

Retries are conservative. By default, Nexus makes no more than two attempts and can choose another target only before output has started. It does not replay ambiguous timeouts or reset a response after content has already reached the application.

Stay responsive under changing load

Smart Balancer uses live operational state to make conservative placement decisions:

• Destination readiness
• Current in-flight and queued work
• Advertised request capacity
• Recent failures and recovery
• Conversation affinity and warm-target usefulness

Retries are bounded. Nexus avoids repeatedly sending the same request through a failing fleet and exposes the decision path in request history.

The Live status page shows active and waiting work, ready and degraded destinations, tokens per second, pool health, and routing decisions as they happen. The Request history page explains:

• Which target was selected
• Why it ranked highest
• Which targets were skipped and why
• Queue time and time to first output
• Whether another target was attempted
• The final completion or failure reason

History contains routing metadata, not prompt or response content.

Deploy models to nodes

Use Deployments when several nodes need the same model before they can join a pool.

1. Choose the target nodes.
2. Select the runtime and model source.
3. Let Nexus install the runtime when needed.
4. Start the deployment and follow each node's progress.
5. Retry only the nodes that failed.
6. Confirm the model appears in node inventory before adding it to a pool.

A deployment prepares capacity; it does not automatically add the resulting models to an existing pool.

Maintain capacity without surprises

Drain a pool or destination before maintenance. Draining prevents new assignments while allowing active requests to finish.

Wait for draining to complete before restarting a Runtime, replacing a model, or taking a machine offline. Resume the destination only after it is healthy again.

Before relying on a pool:

• Validate checks target availability, model compatibility, and configuration warnings.
• Dry run checks how the configured strategy would rank the available targets without changing the pool's traffic path.
• Try traffic sends representative requests through the pool so you can watch live balancing.
• Failure drill measures traffic before, during, and after one target is drained, then reports whether the pool routed around it and recovered.

Traffic tests and failure drills create real inference work. Run them only where model usage, data handling, and provider cost are acceptable.

Keep credentials at the edge

Smart Balancer does not distribute upstream provider keys to callers or other machines. Each destination keeps its own credentials, while applications authenticate to the controller with a Nexus client token.

Verify machine identity, transport, advertised address, and token scope before adding any destination. See Security for network exposure guidance.