Cluster

Run models no single Mac can carry. Msty Nexus Cluster combines multiple Apple Silicon machines into one distributed MLX inference system, making their memory and compute available to a shared workload.

Cluster is designed for larger local models, demanding private workloads, and teams that need more local inference capacity without moving sensitive work to an online provider.

Every participating node requires the explicitly granted Enterprise Cluster entitlement.

One model across several Macs

Cluster and Smart Balancer solve different scaling problems:

• Smart Balancer increases throughput by sending separate requests to destinations that can each run the model independently.
• Cluster divides one model execution across several Macs when no single participating machine should carry it alone.

A running cluster may later serve as one destination behind Smart Balancer. The cluster provides distributed execution; the balancer decides where requests should go.

What Cluster gives you

• Combined local capacity for MLX models that need more memory or compute than one Mac provides
• A single Nexus gateway target for applications using the distributed model
• Planned topology and preflight checks before model preparation or launch
• Peer-assisted model preparation that can avoid downloading the same immutable files separately on every machine
• Live operational visibility into members, ranks, memory, fabric activity, requests, and generation speed

Plan before changing machines

Use the Plan tab to estimate whether a model can run on one Mac or needs a cluster. Enter the model and candidate machine memory to project:

• The model footprint
• Pooled usable memory
• Estimated model share per node
• The minimum number of machines needed
• A recommended distribution mode

Pooled usable memory is not simply the advertised memory of every Mac added together. Nexus reserves headroom for macOS, Runtime, model execution, context, and safe operation.

The planner is a fit estimate, not a speed benchmark. A model can fit and still generate too slowly for the workload. Use the speed and capacity tests after formation to measure the real cluster.

Requirements

Prepare:

• Two or more supported Apple Silicon Macs
• Msty Nexus Runtime installed and running on every Mac
• All participating machines connected through Fleet or the guided Cluster join flow
• Thunderbolt connections between participating Macs
• RDMA enabled on each Mac
• A supported MLX model with enough combined memory
• Cluster entitlement installed on every node

Use a trusted local network. Nexus does not use SSH for the normal cluster workflow, and cluster membership does not make Runtime safe to expose broadly.

For the most reliable fabric:

• Connect the Macs with the Thunderbolt topology shown in Cluster setup.
• Enable RDMA on every participating Mac.
• Keep all machines awake, powered, and on stable network addresses.
• Avoid changing cables, network interfaces, or machine membership while a model is running.
• Use Macs with enough local storage for the selected model preparation mode.

Controller and members

A cluster has one controller and one or more members:

• The controller plans the topology, coordinates preparation, launches the distributed model, and exposes it through the Nexus gateway.
• Members run their assigned workers and report bounded operational telemetry to the controller.

Choose a controller that will remain available for the full time the cluster is serving requests.

The selected coordinator is rank 0 for the active formation. It loads the model, coordinates the other ranks, and provides the gateway route used for inference. If the coordinator stops, the running formation cannot continue serving normally.

Changing the coordinator, distribution mode, model, context length, or membership creates a different formation and requires the cluster to be prepared or started again.

Create a cluster

Cluster setup opens each step after the previous requirement is satisfied:

1. Open Cluster on the controller.
2. Add at least two entitled Apple Silicon Macs from Fleet or the guided join flow.
3. Run Discover fabric so Nexus can verify reachability, RDMA, and Thunderbolt links.
4. Choose an MLX model from the library or enter a compatible Hugging Face repository.
5. Select the coordinator, context length, distribution mode, and storage mode.
6. Run Preflight and resolve every blocking issue.
7. Select Provision runtimes to prepare MLX on every node.
8. Select Prepare model to place the required model files on every node.
9. Select Launch workers to start the ranks and form the cluster in the background.
10. Wait for the running state, then test speed and request capacity.

Use Fleet-enrolled machines whenever possible. Adding a machine from Fleet completes the trusted connection automatically and avoids manually sharing join credentials.

Discover the fabric

Fabric discovery checks whether every cluster member can participate in distributed inference:

• Every node is online and joined
• Nodes can reach one another
• The coordinator is online
• RDMA is enabled everywhere
• The expected Thunderbolt mesh is complete
• No other formation is being started or stopped

If a cable is added or moved after discovery, discover again so Nexus can rebuild the topology from the current links. Do not proceed on the assumption that a normal network connection can replace a missing RDMA-over-Thunderbolt link.

Choose a distribution mode

Distribution mode determines how model work and memory are divided.

Pipeline parallel

Each Mac owns part of the model and processes its assigned layers in sequence. Use pipeline parallelism when the model needs the pooled memory of the cluster and cannot fit comfortably on each Mac by itself.

Pipeline mode supports full-copy or sliced model storage. Its performance depends on a complete, stable fabric because prompt and generation work must move through the ranks.

Tensor parallel

Every Mac keeps the full model and shares the computation for each layer. Use tensor parallelism only when the model fits on every participating Mac.

Tensor mode requires a full model copy on every Mac. It can increase compute available to one generation, but each token requires coordination across the machines.

If the primary goal is to serve more independent requests rather than make one model execution larger or faster, use Smart Balancer instead.

Choose model and context

Cluster accepts supported MLX models from the Nexus library or a compatible Hugging Face repository. Preparation downloads the selected revision before workers start so the first application request does not wait for model transfer.

Context length affects memory use on every request. A model that fits with a shorter context may exceed safe memory at a larger context or with several simultaneous replies. Choose a realistic context for the applications that will call the cluster, then use Preflight and Capacity to verify it.

Run Preflight

Preflight is the launch gate. It checks:

• Nodes are online, joined, and mutually reachable
• The coordinator is available
• Fabric discovery has completed
• The Thunderbolt mesh is complete
• RDMA is enabled
• No formation change is already in progress
• The selected model fits pooled usable memory
• The model structure supports the chosen distribution and storage modes

A warning deserves review, but a failed required check blocks formation. Fix the affected machine or configuration and run Preflight again instead of repeatedly attempting launch.

Choose how models are stored

Cluster supports two preparation strategies:

• Full copy keeps a complete model snapshot on every participating Mac. It uses more storage but makes each node easier to reuse independently.
• Sliced weights keeps only the weight files assigned to each rank when the model layout supports safe slicing. It reduces repeated storage and transfer work.

Nexus can seed model files between enrolled peers so the same immutable file does not need to cross the internet once per machine. The source revision and file identity must match the planned snapshot.

Use Full copy when you value flexibility and easier role changes. Use Sliced when storage and preparation time matter more and the formation is expected to remain stable.

Sliced storage is available only with pipeline parallelism and supported sharded model layouts. If safe slicing is unavailable, Nexus uses a full copy and reports the effective storage mode rather than preparing an incomplete model.

Provision, prepare, and launch

These are separate operations so failures can be retried without repeating completed work:

• Provision runtimes installs the supported MLX runtime on nodes that need it.
• Prepare model downloads or receives the required model files and verifies that every rank has its expected data.
• Launch workers starts each rank and forms the distributed model.

Peer-assisted preparation allows one downloaded file to be shared across the trusted fabric rather than fetched from the internet separately by every Mac.

The coordinator must load rank 0 before the formation becomes ready. Large models can take several minutes to load. Launch workers runs this in the background, and Activity continues to show progress if you leave the Cluster page.

Operate the cluster

The Cluster workspace shows:

• Member readiness and assigned rank
• Model preparation and launch state
• Memory and GPU activity
• Thunderbolt fabric activity
• Request progress and generation speed
• Warnings that need operator action

An idle coordination heartbeat is normal. Investigate sustained saturation, missing members, repeated launch failures, or a topology that no longer matches the connected hardware.

Measure cluster speed

Use Test speed after formation to measure:

• Writing speed in tokens per second
• Time until the first generated token
• Performance across the current number of Macs

The test uses a short request against the active formation. Run it again after changing the model, membership, context, distribution mode, or fabric.

Find the request limit

The Capacity tab measures how many replies the formation can sustain while meeting the response targets you choose.

1. Set the minimum acceptable per-stream speed.
2. Set the maximum acceptable time to first token.
3. Run the capacity test while the cluster is otherwise idle.
4. Review aggregate throughput and per-stream results at each concurrency level.
5. Apply the recommended simultaneous-reply limit when it matches your workload.

The test takes several minutes and creates sustained inference load. Running it alongside real traffic can slow those requests and distort the result.

Applying a new MLX request limit restarts Runtime. Active local replies stop during that restart, so schedule the change accordingly.

Understand degraded states

A cluster is degraded when one or more ranks are no longer participating. Distributed inference depends on the active formation, so losing a required member can interrupt requests rather than merely reduce capacity.

If the cluster is running but the gateway route has not reconnected, wait for automatic recovery or use Reconnect now. If a node or fabric link is missing, stop traffic, restore the machine or cable, discover the fabric again, and reform the cluster.

Stop active traffic before changing membership, storage mode, or topology. Removing a member while a distributed model is active interrupts that model. Leaving a cluster removes the membership relationship but does not erase unrelated Runtime state, external caches, or unmanaged files.

Keep the cluster private

Cluster uses authenticated Runtime-to-Runtime control and a separate distributed MLX transport on operator-selected interfaces. Keep controller and member APIs on trusted addresses; entitlement is not permission to expose Runtime publicly.

Use Fleet enrollment and the Console-brokered join flow so node credentials do not need to be copied between machines manually.