Connect AI Cubes with ConnectX-7: plan a cluster

Start with one AI Cube Pro and expand when required? The AI Cube Pro is designed as a productive single-system starting point. For more users or larger models, we connect and configure additional AI Cubes for the measured workload. Explore the AI Cube Pro and expansion paths

GB10 systems can connect directly through ConnectX-7. This enables two distinct architectures: multiple model endpoints for more concurrent requests, or one model distributed across nodes. Calling both a “cluster” hides different goals and bottlenecks.

Hardware foundation

Each GB10-based AI Cube has 128 GB of unified memory, 10 GbE, and two QSFP connectors attached to the ConnectX-7 NIC. NVIDIA documents up to 200 Gbit/s per QSFP port and recommends compatible cables. These interfaces operate as Ethernet; they do not create shared memory between systems.

Two devices can use a direct cable. The current NVIDIA Sync Cluster Assistant also supports three systems connected directly and up to four through a switch. Larger or different topologies need a custom network design. Cabling, firmware, MTU, addressing, and topology must be documented.

Option A: more independent requests

Each AI Cube loads a model and serves its own requests. A load balancer sends new sessions or API calls to available endpoints. This increases aggregate throughput, allows different models per node, and can isolate failures.

Open WebUI, database, knowledge store, and session state still need a defined location. Duplicated inference alone is not a highly available platform.

Option B: one model across systems

If a model and KV cache do not fit usefully on one node, a supported runtime can distribute weights and computation. Nodes exchange intermediate results over ConnectX-7. Latency and throughput then depend on model architecture, parallelisation, runtime, and network.

Two devices do not become one computer with 256 GB of freely addressable memory. Each provides 128 GB coordinated by distributed software, so the exact model artefact must be tested.

Direct two-node connection

A production setup includes compatible operating-system, driver, and firmware levels; a validated QSFP cable; explicit IP addresses; connectivity and throughput tests; runtime configuration; workload measurement; and monitoring for nodes, network, model, and queues.

NVIDIA notes that one physical port can appear as several Linux interfaces because of the internal topology. Use the documented port mapping rather than guessing from interface names.

What NVIDIA Sync handles - and what it does not

The NVIDIA Sync Cluster Assistant discovers devices, creates a supported ConnectX-7 topology, applies network settings, validates links, and configures SSH between nodes. This establishes the communication layer.

It does not configure model distribution, NCCL or vLLM, container orchestration, load balancing, or high availability. Those layers still need to be designed and tested for the model and operating objective. A successful network test is not yet a working inference cluster.

Decision matrix

WZ-IT expansion path

With the AI Cube Pro, one local workload is first configured and measured. As use grows, we add AI Cubes for throughput or distributed models. When rack mounting, redundant power, several discrete GPUs, or stricter availability are needed, AI Cube Custom or a GPU server is the better path.

Before scaling out, define both the usage and load profile and the specific model artefact for 128 GB. Otherwise, hardware is connected before the actual capacity problem has been identified.

Sources

• NVIDIA: ConnectX-7 networking and clustering
• NVIDIA Sync: inspect a ConnectX-7 network
• NVIDIA: connect two DGX Spark systems
• NVIDIA: DGX Spark hardware