Multi-GPU inference without NVLink: tensor parallelism, pipeline parallelism or replicas

Larger model or more throughput than one GPU can deliver? WZ-IT runs managed GPU servers with NVIDIA RTX PRO 4000 Blackwell (24 GB) and RTX PRO 6000 Blackwell Max-Q (96 GB), including vLLM, Open WebUI and monitoring. We plan multi-GPU configurations to match model and load. View GPU servers · Book a call

A server with two GPUs does not automatically deliver twice the performance. What matters is how the inference software distributes the model across the cards and how much bandwidth is available between the cards. Data centre GPUs such as the H100 or B200 are coupled via NVLink. The RTX PRO Blackwell cards found in many business servers communicate over PCIe only. That shifts the trade-off between tensor parallelism, pipeline parallelism and several independent instances (replicas). This article explains the three approaches, the corresponding vLLM parameters and a decision rule for systems without NVLink. As of October 2026.

Table of contents

• NVLink and PCIe on RTX PRO Blackwell
• Three ways to put one model on several GPUs
• Why tensor parallelism gets expensive without NVLink
• The decision rule: split as little as possible
• vLLM configuration for two GPUs
• What a public measurement shows
• Check and measure the topology

NVLink and PCIe on RTX PRO Blackwell

NVLink is NVIDIA's direct GPU-to-GPU interconnect. According to NVIDIA, the fifth generation (Blackwell) reaches 1,800 GB/s per GPU and the fourth generation (Hopper) 900 GB/s (NVIDIA, NVLink). For comparison, NVIDIA lists 128 GB/s for the PCIe Gen5 connection of an H100 (NVIDIA, H100). That figure covers both directions of an x16 link combined.

The RTX PRO Blackwell cards in the following table have no NVLink connector. Their datasheets list PCIe as the only system interface:

The order of magnitude matters: inside an RTX PRO 6000, memory moves 1,792 GB/s, while PCIe 5.0 x16 between two cards provides a calculated 60-plus GB/s per direction. Any data exchange between the GPUs is therefore more than an order of magnitude slower than an access to local memory. Whether the cards in a server are actually connected with full lane width and without a detour through the second CPU depends on the mainboard (see Check the topology).

Three ways to put one model on several GPUs

vLLM offers three basic approaches that can be combined (vLLM, Parallelism and Scaling):

All of these parameters default to 1 or are off by default in vLLM (vLLM, Engine Arguments). The number of GPUs required is the product: -dp 2 -tp 2 needs four GPUs.

Tensor parallelism is the approach most guides mention first. The vLLM documentation recommends it when a model does not fit on one GPU but fits on one server.

Pipeline parallelism places, for example, the first half of the layers on GPU 0 and the second half on GPU 1. Only the intermediate results at the handover point travel between the cards. A single request does not get faster, because the GPUs work one after the other. With many concurrent requests, both stages work in parallel on different requests.

Replicas do not split the model; they are several complete instances. vLLM provides --data-parallel-size for this, with a shared API endpoint and internal load balancing; it works for dense and MoE models (vLLM, Data Parallel Deployment). Alternatively, two separate vLLM processes run behind a gateway such as LiteLLM.

Expert parallelism only concerns mixture-of-experts models. The experts are distributed across the GPUs instead of slicing every expert with TP; the EP size is TP times DP (vLLM, Expert Parallel Deployment).

Why tensor parallelism gets expensive without NVLink

Tensor parallelism goes back to the method from Megatron-LM: the weight matrices of attention and MLP are split by columns or rows, each GPU computes its part, and after attention and after the MLP the partial results are combined with an all-reduce (Shoeybi et al., Megatron-LM). That makes two synchronisation points per transformer layer. A model with 60 layers therefore synchronises around 120 times for every generated token.

During token generation (decode), the amount of data per step is small, but every synchronisation carries a fixed latency. Processing long inputs (prefill) moves large amounts of data. Over NVLink, both effects barely matter. Over PCIe, the GPU measurably waits for the other card, and the share of waiting time grows with every additional TP GPU.

The vLLM documentation draws an explicit conclusion: if the GPUs in a server lack an NVLink interconnect, for example the L40S, pipeline parallelism should be used instead of tensor parallelism for higher throughput and lower communication overhead (vLLM, Parallelism and Scaling). The RTX PRO Blackwell cards fall into the same category.

This does not make TP over PCIe unusable. With two GPUs the overhead remains manageable, and TP shortens the response time of a single request, which PP does not. The gap widens with TP=4 and TP=8 and with the number of concurrent requests.

The decision rule: split as little as possible

For servers without NVLink the choice comes down to one rule: distribute the model across as few GPUs as needed for it to fit with enough KV cache, and use the remaining GPUs for additional replicas. How much memory weights and KV cache need is worked out in GPU and VRAM sizing for LLMs; which format shrinks the weights is explained in LLM quantization.

An example of the third case: Qwen3.5-122B-A10B in FP8 occupies considerably more than the 96 GB of an RTX PRO 6000 with its weights alone. Two cards with TP=2 hold weights and KV cache together; the setup is documented in vLLM with Qwen3.5-122B on two RTX PRO 6000.

An example of the first case: a model with around 30 billion parameters in FP8 or 4-bit fits on a 96 GB card with plenty of room for KV cache. With two such GPUs, two instances deliver more throughput than one instance with TP=2.

vLLM configuration for two GPUs

The following commands show the three variants with vllm serve. Model name, context length and memory fraction are placeholders that have to match the model.

Tensor parallelism: one model across both GPUs

vllm serve <model> \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90

Pipeline parallelism: layers distributed in blocks

vllm serve <model> \
  --pipeline-parallel-size 2 \
  --max-model-len 32768

Replicas via data parallelism: two copies behind one endpoint

vllm serve <model> \
  --data-parallel-size 2 \
  --max-model-len 32768

Replicas as separate instances

CUDA_VISIBLE_DEVICES=0 vllm serve <model> --port 8000
CUDA_VISIBLE_DEVICES=1 vllm serve <model> --port 8001

Separate instances can be restarted and updated individually. A gateway with load balancing and health checks distributes the requests, for example LiteLLM. With --data-parallel-size, vLLM handles the distribution itself; queue limits then apply to the whole server, not per rank (vLLM, Data Parallel Deployment).

Further parameters that matter on PCIe systems:

The descriptions come from the engine arguments reference (vLLM, Engine Arguments). How vLLM differs from Ollama overall is shown in vLLM vs. Ollama.

What a public measurement shows

Reliable published measurements of TP versus replicas on PCIe systems are rare. One reproducible series is vllm-topology-bench. It compares one instance with TP=4 against two replicas with TP=2 each on four RTX 3090 GPUs without NVLink:

The pattern confirms the rule: with a single request both variants are almost level; as load increases, the variant with more TP GPUs saturates earlier. One copy per card (4 x TP=1) was not possible in this measurement because the weights almost fill the 24 GB cards.

The project names its own limitations: one run per data point, consumer cards, short context length, a single MoE model. The absolute numbers do not transfer to RTX PRO Blackwell or other models. The direction does, and it should be checked on your own system with your own load before committing to a layout.

Check and measure the topology

How well two GPUs communicate depends not only on the card but also on the server. Three checks belong in front of every multi-GPU configuration:

1. Show the connection between the GPUs. nvidia-smi topo -m displays the topology matrix. If the GPUs sit on the same PCIe switch or root complex, the path is short. If the matrix shows SYS between them, traffic crosses the link between two CPUs, which slows TP down further.
2. Check lane width and generation. nvidia-smi -q shows the current PCIe generation and link width. A card connected with only x8 or PCIe 4.0 halves the available bandwidth.
3. Measure with realistic load. Compare throughput and time to first token at the expected number of concurrent users and the actual input length, for TP=2, PP=2 and replicas where the model allows it. vLLM ships its own benchmark tool for this (vllm bench serve).

When starting with TP, NCCL logs which path the GPUs use to communicate. The vLLM documentation describes how to read this output (vLLM, Parallelism and Scaling). Connecting several servers instead of several GPUs in one server raises the same questions at network level; for GB10 systems this is covered in Connect AI Cubes with ConnectX-7.

What this means for your project

Two GPUs without NVLink are a sound basis for production use if the split matches the goal. For more users with a model that fits on one card, replicas are the direct route. For models that exceed one card, TP=2 is the common standard and PP=2 the alternative that can pay off under high load. TP across four or more PCIe GPUs should only be chosen after measuring it yourself.

WZ-IT provides managed GPU servers with a dedicated NVIDIA RTX PRO 4000 Blackwell (24 GB, from €699 net per month) or RTX PRO 6000 Blackwell Max-Q (96 GB, from €1,799 net per month), each with vLLM, Open WebUI and monitoring. We plan multi-GPU configurations, separate instances behind a gateway and redundant model endpoints to match model, context length and user count, and itemise them in the quote. For on-premises operation, the AI Cube is the alternative. Support, consulting and implementation by WZ-IT.

Which model is a candidate in the first place is covered in Which LLM to self-host?. How many users a server can carry is described in Size a local AI server by user count, and the differences between inference engines are compared in vLLM, Ollama and llama.cpp compared.