Which LLMs run on 128 GB of unified memory?

Select the right model rather than merely installing one? For the AI Cube Pro, we agree the local model for language, task, and expected use in advance. Installation and functional testing are included. Explore the AI Cube Pro

With 128 GB of unified memory, GB10 systems can run model classes that do not fit into a typical workstation GPU. The number alone does not determine which LLM runs well. Weights, quantisation, runtime, KV cache, and memory reserved for the system must fit together.

What “fits” means in production

A model only fits when more than its weights can be loaded. The operating system, inference server, activations, and KV cache need memory too. KV-cache demand grows with context and active sequences. A model that starts in a single-user test may be unsuitable for ten concurrent chats.

As a rough calculation, weights use about two bytes per parameter at 16 bit, one byte at 8 bit, and half a byte at 4 bit. Real artefacts include metadata and quantisation structures, and the runtime needs headroom.

Specific model examples for GB10 and 128 GB

As of August 2026, NVIDIA's own DGX Spark playbooks list the following supported configurations among others. The list establishes technical support for those artefacts; it does not make each model suitable for every business workload.

Model names and support status change. A deployment therefore records the exact artefact, runtime, and release.

Useful model classes

Compact models

Models from one to the lower tens of billions leave substantial memory for context and concurrency. They often suit classification, extraction, short drafts, internal search, and narrowly defined assistants.

Mid-sized models

Models in the middle tens of billions often balance quality, multilingual capability, and latency. With suitable quantisation, 128 GB can retain room for several active requests.

Large and mixture-of-experts models

Quantised models in the hundreds-of-billions range can become technically possible. For mixture-of-experts models, total parameters and active parameters per token describe different things. Memory and compute do not follow the same number.

NVIDIA describes support for models up to 200 billion parameters on a DGX Spark and certain larger configurations across two systems. This is not a performance commitment for every model. Production requires testing the exact artefact.

Why a model family is not enough

“Qwen”, “Llama”, or “Mistral” names a family, not a reproducible configuration. Document the exact model ID and release, licence, quantisation and artefact author, runtime and version, typical and maximum context, quality tests, language, and throughput under realistic concurrency.

The broader guide Which LLM should you self-host? adds licensing, language, model-card, and evaluation criteria to this hardware perspective.

Selection for the AI Cube Pro

For the AI Cube Pro, we clarify the first use case before delivery. We then select, install, and test a local model. The goal is not the largest possible parameter count, but a platform that answers quickly enough and handles the intended content effectively.

Open WebUI can later present several local endpoints or deliberately approved cloud models. Model access can be constrained through users and groups. A model can change without rebuilding the user interface and knowledge spaces from scratch.

When two AI Cubes or Custom make sense

Two AI Cubes can increase aggregate throughput for independent requests or distribute a model. The second path needs appropriate software and ConnectX-7 configuration and is not identical to one computer with 256 GB of memory. For high concurrency, several large resident models, or committed availability, AI Cube Custom or a GPU server may be the better architecture.

Sources

• NVIDIA: DGX Spark hardware and model range
• NVIDIA: TensorRT-LLM model matrix for DGX Spark
• NVIDIA: LM Studio on DGX Spark
• NVIDIA: connect two DGX Spark systems
• Hugging Face: model cards and artefacts
• WZ-IT: sizing GPU and VRAM