What is token generation speed in the context of Large Language Models (LLMs)?

Token generation speed describes how quickly a model emits text after the first token. It is measured in tokens per second (tok/s). The number of characters or words represented by a token depends on the language and tokenizer; four characters per token is only a rough visualisation.

Why is simulating token generation speed important?

The simulation makes the visible output speed of one chat tangible. Sizing a business system also requires time to first token, context length and aggregate throughput across concurrent chats.

What exactly is a token?

A token is a text unit produced by a model's tokenizer. It may be a word, part of a word, a number or punctuation. Token counts therefore are not fully comparable across models and languages.

What factors affect token generation speed?

Key factors are model architecture, active parameters, quantisation, context length, inference runtime, cache configuration and concurrent requests. Input length and knowledge retrieval also affect time to first token.

Why is parameter count alone not a sufficient comparison?

Dense models use all parameters for every token, while Mixture-of-Experts models use only a subset. Qwen3.6-35B-A3B, for example, has 35 billion total parameters but activates around 3 billion per token. Runtime and quantisation can therefore matter more than the number in the model name.

Can real LLMs maintain constant speed like in the simulator?

No. The system first processes the prompt and any retrieved documents, then output begins. Concurrency, context and system load change the rate during generation. The simulator deliberately uses a constant rate to make differences visible.

How does token speed affect user experience?

From roughly 20 to 30 tok/s, an answer is generally easy to read as it streams. Higher rates shorten long outputs, but they are not the only quality factor. With multiple users, per-chat speed and waiting time for the first token both matter.

What speed do I need for my use case?

For interactive chat, a stable 30 to 50 tok/s per active session is generally practical. Long reports benefit from higher throughput; RAG applications also need a short time to first token. We therefore set a target against the actual workflow.

How can I optimize token generation speed?

Key levers include a suitable quantised checkpoint, an optimised runtime such as vLLM, sensible context limits, KV-cache configuration, continuous batching and, where appropriate, speculative decoding. Each optimisation must be checked against answer quality and the real workload.

What speeds does the AI Cube achieve?

A publicly reproducible GB10 test with nvidia/Qwen3.6-35B-A3B-NVFP4 measured around 108 tok/s for one chat. Four concurrent chats delivered around 50 tok/s per chat and 189 tok/s aggregate; eight delivered around 43 tok/s per chat and 320 tok/s aggregate. These are reference values, not a guarantee for every workload.

Is local AI inference automatically faster than a cloud API?

No. Large cloud platforms can provide more powerful hardware. AI Cube instead provides dedicated, predictable capacity on the company network without an external model call. Whether speed, data control or cost matters most depends on the use case.

How does speed relate to cost?

External model providers often charge by token or usage. With AI Cube, local compute capacity is part of the device you buy; AI Cube Care is added as monthly maintenance, running with the device for the first twelve months and cancellable monthly after that. Custom integrations may be added. Whether this is more economical depends on utilisation, model choice and operating requirements.

Interactive simulator

LLM Token/s Simulator: Qwen3.6-35B-A3B NVFP4 on GB10

Is one AI Cube enough for your team, and how does local AI feel in daily use? Experience measured output speed for nvidia/Qwen3.6-35B-A3B-NVFP4 in a single chat and under concurrent load on a GB10-based AI Cube.

Try the simulator

IT Beratung Service Consulting Soforthilfe

Simulator

Try tokens per second yourself

Set speed and answer length, start the simulation and read along. The presets match measured values.

Speed108 tok/s

10120 tok/s

Qwen presets show measured per-chat speed in a public GB10 test dated 19 July 2026. The workload used 1,024 input and 256 output tokens. We validate model, context and concurrency separately for your deployment.

Answer length500 Token

503.000 Token

Output0 / 500 Token

Start the simulation to read the answer at the chosen speed.

Estimated: 4.63 s
Elapsed: 0.00 s
Current: 0.0 tok/s

Current measurements

Qwen3.6-35B-A3B-NVFP4 on one AI Cube

Visible speed per chat decreases under concurrent load while total system throughput increases. Both matter when sizing the system.

AI Cube · 1 × NVIDIA GB10 · 128 GB Unified Memory

nvidia/Qwen3.6-35B-A3B-NVFP4

≈ 108 tok/s · Single chat
≈ 43 tok/s · 8 concurrent chats · per chat
≈ 320 tok/s · 8 concurrent chats · aggregate
Architecture: 35B total · 3B active
Checkpoint: NVIDIA NVFP4
Runtime: vLLM · FP8 KV-Cache · MTP-2
Test workload: 1.024 Input · 256 Output

Alternative reference: GPT-OSS 20B reached around 91 tok/s in a single chat. Runtime and quantisation differ, so this is not a direct quality or model comparison.

Output speed under concurrent load
Concurrent chats	Per chat	Aggregate	TTFT p50
1	≈ 108 tok/s	≈ 100 tok/s	178 ms
2	≈ 85 tok/s	≈ 158 tok/s	209 ms
4	≈ 50 tok/s	≈ 189 tok/s	245 ms
8	≈ 43 tok/s	≈ 320 tok/s	282 ms
16	≈ 29 tok/s	≈ 432 tok/s	447 ms

Publicly documented measurement dated 19.07.2026 on a single GB10. Per-chat and aggregate values use different harness aggregations and can therefore differ slightly at low concurrency. NVIDIA model card · Measurement data and configuration

Use cases

Typical use cases: how long an answer takes

From around 30 tokens per second per chat, an answer is easy to read as it streams. How long it takes overall depends on its length.

AI chat
Visible speed for each active conversation
- Tokens: 50-250
- Rec. Speed: 30+ tok/s
- Duration: 0.5-8s
E-Mail
Response time also depends on the requested length
- Tokens: 200-500
- Rec. Speed: 30+ tok/s
- Duration: 2-17s
Report/Article
Aggregate throughput matters most for long outputs
- Tokens: 1000-3000
- Rec. Speed: 30+ tok/s
- Duration: 9-100s

Planning figure for one AI Cube: up to ten answers generated at the same time. It sits between the measured 8- and 16-chat points; we do not derive a falsely precise ten-chat figure from it. For teams of five to twenty people one Cube is usually enough; sizing uses the target model, context and knowledge sources.

Next step

AI inference on your own network: the AI Cube

The AI Cube brings the measured performance into your network, without cloud dependency and without external token costs. Once for the device and its setup, monthly for maintenance.

One-off

The AI Cube

from €6,490 excl. VATone-off, 1 TB version

NVIDIA GB10 with 128 GB unified memory, preconfigured for your network
Tuned language models and Open WebUI as the working interface
Delivered in ten working days, briefing included

The AI Cube in detail

Ongoing maintenance

AI Cube Care

from €349.90 excl. VAT / monththe first twelve months with the device, cancellable monthly after that

Updates for system, inference engine and models in the maintenance window
Monitoring and alerting around the clock
Ticket support with an answer by the next working day

More concurrent load than one Cube carries, or no room on your premises: managed GPU servers in a German data centre, from €699 excl. VAT / month. View managed GPU servers

All prices are net and exclude statutory VAT. The offers are addressed to businesses.

Token Speed FAQ

Tokens, influencing factors, practice and the AI Cube

TopicsBasics Influencing Factors Practical Application Hardware & AI Cube

Basics

Influencing Factors

Practical Application

Hardware & AI Cube

Reviews & projects

What clients say about working with us

From local AI integration to architecture, data sovereignty and ongoing operations.

WZ-IT moved our studio infrastructure from decentralised individual devices to a central platform: every site is securely connected via VPN, new devices are onboarded automatically and an entire site is provisioned from a template, without manual steps on location. What impressed me most is the breadth and depth of their knowledge: Timo and Robin are not a typical IT provider who sets up a server and leaves. The two of them think their way into highly complex infrastructure and software topics, work through every requirement we put in front of them, and build networking, provisioning and operations so that everything fits together in the end. WZ-IT is an excellent partner for complex software, network and architecture projects.

Steve KirchnerManaging Director, nextGYM GmbH

View project

International

Built in Germany's Ruhr Valley. Running worldwide.

WZ-IT designs, develops and operates infrastructure and software for clients in Germany and internationally. We deliver projects remotely and continue supporting them in ongoing operations after go-live.