WZ-IT Logo
Interactive Tool

Token Generation Speed Simulator

Simulate token generation speed for Large Language Models and understand how different speeds affect user experience.

45
10 tok/s500 tok/s
500
503000
Estimated time:11.11s
Progress0 / 500 tokens (0.0%)
Output:
Click "Start Simulation" to begin...
Elapsed Time

0.00s

Current Speed

0 tok/s

Our Hardware

Achieve These Speeds with AI Cube

Local AI inference without cloud dependency. Full data control.

WZ-IT AI Cube

ASUS/NVIDIA GB10 Appliance

GPT-OSS 20B
85 tok/s
GPT-OSS 120B
45 tok/s
Memory128 GB Unified Memory
AI Performancebis 1 PFLOP FP4
Max Model SizeGPT-OSS 20B / 120B
ROI vs. Clouddepends on usage profile
From
5.990,90 €
excl. VAT
Learn More

Performance Comparison

ModelWZ-IT AI Cube
GPT-OSS 20B
~20 Billion Parameters
80-90 tok/s
GPT-OSS 120B
~120 Billion Parameters
35-60 tok/s

* Conservative guidance based on publicly available GB10/DGX Spark benchmarks. Performance depends on model, context length, backend and concurrency.

Typical Use Cases

Chatbot

Tokens:50-200
Rec. Speed:50-100 tok/s
Duration:0.5-2s

Quick responses for fluid conversation

E-Mail

Tokens:200-500
Rec. Speed:30-50 tok/s
Duration:4-10s

Sufficient for email generation

Report/Article

Tokens:1000-3000
Rec. Speed:50-100 tok/s
Duration:10-60s

Longer texts with good performance

Token Speed FAQ

Everything you need to know about token generation

Themen

Basics

Token generation speed (measured in tokens per second or tok/s) indicates how fast an AI model can generate text. One token corresponds to approximately 4 characters or 0.75 words. At 100 tok/s, about 75 words are generated per second.

Simulation helps developers and businesses understand how different speeds affect user experience. Slow generation (under 30 tok/s) feels sluggish, while fast generation (over 100 tok/s) provides a fluid experience.

A token is the smallest unit an LLM processes. It can be a word, word part, or punctuation mark. 'Hello' is one token, 'championship' might be split into 'champion' + 'ship'. Most LLMs use about 1 token per 4 characters.

Influencing Factors

Key factors include: 1) GPU performance and VRAM, 2) Model size (7B, 70B, 120B parameters), 3) Quantization (FP16, INT8, INT4), 4) Batch size, 5) Context length, 6) Inference backend (vLLM, Ollama, TensorRT).

Larger models are slower. A 7B model can achieve 200+ tok/s, a 70B model about 50-100 tok/s, and a 120B model typically 30-60 tok/s on consumer hardware. However, response quality increases with model size.

No, speed varies. At the beginning (prefill phase), generation is often slower, then stabilizes. Query complexity, context length, and system load also affect speed.

Practical Application

Under 20 tok/s: Noticeably slow, frustrating. 20-50 tok/s: Acceptable for most applications. 50-100 tok/s: Good experience, feels fluid. Over 100 tok/s: Excellent, text appears almost instantly.

Chatbots: 50-100 tok/s for fluid conversation. Document generation: 30-50 tok/s sufficient. Real-time translation: 100+ tok/s recommended. Code completion: 100+ tok/s for best developer experience.

1) Use better GPU with more VRAM, 2) Quantization (INT4/INT8) for smaller models, 3) Use optimized inference engines like vLLM, 4) Batch processing for multiple requests, 5) Optimize KV-cache, 6) Enable continuous batching.

Hardware & AI Cube

WZ-IT AI Cube based on GB10: conservative guidance is around 80-90 tok/s with GPT-OSS 20B and around 35-60 tok/s with GPT-OSS 120B. Values depend on context length, quantization, backend and concurrency.

Cloud APIs have network latency, rate limits, and share resources with other users. Local hardware like the AI Cube offers dedicated resources, no network delay, and consistent performance without queues.

Cloud APIs charge per token (e.g., $0.002/1K tokens). With high usage, this adds up quickly. Local hardware has one-time acquisition costs but no ongoing token costs. From about 500,000 tokens/month, local hardware often pays off.

Ready for fast AI inference?

The AI Cube offers local AI inference on owned hardware - without cloud dependency and external token costs.

Discover AI Cube

Let's Talk About Your Idea

Whether a specific IT challenge or just an idea - we look forward to the exchange. In a brief conversation, we'll evaluate together if and how your project fits with WZ-IT.

E-Mail
[email protected]

Leading companies trust WZ-IT

  • Rekorder
  • Keymate
  • Führerscheinmacher
  • SolidProof
  • ARGE
  • Boese VA
  • nextGYM
  • Maho Management
  • Golem.de
  • Millenium
  • Paritel
  • Yonju
  • EVADXB
  • Mr. Clipart
  • Aphy
  • Negosh
  • ABCO Water Systems
Timo Wevelsiep & Robin Zins - CEOs of WZ-IT

Timo Wevelsiep & Robin Zins

Managing Directors of WZ-IT

1/3 - Topic Selection33%

What is your inquiry about?

Select one or more areas where we can support you.