How many users can a local AI server support?

Does the AI Cube Pro fit your user load? We align the model and expected concurrency before delivery. If measured load exceeds the platform, we add AI Cubes or size a GPU server. Configure the AI Cube Pro

A local AI server has no credible fixed limit such as “50 users”. One hundred accounts may be easy when only a few people ask short questions at once. Ten concurrent users can be demanding when they process large documents and expect long answers. Capacity planning therefore starts with concurrency rather than account count.

Short answer: which setup fits which usage profile?

This table is not a fixed performance commitment. It identifies the architecture to test first. A defensible user count only follows from the exact model, representative tasks, and acceptable waiting time.

Five factors that determine throughput

1. Model and quantisation: larger weights use more memory and compute per token. The guide to LLMs on 128 GB of unified memory maps the relevant model classes.
2. Input context: long chats and document extracts increase prefill work and KV-cache use.
3. Response length: a short answer occupies capacity for far less time than a long report.
4. Concurrent requests: inference servers batch work, but every active sequence consumes resources.
5. Expected latency: a team may accept a minute for analysis but only a few seconds to first output in chat.

Three user counts instead of one

Record registered users, daily active users, and users active concurrently during a peak. The third number drives hardware sizing. Also separate chat, document search, transcription, and automated API jobs, as they create different load profiles.

What we measure on the AI Cube Pro

The AI Cube Pro provides 128 GB of unified memory and a local platform prepared for multi-user operation. We agree a model for the use case before delivery. Capacity is then measured with representative requests:

• time to first token;
• visible output rate per request;
• aggregate throughput;
• memory use at typical and maximum context;
• queues, cancellations, and errors under peak load.

Results apply to the documented model release, quantisation, runtime, and test profile. Moving to a larger model can substantially alter capacity.

When a second AI Cube makes sense

For more concurrent users, two AI Cubes can process independent requests. A load balancer sends new work to available capacity. This differs from distributing one large model across two systems. Distributed inference uses ConnectX-7 and adds network and software configuration to the performance profile.

A second node does not automatically provide high availability. Model endpoints, interface, database, knowledge store, identity, and load balancer must also be redundant or recoverable.

When AI Cube Custom or a GPU server follows

A larger deployment is appropriate for consistently larger models, many long concurrent contexts, several resident models, or defined availability. We then size AI Cube Custom or GPU servers from the measured profile.

A practical load-test profile

1. select five to ten typical tasks;
2. define realistic prompt, document, and output sizes;
3. test one, two, four, and eight concurrent workflows;
4. assess quality and speed together;
5. add peak periods and automated jobs;
6. define explicit capacity headroom for growth and updates.

This turns a user count into an auditable capacity decision rather than a marketing number.

Sources and further reading

• vLLM: architecture and continuous batching
• NVIDIA: DGX Spark hardware
• WZ-IT: sizing GPU and VRAM
• WZ-IT: connecting AI Cubes with ConnectX-7