Size the workload before the machine

Hardware conversations tend to start with a card. They should start with a list of tasks. A private AI system for a small firm answers questions over its own documents, drafts text, and summarises files. Each of those tasks sets a different load. The machine follows from the load.

This article works through the order of decisions. The evidence behind it reflects public documentation available in early November 2025. Software versions are named where they matter, because memory behaviour changes between releases.

Write down the work first

Before any model is chosen, four facts shape the design:

1. Who asks, and how many at once. Twelve people who each ask a few questions a day make a different load from a team that runs batch summaries every evening. Peak concurrency matters more than head count.
2. How much text each request carries. A short question over three retrieved passages is small. A request to compare two long contracts is large. Context length drives memory, as the next sections show.
3. How fast an answer must arrive. A drafting assistant can take a few seconds. A tool used live during a client call cannot.
4. What the documents look like. Volume, languages, scans versus native text, and how often they change. These set the load on the embedding model, the index and the storage.

These answers come from the firm, not from a vendor sheet. A short pilot on real questions, with the firm’s permission, turns guesses into observations.

Weights are the fixed cost

In a system served from GPUs, the model’s parameters are loaded into GPU memory. Some runtimes can keep part of a model in system RAM and move it across as needed. That extends capacity but slows every answer, so it suits occasional batch work better than live use. Either way, the size of the weights sets the baseline.

That size depends on numeric precision. Hugging Face’s guide to optimizing LLMs for speed and memory (Transformers v4.46.0, released October 2024) gives the rule of thumb: a model with X billion parameters needs roughly 2X GB in bfloat16 or float16, since each parameter takes two bytes. It cites a 70-billion-parameter model at about 140 GB.

The general form is simple: parameters × bits per parameter ÷ 8 = bytes. The chart at the top of this article applies it to three model sizes at 16, 8 and 4 bits, in decimal gigabytes. Its figures are arithmetic, not measurements. They cover weights alone and leave out everything a running system also needs. Quantized formats also store scaling factors, so real files differ a little from the calculation.

Lower precision is a trade. The same Hugging Face guide explains that quantized weights are dequantized for computation, and warns that 4-bit results can differ from 8-bit or full bfloat16 inference. The only sound test is the firm’s own questions, run at each candidate precision, with answers read by someone who knows the subject.

The cache grows with every conversation

Weights are the fixed cost. The key-value cache is the variable one. While a model generates text, it stores intermediate values for every token in the context. The Hugging Face guide works one example: for a 16,000-token input, the cache of the model it studies needs around 15 GB in float16, roughly half the size of its weights. Newer attention designs shrink that figure a great deal, and the guide describes them, but the principle holds. Longer contexts and more simultaneous users mean more cache.

The PagedAttention paper behind vLLM makes the same point from the serving side. Each request’s cache is large and grows and shrinks as the request runs. Poor management wastes memory and caps how many requests fit at once.

So the question “does the model fit?” has two halves. Do the weights fit, and is enough memory left for the cache at the context lengths and concurrency the firm needs?

The levers, and what each costs

The vLLM v0.11.0 documentation on conserving memory (released October 2025) sets out the main levers:

• Split the model across GPUs with tensor parallelism. This adds cards, power and heat, and the cards must communicate.
• Use quantized weights. Less memory, at some cost in precision.
• Limit context length and batch size. Less memory, but shorter documents per request and fewer users served at once.
• Disable CUDA graph capture. The same page notes that graphs take extra GPU memory. Turning them off saves memory and can slow inference.

Each lever moves cost somewhere else. A written record of which levers were used, and why, helps later when the load changes.

Everything that is not the model

The main model is rarely alone. A retrieval system also runs an embedding model to index documents, often a reranker, a vector index, a database and the application itself. Each needs memory, CPU and disk. Storage must hold the current model files, the previous version for rollback, the index, and backups. System RAM matters too: loading large model files can need substantial host memory before they reach the GPU.

For scale, NVIDIA’s August 2023 announcement of the L40S gave each card 48 GB of memory. Its March 2025 announcement of the RTX PRO Blackwell line gave up to 96 GB for workstation and server cards. On the arithmetic above, a 16-bit 70B model fits on neither alone. A smaller model, or a quantized one, may fit on one card with room for the cache. Whether it answers the firm’s questions well is a separate test. Power draw and cooling needs vary by card and configuration; the current vendor datasheet for the exact part is the reference at purchase.

Room, power and a fallback

A multi-GPU server draws steady power and turns it into heat and fan noise. Planning covers the electrical circuit, a UPS sized for a clean shutdown, cooling, and a room where noise does not reach desks. The RTX PRO 6000 Server Edition, for example, is passively cooled and depends on the server’s airflow. Cards like it suit a ventilated technical room better than an office corner.

Hardware also fails. ANSSI’s recommendations for generative AI systems advise a degraded mode: a way to keep working when the AI system is down. For a small firm, that can be the manual process the system replaced. Planning it at the sizing stage costs little. Deciding it during an outage costs more.

A sizing record

A useful sizing exercise ends with a short document, not a purchase order alone. It lists the tasks, the peak concurrency, the target context length, the candidate models and precisions tested, the memory each used in the pilot, the levers applied, and the room and power plan. When the firm’s work changes, that record shows what to revisit.

The figures here are arithmetic and vendor announcements, not measured requirements for any deployment. To size a private system around your own work, write to contact@desgarde.com.