Local AI in the company: what hardware is needed?
A local language model needs one thing above all: enough fast memory on the graphics card. How much depends on the model size, the quality you want and the number of simultaneous users.
Why graphics memory decides
To run a language model, its parameters, the learned weights, must be held in memory. Graphics processors (GPUs) compute the answers many times faster than ordinary processors, but only if the model fits into their memory (VRAM). If the memory is not enough, the system slows down sharply.
Rules of thumb for model size
Model size is stated in billions of parameters. Quantisation, meaning storing the weights with fewer bits, reduces memory requirements considerably, usually with little loss of quality. As a rough guide at 4-bit quantisation:
| Model size | Graphics memory (approx.) | Typical use |
|---|---|---|
| 7 to 9 billion parameters | 8 to 12 GB | Classifying, summarising, simple assistants |
| 12 to 14 billion parameters | 12 to 16 GB | Good German text, document search |
| 27 to 32 billion parameters | 24 to 32 GB | Demanding texts, more complex analyses |
| 70 billion parameters or more | 48 GB or more, often multiple GPUs | Highest quality, many users |
On top of that comes memory for the context, meaning the amount of text the model processes at once. Long documents and many parallel requests increase the requirement.
Typical configurations
- Small office, few users: workstation or small server with a graphics card with 16 to 24 GB of memory.
- Mid-sized company, departmental use: server with one or two professional GPUs totalling 48 GB or more, 128 GB of memory, fast NVMe SSDs.
- Many users, large models: several GPUs, operation via vLLM with parallel processing.
What matters besides the GPU
- Memory (RAM): at least as much as GPU memory, ideally twice as much.
- SSD: models offer many gigabytes; NVMe SSDs shorten loading times.
- Power supply and cooling: powerful GPUs need several hundred watts and good ventilation.
- Document search: RAG applications additionally need a vector database and a smaller model for retrieval.
Used servers as a basis
Decommissioned servers from data centres are an inexpensive basis for local AI: stable power supplies, plenty of memory, room for several drives. Fitted with a suitable graphics card, many models run reliably on them. We assemble such systems from tested stock.
Software
Ollama and llama.cpp (simple, efficient) and vLLM (many simultaneous users) have proved their worth in operation. Open WebUI, for example, serves as the interface. Whisper is used for speech recognition, locally and without the cloud.
Local AI for your company
We plan the suitable hardware, install the model and connect it to your workflows.
