What decides whether it runs at all
RAM, and nothing else. A quantised model has to fit entirely in memory. If it does not, the system swaps, and inference goes from slow to unusable, minutes per token rather than seconds.
The arithmetic is simple: a 4-bit quantised model needs roughly half its parameter count in gigabytes, plus room for context and the operating system.
| Model size | 4-bit weights | Plan to run it on |
|---|---|---|
| 3B | ~2 GB | 4 GB |
| 7–8B | ~4–5 GB | 8 GB |
| 13–14B | ~9–10 GB | 16 GB |
| 30B+ | 18 GB+ | not on these plans |
Below 4 GB you are limited to 3B-class models. They exist, they work, and they are noticeably less capable, fine for classification and extraction, frustrating for anything that needs reasoning.
What decides whether it is bearable
Generation speed on three Hetzner Cloud servers, Ollama with default 4-bit models, median of three runs:
| Model | 2 shared vCPU, 4 GB | 4 shared vCPU, 8 GB | 4 dedicated vCPU, 16 GB |
|---|---|---|---|
| Llama 3.2 3B | 4.3 tok/s | 7.1 tok/s | 16.3 tok/s |
| Qwen 2.5 3B | 5.0 tok/s | 7.9 tok/s | 17.6 tok/s |
| Llama 3.1 8B | does not fit | 3.6 tok/s | 7.9 tok/s |
| Qwen 2.5 7B | does not fit | 4.0 tok/s | 8.2 tok/s |
| Qwen 2.5 14B | does not fit | does not fit | 4.2 tok/s |
People read at roughly four to six tokens per second, so every cell above is at least close to reading pace. Three things stand out:
- Cores help. Going from 2 to 4 shared vCPUs gave about 1.6x.
- Dedicated cores help more. The 4-core dedicated server was about twice as fast as the 4-core shared one. It also had a newer CPU, so that is not a pure shared versus dedicated comparison, but it is the choice you face when picking a plan.
- The prompt is the real wait. Reading a 720-token prompt took about 40 seconds on the smallest server with a 3B model, and about 45 seconds with an 8B model even on dedicated cores, before the first word appeared. Short chat messages are fine. Long documents are where CPU inference hurts.
The jobs where this is genuinely fine
Slow inference is a problem for chat and almost nothing else. Where latency does not matter, a CPU box is a real answer:
- Classification and tagging. Feeding items through a model on a schedule, support tickets, incoming documents, RSS items, where the queue drains overnight.
- Summarisation in batches. Same shape: the work arrives continuously and the answers are read later.
- Extraction. Pulling structured fields out of unstructured text, which small models do well.
- Embeddings. Cheap on CPU and the basis of search over your own documents. A private RAG setup is largely embedding work, with the slow generation step only at the end.
- Development. Getting a pipeline working before paying for anything faster.
What it is not: a private ChatGPT for daily use. People try that first and conclude the whole idea is broken, when what is broken is the fit.
Making it less slow
The gains available are worthwhile but bounded.
Quantise harder. Q4 is the usual default; Q5 and Q6 are better quality and slower, Q3 is faster and noticeably worse. For batch jobs Q4 is the right compromise.
Pick a smaller model that was trained well over a bigger one squeezed hard. A good 3B beats a badly-quantised 8B on both counts.
Keep the context short. Prompt processing runs at roughly 9 to 40 tokens per second on these servers, so a long system prompt or a pasted document costs tens of seconds on every call.
Keep the model loaded. Ollama unloads after an idle period by default; if requests arrive every few minutes, the reload costs more than the inference.
Do not run anything else on the box. The model wants all of the memory bandwidth, and a database on the same machine will make both slower.
When to stop and rent a GPU
When a human is waiting on long inputs: chat over documents, RAG with big chunks, anything user-facing that needs an answer in a couple of seconds. Short-message chat with a 3B model on dedicated cores is usable. A 14B model reading a long prompt on CPU is not.
Because GPU instances are billed by the hour, the sensible pattern is not to choose once. Develop on the cheap CPU box, and rent a GPU for the hours when you need throughput.
Trying it
The plans that matter here are the 8 GB and 16 GB tiers and the dedicated CPU line, Ubuntu or Debian, full root, billed by the hour from a prepaid balance. The benchmark is one script:
curl -fsSLO https://raw.githubusercontent.com/DanielChernov7/llm-cpu-bench/main/bench.sh
chmod +x bench.sh
sudo ./bench.shIt picks the models that fit your RAM and writes a results table. An hour of testing on your own workload tells you more than any table, including ours.