LLM speed on cloud GPUs
Measured 2026-11-15
On RTX 4000 Ada a Llama 3.1 8B model writes about 65 tokens per second and Qwen 2.5 14B about 36; on L4 about 51 and 27. Both are faster than anyone reads, so the choice is about price and how much memory the model needs.
Tokens per second
| GPU | VRAM | Llama 3.1 8B | Qwen 2.5 14B | Per hour | Tokens per $1, 8B |
|---|---|---|---|---|---|
| RTX 4000 Ada (Linode) | 20 GB | about 65 | about 36 | $0.73 | about 322k |
| L4 (OVHcloud) | 24 GB | about 51 | about 27 | $1.16 | about 157k |
Generation speed, median of 3 runs. Models are the Ollama defaults (Llama 3.1 8B: Q4_K_M, about 5 GB; Qwen 2.5 14B: Q4_K_M, about 9 GB). Tokens per dollar assume the GPU generates without pauses. Exact numbers: gpu-llm-speed.csv (CC BY 4.0, cite NoctHost). Also on GitHub and Zenodo, DOI 10.5281/zenodo.24251441.
How we measured
One fresh server per card, the smallest size with that GPU. We installed Ollama, loaded each model with a warm-up request, then sent the same prompt three times with temperature 0 and a limit of 256 new tokens. The speed is the generated token count divided by the generation time reported by the Ollama API. The runs on each card differed by under 1%. Servers were deleted the same hour. More cards will be added as we measure them.
Check any GPU server yourself:
curl -fsSL https://ollama.com/install.sh | sh ollama run llama3.1:8b --verbose "Explain how a hash table works." # look at "eval rate" in the output
Frequently asked questions
How many tokens per second does Llama 3.1 8B run at on a cloud GPU?
About 65 tokens per second on RTX 4000 Ada and about 51 on L4, with Ollama and the default 4-bit model. A person reads 5 to 10 tokens per second.
Which GPU gives more tokens per dollar?
RTX 4000 Ada: about 322 thousand tokens per dollar on Llama 3.1 8B at $0.73 an hour, against about 157 thousand on L4. L4 has more memory (24 GB against 20 GB) for larger models or longer context.
Can a 14B model run on these cards?
Yes. Qwen 2.5 14B in 4-bit takes about 9 GB and fits on both, at about 36 tokens per second on RTX 4000 Ada and 27 on L4.
How was it measured?
On a fresh server for each card: install Ollama, load the model with one warm-up request, then send the same prompt three times with temperature 0 and 256 new tokens. Speed is eval_count divided by eval_duration from the Ollama API. The three runs differed by under 1%.