LLM speed on cloud GPUs

Measured 2026-11-15

On RTX 4000 Ada a Llama 3.1 8B model writes about 65 tokens per second and Qwen 2.5 14B about 36; on L4 about 51 and 27. Both are faster than anyone reads, so the choice is about price and how much memory the model needs.

Tokens per second

GPUVRAMLlama 3.1 8BQwen 2.5 14BPer hourTokens per $1, 8B
RTX 4000 Ada (Linode)20 GBabout 65about 36$0.73about 322k
L4 (OVHcloud)24 GBabout 51about 27$1.16about 157k

Generation speed, median of 3 runs. Models are the Ollama defaults (Llama 3.1 8B: Q4_K_M, about 5 GB; Qwen 2.5 14B: Q4_K_M, about 9 GB). Tokens per dollar assume the GPU generates without pauses. Exact numbers: gpu-llm-speed.csv (CC BY 4.0, cite NoctHost). Also on GitHub and Zenodo, DOI 10.5281/zenodo.24251441.

How we measured

One fresh server per card, the smallest size with that GPU. We installed Ollama, loaded each model with a warm-up request, then sent the same prompt three times with temperature 0 and a limit of 256 new tokens. The speed is the generated token count divided by the generation time reported by the Ollama API. The runs on each card differed by under 1%. Servers were deleted the same hour. More cards will be added as we measure them.

Check any GPU server yourself:

curl -fsSL https://ollama.com/install.sh | sh
ollama run llama3.1:8b --verbose "Explain how a hash table works."
# look at "eval rate" in the output

Frequently asked questions

How many tokens per second does Llama 3.1 8B run at on a cloud GPU?

About 65 tokens per second on RTX 4000 Ada and about 51 on L4, with Ollama and the default 4-bit model. A person reads 5 to 10 tokens per second.

Which GPU gives more tokens per dollar?

RTX 4000 Ada: about 322 thousand tokens per dollar on Llama 3.1 8B at $0.73 an hour, against about 157 thousand on L4. L4 has more memory (24 GB against 20 GB) for larger models or longer context.

Can a 14B model run on these cards?

Yes. Qwen 2.5 14B in 4-bit takes about 9 GB and fits on both, at about 36 tokens per second on RTX 4000 Ada and 27 on L4.

How was it measured?

On a fresh server for each card: install Ollama, load the model with one warm-up request, then send the same prompt three times with temperature 0 and 256 new tokens. Speed is eval_count divided by eval_duration from the Ollama API. The three runs differed by under 1%.

Run your model by the hour

See all GPU servers and running Ollama on a GPU server.

Get started