GPU math
Cheapest GPU setup for your load, step by step.
1Load
Request rate
reqper
9,000 ÷ 86,400 s / day
Peak RPS
× peak
0.104 req/s × 3
Tokens / request
in+out
Prefix cache
2Latency
total
ms max
10,800ms
1,200 ms + 600 tok ÷ 62.5 tok/s × 1,000
First output1,200Rest9,60010,800 ms
3Budget
GPU budget
$ maxperreq / user
0.02 ÷ 1 req / user × 0.313 peak req/s
budget0.00625$ / s
cost1 × H1000.00181$ / s
4GPUs
Model
Fleet
GPUs needed · $ / user
GPUGPUs$ / user
L424 GB · $0.85/h · speech, embeddings, 8B——
A10080 GB · $4.00/h · 8B–32B, 70B if H100 is unavailable2too slow
H10080 GB · $6.50/h · 70B, long context1$0.00578
B200180 GB · $9.98/h · 70B+ at scale, DeepSeek-class1$0.00887
Llama 3.3 70B1 × H100
0.313 req/slatency 15,000 ms max0.02 $ / user