Why does prefill get slower with longer context?

Prefill is quadratic-ish vs context length. At 64K it peaked at 840 tok/s, but at 256K it dropped to 272 tok/s with higher memory pressure and scheduler overhead.

What is the most practical context size for everyday use?

64K is the sweet spot, good balance of capability vs speed. Beyond 64K you pay steep prefill and KV-cache costs. Use truncation or retrieval for larger docs.

Did you hit OOM before optimization?

Yes, at 110K with the old config. After lowering --n-cpu-moe to 23 and tuning --threads-batch, it became stable up to 251K (tested limit 262144).

Prefill speed on long context inference.