TL;DR — Sep 27–28 arxiv dropped 5 fresh papers on agent-infrastructure compression. The axis shifted from research papers to runtime/proxy-layer improvements: ElasticKV 9.1× P90 TTFT vs vLLM, SANTA++ 16–22% of dense KV reads at 94–99% quality, and LiteLLM PR #41216 turning /utils/token_counter into a 40× faster cached path. Two research papers stand out: DGE 99.8% of FullKV at the same budget (training-free timing-only fix) and KV-streams 2.6–5× wall-clock training speedup for agentic RL. Plus GLM-5.2 IndexShare 75% indexer KV cache + 1.5–1.8× throughput, now productionized in vLLM commit #59081.
1. DGE — eviction timing fix · 99.8% of FullKV at same budget
arxiv 2609.33334v1 · Sep 27 · reduction axis
Tokens saved
44.2 LongBench at FullKV 44.3 (gap = 0.1) at same KV budget
% of baseline
~99.8% of FullKV quality · per-head budget unchanged
Quality delta
−0.1 LongBench vs FullKV · DGE-W (timing-only, no selection change) hits same 44.2
How to deploy
Defer eviction by exactly k=2 draft tokens past prefill (one decode step). Plug into existing SnapKV/PyramidKV/H2O/StreamingLLM without modifying their eviction scores.
Drop-in for any LLM serving stack. The control is when, not what — apply to any eviction code path that currently runs at end-of-prefill. Eliminates the prefill-vs-answer-trajectory mismatch.
2. Cartridges++ — preserves off-context quality at same compression
arxiv 2609.35621v1 · Sep 28 · reduction axis
Tokens saved
Same compression ratio as vanilla Cartridges (compact learned KV from Q/A pairs)
% of baseline
Preserves general knowledge + instruction following + avoids context contamination
Quality delta
Small/negligible vs vanilla Cartridges on on-context queries; fixes the off-context degradation vanilla Cartridges shipped
How to deploy
Add router variant (decides at inference time whether to use learned long-context memory) or data-mixing variant (small fraction of training Q/As outside reference doc).
Drop-in for any RAG/repeated-document workload that compresses a long doc to a small KV cache and then asks off-context questions.
3. KV-streams — 2.6–5× wall-clock training speedup for agentic RL
arxiv 2609.35750v1 · Sep 28 · reduction axis
Tokens saved
2.6–5× wall-clock training speedup · 3 different compaction strategies validated
% of baseline
Same GPU memory as fixed-budget context compaction · 2.6–5× throughput
Quality delta
No evidence of hindering performance · bonus: emergent recurrent-state behavior from RL alone
How to deploy
Stream the KV cache forward through compaction rather than flushing after each. Plug-and-play with any post-training pipeline running context compaction during RL rollouts.
Critical for long-horizon agentic RL where context-compaction is currently the dominant training bottleneck.
Drop-in for any vLLM-based serving stack. The 9.1× P90 TTFT gain comes from eliminating request-stall/preemption when KV memory pressure hits vLLM's target-only contract.
5. SANTA++ — 16–22% of dense reads at 94–99% quality
arxiv 2609.35629v1 · Sep 28 · reduction axis
Tokens saved
16–22% of dense attention's KV reads · 1.69× attention speedup at 32K context
% of baseline
94–99% on LongBench v2 + HELMET RAG subset · 85–91% on RULER (Qwen2.5-7B-Instruct at 32K)
Quality delta
Tunable — accuracy vs reads is one knob (sampling budget)
How to deploy
Organize cached keys into teams, score one representative per team, sample teams with importance-sampling correction. Use 32 or 64 sampled teams at 32K context.
Drop-in for any attention kernel. Particularly complementary to compressed KV representations (MHLA) since SANTA++ reduces reads, not storage.
Zhipu AI GLM-5.2 + vLLM commit e03875f4 · Sep 28 · both axis
Tokens saved
75% reduction in indexer KV cache and computation by sharing indexers across every four layers
% of baseline
~25% indexer KV cache · 1.5–1.8× throughput at fixed quality
Quality delta
GLM-5.3 (IndexShare descendant) scored 84.5% on CyberGym vs GLM-5.2's 77.2% — preserved or improved
How to deploy
Use GLM-5.2 or GLM-5.3 with IndexShare, run on vLLM with fp8 indexer cache commit #59081 (Sep 28) for non-SM100 GPUs.
Productionized model-side pick. The 1.5–1.8× is on top of sparse-attention baseline, not vs dense attention. Stack with KV-cache compression for compound gains.
/utils/token_counter goes from ~0.43s per call to <10ms after first call (~40× speedup cached)
% of baseline
~2.3% of original per-call latency on cached path · first call stays ~0.6s (tokenizer download)
Quality delta
Identical token counts · lru_cache keyed on identifier+revision+auth token
How to deploy
Upgrade LiteLLM proxy to v1.103.0 (Sep 28 stable) or v1.104.0-rc.1. Set model_info.custom_tokenizer on a deployment. Every subsequent request with same body hits the LRU cache.
Direct value for any proxy fronting token-counting to a billing dashboard or routing layer. 1168 focused tests pass. No config change beyond the version bump.