Amazon SageMaker Inference adds prefix-aware routing for LLMs
Amazon SageMaker Inference introduced prefix-aware routing, which directs requests sharing the same prompt prefix to the same instance to keep the KV cache warm. Benchmarks on Llama 3.1 70B showed P50 time-to-first-token reduced by up to 77% and KV cache hit rates rising from roughly 25% to over 80%.
Sources and evidence
Summary last validated Sep 10, 2026
Reader actions
Report an issue
Use this for an incorrect summary, wrong source, duplicate story, or wrong category. Submissions are private and do not change the story automatically.
Related coverage
- models
AWS Adds Bedrock Models, Faster AgentCore Agents, Knowledge Base Sync
AWS's September 2026 recap covers Amazon Bedrock, Bedrock AgentCore, and Strands updates: broader model choice, faster serverless agents with built-in evaluation, and automated knowledge base syncing via native enterprise connectors. The post aggregates the month's releases for AI builders across AWS's model and agent tooling.
- agents
Postman runs Agent Mode for 40 million developers on Amazon Bedrock
Postman and AWS detailed the architecture behind Postman's Agent Mode, which serves 40 million developers on Amazon Bedrock. The post covers controlling tool sprawl, exposing schema-based reads, and treating context as the real bottleneck when running an AI agent at scale rather than in a demo.
- infrastructure
Ai2 replaces priority GPU scheduler with budget-based fair-share system
Ai2's AI Infrastructure team replaced its priority-based GPU scheduler with a system using GPU time budgets, hierarchical fair-share allocation, and time-slicing contracts. The institute manages thousands of NVIDIA H100, B200, and B300 GPUs across 88- to 1024-GPU clusters for about 150 researchers, with demand running 2-3x available capacity. The change moved GPU allocation debates into a transparent administrative budgeting process.
- infrastructure
Ai2 details GPU scheduler using time budgets and fair-share allocation
Ai2 published a blog post explaining its new GPU scheduler for its clusters. The scheduler combines time budgets, fair-share allocation, and time-slicing to prioritize high-impact research, shorten queue waits, and keep GPUs busy.
- infrastructure
Cohere Publishes Guidance on Shared vs Dedicated Inference for Embed and Rerank
Cohere published a blog post explaining how to choose between shared, consumption-based inference and dedicated, provisioned inference for Embed and Rerank workloads. It argues request shape matters more than request volume, that traffic patterns determine cost efficiency, and that break-even depends on request size, token volume, and reranking candidate count rather than infrastructure pricing alone.