vLLM TPU Inference v0.30.0 adds FP8 load-time quantization and prefill context parallelism
vLLM's tpu-inference v0.30.0 release adds FP8 quantization of bf16 linear layers at load time, multi-request prefill context parallelism on the ring cache phase, distributed top-k sampling with sharded vocab logits, Gemma 4 MoE performance fixes, hybrid Mamba/GDN model fixes, and a round-robin routing policy for the data-parallel scheduler.
Sources and evidence
Summary last validated Sep 30, 2026
Reader actions
Report an issue
Use this for an incorrect summary, wrong source, duplicate story, or wrong category. Submissions are private and do not change the story automatically.
Related coverage
- open source
BerriAI litellm AI Gateway Hits GitHub Trending With 60k Stars
BerriAI/litellm appeared on GitHub Trending on October 10, 2026, ranking seventh with 95 stars gained that day. The repository, described as an AI gateway with a Rust core and Python SDK, routes calls to over 100 LLM APIs in OpenAI or native format, adding cost tracking, guardrails, load balancing, and logging. It has 60,658 stars and 12,186 forks.
- agents
OpenAI Python SDK v3.28.0 adds agent environment suspension and expiration
OpenAI released openai-python v3.28.0 on October 9, 2026. The update adds API support for agent environment suspension and expiration, per the release notes. It also includes routine dependency bumps, including a PyJWT security constraint update to 2.14.0.
- open source
Hugging Face swift-transformers 1.3.5 fixes byte_fallback tokenization
Hugging Face released swift-transformers 1.3.5. The update makes Unigram and BPE tokenizers honor the byte_fallback configuration value on encode, reuses a shared byte fallback table, scopes GITHUB_TOKEN permissions per job, and allows swift-huggingface 1.0. Contributors RyanFarrar13 and hf-security-analysis bot made their first contributions.
- agents
Anthropic ships Claude Code v2.1.296 with gateway code policies and subagent auto-compact
Anthropic released Claude Code v2.1.296. It adds a `code` key to the Claude apps gateway's managed policies, applying CLI settings in Claude Desktop's Code tab and enabling gateway mode. Subagent frontmatter and `--agents` gain `autoCompactWindow`, plus new env vars for workflow subagent models and overloaded-request retry delays. Several managed-hook and gateway sign-in bugs were fixed.
- open source
Mistral TypeScript SDK v3.0.0 removes Runs API and changes security error types
Mistral AI released client-ts v3.0.0, a breaking update to its TypeScript SDK. The stable API surface drops six items including Runs, Runs.getRun, Runs.getRunHistory, Runs.listRuns, SecurityErrorCode, and dlv. SecurityError.code is now required, and twelve type signatures changed, including Workflows.runs becoming WorkflowsRuns and several encoding helpers switching to a CharEncoding type.