Cohere releases megakernel serving engine for North Mini Code
Cohere published a serving engine for its 30B North Mini Code model built around a decode megakernel, running BF16 on a single H100. It reaches 292 tok/s at batch size 1, 1.58x faster than vLLM, and 1.25x-1.41x faster end-to-end, with continuous batching, paged attention, ragged sequences, and an OpenAI-compatible endpoint. Code is on GitHub.
Sources and evidence
Summary last validated Oct 5, 2026
Reader actions
Report an issue
Use this for an incorrect summary, wrong source, duplicate story, or wrong category. Submissions are private and do not change the story automatically.
Related coverage
- infrastructure
PyTorch details session-aware agentic inference with NVIDIA Dynamo
PyTorch published a blog post describing session-aware agentic inference with NVIDIA Dynamo. It explains that agentic workloads differ from single-turn chat: an agent session can include a large initial prefill, repeated model calls, and parallel subagents, changing the traffic an inference server handles.
- infrastructure
AWS details multi-team GPU sharing on SageMaker HyperPod
AWS published a reference architecture for sharing one Amazon SageMaker HyperPod EKS cluster across multiple teams. It uses AWS IAM Identity Center for authentication, per-team SageMaker Domains and Kubernetes namespaces for isolation, HyperPod Task Governance for fairness, and namespace-level cost allocation for chargeback.
- infrastructure
MIT's Christina Delimitrou targets data center energy efficiency
MIT Associate Professor Christina Delimitrou is rethinking how large cloud computing systems operate to reduce the environmental impact of data centers. Her work focuses on making these facilities more energy efficient as their environmental threat grows.
- industry
Cohere opens new Ottawa office to expand Canadian footprint
Cohere announced a new office in Ottawa, expanding its Canadian footprint. The company, which offers enterprise AI products including the North platform and Command models, said the location extends its presence in Canada. The announcement did not include details on staffing, investment, or specific government work.
- infrastructure
Amazon Quick and Bedrock Knowledge Bases add real-time document-level access control for RAG
AWS published guidance on enforcing document-level access controls for enterprise RAG using Amazon Quick and Amazon Bedrock Knowledge Bases. The approach verifies user permissions directly with authoritative sources such as SharePoint, Google Drive, and Confluence at query time, rather than relying on pre-indexed permission copies, so answers only draw on documents each user is authorized to see.