AWS Releases Open-Source Benchmarking Harness for OpenAI Models on Amazon Bedrock
AWS published an open-source benchmarking harness for comparing OpenAI models on Amazon Bedrock. It measures cost per correct answer, agent trajectory cost, and rubric-graded deliverable quality, arguing that dollars per million tokens misses what production workloads actually pay for: outcomes.
Sources and evidence
Summary last validated Sep 11, 2026
Reader actions
Report an issue
Use this for an incorrect summary, wrong source, duplicate story, or wrong category. Submissions are private and do not change the story automatically.
Related coverage
- models
AWS Adds Bedrock Models, Faster AgentCore Agents, Knowledge Base Sync
AWS's September 2026 recap covers Amazon Bedrock, Bedrock AgentCore, and Strands updates: broader model choice, faster serverless agents with built-in evaluation, and automated knowledge base syncing via native enterprise connectors. The post aggregates the month's releases for AI builders across AWS's model and agent tooling.
- agents
Postman runs Agent Mode for 40 million developers on Amazon Bedrock
Postman and AWS detailed the architecture behind Postman's Agent Mode, which serves 40 million developers on Amazon Bedrock. The post covers controlling tool sprawl, exposing schema-based reads, and treating context as the real bottleneck when running an AI agent at scale rather than in a demo.
- open source
Mistral TypeScript SDK v3.0.0 removes Runs API and changes security error types
Mistral AI released client-ts v3.0.0, a breaking update to its TypeScript SDK. The stable API surface drops six items including Runs, Runs.getRun, Runs.getRunHistory, Runs.listRuns, SecurityErrorCode, and dlv. SecurityError.code is now required, and twelve type signatures changed, including Workflows.runs becoming WorkflowsRuns and several encoding helpers switching to a CharEncoding type.
- open source
NVIDIA Details Five-Step Workflow for SimReady Robotics Assets
NVIDIA published a five-step workflow for converting CAD assets into SimReady robotics simulation assets using Omniverse libraries, SimReady Foundation specifications, and agentic NVIDIA skills. The process covers configuring and validating materials, collision geometry, joints, and physics properties beyond simple OpenUSD geometry conversion, preparing assets before robot behavior testing.
- agents
BlockRun and Incarna use Amazon Bedrock AgentCore payments for per-inference agent billing
Amazon Bedrock AgentCore payments lets AI agents pay for services on demand with infrastructure-enforced spending limits. Incarna's agents pay BlockRun for model inference one request at a time over x402, reducing the work of adding x402 payment support from months to days, according to an AWS Machine Learning Blog post.