BenchMIRT: Auditing LLM Benchmarks Question by Question
AI2 introduces BenchMIRT, a method to audit LLM benchmarks question by question, revealing which capabilities they actually measure. This helps researchers build smaller, more focused, and easier-to-interpret evaluations, addressing the question of what benchmarks truly assess.
Sources and evidence
Summary last validated Sep 1, 2026
Reader actions
Report an issue
Use this for an incorrect summary, wrong source, duplicate story, or wrong category. Submissions are private and do not change the story automatically.
Related coverage
- research
NIST Study Maps How Toxic Adulterants in Fentanyl Vary Across US
NIST researchers analyzed fentanyl samples nationwide and found that the toxic substances mixed into the drug vary by region and shift over time. The study aims to help first responders and law enforcement identify dangerous adulterants in local supplies, which the agency says can help save lives.
- research
NVIDIA teaches robots to assemble GB300 tester trays
NVIDIA describes how it taught robots to assemble tester trays for its Grace Blackwell GB300 superchip, which powers AI training and inference. The work required skilled physical labor in factories worldwide. NVIDIA says the project offered lessons in robot learning, mechanical intelligence, and traditional engineering.
- research
Ai2's Bolmo byte-level language model technique published in Nature
Ai2 announced that the technique behind Bolmo, its fully open byte-level language models, has been published in Nature. New checkpoints show the approach generalizes beyond the Olmo model family to other model families, according to the Allen Institute for AI.
- research
Microsoft Researcher Jennifer Neville Discusses AI Surprising Failures
Microsoft Research podcast features Jennifer Neville, who discusses her unexpected path into computer science and her work identifying "surprising failures" that make it hard for AI to handle complexity. The episode explores how failure teaches lessons and led to her professional sweet spot in AI research.
- research
Google Research Debuts Earth AI Population Dynamics Foundation Model for Public Health
Google Research introduced the Population Dynamics Foundation Model (PDFM), part of Google Earth AI, which compresses privacy-preserving search trends, mobility, built-environment density, and environmental data into location embeddings. Five partner-driven case studies show these off-the-shelf embeddings matched or improved conventional inputs across disease domains and epidemiological tasks without task-specific fine-tuning.