Apple Studies Scaling Laws for Mixture Pretraining Under Data Constraints
Apple researchers studied the trade-off of mixing scarce target data with abundant generic data in language model pretraining, running over 2,000 training runs. They found that too little target data underexposes the model, while too much leads to repetition and overfitting, providing scaling laws for optimal mixture ratios.
Sources and evidence
Summary last validated Aug 24, 2026
Reader actions
Report an issue
Use this for an incorrect summary, wrong source, duplicate story, or wrong category. Submissions are private and do not change the story automatically.
Related coverage
- research
NIST Study Maps How Toxic Adulterants in Fentanyl Vary Across US
NIST researchers analyzed fentanyl samples nationwide and found that the toxic substances mixed into the drug vary by region and shift over time. The study aims to help first responders and law enforcement identify dangerous adulterants in local supplies, which the agency says can help save lives.
- research
Apple Introduces Normalizing Trajectory Models for Few-Step Generation
Apple researchers introduced Normalizing Trajectory Models (NTM), which model each reverse diffusion step as an expressive conditional normalizing flow with exact likelihood training. NTM combines shallow invertible blocks per step with a deep parallel architecture, addressing few-step generation without sacrificing the likelihood framework that distillation, consistency training, or adversarial methods discard.
- research
NVIDIA teaches robots to assemble GB300 tester trays
NVIDIA describes how it taught robots to assemble tester trays for its Grace Blackwell GB300 superchip, which powers AI training and inference. The work required skilled physical labor in factories worldwide. NVIDIA says the project offered lessons in robot learning, mechanical intelligence, and traditional engineering.
- research
Ai2's Bolmo byte-level language model technique published in Nature
Ai2 announced that the technique behind Bolmo, its fully open byte-level language models, has been published in Nature. New checkpoints show the approach generalizes beyond the Olmo model family to other model families, according to the Allen Institute for AI.
- research
OpenAI Releases 722 Manuscripts of Math Solutions From Unreleased Model
OpenAI released 722 manuscripts containing solutions to hundreds of open mathematics problems, grouped into 372 result families, produced by an unreleased frontier model. The independent advisory group AGMAI said the batch includes solutions to hundreds of open questions. The release has raised questions about research ethics and academic conduct within the mathematical community.