METR update shows AI progress on judgment-heavy tasks lags behind coding gains

METR's revised time horizon plot adds longer and more open-ended tasks, and finds models trail humans on work such as designing AI evaluations, though the newest systems show signs of improvement.

· 2 min read

Robot arm playing chess with a human hand
Photo: Amos K / Unsplash

The AI evaluation nonprofit METR has released an update to its widely shared time horizon plot, and the results suggest some AI capabilities keep improving exponentially while models handle judgment-heavy work less well than coding. MIT Technology Review reported on the update.

The original plot rated models by the length of the tasks they could complete, with each task's difficulty based on how long human testers took to finish it. It showed that the length of tasks frontier models could successfully complete doubled every four months. The plot was often misread because of the complexity of its measures, according to the report.

The new version adds tasks that humans are estimated to need far longer to complete, along with more open-ended ones. This matters because AI systems have become very good at coding and math, where software can automatically check whether an answer works. Reinforcement learning, in which systems repeat a task and are reinforced for successes, is much harder to apply when success is poorly defined or slow to judge, such as when choosing a new research direction.

In one new task, models were asked to design AI evaluations that METR could use in its research, and METR staff scored the results. Even though the most advanced models now beat almost every human at software engineering, none of the evaluations they produced was considered useful by METR employees. In another task, models scored philosophical arguments, and they again fell short of human performance. METR notes that its staff disagree on how relevant this task is to automating AI research and development.

There are signs of progress. Some of the newest models, including Opus 5.5, tend to reach what METR calls borderline above the bar on the evaluation design tasks, meaning the work is not yet usable but is promising. A year ago, every model was rated clearly below the bar.

Experts quoted by MIT Technology Review welcomed the change. Sayash Kapoor, a computer scientist and incoming professor at UC Berkeley, said AI evaluations need to become more creative. He pointed to an August preprint by him and collaborators that found models cannot yet reproduce human computer science research on their own. He also cited a Wednesday OpenAI blog post saying its models do worse on real tasks from its employees than on the easier, verifiable tasks in METR's original suite.

The report also lays out limits. Because the tasks are hard to design and score, METR includes only a few, which makes extrapolation difficult. Najoung Kim of Boston University said she is unsure how the observed improvement would extend. Completion times are mostly estimates, since paying people to complete tasks that may take hundreds of hours is prohibitively expensive. For the philosophical task, one person rated most of the arguments, and models were judged on agreement with that individual.

METR's Matt Putz said he does not expect trends for such tasks to become as clean and robust as those for easy-to-specify ones. Experts agree the new tasks are a step forward, but they say definitive answers on whether and how fast automated AI research and development might arrive remain far off.

Related