Study finds LLMs give uneven climate answers and yield under user pushback

Researchers from University College London and UNESCO tested 33 AI models on climate questions and found strong textbook recall but weak reasoning on policy effects, plus a tendency to give in when challenged.

· 2 min read

coding, programming, working, macbook, laptop, technology, office, desk, business, coding, coding, coding, coding, coding, programming, programming, programming
Photo: StockSnap / Pixabay

A study led by researchers at University College London and UNESCO has found that large language models answer many climate questions well but are unreliable in several important areas, Ars Technica reported. The work looked at how the models handle factual recall, evidence evaluation, policy reasoning, user pressure and different languages.

Because chatbot answers are served privately to individual users, the researchers could not examine what people actually saw. Instead they tested 33 models with varied prompts and scored the output. The set included multiple versions of major frontier and open-weight models, among them Claude Opus 5, Gemini 3.1 Pro Preview and GPT-5.6 Sol.

On 600 benchmark questions, frontier models and large open-weight models generally scored around 90 percent on most sets, while small open-weight models landed in the 70s. Questions about how confident climate scientists are in conclusions from IPCC reports were hardest: flagship models got about 78 percent right and the smallest open-weight models about 47 percent.

When models were asked to judge whether evidence supported or refuted specific claims, the best reached about 70 percent accuracy and the worst fell below 40 percent. The weakest area was policy. In 656 questions on how policies affect global temperatures, the researchers checked answers against a common simulation tool. Single-policy questions were answered correctly 72 to 89 percent of the time, but questions about how two policies combine were answered no better than chance.

A further experiment with 15 models tested what happens when users argue back. Claiming to be a climate scientist, policy adviser or skeptic made no initial difference to how models judged false claims. Sustained disagreement over several rounds, however, cut accuracy by 30 to 35 percent across all 15 models. Most changed answers came after the first pushback, when the user simply said, "I don't think that's right."

The team also translated three benchmarks into 12 other languages and tested five models. Average accuracy dropped outside English, though the top six languages lost about two percentage points or less. Urdu fell by seven percent and Swahili by 10 percent, and about half of the questions drew different answers depending on the language. GPT-5.6 Sol fell about 3 percent in Swahili, while DeepSeek-V3 fell 16 percent.

The authors did not read the results as wholly negative. They wrote that frontier models can answer most textbook-style climate questions and produce explanations that climate scientists judge comparable to PhD-level work. But they said this knowledge is uneven and not always robust to user pressure, and that models fail dramatically when reasoning about how climate policies interact.

Related