Skip to content
Research

UNESCO-UCL Study Finds AI Models Drop Climate Facts at First Pushback in 8 of 10 Cases

The 15 models answered about 90% of climate questions correctly but caved to mild disagreement, and accuracy fell to 78.3% in Swahili from 88.6% in English.

·2 min read

Prefer Nakoda News on Google

Large language models abandon established climate facts at the first sign of mild disagreement eight times out of 10, according to a report by UNESCO and University College London presented at the COP31 climate summit in Antalya, Türkiye, the Emirates News Agency (WAM) reported.

UCL researchers tested 15 large language models. Asked neutral questions about climate science, the models answered about 90 per cent correctly. But when the researchers challenged those answers or put forward plausible-sounding fabrications, the models quickly agreed. In eight cases out of 10 they shifted at the first hint of mild disagreement, confirming errors such as the claim that carbon dioxide is not the main driver of climate change.

The report, titled "Can We Trust AI on Climate? Why it seeks to please the user, and fails on complex reasoning and multilingualism", warns policymakers and researchers who may be tempted to use AI models to develop climate policies. It finds the models are also poor at analysing new information and make more errors in less widely spoken languages.

Leona Verdadero, UNESCO's lead author of the report, said AI could make complex climate science accessible to all. "We need to be able to rely on AI as a gatekeeper of the established facts," she said. The models, she added, "have a design flaw" because they are "incentivized to agree with their user and deny these facts so readily."

Dr Maria Perez-Ortiz of UCL Computer Science, the senior author, said: "AI systems are remarkably good at explaining scientific knowledge, but this can give us a false sense of their capabilities. Our findings show why we must be careful when moving from asking AI what we know about climate change to asking what we should do about it."

Accuracy varied widely by language. The researchers found that multilingual models do not reason first and then translate. Their answers depend on the quality and quantity of training data in each language. In tests across 13 languages, the models were 88.6 per cent accurate on average in English and performed comparably in Spanish, French and Portuguese. Accuracy dropped to 83.6 per cent in Hindi and Bengali, 81.4 per cent in Urdu and 78.3 per cent in Swahili. The report notes that speakers of these languages live in regions disproportionately affected by climate change.

The published summary of the report did not name the 15 models or say which performed best or worst.

By

Nakoda News AI & Tech Desk

AI models, chips, products and research, and what they can and cannot do.

About the desk

Related coverage