Access to an AI assistant built on World Bank reports saved no significant time

Policy professionals need reliable facts, but general AI models often invent information. To address gaps in epistemic humility¹ and in-the-wild² evidence, researchers built AVA, an AI assistant restricted to a curated library of World Bank reports. The system breaks down questions, searches the text, and drafts answers with clickable page citations. If it cannot find enough evidence, it refuses to answer. The researchers deployed the tool for five months to more than 2,200 professionals across 116 countries. They analysed system logs, surveyed users, and interviewed 20 participants to measure trust, abstention, taking the tool into their own working routine, and time saved.

The system’s refusal rate dropped from between 40 and 70 percent in its early weeks to under 10 percent after the researchers expanded the library from about 50 reports to over 4,000. The researchers randomised registrants 80:20 into access and holdout groups. While difference-in-differences estimates associate multiple-session use with 1.8 to 3.3 more self-reported hours saved a week than single-session use, significant only at the 10 percent level, the broader experiment found no significant time savings for the overall group given access. Users trusted the system because it cited specific pages and admitted when it lacked information, though some found the blunt refusals frustrating. Because 83.8 percent of queries were in English, the study provides little evidence on how well the tool handles other languages. The authors argue that specialised AI tools should direct users to general models when a question falls outside their verified library.

An AI system saying it does not know an answer is a measure of its database size rather than a cognitive capability. The system can refuse to answer because a closed document library has a defined edge. When an AI operates without a bounded corpus, it has no edge to measure against, leaving it unresolved whether this refusal mechanism can function on the open web.

Today’s links: Assorted links for 6 September 2026.

Footnotes

1. Epistemic humility is the willingness to realise that one’s knowledge is limited and might be wrong. This concept is normally used to prevent overconfidence and encourage people to learn from their mistakes. ↩

2. In-the-wild refers to observing software in everyday situations rather than in a controlled laboratory. Researchers normally use this approach to gather realistic evidence and familiarise themselves with actual user behaviour. ↩