Google Open-Sources Agile Deliberation, a Tool for Building Classifiers for Gray-Area Images

The company described the work in a blog post on November 4 by research scientist Otilia Stretcu and student researcher Leijie Wang. The underlying paper, "Agile Deliberation: Concept Deliberation for Subjective Visual Classification", appears at CVPR 2026. Its authors include Wang, Stretcu, Ranjay Krishna and Ariel Fuxman. The blog post says the work was done in partnership with Google Ads.

The problem it targets is that most classifier-building tools assume users already know where a concept's boundary lies. The researchers wrote that they built the system after interviewing content moderation experts. Those experts usually started with a vague idea and refined it as they looked at difficult examples, a process the paper calls "concept deliberation". Rather than treating people as "static data labelers", the blog post says, the framework treats building a classifier as an iterative conversation.

The process has two stages. In concept scoping, the system breaks a broad concept into sub-concepts and explicit exclusions. For healthy food, salads, grain bowls and lean proteins might fall inside the definition, while processed food falls outside. In concept iteration, vision-language models find borderline images that test the current written definition. The user decides whether each one fits, and that feedback drives an automatic prompt-optimisation loop. Google says the system picks images by how ambiguous their meaning is, not only by how uncertain the model is, as standard active learning does.

The output is a structured text definition that works both as a policy people can read and as the prompt for the classifier. According to MarkTechPost's analysis of the release, the framework prompts Gemini 2.5 Pro and Gemini 2.5 Flash through the API without fine-tuning, and uses CLIP embeddings with a FAISS index for image retrieval.

The researchers did not use static benchmarks, arguing that subjective concepts have no fixed ground truth. Instead they ran 18 live user sessions of about 90 minutes each on two concepts: "healthy food" and "pay-to-play", which covers gaming and entertainment clickbait that promises misleading financial or digital rewards. According to the paper's abstract, the framework scored 7.5% higher on F1 than automated decomposition baselines and more than 3% higher than manual deliberation.

MarkTechPost reported that the paper's results table gives Agile Deliberation an F1 score of 0.59 on pay-to-play and 0.58 on healthy food, against 0.48 for a zero-shot Gemini 2.5 Flash classifier on both. The gains came at a cost: on pay-to-play, recall fell from 0.80 to 0.63 while precision rose from 0.35 to 0.57.

Participants reported significantly less effort and stress than when they edited prompts by hand. Google said every participant preferred Agile Deliberation to the manual methods.

The code is in Google's google-research GitHub repository. It includes a demo notebook that walks through the healthy food example and needs a Gemini API key to run. MarkTechPost reported that the default image index takes about 260MB, while the full set of 100 shards built on the LAION-400M dataset takes about 13GB.