Apple Researchers Train Small Networks to Speed Up Vector Search for RAG
In an ICML 2026 paper, Apple shows learned query routing that reaches 99% recall up to about eight times faster than searching with the raw query.
By Nakoda News Desk
·2 min read
Prefer Nakoda News on Google
Apple researchers have shown that small neural networks, trained once for a fixed database, can make the vector search behind retrieval-augmented generation faster and more accurate, according to a paper presented at ICML 2026 and summarised on Apple's machine learning research blog.
The work targets maximum inner product search, the step in which a system compares a query vector against millions of stored vectors to find the closest match, such as the document a chatbot should read before answering. Standard approximate methods group the stored vectors into clusters and send each query to the cluster whose centre looks nearest, which saves computation but sometimes sends the query to the wrong place. The Apple team argues those methods ignore what real queries look like: in RAG, short user questions are searched against long, curated passages, and production systems usually serve a stable, predictable stream of queries.
The paper, titled Amortizing Maximum Inner Product Search with Learned Support Functions, builds on a property from convex analysis. The function that returns the best inner product between a query and a set of keys is that set's support function, which is convex, and its gradient points to the best-matching key. The researchers train two kinds of model on queries sampled from the expected distribution, with the true answers computed once by exhaustive search. SupportNet learns the support function itself and is used to route queries to clusters. KeyNet maps a query directly to an estimate of its best-matching key.
On the Natural Questions dataset, about 2.6 million passages split into 128 clusters, the smallest SupportNet model reached 95% routing recall in 1.74 milliseconds against 3.36 milliseconds for standard centroid routing, roughly 1.9 times faster. When scanning 5% of the database, learned routers found the right partition up to 92.6% of the time, compared with 87.5% for the centroid method.
Used with a FAISS IVF index, KeyNet reached 99% recall in 0.23 milliseconds with the largest model, against 1.8 milliseconds for the raw query, and needed about 11 times fewer keys to do so. Apple's figures for KeyNet include the time taken to map each query.
The approach trades an upfront training cost for cheaper searches later, and it depends on queries resembling the ones the networks were trained on. Apple did not say whether the method is used in any of its products.
Sources
- Apple Machine Learning Research · Faster Retrieval for Better, More Efficient RAG
Produced with AI assistance from the sources above. How we work
By
Nakoda News Desk
Nakoda News covers AI and business. Every story links the sources it draws on; how we work is set out in our editorial standards.