The Pulse

Apple Researchers Report Faster Retrieval With Small Neural Networks for Maximum Inner Product Search

Apple describes small neural networks that route queries or rewrite them to speed up maximum inner product search. In tests, the learned approaches improved retrieval performance against centroid routing and raw queries.

AI.info Team · Written with AI · fact-checked · 2026-11-06

A small SupportNet router reached 95% routing recall in 1.74 milliseconds per query, compared with 3.36 milliseconds for centroid routing, Apple reports. The result comes from the company’s work on maximum inner product search, a retrieval method used in systems including retrieval-augmented generation (RAG).

Apple Machine Learning Research published the findings on November 5, 2026, describing a method that trains small neural networks offline for a fixed database of keys. The page lists a paper titled “Amortizing Maximum Inner Product Search with Learned Support Functions,” presented at ICML 2026.

Why centroid routing can miss the best key

Maximum inner product search, or MIPS, finds the key vector with the largest inner product against a query vector. In RAG, the document associated with the best-matching key is typically prefixed to the user’s question.

A common way to reduce search costs divides keys into clusters and compares a query with each cluster’s centroid. The system then searches the chosen cluster. But centroid routing can miss a strong match when a key sits far from its cluster’s centroid.

The research describes a mathematical property of MIPS: the function that returns the largest inner product between a query and any key in a set is that set’s support function. The function is convex, and its gradient at a query returns the best-matching key.

SupportNet scores clusters for each query

SupportNet learns to estimate the support functions of clusters, helping route a query according to how well any key in a cluster matches, not only how close its centroid is. The researchers trained models of four sizes on Natural Questions, a retrieval benchmark that pairs questions with passages from a document database; the reported setup contains roughly 2.6 million unique passages split into 128 clusters.

At a fixed amount of database scanning, the learned routers found the correct partition more often than the centroid baseline. When scanning 5% of the database, the models recovered the true partition about 90–93% of the time, up to 92.6% for the mid-size model, versus 87.5% for centroid routing.

The latency comparison includes the extra computation required to run the neural networks. The small SupportNet reached 95% routing recall about 1.9 times faster than centroid routing, while the large model reached 97% in 1.74 milliseconds.

KeyNet redirects queries into an IVF index

KeyNet predicts the best-matching key vector and supplies that prediction as a rewritten query to an existing inverted file index, or IVF index. The researchers evaluated KeyNets with a FAISS IVF index on the Natural Questions dataset.

With the largest KeyNet, reaching 99% recall took 0.23 milliseconds, compared with 1.8 milliseconds for the raw query. When the index scanned 0.001% of the database, KeyNet-mapped queries retrieved the true key 89–97% of the time; the largest model reached up to 96.7%, compared with 86% for the raw query.

For 99% recall, reaching that target took up to about 11 times fewer scanned keys with KeyNet: 0.009% of the database, compared with 0.10% for the raw query. The examples use Natural Questions served through FAISS, and the page says the same trend appeared on other datasets.

The two networks have different roles

The page recommends KeyNet when inference speed is the bottleneck and the system only needs an approximate best key. SupportNet can provide a calibrated score for re-ranking, thresholding or downstream use, and stays competitive at small model sizes when compute is tight.

For a very large-scale MIPS system with a huge set of keys, the page describes using both: a SupportNet router over all clusters, followed by a KeyNet for each cluster to help its index search for the best key.