NV-Tree: An Efficient Disk-Based Index for Approximate Search in Very Large High-Dimensional Collections
Issue No. 05 - May (2009 vol. 31)
Björn Þór Jónsson , Reykjavik University, Reykjavik
Friðrik Heiðar Ásmundsson , Reykjavik University, Reykjavik
Herwig Lejsek , Reykjavik University, Reykjavik
Laurent Amsaleg , CNRS-IRISA, RENNES
Over the last two decades, much research effort has been spent on nearest neighbor search in high-dimensional data sets. Most of the approaches published thus far have, however, only been tested on rather small collections. When large collections have been considered, high-performance environments have been used, in particular systems with a large main memory. Accessing data on disk has largely been avoided because disk operations are considered to be too slow. It has been shown, however, that using large amounts of memory is generally not an economic choice. Therefore, we propose the NV-tree, which is a very efficient disk-based data structure that can give good approximate answers to nearest neighbor queries with a single disk operation, even for very large collections of high-dimensional data. Using a single NV-tree, the returned results have high recall but contain a number of false positives. By combining two or three NV-trees, most of those false positives can be avoided while retaining the high recall. Finally, we compare the NV-tree to Locality Sensitive Hashing, a popular method for \epsilon-distance search. We show that they return results of similar quality, but the NV-tree uses many fewer disk reads.
High-dimensional indexing, multimedia indexing, very large databases, approximate searches.
Björn Þór Jónsson, Friðrik Heiðar Ásmundsson, Herwig Lejsek, Laurent Amsaleg, "NV-Tree: An Efficient Disk-Based Index for Approximate Search in Very Large High-Dimensional Collections", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol. 31, no. , pp. 869-883, May 2009, doi:10.1109/TPAMI.2008.130