21st International Conference on Advanced Information Networking and Applications (AINA '07) (2007)
Niagara Falls, Ontario, Canada
May 21, 2007 to May 23, 2007
DOI Bookmark: http://doi.ieeecomputersociety.org/10.1109/AINA.2007.60
Odysseas Papapetrou , Leibniz Universitat Hannover, Germany
Wolf Siberski , Leibniz Universitat Hannover, Germany
Wolf-Tilo Balke , Leibniz Universitat Hannover, Germany
Wolfgang Nejdl , Leibniz Universitat Hannover, Germany
Distributed Hash Tables (DHTs) are very efficient for querying based on key lookups, if only a small number of keys has to be registered by each individual peer. However, building huge term indexes, as required for IR-style keyword search, are impractical with plain DHTs. Due to the large sizes of document term vocabularies, joining peers cause huge amounts of key inserts, and subsequently large numbers of index maintenance messages. Thus, the key to exploiting DHTs for distributed information retrieval is to reduce index maintenance. We show that this can be achieved by combining DHTs with peer clustering. Peers are first clustered into communities, each of the communities having a representative super-peer. Then all occurrences of a term in a community are published to the global DHT in a batch by the representative super-peer. Our evaluation shows that this reduces index maintenance cost by an order of magnitude, while still keeping a complete and correct term index for query processing.
O. Papapetrou, W. Balke, W. Nejdl and W. Siberski, "DHTs over Peer Clusters for Distributed Information Retrieval," 21st International Conference on Advanced Information Networking and Applications (AINA '07)(AINA), Niagara Falls, Ontario, Canada, 2007, pp. 84-93.