Second IEEE International Conference on e-Science and Grid Computing (e-Science'06)
Grid Enabling Data De-Duplication
Amsterdam, Netherlands
December 04-December 06
ISBN: 0-7695-2734-5
A Grid based implementation of a system for finding duplicates in large databases is described. The solution is scalable to many nodes and does not suffer the problems found in other implementations that can result of loss of data and/or deadlock. The system may be applied to conventional de-duplication problems such as found in address management as well as more advanced problems such as banned image detection. The system uses the AURA pattern match methods implemented within a service oriented architecture. The approach builds on the PMS and PMC technology developed in the DAME eScience project.
Citation:
Jim Austin, Aaron Turner, Sujeewa Alwis, "Grid Enabling Data De-Duplication," e-science, pp.2, Second IEEE International Conference on e-Science and Grid Computing (e-Science'06), 2006