2015 IEEE 31st International Conference on Data Engineering (ICDE) (2015)
Seoul, South Korea
April 13, 2015 to April 17, 2015
Sebastian Schelter , Technische Universität Berlin, Germany
Juan Soto , Technische Universität Berlin, Germany
Volker Markl , Technische Universität Berlin, Germany
Douglas Burdick , IBM Almaden Research Center, USA
Berthold Reinwald , IBM Almaden Research Center, USA
Alexandre Evfimievski , IBM Almaden Research Center, USA
Meta learning techniques such as cross-validation and ensemble learning are crucial for applying machine learning to real-world use cases. These techniques first generate samples from input data, and then train and evaluate machine learning models on these samples. For meta learning on large datasets, the efficient generation of samples becomes problematic, especially when the data is stored distributed in a block-partitioned representation, and processed on a shared-nothing cluster. We present a novel, parallel algorithm for efficient sample generation from large, block-partitioned datasets in a shared-nothing architecture. This algorithm executes in a single pass over the data, and minimizes inter-machine communication. The algorithm supports a wide variety of sample generation techniques through an embedded user-defined sampling function. We illustrate how to implement distributed sample generation for popular meta learning techniques such as hold-out tests, k-fold cross-validation, and bagging, using our algorithm and present an experimental evaluation on datasets with billions of datapoints.
Training, Partitioning algorithms, Electronic mail, Indexes, Distributed databases, Data models, Predictive models
S. Schelter, J. Soto, V. Markl, D. Burdick, B. Reinwald and A. Evfimievski, "Efficient sample generation for scalable meta learning," 2015 IEEE 31st International Conference on Data Engineering (ICDE), Seoul, South Korea, 2015, pp. 1191-1202.