Parallel and Distributed Processing Symposium, International (2011)
Anchorage, Alaska USA
May 16, 2011 to May 20, 2011
We introduce Dryad Opt, a library that enables massively parallel and distributed execution of optimization algorithms for solving hard problems. Dryad Opt performs an exhaustive search of the solution space using branch-and-bound, by recursively splitting the original problem into many simpler sub problems. It uses both parallelism (at the core level) and distributed execution (at the machine level). Dryad Opt provides a simple yet powerful interface to its users, who only need to implement sequential code to process individual sub problems (either by solving them in full or generating new sub problems). The parallelism and distribution are handled automatically by Dryad Opt, and are invisible to the user. The distinctive feature of our system is that it is implemented on top of Dryad LINQ, a distributed data-parallel execution engine similar to Hadoop and Map-Reduce. Despite the fact that these engines offer a constrained application model, with restricted communication patterns, our experiments show that careful design choices allow Dryad Opt to scale linearly with the number of machines, with very little overhead.
Mihai Budiu, Daniel Delling, Renato F. Werneck, "DryadOpt: Branch-and-Bound on Distributed Data-Parallel Execution Engines", Parallel and Distributed Processing Symposium, International, vol. 00, no. , pp. 1278-1289, 2011, doi:10.1109/IPDPS.2011.121