MapReduce / Nutch Evaluation

24.03.2006

Abstract:
MapReduce is a model for parallel programming. It hides the details concernig distribution of data, parallelizing of computation and failure handling from the programmer. This means even programmers without the knowledge of parallel programming can use it.

However, the model does have som limitations. The way a MapReduce computation can be configured means that there exists an artificial barrier between each job, where the next job depends on the whole previous job. Instead of this we show that by defining dependencies on files instead of jobs we exploit the parallelism of the system resources in a better way. It is also possible to replace a set of identity tasks with sort tasks, and in this way save unnecessary read and write operations. By doing so, we increase the performance of the MapReduce computations.