Table of Contents Author Guidelines Submit a Manuscript
Applied Computational Intelligence and Soft Computing
Volume 2017 (2017), Article ID 5134962, 13 pages
https://doi.org/10.1155/2017/5134962
Research Article

Distributed Nonparametric and Semiparametric Regression on SPARK for Big Data Forecasting

Clausthal University of Technology, Clausthal-Zellerfeld, Germany

Correspondence should be addressed to Jelena Fiosina

Received 22 July 2016; Accepted 22 November 2016; Published 8 March 2017

Academic Editor: Francesco Carlo Morabito

Copyright © 2017 Jelena Fiosina and Maksims Fiosins. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Forecasting in big datasets is a common but complicated task, which cannot be executed using the well-known parametric linear regression. However, nonparametric and semiparametric methods, which enable forecasting by building nonlinear data models, are computationally intensive and lack sufficient scalability to cope with big datasets to extract successful results in a reasonable time. We present distributed parallel versions of some nonparametric and semiparametric regression models. We used MapReduce paradigm and describe the algorithms in terms of SPARK data structures to parallelize the calculations. The forecasting accuracy of the proposed algorithms is compared with the linear regression model, which is the only forecasting model currently having parallel distributed realization within the SPARK framework to address big data problems. The advantages of the parallelization of the algorithm are also provided. We validate our models conducting various numerical experiments: evaluating the goodness of fit, analyzing how increasing dataset size influences time consumption, and analyzing time consumption by varying the degree of parallelism (number of workers) in the distributed realization.