Journal of Electrical and Computer Engineering

Volume 2018, Article ID 6270816, 10 pages

https://doi.org/10.1155/2018/6270816

## Parallel Nonnegative Matrix Factorization with Manifold Regularization

State Key Laboratory of Mathematical Engineering and Advanced Computing, Zhengzhou, Henan 450001, China

Correspondence should be addressed to Fudong Liu; moc.621@ydyfwl

Received 12 November 2017; Revised 7 February 2018; Accepted 15 March 2018; Published 2 May 2018

Academic Editor: Tongliang Liu

Copyright © 2018 Fudong Liu et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

#### Abstract

Nonnegative matrix factorization (NMF) decomposes a high-dimensional nonnegative matrix into the product of two reduced dimensional nonnegative matrices. However, conventional NMF neither qualifies large-scale datasets as it maintains all data in memory nor preserves the geometrical structure of data which is needed in some practical tasks. In this paper, we propose a parallel NMF with manifold regularization method (PNMF-M) to overcome the aforementioned deficiencies by parallelizing the manifold regularized NMF on distributed computing system. In particular, PNMF-M distributes both data samples and factor matrices to multiple computing nodes instead of loading the whole dataset in a single node and updates both factor matrices locally on each node. In this way, PNMF-M succeeds to resolve the pressure of memory consumption for large-scale datasets and to speed up the computation by parallelization. For constructing the adjacency matrix in manifold regularization, we propose a two-step distributed graph construction method, which is proved to be equivalent to the batch construction method. Experimental results on popular text corpora and image datasets demonstrate that PNMF-M significantly improves both scalability and time efficiency of conventional NMF thanks to the parallelization on distributed computing system; meanwhile it significantly enhances the representation ability of conventional NMF thanks to the incorporated manifold regularization.

#### 1. Introduction

Data representation is a fundamental problem in data analysis. A good representation typically uncovers the latent structure of a dataset by reducing the dimensionality of data. Several methods including principal component analysis (PCA), linear discriminant analysis (LDA), and vector quantization (VQ) have addressed this issue. Recently, nonnegative matrix factorization (NMF) [1] incorporates nonnegativity constraint to obtain parts-based representation of data, and thus it has been widely applied in many applications, such as document clustering [2, 3], image recognition [4, 5], audio processing [6], and video processing [7].

However, conventional NMF suffers from a few deficiencies: conventional NMF usually works in batch mode and requires all data to reside in memory, and this leads to tremendous storage overhead as the increase of the data samples and conventional NMF ignores the geometrical structure embedded in data and causes unsatisfactory representation ability. To overcome the first deficiency, either parallel or distributed algorithms have been proposed for NMF to fit for large-scale datasets. Kanjani [8] utilized multithreading to develop a parallel NMF (PNMF) based on multicore machine. Robila and Maciak [9] introduced two thread-level parallel versions for traditional multiplicative solution and adaptive projected gradient method. However, their methods are prohibited for large-scale datasets due to the memory limitation of a single computer. Liu et al. [10] proposed a distributed NMF (DNMF) to analyze large-scale web dyadic data and verified its effectiveness on distributed computing systems. Dong et al. [11] also attempted to design a PNMF based on the distributed memory platform with the message passing interface library. Although all the above parallel NMF algorithms achieve a considerable speedup in terms of scalability, they cannot consider the geometric structure in dataset. To overcome the second deficiency, Cai et al. [12] proposed graph-regularized NMF (GNMF) which extended conventional NMF by constructing an affinity graph [13] to encode the geometrical information of data and enhanced representation ability. Gu et al. [14] further extended GNMF to avoid trivial solution and scale transfer problems by imposing a normalized cut-like constraint on the cluster assignment matrix. Lu et al. [15] incorporated manifold regularization into NMF for hyperspectral unmixing and obtained desirable unmixing performance. These improved algorithms get better representation ability but work inefficiently for large-scale datasets. Liu et al. [16] introduced the geometric structure of data into incremental NMF [17, 18] and utilized two efficient sparse approximations, buffering and random projected tree, to process large-scale datasets. Yu et al. [19] also presented an incremental GNMF algorithm to improve scalability. But these algorithms only performed well for incremental or streaming datasets and could not deal with large-scale batch datasets. In addition, Guan et al. [20] and Liu et al. [21], respectively, introduce Manhattan distance and large-cone penalty for NMF to improve representation and generalization ability.

In conclusion, none of the above works can simultaneously overcome both deficiencies due to the great computation for calculating the decomposition and the storage requirement of the adjacency matrix. In this paper, we take the best of advantages of parallel NMF and manifold regularized NMF and design a parallel NMF with manifold regularization method (PNMF-M) by parallelizing manifold regularized NMF on distributed computing systems. In particular, PNMF-M distributes both data samples and factor matrices to multiple computing nodes in a balanced way and parallelizes the update for both factor matrices locally on each node. Since the graph construction is the bottleneck of the computation of PNMF-M, we adopt a two-step distributed graph construction method to compute the adjacency matrix and obtain an adjacent graph equivalent to that constructed in batch mode. Experimental results on popular text corpora and image datasets show that PNMF-M not only outperforms conventional NMF in terms of both scalability and time efficiency by parallelization, but also significantly enhances the representation ability of conventional NMF by incorporating manifold regularization.

#### 2. Related Work

This section gives a brief review of PNMF, GNMF, and their corresponding popular algorithms. Details are as follows.

##### 2.1. PNMF

Due to working in batch mode, conventional NMF requires to process masses of large matrix operations, which is inevitably confronted with high time overhead. Parallel processing is often an efficient method to speed up computing intensive algorithms. In general, the parallelism of the algorithms is embodied in two aspects. The first aspect is computation flow, but unfortunately conventional NMF is highly correlated among the computing steps and could not be decomposed into uncorrelated tasks. So the second aspect, data structure, is taken into account. A feasible plan is to divide the data into small blocks and assign them to different processing units.

Robila and Maciak [9] implement a common example of parallel NMF. Let the input data matrix , the basis matrix , and the coefficient matrix . Given that there are processing units, both and are exactly divisible by . They divide the above three matrices into equally sized submatrices by column; that is, , and . They still minimize the cost function of conventional NMF and obtain the update rules for and as follows:

where denotes the iteration number, , and .

By parallelizing conventional NMF, PNMF can achieve the same representation ability, while reducing the time complexity.

##### 2.2. GNMF

In common, conventional NMF performs the factorization in the Euclidean space, which cannot get satisfactory solution. Based on manifold assumption [22], GNMF embeds the intrinsic geometric structure of the data space in conventional NMF to overcome the above defect.

One of the representative algorithms for GNMF is proposed by Cai et al. [12]. They define the graph weight matrix , which is obtained by computing the weight relationship between any two data samples in input data matrix . Furthermore, another two matrices are derived from . One is the diagonal matrix whose entries are row or column sum of , that is, , and the other is , which is called graph Laplacian. They utilize Lagrangian multiplier method to solve the above problem and derive the following update rules:

where is the regularization parameter, denotes the iteration number, , and .

By incorporating the geometric structure, GNMF can significantly improve representation ability in comparison to conventional NMF.

#### 3. Parallel Manifold Regularized Nonnegative Matrix Factorization

In conventional GNMF, the intrinsic low dimensional manifold structure embedded in the dataset is expressed by by graph weight matrix . From the update rule (2), we can easily see that the manifold terms in conventional GNMF give rise to a lot of additional computations compared with NMF. Therefore, we describe the proposed PNMF-M in this section, which improves this issue by distributing both data samples and factor matrices to multiple computing nodes while parallelizing the update for both factor matrices and manifold construction.

##### 3.1. Data Partition and Distributed Computing

We still assume , , , , and as the input data matrix, basis matrix, coefficient matrix, graph weight matrix, and number of processors, respectively.

PNMF-M divides and into blocks in the column direction, namely, , , as shown in Figures 1(a) and 1(b); thus each of the first blocks consists of columns and the last one consists of columns. Similarly, PNMF-M divides and into blocks in the row direction as shown in Figures 2(a) and 2(b); that is, , , so each of the first blocks consists of rows and the last one consists of rows.