Abstract
The singular value thresholding (SVT) algorithm plays an important role in the well-known matrix reconstruction problem, and it has many applications in computer vision and recommendation systems. In this paper, an SVT with diagonal-update (D-SVT) algorithm was put forward, which allows the algorithm to make use of simple arithmetic operation and keep the computational cost of each iteration low. The low-rank matrix would be reconstructed well. The convergence of the new algorithm was discussed in detail. Finally, the numerical experiments show the effectiveness of the new algorithm for low-rank matrix completion.
1. Introduction
The problem of completing low-rank and sparse matrices from some of its observed entries occurs frequently in many areas of engineering and applied science such as machine learning [1, 2], model reduction [3], compressed sensing [4], control [5], pattern recognition [6], signal and imaging inpainting [7–10], and computer vision [11]. From the pioneering work on low-rank approximation by Fazel [12] as well as on matrix completion by Candès and Recht [13], there has been a lot of study (see [1–35] and references therein) both from theoretical and algorithmic aspects on the problem of recovering a low-rank matrix from partial entries, also known as matrix completion. There is a rapidly growing interest for this issue. Explicitly seeking the lowest rank matrix consistent with the known entries is mathematically expressed aswhere the matrix is the underlying matrix to be reconstructed and is the associated sampling orthogonal projection operator which acquires only the entries indexed by with is a random subset of indices for the known entries.
The general problem (1), however, is nonconvex and is NP hard [14] due to the rank objective. There are a few of algorithms for solving this model directly. Alternatively, Candès and Rechat [13] switched (1) to another simple convex optimization problem (2) as follows:where the nuclear norm is the sum of all singular values of the matrix .
Furthermore, it has proved that the sequence converges to the unique solution of the following optimization problem closely related to (2) in [15]:
As for the solution of problems (2) and (3), there have been many computational efficient algorithms which are designed for a broad class of matrices such as mainly the accelerated proximal gradient (APG) algorithm [16], the augmented Lagrange multiplier (ALM) algorithm [17], several methods [18–21] resulted in alternating optimization based on the bilinear factorization with and rank , and the singular value theresholding (SVT) algorithm, as well as its improvements [15, 22–24]. However, the computations of partial singular value decomposition (SVD) were required at each iteration in the most direct implementation of these algorithms. The computational cost of computing the SVD has complexity of when the rank and matrix-size are proportional, resulting in computing the SVD to be the dominant computational cost at each iteration and then limits their applicability for large . In view of its outstanding performance and elegant mathematical properties, the SVT algorithm obtains widespread attention [25–27]. The variants and extended applications of the SVT algorithm have been studied later: Candès et al. [28] presented a unbiased risk estimate formula of the SVT for the noisy observations; Chatterjee [29] studied a general method for matrix denoising using SVT, which covers the stochastic block model as a special case; Donoho and Gavish [30] pointed out several ways that these matrix denoising results for singular value soft thresholding (SVST) estimation of low-rank matrices parallel results for soft thresholding of sparse vectors; Dutta et al. [31] proposed an alternative solution to the sensitivity of the classical principal component analysis (PCAs) to the outliers for solving the problemwith the nonsingular weight matrix which is user provided or automatically inferred from the data, called as WSVT problem; Klopp [32] introduced a variant of the SVT iteration; Ma and Xu [33] recovered the received signal strength (RSS) reading and achieve good localization performance based on the SVT theory; Zhang et al. [34] put forward a lower bound guaranteeing exact matrix completion via the SVT algorithm; Zhang et al. [35] have considered the low-rank tensor completion problem through a hybrid singular value thresholding scheme.
This paper develops a modification of the SVT algorithm for approximately solving the nuclear norm minimization problem (2). By using a diagonal-update technique for the approximated sequence at each step, the iteration matrices generated by the new algorithm are approximated to the true solution well, which saves significant computational cost of the SVT algorithm performance. And we also establish the convergence theory in detail. Experimental results show that the new algorithm outperforms the standard SVT and its several variants algorithms, especially when problem (2) comes to large-scale issues.
The rest of the paper is organized as follows. After we provide some notations in this section, we review briefly the standard SVT algorithm, the accelerated singular value thresholding (ASVT) algorithm as well as its modification, and a modified algorithm of the SVT algorithm with diagonal-update (D-SVT) is proposed in Section 2. In Section 3, the convergence theory of the new algorithm is established. Then, numerical experiments are shown and compared in Section 4. Finally, we end the paper with the concluding remarks in Section 5.
Here are some notations. denotes real matrices set, represents the nonnegative and real matrices’ set. The nuclear norm of a matrix is defined by denotes the th largest singular value of the real matrix of rank , and the Frobenius norm of a matrix is . is the transpose of a matrix . denotes the inner product between two matrices. is the indices of the observed entries, and is the complementary set of . is the orthogonal projector onto , satisfying,
2. Related Algorithms
In order to completing comparison subsequently, we briefly review and introduce some algorithms for solving the matrix completion problem (2).
2.1. The Standard Singular Value Thresholding (SVT) Algorithm
Definition 1. The singular value decomposition (SVD) of a matrix of rank iswhere and are two orthogonal matrices, .
Definition 2 (see [15]). For each , the singular value thresholding operator is defined as follows, say the “shrinkage”:where
The standard SVT algorithm proposed in [15] is a solution for solving the convex optimization (2).
| ||||||||||||||||||||||||||||||||||
Remark 1. Due to the ability of producing low-rank solutions with the soft-thresholding operator, the SVT algorithm was shown to be extremely efficient at addressing problems with low-rank optimal solutions such as recommender systems.
2.2. The Accelerated Singular Value Thresholding (ASVT) Algorithm
Introduce the Lagrangian function of problem (3) aswhere is the Lagrangian variable.
In terms of the dual approach, is the dual function of , which is concave. Definewhich is convex. Thus, we can solve problem (3) by firstly minimizing the objective function , namely,
Problem (10) was computed via Nesterov’s method with an adaptive line search scheme. Then, Algorithm 2 has been provided.
| ||||||||||||||||||||||||||||||||||
Based on the above, the accelerated singular value thresholding (ASVT) algorithm has been proposed in [22]. Furthermore, Wang et at. [23] presented the Ne-SVT by replacing the adaptive linear search with Nemirovski’s technique and the M-ASVT algorithms by using the same search technique in the ASVT algorithm. The overall steps of the later can be organized as Algorithm 3.
| ||||||||||||||||||||||||||||||||
It is reported that M-ASVT needs much fewer iterations than ASVT algorithm under the same level of accuracy and the same cost of computing.
2.3. The Singular Value Thresholding with Diagonal-Update (D-SVT) Algorithm
We are now in the position to introduce a modified singular value thresholding algorithm by using a diagonal-update technique, as shown in Algorithm 4.
| ||||||||||||||||||||||||||||||||||||||
Set for short. The difference with the standard SVT algorithm may seem at th step, replacing the iteration matrix with the diagonal-update of its projector , where is obtained by
Equation (11) is easy to compute since it is so simple just some arithmetic operation required, without extra cost. In fact, the exact solution of (11) is given by
Remark 2. The sequence matrices generated by the new algorithm are approximated to the true solution well, which saves significant computational cost of the SVT algorithm performance, without actually extra complexity. It is designed as Algorithm 4 by plugging some steps into the SVT method. It should be seen that Algorithm 4 has three lines (as shown in lines 10–12) more than Algorithm 1. The new algorithm includes Algorithm 1 as special case when .
3. Convergence Analysis
In this section, the convergence theory is discussed for the singular value thresholding which involves diagonal-update algorithm.
Theorem 1. Suppose that is uniformly bounded. Then,
Proof. It follows from Algorithm 4 thatIn term of the assumption of this theorem, such thatholds true.
Let . We note that from Algorithm 4, and by substituting the following inequalitieswe haveHence, is bounded and so is .
Moreover, it is obtained thatfrom the equationMoreover,since . The theorem has been proved.
Theorem 2. Let be the limiting point of the sequence generated by Algorithm 4. Then, is the solution of the optimization problem (3).
Proof. It is obtained that is the optimal solution of (8) for that value of from Algorithm 4.
Hence, for any feasible matrix , it is yielded thatThus, is the unique solution of (3).
Theorem 3. Suppose that . Then, is the optimal solution of the optimization problem (2).
Proof. Note that is the optimal solution of the optimization problem (2) if and only ifFrom Theorem 2, we havewhich implies thatTherefore,since is bounded. Thus,
4. Numerical Experiments
In this section, we provide the performance of our D-SVT algorithm in comparison with the SVT, ASVT, and M-ASVT algorithms mentioned in Section 2 and report the running time in seconds (denoted by “time (s)”), the numbers of iterations (denoted by “IT”) it takes to reach convergence, and the relative errors of the reconstruction (denoted by” Error 1 and Error 2″) as follows:
All the experiments are conducted on the same workstation with an Intel (R) Core (TM) i7-6700 CPU @3.40GHZ that has 16 GB memory and 64 bit operating system, running Windows 7, and Matlab (vision 2016a). For conciseness, the tests presented consider square matrices as is typical in the study. That is to say, suppose to simplify that the unknown matrix is square and that one has available sampled entries , where is a random subset of cardinality . By the way, the iteration fails if the number of iterations is up to 1000.
In our implement, we generate matrices of rank by sampling uniformly at random among all sets of cardinality ; then, denotes the observation ratio. Let . As discussed earlier [15], and the step sizes are constant and the parameter is chosen empirically. And as presented earlier [23], the parameters . As for D-SVT algorithm, we choose , and is the same as the SVT algorithm.
The tested matrix dimensions (denoted by “size (n)”) are from 1,000 to 12,000. The experimental results are shown in Tables 1–6. Our experiments suggest that Algorithm 4 is fast and significantly outperforms the other algorithms in terms of both number of iteration steps and computing time. The new algorithm is especially well suited for problems of very large sizes.
In order to show convergence behave of the algorithms briefly, convergence curves of several algorithms are clearly given, which are shown in Figure 1 for the different parameters. It is easy to see that our algorithm takes much less computational cost from iteration steps and computing time. That is to say, the D-SVT algorithm is more efficient than the other algorithms especially when the size of matrix is large.
(a)
(b)
(c)
(d)
(e)
(f)
5. Concluding Remarks
In this paper, we focus on the problem of completing a low-rank matrix from a small subset of its entries. This model can characterize many applications arising from the areas of signal and image processing, statistical inference, and machine learning. We have proposed a modification of the SVT algorithm for solving the low-rank matrix sparse model. The key step of the algorithm is to update each iteration matrix by a weighting diagonal matrix, without the extra cost. The weighting matrix was determined adaptively in iteration process. This algorithm is easy to implement and surprisingly effective both in terms of computational cost and storage requirement. Consequently, the matrix would be completed well.
Data Availability
We generate a matrice of rank r by sampling randomly in our implement. The readers can access the data supporting the conclusions of the study by MATLAB codes.
Conflicts of Interest
The authors declare that they have no conflicts of interests.
Authors’ Contributions
All authors contributed equally to the writing of this paper. All authors read and approved the final manuscript.
Acknowledgments
This work was supported by the NSF of China (11371275), NSF of Shanxi Province (201901D211423), STIP of Shanxi Provincial Department of Education (2020L0719), and the CSREP in Shanxi (no. 2019KJ035).