Abstract
Many problems in signal processing and statistical inference involve finding sparse solution to some underdetermined linear system of equations. This is also the application condition of compressive sensing (CS) which can find the sparse solution from the measurements far less than the original signal. In this paper, we propose - and -norm joint regularization based reconstruction framework to approach the original -norm based sparseness-inducing constrained sparse signal reconstruction problem. Firstly, it is shown that, by employing the simple conjugate gradient algorithm, the new formulation provides an effective framework to deduce the solution as the original sparse signal reconstruction problem with -norm regularization item. Secondly, the upper reconstruction error limit is presented for the proposed sparse signal reconstruction framework, and it is unveiled that a smaller reconstruction error than -norm relaxation approaches can be realized by using the proposed scheme in most cases. Finally, simulation results are presented to validate the proposed sparse signal reconstruction approach.
1. Introduction
Compressive sensing or compressive sampling (CS) [1–3] is a novel technique that enables efficient sampling below Nyquist rate, without (or with little) sacrificing reconstruction quality. CS converts the high-dimensional sparse signal into a significantly lower dimensional measurement signal. More precisely, let be a sparse vector, let () be a measurement matrix, and suppose the noisy observation vector is given bywhere is the noise vector. Here sparsity means that and ; -norm counts the number of nonzero items of a vector. The goal is to obtain an estimate of given and .
Determining the sparse signal in from measurements is typically an underdetermined problem; obviously, there will be no unique solution without any prior knowledge or constraint imposed on the solution . As the original signal is sparse, the problem of finding the desired solution can be phrased as some optimization problem, where the objective is to minimize the noise between noisy observation vector and real measurement while satisfying the constraint of the sparsity of . As the sparsity of is reflected by the number of its nonzero entries, equivalently its so-called -norm, in the noise-free case of (1), we can seek to solve the following problem: can recover exactly if is sufficiently sparse and the matrix satisfies the requirement of Restricted Isometry Constant (RIC) , which can be defined as the smallest constant such that the matrix satisfies the Restricted Isometry Property (RIP) of order ; namely,whenever . However, the optimization problem in (2) is nondeterministic polynomial (NP) hard which is difficult to be solved in practice. A number of approaches are presented to solve (2). The related approaches can be briefly classified into the following three categories.
(i) Greedy Pursuits. Greedy algorithms attempt to determine the nonzero entry indices, based on the relationship between the columns of the measurement matrix and the signal (or the residual signal), and then to estimate the non-zero entry amplitudes by using the least square method. Its objective is to pursue the minimization of -norm directly. The typical greedy approaches include Orthogonal Matching Pursuit (OMP) [4], Stagewise OMP (StOMP) [5], and Subspace Pursuit (SP) [6]. Basically, the greedy algorithms can only seek an approximate solution to (2) and are sensitive to noise.
(ii) Optimization. This kind of approach attempts to replace -norm minimization problem in (2) with - or -norm (), admitting tractable algorithms. These methods try to obtain the sparse solution by employing a newly reformulated problem as follows:In the noise-free case, can recover exactly as soon as . In the noisy case, the above convex relaxation formulation leads to the following penalized least-square problem [7, 8]:Later, it is shown in [9] that the reformulated one like can better approach the original -norm problem in (2), where denotes the nonnegative weighting coefficient. Iterative reweighting scheme was further investigated in [10]. -magic algorithm in [11], the gradient projection for sparse reconstruction algorithm (GPSR) in [8], and their variants also belong to the convex relaxation categories. -norm minimization problem can be recast aswhere . Chartrand demonstrated that -sparse vectors can be exactly recovered by solving under the assumption that holds for some and [12]. This kind of algorithm is capable of achieving better reconstruction quality and less error; however, it takes more time to solve the optimization problems.
(iii) Bayesian Framework. Given the unknown sparse coefficients a priori probability distribution, the Maximum A Posteriori (MAP) estimation mechanism can be employed to derive the Bayesian framework based sparse signal recovery schemes [13, 14]. It is unveiled in [15] that this framework may lead to a new mixture penalty of - and -norm functions.
-norm sparsity inducing item is very useful in image processing, for example, image restoration, image denoising, and image superresolution. Since -norm is nonconvex, image processing pursues - and -norm or total variation (TV) norm to replace the original -norm and solves the problems. Reference [16] studies a minimization problem where the objective includes a usual -norm data-fidelity term and an overlapping group sparsity total variation regularizer, and a fast algorithm is proposed to solve this problem. This method can avoid staircase effect and allow edges preserving. Reference [17] proposes -norm fidelity term with a total variation regularizer to recover blurred and salt-and-pepper impulse noisy image for higher visual quality. Reference [18] proposes an optimization model for noise and blur removal involving the generalized variation regularization term, the MAP (maximum posterior probability) based data fitting term, and a quadratic penalty term based on the statistical property of the noise. This minimization problem can be solved by a primal-dual algorithm. Reference [19] studies TV regularization in deblurring and sparse unmixing of hyperspectral images. Reference [20] investigates the modified linearized Bregman algorithm (MLBA) for image deblurring problems with a proper treatment of the boundary artifacts and - plus -norm.
Motivated by the utility of sparsity and the efforts in relaxing -norm function with tractable norm functions to get either convex or nonconvex relaxation problem, in this paper, we propose a novel - and -norm joint regularization based reconstruction framework to approach the original -norm sparseness-inducing constraint based reconstruction problem. Although [21] has shown that and measures are theoretically better than -norm to promote sparsity, our analysis shows that the proposed sparse signal reconstruction model can also solve the original problem. Moreover, the upper error limit of the proposed sparse signal recovery model is derived. It is unveiled that the proposed signal reconstruction model can achieve the tradeoff between -norm relaxation and -norm relaxation techniques. Namely, it exhibits similar -norm approximation capability like -norm relaxation. At the same time, like -norm convex relaxation approaches, we can resort to a variety of feasible optimization algorithms to derive the solution.
The remainder of this paper is organized as follows. The proposed sparse signal recovery model will be introduced in Section 2. Nextly, we deduce the error bound of the sparse signal reconstruction in Section 3. The practical algorithm design and experimental results are demonstrated in Sections 4 and 5. Finally, we conclude this paper in Section 6.
2. The Proposed Sparse Signal Recovery Model
2.1. Problem Reformulation
The reconstruction algorithm plays a central role in the sparse signal reconstruction. We propose - and -norm joint regularization to approximate -norm and the sparse signal recovery problem can be reformulated as follows:
We will try to explicate that the joint combination of - and -norm regularization provides a reasonable approximation to -norm, which can bring surprising benefits. Let us show this from the geometry point of view at first. The linear constraint of defines the feasible set of the problem. Geometrically, solving (4) (or (6)) is done by blowing - (or -) norm balloon centered around the origin and stopping its inflation once it touches the feasible set. Figure 1 illustrates some examples with different values of 2, 1.5, 1, 0.5, and 0.1, respectively. One can see that the norms with tend to touch the feasible set at the axes, which leads to the sparse solutions. On the other hand, - or -norm tends to derive the nonsparse solution. One can also observe from Figure 1 that, by using the joint - and -norm in (7) with the suitable ratio of , the intersection will also take place at the axes, which leads to sparse solution as well. Namely, the proposed signal reconstruction model resembles -norm approximation capability like -norm relaxation scheme. The influences of different choices of and will be highlighted in the following discussions.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
2.2. Sparse Solution Derivation Analysis
In this subsection, the analysis will validate that we can derive the same sparse solution by using the joint - and -norm instead of -norm function theoretically; that is to say, the solution to (7) is exactly , when the certain signal sparsity requirement can be satisfied. Fortunately, if the original problem actually has a sufficiently sparse solution, the success of the proposed model in addressing the original objective (2) can also be guaranteed. Some definitions are firstly introduced.
Definition 1 (see [22]). The mutual-coherence of a given matrix is the largest absolute normalized inner product between different columns of . Let denote the th column in ; the mutual-coherence can be calculated as The mutual-coherence provides a metric to assess the dependence between columns of the matrix .
Conclusion 1. For the linear system of equations and , is assumed to be -sparse; namely, ; if a solution exists obeying will be the unique solution to (2) and (7).
Proof. Assume that there are possible solutions different from , which have larger support and can satisfy . Define the following set of alternative solutions: This set contains all the possible solutions different from , which have larger support and can satisfy . Nonempty implies that there is at least one alternative solution to (7), rather than the desired .
Theorem in [22] tells us that if , is necessarily the unique sparsest solution; any other alternative solutions must be denser; that is to say, has more nonzero items. Our requirement on sparsity is , which is less than , no matter how to select the nonnegative parameters of and . It implies that is also the unique sparsest possible solution, so can be removed from . Let us define ; we can rewrite as the following shifted version around :The strategy of the proof we are going to present is to enlarge this set and show that the enlarged set is empty. This will prove that (7) indeed succeeds in recovering . We start with the requirement . Without loss of generality, the nonzeros in are moved to the first positions through a simple column permutation, which will not change the aforementioned inequality; we may have Here we use to denote the element-wise absolute operation for every element of the matrix (vector), and the result of the matrix (vector) is still a matrix (vector); only the elements of the matrix (vector) are now all positive. By using the common inequality , we can relax the above condition as follows: As for , because of the triangle inequality property of vector norm, we have . Then We use to denote the sum of the first entries of the vector , where denotes the column vector with ones followed by N-K zeros; namely, . So we may have Thus we have As stated before, the relaxation of the new alternative will effectively enlarge the initial set . We now turn to handle the requirement by replacing it with a relaxed requirement that expands the set further. We can normalize the matrix such that each column is of unit -norm (this operation will preserve the mutual-coherence, as shown in [22]). Then the multiplication by yields the condition , which does not change the set . By letting , this condition can be further rewritten asTaking an element-wise absolute value on both sides, we can relax the requirement on to obtain where we have used the inequality of . Because every column of has been normalized, one may readily derive the properties of the entries of the resulting Gram matrix as follows: (1);(2).By letting denote an identity matrix and denote a rank-1 matrix filled with ones, we can have Based on the above inequality, we can derive that . Returning to the set , we have The resultant set is unbounded since if , for all . Thus we can restrict our quest for the normalized vector , because we have , where denotes the all-ones column vector. Because , this new set of becomes By substituting into , we have where we have utilized the fact that is at most -sparse because it is assumed that both and are -sparse solutions to and . From the last inequality, we can obtain Let ; we get . Solving this inequality yields and . Since , we have , leading to This implies if is less than , the set will be necessarily empty, which also implies that we can not find such with larger -norm but smaller difference between - and -norm. So our model can find the desired solution.
For BP algorithm, it is known that if a solution exists obeying , the solution will be the unique solution to (2) and (4) as well. In our proposed model, when , the requirement will be reduced to as well. Otherwise, the proposed model in (7) can derive the same solution when , which is more stringent sparsity requirement. And this is the paid cost of the proposed - and -norm joint regularization based sparse signal reconstruction approach.
3. Signal Recovery via - and -Norm Joint Minimization
3.1. Noise-Free Signal Recovery
There exist lots of approaches to recover a sparse signal from a small number of linear measurements; we attempt to recover by solving (7). We begin by considering noise-free case. Our main theorem is similar in flavor to many previous ones; in fact, its proof is inspired by those previous works. We start by illustrating Theorem 2 in the special case of -sparse vectors that are measured with infinite precision, which means that the signal is exactly -sparse and the relative measurement error is equal to zero. For noise-free sparse signal recovery, we have the following conclusion about the signal recovery error upper limit.
Theorem 2. Given and , if satisfies the RIP of order with ; then every -sparse vector is exactly recovered by solving .
Proof. Firstly, we introduce Lemma 3.
Lemma 3. Suppose that satisfies the RIP of order with ; we obtain the measurement . Let and be given, and define . Let denote the index set corresponding to entries of with the largest magnitude and the index set corresponding to entries of with the largest magnitude, where stands for the complement set of . Set . If , thenwhere , , and .
Lemma 3 establishes an error bound for the proposed signal recovery model described by (7), when the measurement matrix satisfies the RIP constraint. The detailed proof can be found in Appendix. In the case of noise-free measurement, that is, , we can apply Lemma 3 to derive that, for , yields , then the term vanishes, and is -sparse signal, leading to ; we can obtain the following result: So every -sparse vector can be exactly recovered by solving problem.
Theorem 2 is an immediate consequence of Theorem 4 to be given in the next section. One may readily note that, in noise-free case, by setting appropriate parameter of and , we may achieve reasonable signal restoration performance, as will be illustrated by the numerical results in Section 4. Compared to the result of Theorem in [23] with , where , we can obtain the same result when . We will reveal how the different choices of and influence the upper error bound of the reconstructed signal in the following section.
3.2. Noisy Signal Recovery
In practical applications, the measurements are likely to be contaminated by various types of noise, such as quantization error and random noise. We supposeNote that represents a relative error between accurate and inaccurate measurements. In order to recover the original vector from the measurement of , we can solve the minimization of the above objective function.
It is obvious that the minimization of is equivalent to where is the minimum of . Because the set is compact and is a continuous function, we can conclude that the minimization of must exist. So we have the following conclusion.
Theorem 4. Given and , if satisfies the RIP of order with , then the solution of approximates the original vector with errorwhere , , and .
Proof. Since , we have So, with Lemma 3,
Theorem 4 shows us that it is possible to stably recover sparse signals from the noisy measurements. Compared to the result of Theorem in [23] with , where and , we may achieve a smaller restoration error by using the proposed sparse signal recovery model when appropriate values of and are utilized.
4. Realization Algorithm
For simplicity, in this paper we resort to the simple computational approach of conjugate gradient (CG) algorithm for computing the local minimizer of problem (7). The CG algorithm is an effective approach to solve the linear system equation from the measurements for the given measurement matrix . It is straightforward to generalize the proposed model with other advanced algorithms.
Let us denote the initial guess of as (without loss of generality, we assume in this paper). The pseudocode of the CG algorithm is illustrated as follows.
The Conjugate Gradient Algorithm(1)Initialization: Initial point , , , , iterations , search direction ;(2)Loop until the specified accuracy or iteration numbers:(3)Use the line search technique to determine the search step length ;(4)Update the signal ;(5)Update the gradient ;(6)Update ;(7)Update the search direction ;(8)Iteration number increases by 1 as .Here denotes the element-wise sign function of the vector , which is defined as follows: for every element , if ; if ; and if .
5. Numerical Results
Simulations are performed to testify the applicability of the proposed signal reconstruction model in different settings. Some representative sparse signal restoration algorithms, such as () the greedy algorithm of the OMP [4], SP [6], and MP [24] for the problem; () the -norm optimization of GPSR-Basic and GPSR-BB [8], LMS (Least Mean Square) [25], [26], IRLS (Iteratively Reweighted Least Square) [27], and LARS (Least Angle Regression Stagewise) [28]; () the Bayesian framework of BCS (Bayesian Compressed Sensing) [14], are included in our simulations. For fair comparison, we considered the commonly assumed experimental setup in the literatures. All of our simulations are performed in MATLAB 2009 environment over Microsoft Window 7 operating system; the simulation platform is provisioned with Inter Core i5-480, ATI HD 5470 processor with 2 GB of memory. Gaussian measurement matrix is assumed in all simulations for simplicity, which satisfies the RIP conditions with high probability. The entries of are independently generated from normal distributed random variables with zero mean and variance of . In all experiments, we consider the fixed signal length of , while the number of measurements and sparsity may change in different experiment settings. The locations of nonzero coefficients of sparse signal are assumed to be uniformly distributed within . The corresponding nonzero coefficients are also generated from a Gaussian distributed random variable. Normalized reconstruction error will be utilized as a metric to evaluate the reconstruction quality, where and represent the reconstructed and original signal vectors, respectively. For each experiment, 1000 simulations are conducted to calculate the normalized reconstruction error in different algorithms.
5.1. The Influence of Parameters of and
The first problem we encounter is the choice of and . In this experiment, we fix and the sparsity degree parameter , and , respectively. We use the simple CG algorithm to solve our proposed model. Figure 2 shows the normalized reconstruction error as a logarithmic function of the ratio ; each point in the curve is the outcome of 100 iterations of the CG algorithm. Figure 2 can help us determine the appropriate ; we can find that when , the reconstruction results are terrible. A larger ratio leads to a more satisfying solution. And from the four curves, when , the reconstruction performance tends to be stable, so we select in the following experiments.
5.2. Noise-Free Measurement Case
We start with the noise-free measurement case as .
Experiment 1 (effect of sparsity on the performance). This experiment explores the answer to such a question on how the sparsity degree of the signal will affect the reconstruction quality with a given number of measurements. In this experiment, we fix , while the sparsity degree parameter will change from 5 to 40. We assume the simple CG algorithm to solve our proposed model. One may readily observe from Figure 3 that the proposed scheme (labeled with CG) far exceed GPSR-Basic, GPSR-BB, LMS, , and IRLS. With the increase of sparsity, there will be larger reconstruction error for all algorithms; however, the proposed reconstruction scheme can still achieve a comparatively low reconstruction error, which complies with our previous analysis. It can be observed that BCS and LARS algorithms are always better than our proposed scheme. Our scheme is comparable to -norm relaxation techniques but is inferior to the inference method.
Experiment 2 (effect of measurement number on the performance). This experiment tries to answer the question on how different number of measurements will affect the achieved reconstruction performance. In the simulations, we fix , while the number of measurements changes from 140 to 210. The signal reconstruction errors are illustrated in Figure 4. We may observe similar results in the first experiment. BCS and LARS algorithms always achieve the best performance. The proposed method is comparable to the LMS algorithm, both of which are superior to other algorithms.
5.3. Noisy Measurement Case
Experiment 3 (robustness against noise). In this experiment, we testify the noisy sparse signal reconstruction performance by fixing . The noisy measurement model is given by , where is an additive white Gaussian noise with covariance matrix ( is identity matrix). From Figure 5, we may observe that the proposed reconstruction algorithm can achieve superior restoration quality compared to greedy approaches, such as OMP and MP, in terms of the reconstruction errors. The performance of BCS and LARS algorithms is satisfactory in the first two experiments, but their antinoise performance is no match to our scheme. In addition, our scheme is slightly inferior to -. Obviously, with the increase of noise variance, our scheme has smaller reconstructed error than LMS which shows the good antinoise performance.
Experiment 4 (robustness against noisy measurement matrix ). In this experiment, we testify the stability of sparse signal reconstruction performance when the measurement matrix is disturbed by Gaussian noise with fixed . It is assumed that the measurement matrix is contaminated by zero-mean Gaussian noise with different variances at the receiver. From Figure 6, we can observe that the proposed reconstruction model exhibits reasonable ability against the noisy measurement matrix except for LMS algorithm. With the increase of noise, our antinoise ability is worse compared to and GPSR-Basic algorithms; however, when the noise is small, we can show competitive power. Our proposed scheme outperforms OMP, MP, BCS, LARS, and GPSR-BB.
By comparing the simulations results in Figures 3–6, we may observe that the proposed scheme outperforms the approaches of GPSR-BB, GPSR-Basic, and LRLS in terms of the achieved sparse signal reconstruction performance under different signal sparsity, different measurements, and robustness against background noise as well as the contamination of the measurement matrix. Although BCS and LARS algorithms exhibit better performance against different sparsity and number of measurements , it is sensitive to noise. LMS algorithm shows stable antinoise performance but is inferior to our proposed model with different sparsity and number of measurements . Greedy algorithms can achieve satisfactory performance given different signal sparsity and different measurements, but our proposed model exhibits better robustness against noise. Meanwhile, our proposed model outperforms most optimization based algorithms. In a word, the proposed reconstruction model provides an effective choice in the sparse signal reconstruction.
6. Concluding Remarks
In this paper, motivated by the efforts in relaxing -norm function with tractable norm functions to obtain either convex or nonconvex problem, we developed a new - and -norm joint regularization reconstruction framework to approach the original -norm sparseness-inducing constraint reconstruction problem. Our analysis shows that the proposed sparse signal reconstruction model can derive the same solution of the original sparse problem with -norm regularization. Moreover, the upper error bound of the proposed sparse signal recovery model is derived. It is shown that the proposed signal reconstruction model can achieve an interesting tradeoff between both -norm relaxation and -norm relaxation techniques. For instance, it exhibits the similar -norm approximation capability like -norm relaxation, while, on the other hand, straightforward and simple algorithms (such as the CG algorithm considered in this paper) can be utilized to derive the solution, just like -norm convex relaxation approaches. Numerical analysis is presented to validate that the proposed sparse signal reconstruction framework provides the exciting and satisfying method for the sparse signal reconstruction problem. Nextly, we will find other effective algorithm to solve this new model for better performance and popularize the proposed new norm model to image processing, such as image restoration in [16].
Appendix
Proof of Lemma . Firstly, we introduce Lemma A.1.
Lemma A.1 (see [23]). Let be an arbitrary subset of such that . For any vector , define as the index set corresponding to largest entries of (in absolute value), as the index set corresponding to the next largest entries, and so on. ThenThe proof is shown in [23]. Lemma A.1 immediately yieldswhere are defined as in Lemma A.1; that is, is the index set corresponding to largest entries of (in absolute value), is the index set corresponding to the next largest entries, and so on.
Since , and , we have . Because of the triangle inequality property of norm, we have ; then ; we can have Let ; it results inCombining this with (A.2), we can obtainwhere we have used the fact that, for any -sparse signal , there is .
Because , from the triangle inequality, we getCombining this with (A.5) and observing that will yield Because , let ; we obtainWhat we need to do is to establish a bound for . Considerwhere denotes the inner product. We can renormalize the vectors and so that their -norms equal one by setting and . We can obtain Substituting by , we can obtain similar results: ThenBecausesubstituting (A.12) and (A.13) into (A.9) will yield Let and ; the above inequality implies This inequality leads to the following two results:So combining (A.2) and (A.4), the fact that and will yieldFrom (A.18), we get Combining this with (A.8) and because , we can obtainwhere and .
Competing Interests
The authors declare that they have no competing interests.
Acknowledgments
This work was supported by the NSFC under Grants no. 61271246 and no. 61371165.