#### Abstract

Sparse arrays, which can localize multiple sources with less physical sensors, have attracted more attention since they were proposed. However, for optimal performance of sparse arrays, it is usually assumed that the circumstances are ideal. But in practice, the performance of sparse arrays will suffer from the model errors like mutual coupling, gain and phase error, and sensor’s location error, which causes severe performance degradation or even failure of the direction of arrival (DOA) estimation algorithms. In this study, we follow with interest and propose a covariance-based sparse representation method in the presence of gain and phase errors, where a generalized nested array is employed. The proposed strategy not only enhances the degrees of freedom (DOFs) to deal with more sources but also obtains more accurate DOA estimations despite gain and phase errors. The Cramer–Rao bound (CRB) derivation is analyzed to demonstrate the robustness of the method. Finally, numerical examples illustrate the effectiveness of the proposed method from DOA estimation.

#### 1. Introduction

Superresolution direction finding is a key branch of signal processing, which has received much attention in many fields like radar systems, communication, and navigation [1, 2]. In the last decades, the research of direction of arrival (DOA) estimation has successively gone through three stages: adaptive beamforming, subspace decomposition (such as multiple signal classification (MUSIC) [3], estimation of signal parameters via the rotational invariance technique (ESPRIT) [4], etc.), and subspace fitting (such as maximum-likelihood (ML) algorithm [5], weighted subspace fitting (WSF) algorithm [6], etc.). With the development of the algorithms, issues such as aperture expansion, optimization of hardware resource requirements, array structure design, and resolution accuracy improvement [7, 8] have been gradually solved, laying a solid foundation for further broadening the application prospects.

However, the uniform linear array (ULA) is always applied in most of the traditional DOA estimation methods due to modeling and computation convenience. For ULAs, most sources can be detected with physical sensors. Besides, the arrangement of ULAs will make it difficult to achieve in some scenarios and increase the system cost. Several sparse arrays are designed in this context. Nested array [9] and coprime array [10] are the most representative geometries, which can resolve sources with only physical sensors. Attracted by the effectiveness of sparse array (e.g., enhancing degrees of freedom, reducing mutual coupling, and eliminating angle ambiguity), a series of DOA estimation algorithms have been developed [11–19]. A spatial smoothing method was proposed in [8], which achieved superresolution direction finding for sparse arrays. However, discrete virtual elements limit the ability to detect more sources by utilizing the spatial smoothing method. In order to improve the utilization of the discrete virtual elements, the multifrequency high-order cumulant algorithms were proposed in [13] and virtual array interpolation methods were introduced to build a nonuniform virtual array by the idea of array interpolation [14, 15]. To achieve better estimation performance, the category of sparse reconstruction algorithms, which can also be utilized to deal with the coherent sources, was extended to sparse arrays [16–19]. Another method of taking advantage of difference coarray is array geometry optimization method; the super nested array (SNA) and generalized nested array (GNA) were designed in [20, 21]. All the new forms of sparse arrays aimed to construct the optimal array geometry with largest DOFs and least mutual coupling.

However, the foregoing strategies are highly sensitive to the accuracy of the manifold matrix. In practice, the DOA estimation system will suffer from one or more model uncertainties inevitably like gain and phase error, mutual coupling, and sensor location error, which will lead to severe performance degradation or even failure of the direction of arrival estimations [22, 23]. Various corresponding algorithms have been proposed to eliminate the influence of gain and phase error, such as Hadamard product method [24], eigenstructure method [25], and ESPRIT-like method [26]. Unfortunately, most of the calibration strategies were proposed based on the assumption of ULA, which was invalid for sparse arrays in underdetermined cases. Partial Toeplitz structure of the covariance matrix and the sparse total least squares (STLS) method were utilized to estimate the parameters for nested array in [27]. But the prior knowledge of the noise power was required. Tian aimed to calibrate the error in the underdetermined case by applying a partly calibrated nested array and a high-power calibrated source [28]. Though the mathematical derivation and simulations demonstrated the effectiveness of the method, it was difficult to find sensors with accurate calibration in practice.

Motivated by the sparse arrays and the proposed methods, a novel strategy for an underdetermined case is proposed in this paper. Firstly, a GNA is applied to enhance the DOFs which enable us to deal with more sources than traditional sparse arrays. To achieve better DOA estimation performance and reduce the adverse effect of gain and phase error, we transfer the DOA estimation into a sparse reconstruction problem with nonnegativity constraint by exploiting a covariance-based sparse representation method. To further demonstrate the robustness of the strategy, the Cramer–Rao bound is derived subsequently.

The remainder of this paper is organized as follows. In Section 2, the geometry and signal model of GNA are constructed, respectively. Next, it is proved that the traditional eigenstructure method is invalid in the underdetermined case and a covariance-based sparse representation method is introduced in Section 3. In Section 4, we derive the Cramer–Rao bound (CRB) for further demonstration. Numerical results are provided to evaluate the effectiveness of the proposed method in Section 5, and the conclusion is drawn in Section 6.

Notations: throughout this paper, , , and represent the Kronecker product, Khatri–Rao product, and convolution product, respectively. , , and denote the transpose, conjugate transpose, and complex conjugate operations, respectively. and denote the real part and imagery part of a complex number. , , and represent the diagonal matrix operation, vectorization operation, and expectation operation, respectively.

#### 2. Problem Formulation

##### 2.1. Generalized Nested Array

Generalized nested array is a flexible sparse array with more DOFs and less mutual coupling. As shown in Figure 1, the GNA is constructed by two concatenated ULAs, where the inner subarray is an -elements ULA with interelement spacing of and the outer is an -elements array with spacing . Herein, equals , where represents the wavelength of the signal. Different from the traditional nested array, the element spacings and are two arbitrary coprime integers.

Figure 1 indicates that the sensor position set as follows:

According to (1), it is easy to find that the GNA has two special cases. When and , the GNA becomes the nested array. When and , the GNA can be interpreted as Coprime Array with Displaced Subarrays (CADiS) [19].

Based on the sensor position set , we can obtain the difference coarray of GNA by the following equation:where and denote the positive and negative part of the difference coarray, respectively. And the values in and are symmetric to zero.

By analyzing the vectorized covariance matrix, only when or can the values in difference coarray be contiguous. The larger and are, the more inconsecutive lags exist. According to the proof in Ref. [13], when and , the range of contiguous lags in the positive part of is . In addition, if we define , the number of unique lags in can reach in the same assumption.

Consider a GNA with sensors, the DOFs of the array under the constraints of and can be obtained by the following equation:

With the optimal factors of , , or , . In that case, the DOFs of GNA are the same as the nested array or SNA.

The nested array and coprime array with the same elements as GNA are selected for direct comparison. Sensors location and DOFs of the three kinds of arrays are listed in Table 1.

Figure 2 shows the coarray location of the sparse arrays. Compared with nested array and coprime array, both the GNA and nested array have the same DOFs, while the coprime array has the least DOFs.

**(a)**

**(b)**

**(c)**

In general, the GNA possesses the advantages of both nested array and coprime array. The GNA not only provides the larger DOFs but also reduces the mutual coupling owning to the large element spacing. However, the large factors lead to short contiguous lags, which makes spatial smoothing MUSIC (SS-MUSIC) algorithm inapplicable. In practice, the compressive sensing (CS) algorithm can be widely utilized to estimate the DOA in this case.

##### 2.2. Array Signal Model

Consider a GNA with sensors, which receives uncorrelated far-field narrow-band sources from . Then the received signal under ideal condition is denoted by the following equation:where denotes the signal vector and is the manifold matrix. represents the steering vector, where denotes the -th sensor’s position. is the noise vector, which is assumed to follow the Gaussian distribution and uncorrelated with the sources.

Now consider the scenario that each sensor is affected by gain and phase error, then (4) can be rewritten as follows:where and denote the gain error matrix and phase error matrix, respectively. and are both diagonal matrices, whose diagonal entries are given by the following equation:

Then, we obtain the covariance matrix of the received signal based on (5)where denotes the source covariance matrix and represents the power of the -th source. In addition, denotes the power of the noise signal and is the number of snapshots.

Define the covariance matrix without gain and phase error or noise, then we have the following equation:

Rewriting the relation in (7), it can be obtained as follows:

As we all know, gain and phase errors will lead to severe performance degradation or even failure of traditional DOA estimation algorithms. For ULA conditions, the covariance matrix has Toeplitz structure, which makes it easy to correct the model errors. However, the nonuniformity of GNA destroys the Toeplitz structure and increases the difficulty of DOA estimation.

#### 3. DOA Estimation Strategy

Although the Toeplitz structure of the covariance matrix will be destroyed by the nonuniformity of sparse arrays, it still can be utilized to estimate some parameters of the gain and phase error. Define a function to indicate the positions of the virtual elements.

It is obvious that the -th element of can be defined as follows:

Then, we rewrite the covariance matrix without noise as follows: denotes the noise power, which can be obtained by estimating the minimum eigenvalue of . The element of can be defined as .

Considering the gain error, a series of equations can be given by the following equation:where denotes a quantity determined from .

Similar to the ULA case [27], select the elements which satisfy . Therefore, (12) can be simplified as follows:

In order to estimate the gain error, no less than equations are required; otherwise, the parameter estimation will be underdetermined. However, by analyzing the elements of , the number of equations like (13) is as follows [29]:where denotes weight function calculated by convolution. is a conditional function. If there is a virtual element located at , the value of is 1; otherwise it is 0 and . It is obvious that when ; thus, the number of equations provided similar to (13) is . That means the gain error can be estimated by an overdetermined equation set.

Taking all the nonredundant relations satisfy and constructing an equation set, it can be described compactly as follows:where is a matrix that the row vector is its lone null space spanning vector, denotes the result of (14).

However, due to the estimation error of , (15) is not strictly correct. The singular value decomposition (SVD) method can be utilized to obtain the least squares solution.where represents the pseudoinverse of .

Equation (16) provides a minimum norm least squares solution, and a general solution can be regarded as adding an arbitrary scalar to the null space spanning vector. Then, the gain error can be estimated by an arbitrary multiplicative constant; i.e., , where is an arbitrary scalar.

Hence, the previous conclusion means that the gain error will only affect the amplitude of the spectrum.

For phase error, we utilize a similar strategy. Definingwhere and is a quantity determined from .

If is satisfied, (17) can be simplified as follows:

Since the elements in the main diagonal are inapplicable as they have no phase error information, the number of meaningful equations provided by (18) is as follows [29]:

Unfortunately, the large coprime factors of GNA lead to a few repeated virtual sensors, making the number of meaningful equations even less than the estimated parameters. In other words, we cannot construct an overdetermined equation set like (15) to estimate the phase error in most scenarios.

To deal with the situation, a covariance-based sparse representation method is introduced for DOA estimation.

Based on (9) and , the covariance matrix after gain error calibration can be obtained by the following equation:

Then, vectorizing the covariance matrix with calibrated gain error, we have the following:where and . is a column vector whose -th element is 1 and the rest elements are all zeros. As there are only *N* nonzero values in specific positions of , an selection matrix to remove the nonzero values can be defined as follows [30]:where . is an column vector of all zeros except a 1 at the -th position. Therefore, (21) can be rewritten as follows:

After this arithmetical operation, the noise item can be completely eliminated.

Directions of source signals impinging on the GNA range from as the assumption, then the grid sampling over the space domain is utilized. Next, we can construct a grid set which contains all the origin signal directions.

The equivalent expression of (23) is denoted by the following equation:where represents the overcomplete dictionary. denotes the -sparse vector whose -norm is , and its nonzero values correspond to the exact DOAs in . Based on the analysis, the DOA estimation can be transformed into the recovery of and the location detection of nonzero values.

For the convenience of calculations, it would be better to transform the high-computation -norm minimization problem into the -norm minimization problem. Thus, the equivalent expression is represented as follows:where denotes the -th elements of .

However, as the number of snapshots is limited, the estimated covariance matrix cannot be equal to the exact covariance matrix, and the error exists. The relations can be expressed by the following:

Therefore, we can transform (25) into the following optimization problem:where denotes the threshold parameter on the upper bound of . By applying the Orthogonal Matching Pursuit (OMP) algorithm or other sparse reconstruction algorithms like L1-SVD algorithm [31], the optimal solution is obtained, and the DOA can be estimated accurately. It should be noted that has no influence on the locations of nonzero value in .

#### 4. The Cramer–Rao Bound

The Cramer–Rao bound provides a theoretical lower bound on the variance of any unbiased estimation, which can be applied for the evaluation of the optimal performance of the estimation algorithms. However, traditional mathematical derivations are based on the assumption of the overdetermined system (the number of sensors is more than sources), which may be invalid for underdetermined cases like a nested array, coprime array, and GNA.

Assuming that the signal model is unconditional, we derive the CRB for DOAs based on the Ref. [32,33], the expression is shown as follows:where represents the orthogonal projection matrix of .

The detailed derivation is presented in the Appendix.

#### 5. Simulation Results

In order to verify the DOA estimation performance of the proposed strategy, several numerical simulations are provided in this section. For DOA estimation performance comparison, the nested array and coprime array are utilized. Meanwhile, sparse total least squares (STSL) algorithm and CRB are provided for error calibration performance comparison. Throughout the experiments, we assume that the physical sensors of GNA are 8 and the coprime factors are and , respectively. Further, the noise term satisfies zero-mean Gaussian distribution.

##### 5.1. Effect of Gain and Phase Error on DOA Estimation

In this simulation, we study the effect of gain error and phase error, respectively. In the first case, only gain error exists. Similarly, only phase error affects in the second case. Assume that the gain error matrix and phase error matrix are and , respectively. We choose 9 sources ranging from to , with a step . The snapshot number is . Simulation results when are given by Figure 3. In the following figures, the dotted red line represents the real DOAs.

**(a)**

**(b)**

Compared with Figures 3(a) and 3(b), it can be found that only phase error affects the performance of DOA estimation, leading to large performance degradation or even failure, while the gain error only leads to influence on the amplitude of the spectrum.

##### 5.2. DOA Estimation Performance under Gain and Phase Error Condition

Firstly, assume that the gain error matrix and phase error matrix in this experiment are the same as the matrices in Section 1. Then, we plot the CS spectrum [34] of GNA in Figure 4. We observe that the effect of gain and phase error can be eliminated and all the sources can be identified by the proposed method at the same time.

For further evaluations, we discuss the effect of SNR and snapshot numbers on DOA estimation performance. Therefore, the root mean square error (RMSE) is introduced for quantitative analysis. The STSL algorithm is selected as a comparison for DOA estimation performance simulations under the gain and phase error. In this section, it is assumed that two uncorrelated sources impinge on the array from directions of and . Then, we provide the estimation performance curves versus SNR and snapshot numbers by fixing snapshot numbers or SNR, respectively. The RMSE via 200 Monte Carlo trails is denoted by the following equation:where represents the estimated value of in the -th trail.

Firstly, assume that SNR ranges from to , while the snapshot number is 200. The DOA estimation performances of two strategies and CRB versus SNR under gain and phase error condition are shown in Figure 5. Similarly, the SNR is set to and the snapshot number varies from 20 to 400. Then, we obtain the DOA estimation performance of three strategies and CRB versus snapshot numbers under the influence of gain and phase error as shown in Figure 6. It can be found from Figures 5 and 6 that the DOA estimation of each strategy will be more accurate as SNR or snapshot number increasing. However, compared with the STSL method, the proposed method in this paper has the lower RMSE curve and becomes closest to the CRB.

#### 6. Conclusions

This paper investigates into sparsity-based strategy for DOA estimation in the case of gain and phase errors via generalized nested array. With the advantage of sparsity, the generalized nested array is utilized to enhance the degrees of freedom and decrease the influence of mutual coupling firstly. Then, the signal model and error model are established, and a covariance-based sparse representation method is provided to estimate the DOAs. Besides, the Cramer–Rao bound is derived, and the robustness of DOA estimation is analyzed. Although the covariance matrix no longer has the Toeplitz structure in the case of gain and phase errors, which makes DOA estimation become an underdetermined problem, the proposed strategy still has excellent parameter estimation performance. Numerical simulations verify the advantages and effectiveness of the theoretical analysis.

#### Appendix

#### CRB for DOA

Under the assumption of unconditional signal model assumption, a parameter vector can be defined as follows:

Thus, the entries of the Fisher Information Matrix (FIM) is formulated by the following equation:where , represents the trace of a matrix, is the -th element of the vector , and is the -th element, respectively.

Assume thatwhere

Then, (A.3) can be further transferred to the following equation:

Since is positive semidefinite, it is easy to obtain that the FIM is positive semidefinite.

Therefore, we can obtain the following:where

According to (A.8), the CRB for DOAs can be calculated by the following:where denotes the orthogonal projection matrix of .

#### Data Availability

The data sources from the references are all marked in this paper. The parameters designed are all described in this paper. Therefore, the data in this paper have been fully covered and can be obtained.

#### Conflicts of Interest

The authors declare that there are no conflicts of interest regarding the publication of this paper.

#### Acknowledgments

This research was funded by the National Natural Science Foundation of China (grant no. 61601504).