#### Abstract

A partial dictionary based direction of arrival (DOA) estimation method which addresses the off-grid problem and exploits combined coprime and nested array (CCNA) is proposed. Compared to general coprime array, CCNA yields two sparse coprime subarrays in the coarray domain by adding a third subarray in the physical-array domain. To ensure the DOA estimation performance, the subarray with larger aperture is chosen, and the cyclic phase ambiguity caused by the sparse subarray allows partial dictionary covering arbitrary cycle to represent the whole atoms, and then, the off-grid sparse reconstruction method is developed to amend the grid mismatch. After the sparse recovery and off-grid compensation, ambiguous DOA estimations can be eliminated by substituting the estimations into the whole virtual array. Multiple simulations verify that the proposed algorithm outperforms the other state-of-the-art methods in terms of DOA estimation accuracy and angular resolution.

#### 1. Introduction

Direction of arrival (DOA) estimation using antenna array is an important issue in many systems, e.g., radar, sonar, and wireless communication [1–5]. Compared to conventional subspace based methods, sparse representation based DOA estimation methods have been attractive since they can provide higher resolution and require fewer samples [6], and many effective sparse representation based methods have been proposed. The greedy methods [7, 8] require the prior information of source number and are sensitive to the noise, and the -norm based algorithms, such as the -norm singular value decomposition (-SVD) method [9], sparse recovery using weighted subspace fitting (SRWSF) method [10], sparse representation of array covariance vector (SRACV) method [11], and sparse iterative covariance-based estimation (SPICE) method [12], can reduce the sensitivity to noise and estimate the angles via convex optimization. However, these methods discretize the whole spatial range into a grid, which will result in performance degradation when the sources are not exactly located on the grid, i.e., the grid mismatch problem [13]. In [14], the off-grid sources were considered and estimated by introducing grid offsets in the sparse Bayesian inference (SBI). Based on the joint sparsity between original signal and the grid mismatch variables, joint sparse recovery method was proposed in [15]. Meanwhile, the grid-less methods are developed to directly recover the covariance matrix based on atomic norm or nuclear norm minimization [16, 17]. However, these methods only concentrate on the physical array model, which has limited degree of freedom (DOF).

Sparse array design has been developed to increase the virtual DOF in the difference coarray domain. Nested array was proposed in [18], which can generate DOF in the difference coarray domain with only physical antennas [19, 20]. Nested array can also be applied in radar system to increase the virtual DOF and enhance the spatial resolution [21]. However, the nested array has a dense subarray, which suffers from the mutual coupling problem. Coprime array [22], another well-known sparse array, was proposed to reduce the mutual coupling influence. With sparsely spaced physical antennas, coprime array can achieve DOF [23], which is generally nonuniform, and the coarray has more holes compared to that of nested array. Therefore, many works are developed to modify the coprime array to generate more continuous virtual elements in the coarray domain, such as the augmented coprime array (ACA) [24], generalized coprime array (GCA) [25], and thinned coprime array (TCA) [26]. To deal with the one-snapshot situation in the coarray domain, sparse representation based methods have also been introduced for sparse array [27–29]. For off-grid sources, a joint reconstruction method named joint LASSO (JLASSO) was proposed in [30], which can exploit the large DOF in the coarray domain of coprime array and amend the grid mismatch via joint sparse recovery. However, the computation complexity is very high due to the dictionary covering whole spatial range.

There is also another way to utilize coprime array, i.e., the separate processing of the two subarrays of coprime array, and the unique estimation is determined from the coincide results from the two subarrays, such as the combined multiple signal classification (MUSIC) method [31], partial search MUSIC method [32], root MUSIC method [33], and combined estimation of signal parameters via rotational invariance technique (ESPRIT) based method [34]. However, as the subarrays are processed separately in the physical-array domain, the exploited DOF is limited, and an alternative way is to transform the coprime relationship into the coarray domain [35]. A combined coprime and nested array (CCNA) geometry, which is obtained by adding a third subarray nested to both of the two subarrays, was proposed in [36], where the coprime subarrays are transformed into the coarray domain to achieve large aperture and DOF. However, the utilized MUSIC method results in aperture loss.

In this paper, we propose an off-grid DOA estimation method, which requires only partial dictionary based on CCNA. Due to the nested relationship within the subarrays, the two virtual subarrays are still sparse but uniform after the vectorization of the covariance matrices. Different with conventional schemes, we adopt the subarray with larger aperture for DOA estimation to avoid the negative effective brought by the smaller subarray. The sparsity of the virtual array enables partial dictionary covering partial spatial range to represent the whole-range atoms. Meanwhile, the off-grid sparse reconstruction method is developed to amend the grid mismatch. Finally, ambiguous DOA estimations can be eliminated based on coprime-ness by substituting the estimations into the whole virtual array. Numerical simulations show that the proposed algorithm outperforms the ACA method [24], partial search (PS) MUSIC [32], root MUSIC method [33], and CCNA with root MUSIC [36] in terms of estimation accuracy and angular resolution.

*Notation*: , , , and denote transposition, conjugation, conjugate-transposition, and pseudo-inversion, respectively. and denote the operations of expectation and vectorization, respectively. is a diagonal matrix with vector **a** being the diagonal elements, and is a identity matrix. means norm, and means the phase of . , , and ./ denote kronecker product, Khatri_rao product, and element-wise division, respectively.

#### 2. Data Model

Figure 1 shows the structure of CCNA, which is composed of three subarrays. Subarray 1 and subarray 2 form the original coprime array, where and are coprime integers. The third subarray is arranged along the negative side with elements and interelement spacing being , where is the unit spacing, which is generally set as half-wavelength. It is also indicated that subarray 1 and subarray 3 form a nested array with the minimum interelement spacing being , and subarray 1 and subarray 3 form another nested array with the minimum interelement spacing being . The total antenna number of CCNA is as the subarrays share the same element in the origin.

Assume that there are plane waves impinging upon the array with DOAs being , which is angle between the wave line and axis. Then, the outputs of the subarrays are expressed as where is the signal vector. , , and are the additive white Gaussian noise (AWGN) vectors with the same noise power . , , and denote the direction matrices of subarray 1, subarray 2, and subarray 3, respectively. The columns are the corresponding steering vectors, which are expressed as

#### 3. Partial Dictionary Based Off-Grid DOA Estimation Method

##### 3.1. Sparse Representation Using Partial Dictionary

Combine the outputs of subarray 1 and subarray 3 to form the first nested array , whose covariance matrix is where is the combined direction matrix and is a diagonal matrix containing signal powers. To obtain the virtual array in the coarray domain, the vectorization of the covariance matrix is where . Due to the nested relationship, there are continuous elements located from to in the virtual array with interelement spacing being [36]. After selecting continuous elements from , then, we obtain where is the selecting matrix and denotes the column vector after the same selecting operation from . is the direction matrix of the continuous part, where

Similar with the steps from Eq. (3) to Eq. (5), we can obtain another virtual array from the overall output of subarray 2 and subarray 3, which can be expressed as where denotes a column vector after the selecting operation and is the direction matrix corresponding to an -element array located from to with interelement spacing being . The steering vector is

Now, the two virtual coprime subarrays in Eq. (5) and Eq. (7) are obtained, and the large interelement spacing will result in parameter estimation ambiguity problem. However, our method will in turn exploit the phase ambiguity to reduce the complexity and then eliminate the ambiguity based on the coprime-ness between and . Suppose , then the first subarray in Eq. (5) achieves larger aperture than that in Eq. (7), so we choose to estimate the DOA for better estimation performance.

As the virtual output has only one snapshot, sparse representation framework will be established to avoid the aperture loss caused by the spatial smoothing [24]. Discretize the whole spatial range as a grid , and suppose that all the true DOAs fall in the grid, i.e., the dictionary contains the columns of , then Eq. (5) can be rewritten in a sparse form as where is a sparse vector, whose elements corresponding to the true DOAs keep the same with those in and last element is noise power . After sparse recovery, the positions of nonzero elements (except the last element) in will give the estimations of the DOAs. However, as the interelement spacing of the virtual array is , which is larger than half-wavelength, then there are phase ambiguities in . To clearly elaborate this problem, let ; then, determines the uniqueness of due to the Vandermonde structure, i.e., if , then .

As , there is a cyclic phase ambiguity in . Except for , there are other angles satisfying

It can be derived from Eq. (10) that the relationship between and is where is an integer making locate at the range [-1, 1]. If angles are also located in the grid , then and will provide identical columns in the dictionary matrix due to Eq. (10). This will not only cause the estimation ambiguity but may also make the sparse recovery fail. However, we can in turn exploit the ambiguity to reduce the size of the dictionary and then reduce the complexity of sparse recovery accordingly.

From Eq. (11), it is shown that the solutions and are uniformly distributed among the range [-1, 1] following a circle . An example is shown in Figure 2, where and , then other two solutions are and , respectively. It is also shown by the dashed lines in Figure 2 that if we divide the whole range into cycles with width being , then there is only one solution in one cycle based on Eq. (11). As these solutions provide identical atoms in the dictionary, we can choose one cycle as a representative to construct the dictionary. Without loss of generality, we choose range , and the angle range is , whose corresponding dictionary is denoted by ; then, the sparse form in Eq. (9) becomes where is a -sparse vector, whose elements corresponding to the true steering vectors (maybe not the true DOAs) keep the same with those in , and the others are zero (except the last element). For example, as shown in Figure 2, the true solution is 0.5, and the representative solution in the partial dictionary is -1/6.

Due to Eq. (12), the phase ambiguity in the dictionary can be avoided now. Meanwhile, as now the size of the dictionary is only of its original size, the computation complexity of sparse recovery can be reduced.

##### 3.2. Off-Grid Sparse Representation Framework

Now a partial dictionary based sparse representation framework is established, but it is built based on the assumption that the true DOAs or their representative angles are located in the grid. However, the angles are very likely to lie off the discretized grid, no matter how fine the grid is defined. Off-grid sources will bring in grid mismatch problem and degrade the sparse recovery performance significantly. In this section, we take the off-grid problem into account and reformulate the sparse representation to enhance the robustness to grid mismatch.

Within the range , we denote the uniformly sampled grid as with adjacent interval being . Then, the true DOA or its representative angle can be represented by a nearest grid plus an offset , which is among the range . Based on first order Taylor expansion around the grid [14], the true steering vector can be approximately expressed as where . Then, Eq. (12) is revised as where , , and Let , then it is easy to verify that and are joint sparse [30]. So the off-grid sparse formulation can be expressed as where , and , where means the -th element of . It should be noted that the covariance matrix in Eq. (3) can only be estimated via finite snapshots where denotes the snapshot number. So the sparse form in Eq. (16) is not robust due to the residual error. Use to denote the deviation of in Eq. (6), then according to [37], follows asymptotic normal distribution with zero mean and covariance matrix being

After selecting operation

Define weight matrix , then

From Eq. (20), follows an asymptotic chi-square distribution with DOF. Consequently, the enhanced sparse recovery problem can be formulated as where is the approximate weight matrix; is the up bound of the fitting error, which can be set as [11], where denotes the inverse cumulative distribution function that makes the inequality holds with a probability . Generally, it is enough to set to make it nearly a sure event.

After solving Eq. (21) via CVX [38, 39], we can obtain the estimations of and , which are denoted as and , respectively.

##### 3.3. Ambiguity Elimination

The positions of nonzero elements in the first elements of and give the initial DOA estimations , which are the grids nearest to the true DOAs or their representative angles. Besides, the offset vector can be obtained via where ./ means element-wise division. Then, the angles are obtained via where the offsets are obtained from the first elements of in Eq. (22).

Now, the angles are estimated with offsets being compensated, but the angles in Eq. (23) may be true DOAs and also may be representative angles. As been discussed in Eq. (10) and Eq. (11), there are totally angles including sharing the same atom, and their relationship is where is an integer making locate at the range [-1, 1].

To determine the unique DOA without ambiguity, we substitute the angles in Eq. (24) into the whole virtual array where and . Due to the coprime relationship between the two subarrays, the unique DOA can be determined from the coincide results from the two subarrays. Consequently, if the whole array containing both two subarrays is exploited, unique angle is determined by finding the maximum value in Eq. (25).

For the complexity, the proposed method only requires partial dictionary, and the main complexity lies in the construction of two covariance matrices and sparse recovery. The total number of complex multiplications is about , where denotes the dictionary size. The proposed method has lower complexity than peak search method [24] and other sparse representation methods that require whole dictionary [27–30]. Compared to DOA estimation methods with closed-form solution, e.g., ESPRIT and root-MUSIC, the proposed method costs more but achieves better estimation performance, which will be verified in the simulation section below.

#### 4. Simulation Results

In the simulations, the CCNA is configured with , , and . snapshots are collected to estimate the covariance matrix, and the root mean square error (RMSE) is defined below to measure the DOA estimation performance where denotes the estimations of of the -th Monte Carlo trial and trials are carried out.

With the measurement of RMSE, Figures 3 and 4 present the angle estimation accuracy comparisons between the proposed algorithm and other methods versus SNR and snapshot number, respectively. The ACA method [24], PS-MUSIC using prototype coprime array [32], root-MUSIC method using prototype coprime array [33], and root-MUSIC using CCNA [36] all adopt the same number of physical antennas with the proposed method for fair comparison. It is indicated from Figures 3 and 4 that the proposed algorithm outperforms the other methods, and the main reasons include (1) the virtual subarray with larger aperture is chosen to avoid the negative effect from the smaller subarray, and (2) the off-grid sparse representation is established to amend the grid mismatch problem. The PS-MUSIC has the worst performance, especially with low SNR, as it has the additional pairing problem, and it utilizes the data from the physical array, which has limited DOF.

To test the resolution performance of the algorithms, we choose two closely spaced sources with angles being and , respectively. Figure 5 shows the estimation results of the algorithms over 20 trials with dB. It is indicated that the proposed method can always clearly identify the two sources, while the other methods have big deviations. Consequently, the proposed method achieves the best angular resolution.

#### 5. Conclusions

An off-grid DOA estimation method exploiting CCNA is proposed. Based on the nested relationships within the three subarrays, two virtual coprime subarrays are obtained firstly in the coarray domain. Thereafter, subarray with larger aperture is chosen for enhanced estimation performance, and cyclic phase ambiguity is exploited to reduce the size of the dictionary. Meanwhile, off-grid sparse reconstruction method is established to amend the grid mismatch. Finally, DOA is uniquely determined by substituting the ambiguous into the whole array. Compared to other methods with simulations, the proposed approach is verified that it has better DOA estimation performance and angular resolution.

#### Data Availability

Data are available in the manuscript.

#### Conflicts of Interest

The authors declare no conflict of interest.

#### Acknowledgments

This work is supported by the fund of state key laboratory of complex electromagnetic environment effects on electronics and information system (CEMEE 2021Z0101B), National Science Foundation of China (61631020, 61601167), the fund of Sonar technology key laboratory (Range estimation and location technology of passive target via multiple array combination), the Jiangsu Postdoctoral Science Foundation (2020Z013), and the China Postdoctoral Science Foundation (2020M681585).