#### Abstract

The spatial smoothing-based and matrix reconstruction-based decorrelation algorithms are widely used to realize the accurate direction-of-arrival (DOA) estimation with the coexistence of both uncorrelated and coherent sources. However, these methods apply only to the traditional uniform linear array (ULA), of which the manifold matrix is a Vandermonde structure. In this paper, to achieve the DOA estimation of mixed signals on coprime array, the coprime array is decomposed into two independent sub-ULAs, and the received data is processed, respectively. An improved spatial differencing technique is exploited to estimate the uncorrelated and coherent sources separately, and the lost degree of freedom (DOF) caused by array decomposition is recovered. The final high-precision DOA estimation is obtained by conducting an ambiguity resolution method to the estimation results of the decomposed subarrays. Furthermore, double-phase partial spectral search scheme is introduced to diminish the computational complexity. The feasibility and superiority of the proposed algorithm are validated through detailed experimental simulations.

#### 1. Introduction

DOA estimation technology obtains the high-resolution azimuth information of targets through array signal processing, which is widely used in self-driving, medical imaging, seismic survey, and precision attack on battlefield targets [1, 2]. However, coherent sources are frequently found in space due to multipath propagation and malicious smart jammers. In this way, the signal subspace and noise subspace infiltrate each other, and the traditional subspace-based methods might fail since the data covariance matrix is rank-deficient. Therefore, the preprocessing of decorrelation is necessary in these situations.

In accordance with or without the loss of DOF, the mainstream decorrelation algorithms are categorized into dimensionality reduction methods and others. The latter mainly includes Maximum Likelihood (ML) algorithm [3], weighted subspace fitting (WSF) [4], and sparse reconstruction algorithms based on compressed sensing [5]. These methods have broad application scenarios as they demand no correlation of signals and possess high utilization ratio of array antenna. Unfortunately, the multidimensional search and the construction of an overcomplete basis involve huge computational complexity, which seriously prevents them from practical application.

In contrast, the subspace decorrelation methods for dimensionality reduction class have simple models and moderate operands, and are better suited to handle coherent signals. Specifically, spatial smoothing-based algorithm [6] divides the array into a series of overlapping subarrays and restores the rank of matrix by the superposition of subarray covariance. In this way, however, nearly half of the DOF is lost, and only the autocorrelation information of main diagonal accessory entries in the full covariance matrix is exploited, leading to low estimation accuracy. In response, Pillai and Kwon [7] and Grenier and Bosse [8] propose Forward and Backward Spatial Smoothing (FBSS) and Weighted Spatial Smoothing (WSS) algorithms, respectively. An ESPRIT-like decorrelation algorithm termed Single-column Toeplitz (STOEP) method [9] reconstructs a Toeplitz matrix by randomly selecting a column of the covariance matrix. Since the reconstructed matrix of STOEP is insensitive to the coherency between signals, it can also effectively detect coherent signals. And the Multi-column Toeplitz (MTOEP) method [10] enhances the robustness of STOEP by extracting the complete covariance matrix. To meet the requirement of DOA preinstallation, Zhang et al. [11] further propose

Partial Toeplitz Matrices Reconstruction (PTMR) based on the square of covariance matrix.

To detect more sources and improve the array utilization, Rajagopal and Rao [12] propose the difference algorithm, which exploits the Toeplitz property of the covariance matrix of uncorrelated signals to separate them from the correlated signals and stepwise estimated the DOAs. Furthermore, the papers [13–15] combine oblique projection and matrix transformation for a thorough purge of uncorrelated signals in the difference process.

Nevertheless, all subspace decorrelation algorithms suffer from a strong dependence on the Vandermonde structure of the array manifold, indicating the uniform distribution of physical sensors is required. In practice, the sensors are usually set sparsely to receive more information in a broader space. Since sparse arrays cannot be directly divided into identical overlapping subarrays as uniform arrays do, the spatial transformation processing is inevitable before the decorrelation operation. For instance, Li eta al. [16] and Belloni et al. [17] transformed the sparse planar array into virtual uniform array via array interpolation and manifold separation, respectively, while the signal type and array structure are specially required.

Recently, one representative kind of the sparse array, the coprime array with enhanced DOF and restrained mutual coupling becomes one of the research highlights in array signal processing [18, 19]. The excellent direction-finding performance of coprime array benefits from the construction of an augmented uniform virtual array based on the difference coarray. Distinguished from traditional ULA, however, the reconstructed virtual array is actually equivalent to the single snapshot of second-order statistics, which contains no waveform information of the original signals. Thus, it cannot be applied to the subspace decorrelation algorithm.

To achieve the DOA estimation of coherent signals based on coprime array, the characteristic of prime number is exploited. The coprime array is decomposed into two sparse ULAs with spacing beyond half wavelength. Note the uniform distribution of the decomposed subarrays, the subspace decorrelation algorithms can be extended to it. An optimized spatial differencing method is employed solely for each subarray; the uncorrelated and coherent sources are separated and estimated stepwise. The multiple signal classification (MUSIC) is combined to get the high-accuracy but ambiguous estimation results. Finally, an improved ambiguity resolution algorithm is presented back to the two groups of estimation results, and the unique DOAs are obtained. Although the proposed approach includes two spectrum peak searches, we have opted to discard the global search scheme on MUSIC spectrum in favor of a partial search scheme [20], which reduces the computational cost and guarantees real-time performance in practical applications.

#### 2. System Model

Assume *D* narrowband quasistationary targets imping on the coprime array with *N* sensors from the direction of . As depicted in Figure 1, the coprime array is the superposition of a pair of sparse ULAs. Subarray 1 possesses elements with the spacing of units, and subarray 2 possesses elements with the spacing of units ( and are coprime numbers, and ). Let represents the location of *n*th sensors and denotes the distribution set. The first marginal element is chosen to be the reference with appropriate generality ().
where d is the unit spacing and , denotes the wavelength of the transmitting source.

Suppose there are uncorrelated signals corresponding to the propagation of the independent sources with power ; while the rest coherent signals correspond to the propagation of the *L* groups of coherent sources with power . Besides, each coherent group contains multipath signals.

The th snapshot of the received signal is where the is the global manifold matrix, is the manifold matrix of uncorrelated signals, and is the manifold matrix of coherent signals. denotes the entire signal waveform vector, denotes the waveform vector of uncorrelated signals, and denotes the waveform vector of coherent signals. represents the steering vector and is the additive Gaussian white noise vector. and , respectively, denotes the DOA and complex attenuation coefficient of the th multipath signals for the th coherent sources.

The sample covariance matrix can be expressed as where is the covariance matrix of uncorrelated signals and denotes the covariance matrix of coherent signals. The global signal covariance matrix is a block diagonal matrix and , is the -dimensional identity matrix.

Limited to the finiteness of snapshot, mathematical average is substituted with statistical average as follows:

#### 3. Proposed Method

In this section, the coprime array is decomposed as shown in Figure 2 and a detailed introduction of the proposed method is exhibited. Moreover, the available DOF and computational complexity are analyzed.

##### 3.1. Improved Ambiguity Resolution Algorithm

As is accepted, the phase ambiguity occurs when the interelement spacing of ULA is times of the unit spacing, and the real DOAs lost the unique corresponding peaks of the MUSIC spectrum. Considering as the real DOA and as the corresponding ambiguous angle, the following relationship is satisfied. where the nonzero integer represents all possible ambiguous angles. If the sensors are spaced no more than half-wavelength apart , there is no such fulfills (6) since , and phase ambiguity is absent. If , in contrast, things can be different. Taking the subarray 1 with for example, to satisfy (6), we have , and all the possible values of are . Since and are interchangeable, have possible values, indicating there are peaks for including the real peaks. Similarly, there are peaks in the MUSIC spectrum of the subarray 2 with sensors, among which only one peak points to the actual DOA.

Since the peak corresponding to real DOA appears at the same position in the spatial spectra of the two coprime subarrays, while the pseudo peaks corresponding to the ambiguous angles do not appear at the same position, the exact unambiguous DOA can be determined by contrasting the estimation results of the subarrays.

and are chosen to denote the angles set corresponding to the spectral peaks of subarray 1 and subarray 2, respectively. Then, the estimation results of the two subarrays, and , are resolved by

To acquire the signal information in a wider range and improve the estimation accuracy, the estimation results of the subarray with larger aperture are selected as the final DOA estimation result

##### 3.2. The Estimation of Uncorrelated Sources

The MUSIC algorithm is exploited to estimate the uncorrelated signals first. The Eigendecomposition (EVD) of the whole covariance matrix is expressed as where is the diagonal matrix of eigenvalues and . consists of signal eigenvalues, and comprises the noise eigenvalues. , , and are the matrixes whose columns are the corresponding eigenvectors of and .

Given that the signal subspace and the noise subspace are orthogonal, and the steering vectors span the same vector space as do, therefore,

Since is the linear combination of the column vector of a Vandermonde matrix, the steering vectors of the single multiple signals in each coherent group cannot satisfy Equation (10) like that of uncorrelated signals. Consequently, the DOAs of the uncorrelated sources can be obtained by searching the peaks of

After obtaining the estimation results on the two subarrays, the unique DOAs of unrelated signals are obtained by applying the ambiguity resolution algorithm shown in Section 3.1.

##### 3.3. The Estimation of Coherent Sources

In this section, an improved spatial differencing method [14] is used to eliminate the uncorrelated components in the covariance matrix, and the estimation of coherent signals is possessed by the differencing matrix.

As demonstrated in Figure 3, subarray 1 is further divided into overlapping identical secondary subarrays, each of which has elements and the spacing of .

The covariance matrix of the secondary subarray meets the following expression: where the selection matrix is given by

To dissociate the coherent sources from uncorrelated signals and resolve the rank deficiency, the equivalent spatial smoothing is performed by averaging the difference between and . where is the dimensional antidiagonal matrix and the order spatial differencing matrix owns the following essential properties: (1) only the information of coherent components is remained in . (2) The rank of equals to the total number of coherent sources when and . Detailed proof can be found in [14].

is Eigen decomposed as where is the diagonal matrix of signal eigenvalues, is the diagonal matrix of noise eigenvalues. and are the subspaces composed of the corresponding eigenvectors of signal eigenvalues and noise eigenvalues, respectively. According to the orthogonality of the steering vectors and the noise subspace, the coherent signals can be detected by searching the spectrum peaks of:

Next, the ambiguity resolution algorithm is used again to get the unique DOAs of coherent signals.

##### 3.4. Biorder Partial Spectral Search Scheme

Since the proposed algorithm involves two spectral peak searches, the selection of search step is crucial: the computational complexity is expensive when is small, while the estimation accuracy will decrease when goes higher. The biorder partial spectral search scheme [21] is introduced to improve the precision without costing much computational complexity. As shown in Figure 4, the first search is performed with initial step size to obtain a roughly estimated angle , and then the second search is performed during the range of with step size to obtain the refined estimated angle .

The detailed steps of the proposed algorithm are listed as follows: (1)The received data of subarray 1 is collected and the covariance matrix is calculated via Equation (5)(2) is eigendecomposed to plot the spatial spectrum of uncorrelated signals via Equation (12). The biorder partial spectral search scheme is applied to find the ambiguous DOAs of the uncorrelated signals(3)The order spatial differencing matrix is generated via Equation (15).(4) is eigendecomposed to plot the spatial spectrum of uncorrelated signals via Equation (17). And the biorder partial spectral search scheme is applied again to find the ambiguous DOAs of the coherent signals(5)Steps 1-4 are repeated on subarray 2 to obtain the ambiguous estimation results of the uncorrelated sources and coherent sources(6)The results of the two subarrays are deblurred via Equations (7) and (8) to attain the unique DOAs

##### 3.5. DOF

Since , the maximum number of detectable sources, namely the DOF, is depending on subarray 1. To correctly identify the uncorrelated sources using MUSIC algorithm, the signal subspace cannot fulfill the eigenspace and the demanded condition is presented as

For the same reason,

when , , and the maximum value of equals to . Then,

is substituted into Equation (18) as follows:

Combining Equation (20) with Equation (21), the DOF of the proposed method is derived as

##### 3.6. Computational Complexity

The computational complexity of the proposed algorithm is investigated in this section, which mainly focuses on the eigenvalue decomposition and spectral peak search.

Subarray 1 is considered first. For the estimation of uncorrelated signals, the computational complexity of EVD of covariance matrix is , and the computational complexity of the peak-finding searching is . Here the is the total number of search points, which is inversely proportional to search step . and are the search points of the rough search and refined search, respectively. For the estimation of coherent signals, the computational complexity of EVD of spatial differencing matrix is , and the computational complexity of the peak-finding searching is and . Hence, the computational complexity of subarray 1 is concluded as . Similarly, the computational complexity of subarray 2 is , , and . The total computational complexity is calculated as follows:

#### 4. Simulation Results

In this section, the effectiveness of the ambiguity resolution algorithm is verified, and the comparisons of the computational complexity and DOA performance are presented. Suppose the coprime array has a total of 10 sensors, which is decomposed to the five-element subarray 1 with the spacing of , and the six-element subarray 2 with the spacing of .

##### 4.1. Experiment 1: Ambiguity Resolution Algorithm

Assume that two coherent signals come from and with the complex attenuation coefficient of and , respectively. Furthermore, there are two uncorrelated signals at and . The Signal to Noise Ratio (SNR) is set to be , and the snapshot .

The MUSIC spectrums in Equation (12) for both uncorrelated signals and coherent signals are shown in Figures 5 and 6. Note that there are 12 peaks presented in the spectrum corresponding to subarray 1, and 10 peaks corresponding to subarray 2, which is identical to the theory in Section 3.1.

The reason why peaks outnumber sources is the existence of ambiguity. Due to the characteristic of prime numbers, only the spectral peaks corresponding to the real DOA present in the same position of both MUSIC spectrums, while the spectral peaks corresponding to do not. Hence, the unique DOAs can be derived by finding the overlapping peaks. The proposed algorithm can accurately detect all four signals as exhibited in Table 1, where the red marks represent the overlapping angles of the two subarrays, namely the estimated results.

Furthermore, by estimating uncorrelated signals and coherent signals separately, it can break through the resolution limit of conventional spatial spectrum estimation algorithms and detect two distinct signals from the same angle.

##### 4.2. Experiment 2: Computational Complexity

To reflect the advantages of the biorder partial spectral search strategy, the computational complexity of regular spectral searching method (RSS) is compared with the proposed search strategy versus different numbers of sensors.

As can be seen from the comparison in Figure 7, the computational complexity of the optimized searching strategy, the red line lines, is much lower than that of rough RSS and close to that of fine RSS , no matter for coherent signals or uncorrelated signals. Since the accuracy of the subspace MUSIC algorithm depends on the length of searching step, the simulation results reveal that the biorder search strategy can maintain precise DOA estimation results while taking into account the computational complexity.

##### 4.3. Experiment 3: The Performance of DOA Estimation

The forward spatial smoothing (FOSS [6]), forward-backward spatial smoothing (FBSS [7]), Single Toeplitz method (STOEP [9]), Multiple Toeplitz method (MTOEP [10]), and Partial Toeplitz square Matrix Reconstruction (PTMR [11]) are compared with the proposed algorithm. Root mean square error (RMSE) and the estimation accuracy (Acc) are chosen to reflect the magnitude of angle estimation bias, which is defined as follows:
where is the number of Monte Carlo simulations, denotes the estimation angle of in the *w*th Monte Carlo simulation, and counts all occurrences of . For the coprime array, . Assume three sources imping on the antenna array from , of which the first two sources are coherent sources with complex attenuation coefficients of , and the last source is independent from other sources. 1000 Monte Carlo trials are repeated to eliminate the randomness of the data.

###### 4.3.1. DOA Performance versus SNR

The number of snapshots is set to be 200 and SNR is evenly evaluated at intervals of 2.5 within the range of . The performance comparisons of RMSE and Acc versus different SNRs are exhibited in Figures 8 and 9.

Consistent with expectations, with the increase of SNR, the RMSE gradually decreases and the Acc gradually increases. It is evident that the blue stars line always corresponds to the lowest RMSE and the highest Acc, indicating that the proposed algorithm outperforms the others. Although the reconstruction of full-rank covariance is simpler for matrix reconstruction-based decorrelation algorithms, the reconstructed noise power becomes the square of original noise power, thus the DOA performance of MTOEP is inferior to FOSS and FBSS.

In contrast, although the proposed algorithm loses nearly half of the array DOF due to the decomposition of coprime array, spatial differencing technique is employed to recoup this loss by estimating the uncorrelated sources and coherent sources in two consecutive steps. Compared with other algorithms with the same DOF, the proposed algorithm can obtain signal information in a larger range and effectively suppress the mutual coupling due to sparse layouts of array sensors, hence achieving the optimum DOA estimation performance.

###### 4.3.2. DOA Performance versus Snapshot Number

The number of snapshots is another crucial factor to determine the DOA performance except for SNR. As shown in Figures 10 and 11, Monte Carlo experiments study the comparison of RMSE and Acc versus different snapshot numbers . The SNR is fixed at 0 dB, and other experimental parameters remain unchanged.

As demonstrated above, the performance curve of the RMSE is continuing to decline with the addition of snapshots. Apparently, the proposed algorithm can obtain the lowest RMSE and the highest Acc for each snapshot number. The proposed algorithm can reduce RMSE to the level of nearly zero and achieve an accurate rate of approximately 100% with only 100 snapshots.

#### 5. Conclusion

In this paper, the joint estimation of both uncorrelated and coherent sources on sparse array is achieved based on the decomposed coprime subarrays and spatial differencing technique. Since uncorrelated and coherent sources are estimated separately, the number of detectable sources exceeded that of physical sensors; even two sources from the same azimuth can be distinguished for some specific situations. Furthermore, the biorder search strategy is introduced to reduce computational complexity. The available DOF is calculated and computational complexity is analyzed. Exhaustive simulations confirm the superior performance of the proposed approach under harsh conditions with limited snapshots and low SNR.

#### Data Availability

The data used to support the findings of this study are included within the article.

#### Conflicts of Interest

The authors declare that they have no conflicts of interest.