#### Abstract

In array, mutual coupling between the antennas is inevitable, which has an adverse effect on the estimation of parameters. To reduce the mutual coupling between the antennas of distributed nested arrays, this paper proposes a new array called the distributed super nested arrays, which have the good characteristics of the distributed nested arrays and can reduce the mutual coupling between the antennas. Then, an improved multiscale estimating signal parameter via rotational invariance techniques (ESPRIT) algorithm is presented for the distributed super nested arrays to improve the accuracy of direction-of-arrival (DOA) estimation. Next, we analyze the limitations of the spatial smoothing algorithm used by the distributed super nested arrays when there are multiple-source signals and the influence of the baseline length of distributed super nested arrays on the accuracy of DOA estimation. The simulation results show that the distributed super nested arrays can effectively reduce the mutual coupling between the array antennas, improve the DOA estimation performance, and significantly increase the number of detectable source signals.

#### 1. Introduction

Direction-of-arrival (DOA) estimation is a major application of the antenna array [1], whose accuracy is related to the aperture of the array and the mutual coupling between the array antennas. The aperture of the array is an important factor affecting the accuracy of DOA estimation. Therefore, it is necessary to increase the aperture of the array to improve the accuracy of DOA estimation. The distributed arrays are usually composed of multiple subarrays with a large baseline length that can effectively increase the aperture of the array and significantly improve the accuracy of parameter estimation. However, it cannot increase the number of detectable source signals. For example, the maximum number of detectable source signals that can be resolved with an N antenna uniform linear arrays (ULA) using traditional subspace-based methods like multiple signal classification (MUSIC) [2] is N-1. It is necessary to increase the number of array antennas to increase the number of detectable source signals that greatly increases the hardware cost. To effectively solve this problem, sparse arrays [3–6], such as nested arrays [7–10], minimum redundancy arrays (MRAs) [11], and coprime arrays [12, 13], are proposed. These sparse arrays are capable of providing a dramatic increase in the degrees of freedom (DOF) and can resolve more source signals than the actual number of physical antennas. However, MRAs do not have simple closed-form expressions for the array geometry, and the antenna locations are determined by computer simulation and complex iterative calculations. When the number of array antennas is large, the amount of calculation is large, and in some arrays with limited aperture, MRAs may not be the best array layout. Coprime arrays have holes in the difference coarray [7], resulting in the maximum number of recognizable source signals lower than that of MRAs and nested arrays. Compared with MRAs and coprime arrays, the nested arrays have closed-form of expressions, and the number of resolvable source signals is greater than that of coprime arrays under the same number of array antennas. Therefore, nested arrays have become a research hotspot.

In 2010, Piya Pal proposed the concept of nested arrays and simulated the DOA estimation characteristics of nested arrays [7]. Piya Pal et al. applied high-order cumulants to multilevel nested arrays for DOA estimation in 2012 [8]. The simulation results show that the nested arrays can significantly improve the DOF of the linear array, and the maximum number of detectable source signals is close to the square of the number of array antennas. To improve the DOA estimation accuracy and increase the number of detectable source signals, combined with the advantages of distributed arrays and nested arrays, a lot of researches have been done on the distributed nested arrays. In 2015, Yi Wang combined the distributed arrays with the nested arrays, proposed the distributed nested arrays, and used spatial smoothing algorithm to improve the accuracy of DOA estimation and increase the number of detectable source signals [14]. In 2016, Yufeng Xie extended the one-dimensional distributed nested arrays to two-dimensional, which improved the accuracy of azimuth and elevation angle estimation [15]. In 2019, Yanping Liao proposed an improved distributed nested array to improve DOF and the accuracy of DOA estimation [16].

The research of [7–16] did not consider the mutual coupling between the array antennas. However, in practical applications, the influence of mutual coupling cannot be ignored when the distance between the array antennas is less than half the wavelength. The electromagnetic characteristics cause mutual coupling between the antennas, making the antenna responses interfere with each other, leading to the decrease in the accuracy of DOA estimation. Using appropriate mutual coupling models to decouple the received data can reduce the effect of mutual coupling between the array antennas [17–21]. However, these methods are usually computationally expensive and sensitive to model mismatch. Nested sparse circular arrays are proposed to reduce mutual coupling between the array antennas, but they still contain a dense subarray [22].

In 2016, the super nested arrays that could reduce the mutual coupling between the array antennas by redistributing the dense subarray of the nested arrays were proposed, and they had all the good characteristics of the nested arrays [23–26]. Besides, the generalized nested array and thinned coprime array were proposed to reduce mutual coupling, but they had holes in the difference coarray [27, 28]. To reduce the mutual coupling between the array antennas and make it easy to apply to actual projects, we introduce the super nested arrays and propose the distributed super nested arrays. An improved multiscale estimating signal parameter via rotational invariance techniques (ESPRIT) algorithm is presented for the distributed super nested arrays to improve the accuracy of DOA estimation. Numerical simulations prove the superiority of the proposed arrays using the improved multiscale ESPRIT algorithm.

#### 2. Concept of Distributed Super Nested Arrays

##### 2.1. Array Structure

The antenna positions of *K*-level nested arrays are given by the set [7],where and *d* denotes the minimum distance between the dense ULA antennas.

Figure 1 shows the structure of two-level nested arrays, and denote the number of first-level and second-level array antennas, respectively. . Bullets denote the actual array antennas and the cross indicates empty locations. The minimum distance between the dense ULA antennas is *d*. The number under the bullets representing an integer multiple of *d* is the position of the array antenna.

Since the antenna spacing *d* of the first-level subarray of the nested arrays is less than or equal to half the wavelength, the mutual coupling between the antennas is relatively serious. In practical applications, to reduce the mutual coupling between the antennas, super nested arrays are proposed. The antenna positions of the two-level super nested arrays can be expressed by the set as [23]where and denote the number of first-level and second-level array antennas, respectively. They are both positive integers, , , and *l* is a positive integer.

The parameters , , , and are expressed aswhere *r* is a positive integer. The structure of two-level super nested arrays is shown in Figure 2.

The distributed two-level super nested arrays are composed of two identical super nested arrays, and each super nested subarray satisfies formulas (2) and (3). The structure of the distributed two-level super nested arrays is shown in Figure 3, where . *D* is the baseline length. The array antenna position sets of the two super nested subarrays are represented by and , respectively. The array antenna position set is . The number of antennas in each subarray is , the total number of antennas is , and and are the positive integers.

##### 2.2. Signal Model

In this section, we shall firstly discuss the concept of difference coarray, degrees of freedom, and weight function [23].

*Definition 1. *(difference coarray). For a sparse array specified by an integer set , its difference coarray is defined as

*Definition 2. *(degrees of freedom). For a sparse array specified by an integer set , its degree of freedom is equal to the number of nonredundant elements in the difference coarray .

*Definition 3. *(weight function). For a sparse array specified by an integer set , its difference coarray is , integer , the weight function represents the number of occurrences of element *n* in the difference coarray .

It can be seen from the above definitions that the difference coarray can expand the physical array aperture of the original array and increase the DOF of the array. The DOF is closely related to the maximum number of detectable source signals, and the weight function is closely related to the mutual coupling between the array antennas.

Consider the distributed super nested arrays composed of two identical subarrays as shown in Figure 3. Both of the two subarrays are two-level super nested arrays composed of antennas, with a uniform spacing of , where denotes the signal wavelength. The total number of antennas is . The distance between the phase centers of the two subarrays, named baseline length, is . Assume that K arriving uncorrelated narrowband far-field sources impinge on the distributed super nested arrays with an unknown DOA, and use to represent the direction of the k^{th} source signal, . Considering the mutual coupling of the array antennas, taking the first antenna of the first subarray as the reference antenna and the first subarray as the reference array, the received signal of the array at time *t* can be expressed aswhere is the received signal, is the received signal of the first subarray, and is the received signal of the second subarray. is the source signal vector, and is the zero-mean additive white Gaussian noise vector. is the array manifold matrix of the first subarray, in which , and . The distance between the n^{th} antenna and the reference antenna is . is the array manifold matrix, , and **C** is mutual coupling matrix.

When the array sensor is a dipole antenna, the mutual coupling matrix calculation formula [29] iswhere and are the element impedance and load impedance, respectively.

In formula (6), the calculation of **C** is very complicated. For the convenience of research, the simple mutual coupling model in [27] is used to calculate the mutual coupling matrix, and the formula iswhere B is the maximum position of the array antenna.

For evaluating the strength of the mutual coupling matrix of different arrays, the coupling leakage [23] is defined asThe larger the *L*, the greater the influence of the mutual coupling effect on the array.

Then, the covariance matrix of the received signal is

#### 3. Proposed Algorithm

##### 3.1. Spatial Smoothing-Based DOA Estimation

According to formulas (5) and (9), the covariance matrix of the received signal iswhere is the noise power, represents the conjugate transpose, and is the covariance matrix of the source signal vector. If the signals are independent of each other, then , where represents the power of the *k*^{th} incident source signal.

Use the Khatri–Rao product property [30] to convert the obtained covariance matrix into a column vectorwhere represents the vectorization of the matrix, is the Khatri–Rao product, is the power of source signals, and represents the conjugate. Remove the redundant rows in and reorder them to obtain the new signal data

For the sake of convenience, let , , and , is a column vector whose middle element is one, and the other elements are zero.

After the data received by the array is processed, a distributed virtual uniform linear array composed of three identical ULAs is obtained as shown in Figure 4. This array is called the adjoint array of the original distributed super nested arrays. After deredundant processing on , the manifold matrix of the array is obtained. After the redundancy is removed, can be regarded as the received data of the adjoint matrix.

By vectorizing the covariance matrix of the received data of the physical array, the equivalent received data corresponding to the difference coarray can be obtained, but the covariance matrix of the received data has a rank of one. To restore the rank of the covariance matrix, spatial smoothing algorithm [7] and Toeplitz matrix reconstruction algorithm [31, 32] are used for processing. Here, we use spatial smoothing algorithm for processing.

Dividing the adjoint array in Figure 4 into overlapping subarrays, where the *i*^{th} subarray has antennas located atthe corresponding received data can be expressed aswhere is a matrix consisting of the *i*^{th} to ^{th} rows of and is a column vector of all zeros except a one at the ^{th} position.

The covariance matrix of the received data of the *i*^{th} subarray can be expressed as

By summing and averaging , the smoothed covariance matrix is

The array manifold matrix after spatial smoothing iswhere is the array manifold matrix of virtual ULA 2, in which .

Perform eigenvalue decomposition on to obtain the signal subspace . Since the space formed by the array manifold matrix is the same space as the signal subspace , there must be a unique nonsingular matrix with .

So far, the adjoint array of the distributed super nested arrays is smoothed to obtain a distributed array containing three ULAs, and the number of antennas of its subarray is *Q*. Next, an improved multiscale ESPRIT algorithm is used for DOA estimation.

##### 3.2. Improved Multiscale ESPRIT Algorithm

The first *Q*-1 and the last *Q*-1 antennas of every subarray in the smoothed distributed array are used to constitute the subarrays **C1** and **C2**, respectively. The amount of translation between the subarrays **C1** and **C2** is *d*. Therefore, the invariance relation between **C1** and **C2** has the formwhere is the selection matrix of the subarray **C1**, is the selection matrix of the subarray **C2**, represents the third-order unit diagonal matrix, , , and represents the directional cosine.

Similarly, the first two ULAs are selected to form the short baseline subarray **Fs1**, and the last two ULAs are selected to form the short baseline subarray **Fs2**. The amount of translation between the subarrays **Fs1** and **Fs2** is D. The first ULA is selected to form the subarray **Fl1**, and the last ULA is selected to form the subarray **Fl2**. The amount of translation between the subarrays **Fl1** and **Fl2** is twice that of D. Therefore, the following equations are established:where and are the selection matrices of the subarrays **Fs1** and **Fs2**, respectively, , in which ; and are the selection matrices of the subarrays **Fl1** and **Fl2**, respectively; , in which .

Substitute the signal subspace into formulas (18)–(20), respectively. The equations are as follows:where , , and are the rotation-invariant relationship matrix of coarse estimation, short baseline fine estimation, and long baseline fine estimation, respectively. , , and are the similarity matrices with , , and , respectively. Apply the least squares rule to solve equations (21)–(23). , , and can be obtained as follows:

Eigenvalue decomposition was performed on the matrices , , and , and the pairing algorithm is used to pair the eigenvalues corresponding to the same incident source signal. The coarse estimation , short baseline fine estimation , and long baseline fine estimation of the incident source signals are as follows:

Since the baseline length is , both the short baseline fine estimation and the long baseline fine estimation have periodic ambiguity. Therefore, the phase ambiguity must be resolved to calculate the fine estimation without ambiguity. The actual unambiguous short baseline fine estimation and long baseline fine estimation have the following relations with and , respectively.where and are the number of ambiguity periods of the phases of and (taking as the period), which can be searched by equations (27) and (28), respectively.

The value range of is and the value range of is . and indicate the rounding operations in the direction of negative infinity and positive infinity, respectively. Through the mentioned deblurring method, a precise estimation of the direction cosine with high accuracy and no ambiguity can be obtained. Then, the DOA estimation values of the incident source signals without ambiguity can be obtained as follows:

#### 4. Simulation Results

This section will verify the DOA estimation characteristics of the proposed distributed super nested arrays and algorithm from the following four parts. To illustrate that the distributed super nested arrays proposed in this paper can effectively reduce the mutual coupling between the antennas, the uniform spacing *d* is set as , , and for the simulation experiments.

##### 4.1. Weight Function and Coupling Leakage

This part simulated and analyzed the weight function and coupling leakage. Experiment one simulated the weight functions of ULA, distributed uniform arrays, nested arrays, distributed nested arrays, super nested arrays, and distributed super nested arrays. In the simulation, the number of antennas is 28. The simulation results are shown in Figure 5.

**(a)**

**(b)**

**(c)**

**(d)**

**(e)**

**(f)**

It can be seen from Figure 5 that the shape of the middle part of the weight functions of ULA, distributed uniform arrays, nested arrays, and distributed nested arrays is triangular, which is caused by the uniform arrangement of the array antenna, especially when the value of *n* is smaller and the weight function is relatively large. However, when *n* takes a small value, the weight functions of the super nested arrays and distributed super nested arrays are relatively small. Therefore, it shows that the mutual coupling between the antennas in super nested arrays and distributed super nested arrays is smaller.

To quantitatively study the mutual coupling effect between the antennas of ULA, distributed uniform arrays, nested arrays, distributed nested arrays, super nested arrays, and distributed super nested arrays, we choose , the remaining coupling coefficients are given by . According to formula (7) and formula (8), the coupling leakage values of ULA, distributed uniform arrays, nested arrays, distributed nested arrays, super nested arrays, and distributed super nested arrays are 0.169, 0.162, 0.120, 0.117, 0.069, and 0.069, respectively. The calculation results indicate that the mutual coupling between the antennas of super nested arrays and distributed super nested arrays is the smallest, followed by the distributed nested arrays and the nested arrays. The distributed uniform arrays and ULA suffer the most severe mutual coupling effect.

##### 4.2. Root-Mean-Square Error

The estimation accuracy is measured by the root-mean-square error (RMSE), and the formula iswhere *K* is the number of source signals, *M* is the number of Monte Carlo trials, is the true DOA of the *k*^{th} source signal, and is the estimated DOA of the *k*^{th} source signal obtained from the *m*^{th} Monte Carlo trial.

Experiment two simulated the DOA estimation characteristics of the distributed super nested arrays, super nested arrays, distributed nested arrays, nested arrays, distributed uniform arrays, and the ULA with the same number of array antennas. The simulation conditions are as follows: the source signal angle is 30°, the number of array antennas is 28, the number of snapshots is 500, and the basic uniform spacing *d* is equal to , , and , respectively. When the basic uniform spacing *d* is equal to , we assume . Since the mutual coupling coefficient is approximately inversely proportional to the antenna spacing, the smaller the antenna spacing, the more severe the mutual coupling between the antennas. Calculated according to the proportional relationship, when *d* is equal to , . When *d* is equal to , , where and are random values on . According to formula (7), the mutual coupling matrix can be calculated. Perform 2,000 Monte Carlo trials in the simulation. For comparison, the RMSE as a function of signal-to-noise ratio (SNR) under the same conditions is simulated without considering the mutual coupling between the antenna elements.

Figures 6–8 are the simulation results of , , and , respectively. DSNA, SNA, DNA, NA, and DUA stand for the distributed super nested arrays, super nested arrays, distributed nested arrays, nested arrays, and distributed uniform arrays, respectively.

**(a)**

**(b)**

**(a)**

**(b)**

**(a)**

**(b)**

Simulation results show that when the mutual coupling between the antennas is not considered, the RMSE of the distributed nested arrays and the distributed super nested arrays are consistent, the estimation error is much better than that of the distributed uniform arrays, and the ULA is at low SNR. When SNR is greater than the ambiguity threshold [33], their estimation error is much better than that of the super nested arrays and nested arrays. When considering the mutual coupling between the array antennas, the estimation error of the distributed super nested arrays is better than that of the distributed nested arrays, and the performance advantage is more obvious when the mutual coupling is more severe. When SNR is greater than the ambiguity threshold, the estimation error of the distributed super nested arrays is much better than that of the super nested arrays and nested arrays. Because of the influence of mutual coupling between the antennas, the estimation accuracy of the distributed uniform arrays and the ULA deteriorates sharply and tends to a fixed value. In addition, the DOA estimation accuracy is also closely related to the baseline length. If the ambiguity threshold of the baseline is exceeded [33], the estimation accuracy will deteriorate rapidly. Therefore, to obtain an accurate DOA estimation, an appropriate baseline length must be selected. In Section 4.3, the influence of the baseline length of the distributed super nested array on the accuracy of DOA estimation is analyzed.

Experiment three simulated the characteristics when there are multiple source signals. In the simulation, the number of source signals is six, the angle values are , other conditions are the same as experiment two. The simulation results of , , and are shown in Figure 9–11.

**(a)**

**(b)**

**(a)**

**(b)**

**(a)**

**(b)**

It can be seen from the simulation results that in the case of multiple source signals, considering the mutual coupling between the array antennas, the estimation accuracy of the distributed super nested arrays is better than that of the distributed nested arrays, super nested arrays, nested arrays, distributed uniform arrays, and ULA. The simulation results are consistent with experiment two.

In addition, when the mutual coupling between the antennas is not considered, the estimation error of the distributed nested arrays and distributed super nested arrays is much better than that of the distributed uniform arrays and ULA under low SNR. However, as the SNR increases, the estimation error tends to be a fixed value. When a certain SNR is reached, the estimation error of the distributed uniform arrays is better than that of the distributed nested arrays and distributed super nested arrays. This is because the number of snapshots is small, which cannot satisfy the uncorrelation between the source signals. The spatial smoothing algorithm is used to process the covariance matrix of the received signal. The covariance matrix is vectorized and the redundant information is removed, resulting in an incomplete utilization of the received data. The spatial smoothing algorithm for multiple-source signals DOA estimation, as the source signals cannot satisfy the uncorrelation, the covariance matrix of the signal vector is a square matrix close to the diagonal matrix. The more the number of source signals, the greater the deviation from the diagonal matrix. Therefore, the redundant information removed by the spatial smoothing algorithm is not really redundant information. Failing to use all the useful information of the received signal, the estimation accuracy is worse than that of the distributed uniform arrays, and it tends to be a fixed error that no longer improves as the SNR increases. The super nested arrays and nested arrays also have the same trend as the distributed super nested arrays, but the estimation accuracy of the distributed super nested arrays is better than that of the super nested arrays and nested arrays.

To verify the superiority of the proposed algorithm, experiment four simulated the estimation performance of the distributed nested arrays using the dual-scale ESPRIT algorithm, multiscale ESPRIT algorithm, and the proposed algorithm. The basic uniform spacing *d* is equal to , the baseline length *D* is set to 130 times the wavelength, and other conditions are the same as experiment two. Figure 12 shows the performance comparison of the three algorithms.

The dual-scale ESPRIT algorithm, multiscale ESPRIT algorithm, and the proposed algorithm use the coarse estimation as the reference to disambiguate the fine estimation. The difference is that the dual-scale ESPRIT algorithm performs defuzzification once. When the baseline length is large, to obtain a precise estimation without ambiguity, the accuracy of the coarse estimation needs to meet the requirements of correct defuzzification, which requires a higher SNR. The multiscale ESPRIT algorithm increases the intermediate estimation, which uses the higher precision intermediate estimation as the reference to disambiguate the fine estimation. The algorithm performs defuzzification twice, which greatly reduces the requirement on the SNR. The improved algorithm in this paper first uses the spatial smoothing algorithm for preprocessing to obtain a more precise coarse estimation, and then, it uses the coarse estimation as the reference to obtain a higher accuracy short baseline fine estimation. Then, it uses the short baseline fine estimation as the reference to obtain a long baseline fine estimation with twice the length of the baseline. The simulation results of Figure 12 show that when the SNR is less than 0 dB, the coarse estimation of the dual-scale ESPRIT algorithm no longer fully meets the requirements for defuzzification, and hence, the estimation accuracy deteriorates rapidly. However, because of the addition of intermediate estimation in the multiscale ESPRIT algorithm, its estimation accuracy in the range of −12 dB to 0 dB can still meet the requirements for defuzzification of fine estimation. When the SNR is less than −12 dB, the accuracy of the intermediate estimation no longer fully meets the accuracy requirements of defuzzification. At this time, the estimation error of fine estimation begins to increase, obviously. The improved method uses a spatial smoothing algorithm for preprocessing, which improves the estimation error of coarse estimation and then improves the estimation error of the short baseline fine estimation and long baseline fine estimation. Therefore, this method has better estimation performance when the SNR is greater than −18 dB. It is observed from the simulation results that the improved algorithm has better estimation accuracy than the other two algorithms when the SNR is low.

From the simulation results of experiment two, experiment three, and experiment four, it is observed that the proposed distributed super nested arrays and the improved algorithm in this paper can effectively reduce the mutual coupling between the antennas and improve the accuracy of DOA estimation.

##### 4.3. Ambiguity Threshold of Baseline

The bias of DOA estimation decreases with the increase in baseline when meeting the requirements of correct defuzzification. However, when the baseline increases up to a certain length, the bias increases rather than decreases, and this length is called the ambiguity threshold of the baseline [34]. Therefore, the influence of the baseline length of the distributed super nested array on the accuracy of DOA estimation is analyzed in this part.

Experiment five simulated the RMSE as a function of baseline length under different SNR. We assume the source signal angle is 30°, the number of array antennas is 28, the number of snapshots is 500, the basic uniform spacing *d* is equal to , and . The baseline length is set from 60 times the wavelength to 540 times the wavelength, and the interval is 20 times the wavelength. The SNR is equal to -5 dB, 0 dB, and 5 dB, respectively. Perform 2,000 Monte Carlo trials in the simulation. Figure 13 shows the simulation results of the distributed super nested arrays.

Figure 13 shows that within the ambiguity threshold of the baseline, the greater the baseline length, the higher the estimation accuracy. When the baseline is greater than the baseline ambiguity threshold, the estimation performance drops rapidly. Besides, the ambiguity threshold of the baseline also increases with the increase of SNR. Increasing the baseline length within the ambiguity threshold will improve the accuracy of the angle measurement. However, increasing the baseline length will also require a corresponding increase in SNR, and hence, selecting a suitable baseline length is crucial for the distributed super nested array.

##### 4.4. Maximum Number of Estimable Source Signals

Distributed super nested arrays can not only improve the accuracy of DOA estimation but also significantly increase the number of detectable source signals. For verifying that the proposed distributed super nested arrays can still improve the DOA estimation accuracy when the number of detectable source signals is greater than the number of array antennas, experiment six simulated the DOA estimation performance of the distributed super nested arrays and distributed nested arrays. In the simulation, the number of source signals is set to 40, the angle values are , the number of array antennas is 28, the number of snapshots is 5000, *d* is equal to , the SNR is 0 dB, and the baseline length *D* is set to 100 times the wavelength, . Use the improved multiscale ESPRIT algorithm and perform 2,000 Monte Carlo trials in the simulation. Figures 14 and 15 show the simulation results of the distributed super nested arrays and the distributed nested arrays, respectively.

**(a)**

**(b)**

**(a)**

**(b)**

It can be seen from Figure 14 that when the number of source signals is greater than the number of antennas, the distributed super nested arrays can estimate all targets under the condition of considering the mutual coupling between the array antennas. As shown in Figure 14(b), because of the mutual coupling between the antennas, the coarse estimation has an upward shift, which increases the estimation deviation, while the fine estimation is hardly affected, and its estimation error is still far better than the coarse estimation. Figure 15 shows that because of the mutual coupling between the antennas, the distributed nested arrays have false targets and large angle deviations when estimating the angle of the source signal. Especially at the edge angle, its estimation performance deteriorates rapidly, and it cannot estimate all targets. The simulation results of experiment six show that considering the mutual coupling between the array antennas, the distributed super nested arrays can significantly increase the number of detectable source signals and improve the DOA estimation accuracy, and its DOA estimation performance is better than that of the distributed nested arrays.

Experiment seven simulated the characteristics of RMSE changing with the number of source signals when considering mutual coupling between array antennas. In the simulation, the number of array antennas is 28, the number of snapshots is 500, *d* is equal to , the SNR is 0 dB, the baseline length *D* is set to 210 times the wavelength, and 2000 Monte Carlo trials are performed. The simulation results are shown in Figure 16.

Figure 16 shows that the maximum number of detectable source signals of the distributed uniform arrays is 13, which is determined by the number of its subarray antennas. When the number of source signals is small, the estimation accuracy is better than that of ULA. As the number of source signals increases, its estimation accuracy is worse than that of ULA. This is because the DOA estimation performance of the distributed uniform arrays is not only related to the mutual coupling between antennas but also closely related to the baseline length between the subarrays. As the number of source signals increases, the ambiguity threshold of the baseline becomes smaller and the estimation accuracy becomes worse. The maximum number of detectable source signals of ULA is 27, and when the number of source signals is greater than 22, the estimation error deteriorates rapidly. When the number of source signals is less than 41, the RMSE of the distributed super nested arrays and the distributed nested arrays is smaller than that of ULA. At the same time, it can be seen that when the number of source signals is less than 36, the RMSE of the distributed super nested arrays is obviously better than that of the distributed nested arrays. Table 1 shows the maximum number of detectable source signals when the number of array antennas of the distributed super nested arrays, distributed nested arrays, distributed uniform arrays, and ULA is 28.

Table 1 and simulation experiments further verify that the distributed super nested arrays can significantly increase the number of detectable source signals and improve the DOA estimation accuracy, and its DOA estimation performance is better than that of the distributed nested arrays. This shows that the distributed super nested arrays have a huge advantage and can be better adapted to practical engineering applications.

#### 5. Conclusions

This paper proposes a distributed super nested arrays model and an improved multiscale ESPRIT algorithm to reduce the mutual coupling between array antennas and improve the accuracy of DOA estimation. Besides, we analyze the limitations of the spatial smoothing algorithm used by the distributed super nested arrays when there are multiple source signals and the influence of the baseline length of the distributed super nested arrays on the accuracy of DOA estimation. Numerical simulations show the superiority of proposed arrays and algorithm. The distributed super nested arrays can not only improve the accuracy of DOA estimation but also significantly increase the number of detectable source signals, which can better meet the requirements of actual engineering. It has broad application prospects in the research of distributed high frequency radar.

#### Data Availability

The data used to support the findings of this study are available from the corresponding author upon reasonable request.

#### Conflicts of Interest

The authors declare that there are no conflicts of interest regarding the publication of this study.

#### Acknowledgments

This research was funded by the National Natural Science Foundations of China, grant number 61701140. The authors thank the Key Laboratory of Marine Environmental Monitoring and Information Processing, Harbin Institute of Technology, China, for data support.