#### Abstract

This paper addresses the problem of direction-of-arrival (DOA) estimation of coherent signals in the presence of unknown mutual coupling, and an autoregression (AR) model-based method is proposed. The effects of mutual coupling can be eliminated by the inherent mechanism of the proposed algorithm, so the DOAs can be accurately estimated without any calibration sources. After the mixing matrix is estimated by independent component analysis (ICA), several parameter equations are established upon the mixing matrix. Finally, all DOAs of coherent signals are estimated by solving these equations. Compared with traditional methods, the proposed method has higher angle resolution and estimation accuracy. Simulation results demonstrate the effectiveness of the algorithm.

#### 1. Introduction

Direction-of-arrival (DOA) estimation is very important in a variety of wireless communication applications, such as mobile communication, radar, and distributed sensor networks. In particular, many effective high-resolution DOA estimation algorithms have been developed and deeply investigated in the last decades [1]. Since then, the attention of the signal processing community has focused on the factors that block the practical application of those algorithms. The first factor is the unknown mutual coupling, which will affect the array manifold of the array and result in poor accuracy of DOA estimation [2]. The other factor is that there may be highly correlated or coherent signals because of multipath propagation [3, 4]. When the incident signals are highly correlated or coherent in the presence of unknown mutual coupling, the performance of conventional high-resolution DOA estimation methods will deteriorate significantly.

In the last years, many array calibration algorithms have been proposed with respect to the mutual coupling effect [5–13]. Hung [5] uses an iterative least mean-square approach to estimate the calibration matrix, but it requires a preliminary calibration. The above algorithms may not be easily carried out in practice, because of the additional calibration sources or sensors. An iterative algorithm is given to compensate the mutual coupling and perturbation of gain and phase in [6]. However, the convergence rate is slow, and computational cost is very expensive. In [7], a novel online mutual coupling compensation algorithm is presented to estimate coupling parameters through an alternating minimization technique, but the convergence is not well guaranteed. In [8], an algorithm that applies a group of auxiliary sensors in uniform linear arrays (ULAs) has been proposed to estimate the DOAs, but the algorithm requires a large number of sensors, and it is difficult to be satisfied in practice. References [9, 10] present a unified framework and sparse Bayesian perspective for array calibration and DOA estimation. Moreover, Dai et al. proposed a sparse representation method to eliminate the effect of mutual coupling by its inherent mechanism [11]. However, the computational cost is expensive. Many studies have been made to reduce the computational complexity of the calculations by using a certain unitary transformation that converts complex-valued manifold matrices of uniform linear arrays (ULAs) into real ones [12, 13].

On the other hand, many techniques have been proposed to deal with the correlated or coherent situation. A forward/backward spatial smoothing (FBSS) method that can solve the coherent problem is presented in [3]. Malioutov et al. propose the method of L1-SVD to address the general DOA estimation problem [14]. L1-SVD first decomposes the array output and extracts the signal energy into (the signal number) singular vectors and then represents them under sparsity constraint to estimate the signal directions. The method in [15] estimates the uncorrelated and coherent signals separately but encounters the differencing matrix power loss and needs extra processing to recover the rank. Recently, independent component analysis (ICA) has been utilized to solve the DOA estimation problem [16, 17]. These methods can estimate the real steering vectors with unknown mutual coupling. However, owing to the complex structure of the MCM, all the methods taking care of the correlated or coherent situation cannot be utilised to estimate the DOAs in the presence of unknown mutual coupling.

However, it is more difficult to estimate DOAs of coherent signals in the presence of unknown mutual coupling. Dai and Ye [18] propose an improved spatial smoothing algorithm for DOA estimation of coherent signals in the presence of unknown mutual coupling, but it significantly deteriorates while angle interval is not large enough or several groups of coherent signals coexist. Inspired by [19] and based on the estimation of the real steering vectors by ICA, we develop a spatial AR model-based algorithm for coherent DOA estimation of ULA in the presence of mutual coupling. Simulations illustrate that the DOA estimation accuracy of our approach is higher than the improved spatial smoothing algorithm in [18].

The paper is organized as follows. The data model of the ULA is given in Section 2. The spatial AR model algorithm is described in detail in Section 3. Computer simulations and conclusions follow in Sections 4 and 5.

#### 2. Data Model

Consider narrowband non-Gaussian signals impinging on a uniform linear array (ULA) with array elements, where the distance between adjacent sensors is equal to half of the wavelength. Assume that there exist groups of coherent signals because of multipath propagation, and the signals within the same group are coherent and independent in different groups. In the th group, suppose the coherent signal coming from the direction is corresponding to the *l*th multipath propagation of the source , and . The total number of coherent signals can be denoted as . The array output vector is expressed as
where is the steering vector of the direction , is the wavelength of the signal, is the transpose operator, and are corresponding to the amplitude and phase fading coefficients of the *l*th signal in the th group, , , , and is zero mean additive white Gaussian noise vector. The real steering vector of the th group coherent signals is given by .

In the presence of mutual coupling, the true steering vector should be modified as where is the mutual coupling matrix (MCM) and can be expressed as a banded symmetric Toeplitz matrix with just a few nonzero coefficients

The covariance matrix of the received signals is defined by where is the expectation operator, the superscript denotes transpose complex conjugate operation, is the sources covariance matrix, is the variance of the additive noise, and is an identity matrix.

#### 3. Novel AR Model-Based DOA Estimation Algorithm

We assume denotes the number of coherent signals in one group. Referring to [19], the mappings of the signal directions on the unit circle are distinct roots of an th order equation if no ambiguity occurs. The relationship between and the direction of the *k*th signal is . Assuming that the coefficients of the unified equation are and the unknown parameter is , the equation is then given by
Equation (5) presents the relationship of the coherent signals directions. For one group of coherent signals without mutual coupling, the real steering vector is a linear mixture ofideal steering vectors. The special relationship is then given by
where is the corresponding fading coefficient in the *k*th multipath propagation and is the corresponding scaling ambiguity coefficient caused by ICA processing.

According to (6), the *J*th element of is

Because are the roots of (5), the following equations hold: According to the idea of [19], multiplying both sides of (8) with and then summing up the equations yield

Substituting (7) into (9), we obtain the relationship between the equation coefficients and vector

The real steering vectors in the presence of unknown mutual coupling can be given by Each of contains all the spatial information of one group of coherent signals. The real steering vectors can be estimated by ICA [16, 17].

Then we will introduce the proposed method to solve the problem of DOA estimation of coherent signals in the presence of unknown mutual coupling.

##### 3.1. The Case of

For any given column vector , we get

For the second equation to the ()th equation, multiplying both sides of (8) with and then summing up the equations yield

Using the same principle to process all the adjacent equations, we get

In addition, as the signals are of complex value, we utilize their conjugate information to improve the precision of the proposed method as [19]. Similarly, the following equations hold:

Combining (10) and (15), we establish the following equation set: in which the superscript * denotes the conjugate operation.

##### 3.2. The Case of

From the case of , we note that the elements of real steering vectors can be utilized adequately due to the mutual coupling. If , the elements and are useless to estimate the DOAs. The number of useful elements is

According to (16), the new formulation is given by

By comparing (16) with (18), we know that (16) is a special case of (18). The right vectors of (18) can be estimated by ICA. So, the coefficient vector can be estimated under the least mean-square error criterion. The least square solution is then given by where denotes Moore-Penrose inverse.

All roots of (8) can be estimated with the known equation coefficients . The DOAs of coherent signals in one group are given by where are the roots of (8) and denotes the phase angle of complex value. Thus, the DOAs of coherent signals in one group are estimated according to (20).

##### 3.3. Discussion

Due to (18), in order to make sure the equation has a unique solution, the left matrix of (18) must be of full rank. Therefore, the following inequality must be satisfied:

It is not difficult to see that the number of coherent signals in one group . As we know, the maximum number of independent sources resolved by ICA is equal to the number of sensors [17]. So, the maximum detectable number of source signals by the proposed method is

For the improved FBSS, the detectable number of source and the length of subarrays must satisfy

When , (23) can be rewrite as

#### 4. Simulation Experiment

In this section, some computer simulations are reported to illustrate the performance of our proposed method. In the following simulations, we will compare the proposed method to FBSS [3] and the improved FBSS algorithm in [18] for DOA estimation.

In the first simulation, we consider one group of two coherent signals impinging on an 8-element ULA from the directions, and the number of the mutual coupling coefficients is with [18]. The amplitude fading factor is . Figure 1 shows the root mean square error (RMSE) of each DOA estimate against input SNR computed via 200 Monte Carlo runs for each SNR and 500 snapshots of data for each run. As shown in Figure 1, the proposed method outperforms the improved FBSS method, and achieves similar performance as the original FBSS method with known mutual coupling.

In the second simulation, we consider the more complicated situation: two groups of two coherent signals impinge on an 8-element ULA from the directions and , and the number of the mutual coupling coefficients is with and . The amplitude fading factors are and . Figure 2 shows the RMSE of each DOA estimate against input SNR computed via 200 Monte Carlo runs for each SNR and 500 snapshots of data for each run. The results illustrate that our method can get high-resolution in the complex structure of the MCM when the level of SNR is large enough. However, the improved FBSS maintains a coarse accuracy no matter what the levels of SNR are.

In the third simulation, we will validate the high spatial resolution of the proposed method. Consider one group of two coherent signals with SNR of 10 dB impinging on an 8-element ULA, where the amplitude factors are the same as simulation 1. Assume that the directions of two coherent signals are and , where denotes the angle interval between the two source signals. We have 200 Monte Carlo trials with 500 snapshots to demonstrate the performance of the proposed method. Figure 3 shows the RMSE of DOA estimation using different algorithms versus the angle interval . It can be seen from Figure 3 that the proposed method outperforms the other two methods when the angle interval is not large enough.

To verify the maximum detectable number of source signals by the proposed method, in the forth simulation, we consider five groups of two coherent signals impinging on a 5-element ULA from the directions, , , , and , and the number of the mutual coupling coefficients is with . According to (22), the number of source signals is the maximum detectable number of our method . However, the improved FBSS and other sparse representation methods are incapable of processing the 10 sources with 5-element ULA due to their conditions. Figure 4 shows the RMSE of each DOA estimate against input SNR computed via 500 Monte Carlo runs for each SNR and 500 snapshots of data for each run. The result illustrates that the maximum detectable number of sources by the proposed method is consistent with (24).

#### 5. Conclusion

In this paper, an AR model-based DOA estimation algorithm is proposed for coherent signals in the presence of unknown mutual coupling. The effects of mutual coupling can be eliminated by solving a mathematical equation. Simulation results demonstrate that the proposed method has high spatial resolution and DOA estimation accuracy compared to the improved FBSS algorithm. Furthermore, the number of signals resolved by our method is larger than that of others.

#### Acknowledgments

The authors would like to thank the anonymous reviewers for their valuable comments and suggestions which vastly improved the content and presentation of this paper. This research was supported by the National Natural Science Foundation of China (no. 61072120) and the Program for New Century Excellent Talents in University of China (NCET).