#### Abstract

The classical detection techniques for multiple-input multiple-output (MIMO) systems are usually designed with the assumption that the additive complex Gaussian noise is uncorrelated. However, for closely spaced antennas, the additive noise is correlated due to the mutual antenna coupling. This letter analyzes an improved zero-forcing (ZF) technique for MIMO channels in colored environments. The additive noise is assumed to be correlated and the Rayleigh MIMO channel is considered doubly correlated. The improved ZF detector, based on the generalized least squares estimator (GLS), takes into account the noise covariance matrix and provides an unbiased estimator of the transmitted symbol vectors. We introduce some novel bounds on the achievable sum rate, on the normalized mean square error at the receiver output, and on the outage probability. The derived expressions are compared to Monte Carlo simulations.

#### 1. Introduction

Multiple-input multiple-output (MIMO) systems have great potential for increasing spectral efficiency [1]. To achieve the promised capacity, efficient and reliable reception techniques are required. The linear zero-forcing (ZF) technique is one of the most popular receivers [2–6]. In fact, despite its low-complexity, the ZF detector yields suboptimal performances [2]. The analysis of the MIMO-ZF detector has attracted considerable attention in the literature [2–6]. The effect of channel estimation error of the MIMO-ZF detector performance has been investigated in [2–4]. The SNR loss due to the spatial correlation and its effect on bit error rate (BER) of the MIMO-ZF detector are discussed in [4–6]. Although previous works present important contributions concerning the MIMO-ZF analysis, they assume that the channel additive complex Gaussian noise is uncorrelated. The noise vector entries are then modeled as independent and identically distributed (i. i. d.) random variables. This assumption is valid when the antennas are sufficiently spaced and thus coupling between them is negligible. However, for MIMO systems with closely spaced antennas, there is a mutual coupling between antenna elements [7]. This mutual coupling leads to a correlated thermal noise at the receiver front-end.

Motivated by the discussion above, this letter proposes and analyzes a ZF detection technique for MIMO systems in correlated environments. We assume that the additive noise is correlated and the Rayleigh MIMO channel has both transmit and receive correlations. We also assume that the fading channel matrix is perfectly known at the receiver. When the noise covariance matrix is known, the ZF detector corresponds to the generalized least squares estimator (GLS) of the transmitted symbols. When the noise covariance is unknown, we propose to replace the true matrix by the estimated one. The ZF detector is then called the feasible generalized least squares estimator (FGLS). This letter presents some bounds for the achievable sum rate (ASR), the normalized mean square error (NMSE) at the GLS receiver output, and the global outage probability (GOP). It is noteworthy that the scenario with doubly correlated channel has been rarely studied [2]. In fact, for this case, it is relatively difficult to describe the statistical properties of the postprocessing SNR [2]. The few relevant studies are given in [8] for MIMO systems and several fading models in [2]. Some results on the eigenstatistics for doubly correlated MIMO channels are reported in [9].

#### 2. System Model and ZF Receiver

##### 2.1. System Model

Consider an uncoded MIMO system with transmit and receive antennas with . The received signal vector at time is where is the number of received samples in the observation window or the frame length, is the transmitted vector, is the channel matrix, and is a circularly symmetric complex Gaussian noise vector with zero mean and covariance matrix . We assume that the noise covariance matrix is given by the widely used noise model (see [10] and the references therein): where is a positive definite matrix, with the th entry The transmitted symbols are uncorrelated with covariance . We consider a doubly spatially correlated Rayleigh flat-fading channel. With this assumption, spatial correlation occurs on both transmit and receive sides and the channel matrix is modeled as where is an matrix with i. i. d. zero-mean complex Gaussian entries with unit variance. The spatial correlation matrices and are defined as and .

When the channel matrix is available at the receiver, the classical ZF detector estimates the transmitted vector as where is the pseudoinverse of the channel matrix. The estimator is an unbiased estimator with covariance matrix [11]

The estimator is called the ordinary least squares (OLS) estimator [12]. If the noise is uncorrelated, that is, , the OLS estimator corresponds to the best linear unbiased estimator (BLUE) [12]. However, when the noise is correlated there is no guarantee that the OLS estimator is the BLUE.

##### 2.2. Generalized Least Squares Estimator with Known

Since the OLS estimator is no longer BLUE, the ZF receiver degrades the system BER. Using the Gauss-Markov theorem and assuming that is the full rank matrix, we can prove that the BLUE estimator of is given by [12] The GLS estimator is unbiased with covariance matrix [12] We can easily show that (the matrix inequality means that is positive semidefinite) As can be seen in (7), the GLS estimator needs the knowledge of the noise covariance matrix . By combining (7) and (2), we can write the GLS estimator as which does not require the knowledge of .

#### 3. Performance Analysis

##### 3.1. Derivation of the Sum Rate

The analysis of the ergodic sum rate of the ZF receiver has been widely discussed [13, 14]. However, previous works have considered systems with uncorrelated noise. In this section, we introduce a novel upper and lower bounds on the ASR of the improved ZF detector. Assuming independent decoding at the receiver, the ASR is
where is the SNR of the th stream. Since the covariance matrix of (8) is the MSE of the data estimation, we have
where and . When is a complex Gaussian matrix, it is well known that follows the central complex Wishart distribution. However, it is generally hard to derive the distribution of . In this case, we will use the decomposition and the first moment of the generalized variance given in [13]
where is an arbitrary Hermitian positive definite matrix, are the eigenvalues of **,** and is defined by
Let us define .

Proposition 1. *The ASR of the GLS receiver in a correlated environment is upper bounded by
**
where is Euler’s digamma function, is the th diagonal entry of , is defined as
**
and is an matrix with elements
**
Furthermore, the ASR is lower bounded by
*

*Proof. *The proof is based on a similar approach as that in [13, Appendix III], more specifically, by assuming instead of .

##### 3.2. Normalized Mean Square Error

Let us define the conditioned normalized MSE (NMSE) at the ZF receiver output, given the channel matrix, as The NMSE is obtained by averaging (19) with respect to .

Proposition 2. *The NMSE at the GLS receiver output in a correlated environment is lower bounded by
*

*Proof. *Since trace and expectation commute, we can write
where is the th minor of and (22) is obtained from matrix properties, while (23) is obtained thanks to Jensen’s inequality since is a convex function. The proof follows by invoking a lemma form [15, Lemma 4] and a result from [13, equations (15), (46)] which give
Substituting (24) into (23), we obtain (20).

##### 3.3. Global Outage Probability

There are several definitions of the GOP which depend on the application of interest [16]. In this letter, we assume that the GOP is defined as the probability that all instantaneous SNRs fall below a fixed threshold . This means that the system is considered available if at least one of the substreams achieves the desired performance. This definition, called all-outage probability in [16], can be expressed as where .

Proposition 3. *The GOP of the GLS receiver in a correlated environment is upper bounded by
*

*Proof. *We start by writing since is a strictly monotonic increasing function and using the Markov’s inequality [17]
Then, we can write the following:
where (29) results from Jensen’s inequality since is concave function. The bound in (26) is obtained using (24).

#### 4. Improved ZF Receiver with Unknown

When is unknown at the receiver, we estimate the covariance matrix and use the estimator (the subscript refers to the frame length) to evaluate the feasible generalized least squares estimator (FGLS) [12] To obtain the FGLS estimator, we propose to use the following iterative process (with iterations).(1)Perform the OLS estimation .(2)Compute the residual vector for where is the hard estimate of obtained by mapping to the closest constellation point using the classical decision rule for the considered modulation.(3)Estimate by .(4)For any iteration , compute the FGLS (5)Update the residual error vector as .(6)Update the covariance matrix estimation (7)Iterate Steps (3), (4), (5), and (6).

When is replaced by , the Gauss-Markov theorem no longer holds [12]. This means that the estimator is not necessarily BLUE. In general, it is difficult to evaluate the statistical properties of . Monte Carlo simulations can be used to check the accuracy of the FGLS estimator.

#### 5. Numerical Results

In this section, Monte Carlo simulations are employed to compare the OLS, GLS, and FGLS estimators and support our derivation of the bounds. We consider a MIMO system with frame size . All simulation curves are obtained over independent MIMO channel realizations. The transmitted symbols are 16-QAM modulated. The correlation coefficients are fixed to .

In Figure 1, the simulated ASR for the three detection techniques are plotted. For the FGLS case, the plotted curve is obtained after the fifth iteration. For the GLS method, the ASR obtained by the simulation is also compared against the bounds of (15) and (18). It is shown that the GLS and the FGLS outperform the OLS receiver. For higher SNRs, the ASR obtained when is unknown is nearly the same with that when is known. Furthermore, we can see that the analytical bounds exhibit a negligible difference with the simulation curves. However, the lower bound is tighter across all SNRs.

Figure 2 compares the NMSE performance of the OLS, GLS, and FGLS approaches. It shows that the lower bound of (20) works well for dB. For the FGLS method, we have plotted the NMSE after each iteration. It can be seen that, from the first iteration to the fourth one, there is an important improvement in the NMSE. However, after the fourth iteration, the NMSE improvement is not noticeable. It is thus possible to shut down the iteration process after the fourth iteration.

The GOPs of the different receivers are plotted in Figure 3. The upper bound of (26) is also provided. The threshold is fixed in order to assure a target BER of . We can see that the upper bound of (26) is fairly tight and can well predict the GOP. At a GOP of , the GLS technique allows a significant gain of about 3 dB with respect to OLS estimator. The GOP of the FGLS receiver after the fifth iteration is close to that of the GLS.

#### 6. Conclusion

A ZF detector is analyzed in the presence of correlated noise and doubly correlated channel. When the noise covariance matrix is known, the GLS algorithm allows a BLUE estimation of the transmitted sequence. In this case, we provide upper and lower bounds of the ASR, a lower bound of the NMSE, and an upper bound of the GOP. When the noise covariance matrix is unknown, we propose to use the iterative FGLS algorithm. Computer simulations have demonstrated the efficiency of the proposed FGLS detector which leads to a substantial improvement in the system performance. After four iterations, the FGLS performances approach those of the GLS. We have validated the derived analytic bounds by computer simulations.

#### Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.