#### Abstract

A fundamental performance trade-off of multicell multiuser multiple-input multiple-output (MU-MIMO) systems is explored for achieving intercell and intracell interference-free conditions. In particular, we analyze the three-dimensional diversity-multiplexing-nulling trade-off (DMNT) among the diversity order (i.e., the slope of the error performance curve), multiplexing order (i.e., the number of users that are simultaneously served by MU-MIMO), and nulling order (i.e., the number of users with zero interference in a victim cell). This trade-off quantifies the performance of MU-MIMO in terms of its diversity and multiplexing order, while nulling the intercell interference toward the victim cell in the neighbor. First, we design a precoding matrix to mitigate both intercell and intracell interference for a linear precoding-based MU-MIMO system. Then, the trade-off relationship is obtained by analyzing the distribution of the signal-to-noise ratio (SNR) at the user terminals. Furthermore, we demonstrate how DMNT can be applied to estimate the long-term throughput for each mobile station, which allows for determining the optimal number of multiplexing order and throughput loss due to the interference nulling.

#### 1. Introduction

Multiuser MIMO (MU-MIMO) scheme, allowing a base station (BS) to communicate with multiple users simultaneously, provides an opportunity to boost the sum capacity through precoding, even when each user has only one antenna. For example, zero-forcing transmit beamforming (ZFBF) is one of the practical multiuser transmission strategies for MU-MIMO systems [1]. By designing one user’s beamforming vector to be orthogonal to other selected users’ channel vectors, ZFBF can completely eliminate the multiuser interference corresponding to intracell interference in cellular systems. Furthermore, using more transmit antenna can increase the number of users simultaneously served by MU-MIMO or enhance the error performance of each link between the BS and user.

Despite the theoretical attractiveness, the capacity gain promised by MIMO techniques has been shown to degrade severely in a multicell environment. To suppress the intercell interference, the authors in [2–5] investigated a coordinated beamforming scheme using multiple antennas at the BS. The achievable rate region of the MISO interference channel, in the case where the full channel information is shared among BSs, was derived in [2, 3], with instantaneous and statistical CSI, respectively. Distributed beamforming with a virtual SINR framework was proposed in [4]. The theoretical results in [2–4], however, are limited to only one user in the victim cell. The authors in [5] assumed that the interference experienced by multiple users in the victim cells is suppressed. Some studies on the interference mitigation in the cooperative beamforming for multiuser systems have been studied from the perspective of scheduling issues [6–9]. A low-complexity random beamforming method, which only requires sharing of user indices, has been suggested with analytic throughput expressions [6]. In [7], the authors provided a transmission beamforming scheme for interference nulling with user selection. Also, the reduced complexity algorithms for joint user selection in adaptive coordination scheme were designed in [8]. However, the unrealistic special homogeneous case, in which all users have the same average SNR, is assumed in [8]. In [9], we have considered a generalized intercell interference coordination problem and proposed a two-step coordination procedure to choose a cell-edge user and decide the coordination. However, the research in [6–9] was not extended to MU-MIMO, that is, only dealing with the multiple users in the serving cell.

In this paper, we analyze the three-dimensional DMNT among the diversity order (i.e., slope of error performance curve), multiplexing order (i.e., the number of users simultaneously served by MU-MIMO scheme), and nulling order (i.e., the number of other cell users subject to zero interference in a victim cell), while providing the victim cell with intercell interference nulling. We consider an interference-free environment in which the BS in each cell employs a precoding matrix with antennas, so as to null the intracell interference while mitigating the intercell interference. It is assumed that all users are equipped with a single antenna. Our contribution is to reveal the fundamental property of performance trade-off, given by , in multicell MU-MIMO subject to intercell and intracell interference-free conditions. To the best of our knowledge, there has never been any rigorous justification for this particular property in previous works. Note that the current diversity-multiplexing-nulling trade-off (DMNT) is quite different from the well-known diversity-multiplexing trade-off (DMT) that deals with the multiple antenna gain to be achieved simultaneously by any coding scheme (e.g., space-time coding) in the point-to-point MIMO system [10]. Meanwhile, we demonstrate that our DMNT can be applicable to estimating the long-term user throughput, which allows for determining the optimal multiplexing order and throughput loss due to interference nulling.

The rest of this paper is organized as follows. We present some preliminaries for our analysis in Section 2. Section 3 presents a system model and the precoding matrix design under consideration. Our analysis results for the diversity-multiplexing-nulling trade-off are given in Section 4. Section 5 demonstrates how DMNT can be applied to estimate the long-term throughput for each MS. Finally, concluding remarks are given in Section 6.

#### 2. Preliminaries

We first introduce some results from previous works, which are useful for our analysis.

*Definition 1. *Let be independent complex Gaussian random vectors with zero mean vector (i.e., ) and identity covariance matrix (i.e., ). If , where is the matrix, then is said to have a* complex Wishart distribution* with degrees of freedom [11], that is, .

By using vectors, we may define the matrix as The probability density function of for is given aswhere . If , that is, is defined as a random variable, then it has a Chi-squared distribution with degrees of freedom, that is, . This result immediately follows by substituting into the probability density function (pdf) of the Wishart distribution.

Lemma 2. *If and is partitioned aswhere is and the Schur complement of block is also a Wishart matrix with a distribution of . In [12], the Schur complement of block is given as *

*Proof. *See proof of Theorem in [13].

#### 3. Signal Model and Precoding Matrix Design

We consider MU-MIMO downlink systems in which the BS serves a set of selected mobile stations (MSs) simultaneously in a serving cell, while imparting interference to the MSs in victim cells, as illustrated in Figure 1. We assume that there are and MSs in the serving cell and victim cells, respectively. Let be a subset of indices for users that are intended for transmission by the BS . The user set is dynamically selected by a scheduler in the BS. At the serving cell, we design a wireless link equipped with transmit antennas at the BS and a single receive antenna at each MS. The MSs in the victim cells also employ a single receive antenna and do not perform any type of interference mitigation. Let us denote and as complex Gaussian channel vector and beamforming vector for the th MS, respectively. For a subset , we define and . The received signal at the th MS in the subset is represented bywhere and are the data symbol and the Additive White Gaussian Noise (AWGN) with variance of , respectively. We impose average power constraints, and . The received signals in (5) are rewritten by the aggregated received signal vector aswhere and . Let denote an channel vector from the serving BS to the th MS in the victim cell. Note that MSs in the victim cells are those subject to intercell interference. In our current system model in Figure 1, MSs in the virtual victim cell can be considered as those multiplexed with MSs in the serving cell while satisfying the intercell interference-free condition. For the MSs in the victim cell, the aggregated received signal vector is given bywhere , which can be known to the serving BS by a sounding signal [14]. Note that a desired signal of the victim MS is not represented in (7); that is, is just an intercell interference vector for the victim MSs, which would be controlled by the BS in the serving cell.

Our proposed precoding matrix design focuses on achieving the interference-free communication for both interferences from the other cell and other user signal from the serving BS. First, the intercell interference-free condition is given aswhere denotes an column vector with all-zero elements. Our objective is to rigorously show how much the spatial degrees of freedom are lost in this situation. Note that the intercell interference-free condition (8) leads to the following proposition:

Proposition 3. *To satisfy the intercell interference-free condition, should lie in the null space of , that is, the orthogonal complement of the subspace spanned by column vectors of .*

From Proposition 3, the precoding matrix can be a cascade of matrices, and , that is, , where eliminates the intercell interference by adopting a projection matrix onto the orthogonal complement (PMOC) of . If is a PMOC of , lies in the null space of , regardless of . Therefore, we may design two precoding matrices, and , independently, so as to meet each design constraint.

To produce a PMOC of , consider the following QR decomposition of** G**:where is an unitary matrix and is an upper triangular matrix. As the bottom rows of consist of entire zeroes, it is often useful to partition and as follows:where is an upper triangular matrix, is orthogonal matrix, is , and both and have orthogonal columns. Let denote a PMOC of . Then, we can obtain by using or as follows:Using and , alternative form can be represented as

Let us now design , whose purpose is to avoid interuser interference. Denoting for ZFBF, beamforming vectors are selected such that they satisfy the zero-interference condition for ; that is, the beamforming vector for user lies in the null space spanned by . One easy choice of the precoding matrix that gives zero interuser interference is the pseudoinverse; that is, The precoding matrix is formed by the unit-normalized columns of ; that is, . In matrix form, is given byFinally, the aggregated received signal vector is given bySince , the received signal is given as

#### 4. Analysis of Diversity-Multiplexing-Nulling Trade-Off (DMNT)

The effective SNR at the th MS is given by , where denotes the average SNR. As , we havewhere denotes an -diagonal element of . Using (17) and , we haveNow, defining , we haveMeanwhile, from and , which givesThe last step follows from the properties of the projection matrix; that is, and . Finally, we haveLet . Without loss of generality, we obtain the SNR for as follows:The last equality in (22) follows from Cramer’s rule [14] and the -element of is given bywhere is the matrix formed by deleting the th row and th column of . When is partitioned as from (23). Thus, (22) can be expressed asTo proceed, we use the following property:This in turn yieldswhere ; that is, is the Schur complement of block . We can show that the distribution of is given by the following lemma, which will be useful for analyzing the DMNT.

Lemma 4. * has a chi-squared distribution with degrees of freedom.*

*Proof. *See Appendix.

It is well-known that the diversity order (corresponding to a slope of an error performance curve) is when the SNR is distributed as a chi-squared distribution with degrees of freedom [15]. By Lemma 4, therefore, the diversity order of the interference-free MU-MIMO system under consideration is given by , which leads to the following result for the DMNT.

Theorem 5. *Let , , and denote the nulling order, multiplexing order, and diversity order for the downlink BS in the multicell MU-MIMO system. Assuming a BS equipped with antennas and MSs equipped with a single antenna, the trade-off among the nulling order, multiplexing order, and diversity order for the interference-free condition is given by*

#### 5. Application of DMNT: Asymptotic Throughput Analysis

In this section, DMNT in Theorem 5 can be applied to estimate the long-term throughput for each MS when an opportunistic ZFBF scheduling is applied. It deals with an asymptotic throughput analysis, from which the optimal number of multiplexing order and the throughput loss due to the interference nulling can be achieved. In fact, it will be a useful analytical framework that can predict how DMN must be traded off so as to maximize the cell throughput or to maintain a required level of cell throughput.

Let represent the effective SNR for the th user when is a set of the selected users that are transmitted at the same time, as defined in Section 3. The objective of the opportunistic scheduling is to determine a subset of users, , such that the sum rate is maximized; that is, where . Assuming (scalable to with the same principle), the long-term throughput is given bySince the channel vectors have independent and identical distribution, and thus (30) can be expressed asFurthermore, since , we haveAs , there will be multiple MSs (i.e., th MS and others) who have near orthogonal channel vector with respect to MS 1. Thus, we can assume that for sufficiently large , reducing (32) intoEquation (33) implies that, for large , the long-term throughput can be obtained by a single user scheduling problem in which each MS has a diversity order of , as dictated by Theorem 5 and therefore, it follows the chi-squared distribution with degrees of freedom. We note that the long-term throughput for the single user scheduling can be obtained by using the extreme value theory [16]. Based on the results in [16], the analytic throughput result is obtained as where is the CDF of in Lemma 4 and , which is the Euler constant [17].

In order to validate the accuracy of our asymptotic throughput analysis, we simulate the case with , , and , and uniformly distributed 30 MSs; that is, . Since all users have the different average SNR in the simulation, the different average throughput is observed for individual user as shown in Figure 2. We find that throughput measurement for each MS is acceptably close to the analytical result. Furthermore, the average cell throughput can be determined by the asymptotic throughput analysis for and , as shown in Figures 3 and 4. Figure 3 shows that there exists a multiplexing order to maximize the cell throughput for the given nulling order, demonstrating the optimal number of multiplexing order and the throughput loss due to the interference nulling. For example, when two other cell users are nulled (i.e., ), the optimal number of multiplexing gains is 7; that is, 7 active users must be served in the reference cell (). Furthermore, it is obvious from Figure 3 that the average cell throughput is reduced as increasing the nulling order. The current analysis implies that the multiplexing order should be adjusted adaptively for the different nulling order. Furthermore, it is observed that the cell throughput gained by reducing the interference-nulling effect subject to the same diversity gain becomes more conspicuous as the diversity order increases; that is, the multiplexing order decreases. Figure 4 shows the same observation for varying the nulling order for the given multiplexing order. Unlike Figure 3, there is no optimal operating point between and . Furthermore, it is observed that the multiplexing gain contributes to the cell throughput differently, depending on the nulling order. For example, spatial multiplexing is more important than diversity in improving the cell throughput if the nulling order is small. Otherwise, diversity gain is more contributing to the cell throughput than multiplexing gain. Meanwhile, the maximum nulling order that is required to maintain the minimum target cell throughput can be determined by the curves in Figure 4.

#### 6. Conclusion

In this paper, we have investigated a fundamental property of performance trade-off in the multicell multiuser multiple-input multiple-output (MU-MIMO) system when the intercell and intracell interference-free conditions must be satisfied. Assuming a BS equipped with antennas and MSs equipped with a single antenna, the trade-off among the nulling order (), multiplexing order (), and diversity order () for the interfere-free condition is given by . By characterizing the three-dimensional diversity-multiplexing-nulling trade-off (DMNT), our analysis provides a quantitative framework for dealing with the intercell interference coordination in a multicell MU-MIMO system. Finally, we have demonstrated that our DMNT result can be used to perform an asymptotic throughput analysis, which predicts how DMN must be traded off so as to maximize the cell throughput or to maintain a required level of cell throughput in the system. The overall system performance can be optimized by selecting a set of users and the precoders at the same time. As the joint optimization involves enormous complexity, joint optimization seems to be unrealistic in practice. It will be worth investigating the suboptimal approach that can be implemented in practice. However, user selection and codebook design subject to the limited feedback is beyond our scope in this paper. In other words, even if the overall system performance is governed by user selection and codebook design, along with channel estimation error, we aimed at demonstrating Diversity-Multiplexing-Nulling Trade-off (DMN) as an ideal performance characteristic, which would play a fundamental design guideline.

The interference-free environment (realized through the interference-nulling effect) may not be straightforward to achieve in practice. Since we assume that a full CSI is available at BS, an interesting venue for future work is to design a limited feedback system and, furthermore, to analyze the performance subject to channel estimation error. Whereas the current asymptotic analysis illustrates throughput of the reference cell only, we may need a multicell coordination framework in which all neighboring cells are coordinated to determine the optimal DMNT for the overall throughput maximization in practice.

#### Appendix

#### Proof of Lemma 4

Since , can be expressed asIn (A.1), can be eigen-decomposed as where is a unitary matrix and is the diagonal matrix whose diagonal elements are the corresponding eigenvalues. Using , (A.1) can be expressed asGiven the orthogonality of , is also an i.i.d. complex Gaussian channel matrix. Defining , we haveMeanwhile, consists of orthonormal vectors. In other words, eigenvalues of have a value of 1, while the others are zero. Without loss of generality, we assume that the diagonal entries are sorted in increasing order. In matrix form, is given byApplying this to (A.3), we haveFrom the definition of the Wishart distribution in (1), we conclude that has a Wishart distribution with degrees of freedom. Furthermore, we also find by Lemma 2 that in (27) has a Wishart distribution. Therefore, it is clear that . By Definition 1, therefore, is a chi-squared random variable with degrees of freedom.

#### Conflicts of Interest

The authors declare that there are no conflicts of interest regarding the publication of this article.

#### Acknowledgments

This work was supported in part by Institute for Information & Communications Technology Promotion (IITP) grant funded by the Korea government (MSIT) (no. 2014-0-00282, Development of 5G Mobile Communication Technologies for Hyperconnected smart services) and in part by the Brain Korea 21 Plus Project in 2017.