Abstract
During the March 2011 meeting of the CTIA MIMO OTA Subgroup (MOSG), the members agreed that the subgroup should first determine “what” aspects of a MIMO-capable device require evaluation; then the group should determine “how” to go about making these measurements. In subsequent meetings of MOSG, new yet-unnamed figures of merit were asked for in order to provide a solution to the carriers' requirements for LTE MIMO OTA evaluation. Furthermore, the December 2011 3GPP RAN4 status report on LTE MIMO OTA listed the evaluation of the use of statistical performance analysis in order to minimize test time and help ensure accurate performance assessment as an open issue. This contribution addresses these petitions by providing four new figures of merit, which could serve the purpose of evaluating the operators' top priorities for MIMO OTA compliance testing. The new figures of merit are MIMO Throughput Effectiveness (MTE), MIMO Device Throughput Effectiveness (MDTE), MIMO Throughput Gain (MTG), and MIMO Device Throughput Gain (MDTG). In this paper, MTE is evaluated using the recently available LTE MIMO OTA RR data from 3GPP.
1. Introduction
Two levels of Multiple Input Multiple Output (MIMO) testing have been established: Basic (verification of MIMO Over-The-Air [OTA] performance in an isotropic environment) and Advanced (Evaluation of MIMO OTA performance in a spatial test environment). In March, 2010, the MACSG (anechoic chamber subgroup) and RCSG (reverberation chamber subgroup) chairs held a conference call with several United States (US) operators to review Third Generation Partnership project (3GPP) Radio Access Network 4 (RAN4)’s OTA proposals at a high level and request independent input concerning each operator’s MIMO test priorities. Operators were asked to rank their MIMO OTA test priorities in descending order, with a limit of six priorities. For all intents and purposes, the top priority of all carriers was to determine whether or not a MIMO Device Under Test (DUT) actually provides gain over a comparable Single Input Single Output (SISO) device. In second place operators wanted to look for information concerning the device’s transmitting radiated performance, particularly at strong signal level compared to SISO.
The task of identifying new Figures of Merit (FoM) was then open as some of the traditional definitions of Single Input Multiple Output (SIMO) FoMs have been reported to be not suitable for characterizing the mobile terminal’s MIMO antenna radiation performance. In a recent communication [1], CTIA (The Wireless Association of US-based Industries MIMO OTA subgroup (MOSG)) proposed to search for a yet-unnamed metric that provides a ratio between the DUT’s radiated and conducted throughput performance in each test condition. This metric requires that the test solution provider has to provide a ratio and show traceability between the DUT’s radiated SIMO and radiated MIMO throughput performance tests. Similarly, the December 2011 3GPP RAN4 status report on Long-Term Evolution (LTE) MIMO OTA listed the evaluation of the use of statistical performance analysis in order to minimize test time and help ensure accurate performance assessment as an open issue.
In this contribution, the authors introduce novel figures of merit with potential for determining these aspects above in a simple and cost-effective way within the Basic MIMO Testing level using the commonly agreed throughput figure of merit and some statistical analyses. As preliminary examples of the implementation of these FoMs, one of the new FoMs (MIMO Throughput Effectiveness or MTE) is evaluated using the recently available LTE MIMO OTA RR data from 3GPP.
Results show that the new figure serves its purpose to clearly distinguished good from bad MIMO device in a simple and cost-effective way. For Reverberation Chamber-(RC-) based and Anechoic Chamber-(AC-) based methods, the new metric results also show a very low standard deviation between labs (below 0.1). Results for 2-stage methodology were not fully available for this study and will be studied in future contributions if provided. In addition, results also show that intermediate MTE values (MTE()) can provide a clear indication of the ability (or disability) of the MIMO device to maintain throughput when under a variety of situations (under stress) and therefore also serve the different goals of such complex MIMO OTA performance assessment. Finally, results demonstrate that the reverberation chamber technique is a solid methodology for LTE MIMO OTA Basic evaluation, and that standard deviations between labs when using the new evaluated FoM (MTE) fall below 0.08. RC-based methods are very fast in comparison to other existing alternatives. Since test time has been proposed to be added as a metric for LTE MIMO OTA compliance testing [2], RC-based methods represent a promising test methodology for LTE MIMO OTA compliance testing.
2. Proposal
The CTIA MIMO OTA test prerequisites outlined in MOSG110707R2 are assumed. MIMO radiated performance can be assessed in terms of(A)measured throughput relative to the theoretical MIMO maximum throughput for a given reference channel with no radio channel impairments,(B) measured throughput relative to the DUT-measured MIMO maximum throughput for a given reference channel with no radio channel impairments,(C) measured throughput relative to the theoretical SIMO maximum throughput for a given reference channel with no radio channel impairments,(D)measured throughput relative to the DUT-measured SIMO maximum throughput for a given reference channel with no radio channel impairments.
Options (A) and (C) allow for inter-DUT analyses, while options (B) and (D) serve intra-DUT performance evaluation. For example, according to 3GPP TS 36.101, Annex A.3.3.2.1-1, the maximum theoretical throughput for Reference Measurement Channel (RMC) R.11 (10 MHz Frequency Division Duplex [FDD], 16 Quadrature Amplitude Modulation [QAM]) is 11,664 kbps per stream, for a total throughput of 23,328 kbps. This throughput is based on a nonfading channel with no multipath and a receiver correlation coefficient of zero. Measurement of the DUT under ideal conducted conditions allows the CTIA-Authorized Test Lab (CATL) to ensure that the DUT and the test equipment are performing as expected prior to beginning actual OTA testing.
Based on the prerequisites above, we propose that the following steps be followed to determine the new metrics, which could be integrated within the Basic MIMO OTA performance evaluation process.(1)Select at least 4 different RMCs (i.e., different reference measurement channels) based on the Channel Quality Indicator (CQI), Rank Indicator (RI), and Precoding Matrix Indicator (PMI) reported by the User equipment (UE). The RMC used shall be the RMC for sustained downlink data rate per section A.3.9 of 3GPP TS 36.521-1 and based on the UE’s device category.(2)Measure the DUT’s maximum MIMO Downlink (DL) throughput using a base station simulator connected directly to the DUT with no radio channel impairments, at a downlink power level (RS_EPRE) of −80 dBm/15 kHz using 2-layer, closed-loop spatial multiplexing for the selected RMCs in step 2. The measured MIMO throughputs at this downlink power level shall be considered the MIMO maximum throughputs for this DUT and eNodeB simulator under unimpaired radio conditions for each selected RMC and shall be >97% of the theoretical throughput specified by 3GPP TS 36.521-1, A.3.9.(3)Measure the DUT’s maximum SIMO DL throughput using a base station simulator connected directly to the DUT with no radio channel impairments, at a downlink power level RS_EPRE of −80 dBm/15 kHz (using single-layer transmit diversity). The measured SIMO throughput at this downlink power level shall be considered the SIMO maximum for this DUT and eNodeB simulator under unimpaired radio conditions.(4)Place the DUT into a test chamber capable of emulating an isotropic Rayleigh-fading environment (NIST channel model). It is generally agreed that the isotropic environment does give kind of a “global view” on the device’s performance, and it is therefore the best a priori selection for the Basic MIMO OTA tests. For Single Antenna transmission mode, measure the MIMO DL throughput using a base station simulator connected to the test chamber while emulating the National Institute of Standards and Technology (NIST) channel model and an RMC appropriate to the DUT’s capabilities. The average downlink power applied to the DUT has to be set from −85 dBm/15 kHz (RS_EPRE) down to −120 dBm/15 kHz in 1 dBm/KHz steps. The nature of the noise profile (e.g., Additive White Gaussian Noise [AWGN] or other) is a topic For Future Study [FFS].(5)Evaluate the channel-model averaged MIMO Throughput Effectiveness (MTE), MIMO Device Throughput Effectiveness (MDTE), MIMO Throughput Gain (MTG), and MIMO Device Throughput Gain (MDTG) for each averaged channel power step value by where indicates the number of selected RMCs, indicates a dependence on channel power, indicates the th selected channel model for performance evaluation, indicates the th selected Reference Measured Channel, TPUTMIMOtheor_max indicates the MIMO maximum theoretical throughput, TPUTMIMOmeas_max indicates the MIMO maximum DUT-measured throughput for this DUT and eNodeB simulator under unimpaired radio conditions as defined in step 2, TPUTSIMOtheor_max indicates the SIMO maximum theoretical throughput, TPUTSIMOmeas_max indicates the SIMO maximum DUT-measured throughput for this DUT and eNodeB simulator under unimpaired radio conditions as defined in step 3, and indicates the number of selected Reference Measured Channels for MTE/MDTE/MTG/MDTG performance evaluation.(6)Repeat the DL throughput measurements in steps 4 to 5 using the channel models (i.e., the delay and power profiles only) below:(a)Spatial Channel Model (SCM) Urban Micro (UMi) channel model,(b)SCM Urban Macro (UMa) channel model.(7)Repeat the DL throughput measurements in steps 4 to 6 using the total of transmission modes below:(a)open-loop Spatial Multiplexing (SM),(b)closed-loop SM,(c)closed-loop SM with single layer. This makes a total number of tested transmission modes . Other modes can of course be tested and added to the proposal in the future.(8)Evaluate the transmission mode-averaged MTE/MDTE/MTG/MDTG for each averaged channel power step value by The obtained figure can be compared between DUTs, where an would indicate radiated performance identical to MIMO maximum theoretical throughput performance. In this way, an intermediate metric for evaluation of performance and comparison between DUTs due to transmission mode with respect to a comparable MIMO is obtained.
The obtained figure can be used to compare the relative device’s behavior for different transmission modes with respect to its optimum MIMO performance, where an would indicate radiated performance identical to the device’s maximum MIMO throughput performance.
The obtained figure can be compared between DUTs, where an would indicate radiated performance identical to SIMO maximum theoretical throughput performance. In this way an intermediate metric for evaluation of performance and comparison between DUTs due to transmission mode with respect to a comparable SIMO is obtained.
The obtained figure can be used to compare the relative device’s behavior for different transmission modes respect to its optimum SIMO performance, where an would indicate radiated performance identical to the device’s maximum SIMO throughput performance.
Intermediate values of MTE/MDTE/MTG/MDTG at high signal level ( dBm/15 kHz) can be calculated by The obtained MTE at −85 dBm/15 kHz figure can be compared between DUTs, where an MTE (−85 dBm/15 kHz) = 1 would indicate radiated performance identical to MIMO maximum theoretical throughput performance. In this way, an intermediate metric for evaluation of performance and comparison between DUTs in strong signal level with respect to a comparable MIMO is obtained.
The obtained MDTE at −85 dBm/15 kHz figure can be used to compare the relative device’s behavior for different transmission modes respect to its MIMO performance at high signal level, where an MDTE (−85 dBm/15 kHz) = 1 would indicate radiated performance identical to the device’s maximum MIMO throughput performance.
The obtained MTG at −85 dBm/15 kHz figure can be compared between DUTs, where an MTG (−85 dBm/15 kHz) = 1 would indicate radiated performance identical to SIMO maximum throughput performance. In this way, an intermediate metric for evaluation of performance and comparison between DUTs in strong signal level with respect to a comparable SIMO is obtained.
The obtained MDTG at −85 dBm/15 kHz figure can be used to compare the relative device’s behavior for different transmission modes with respect to its SIMO performance at high signal level, where an MTG (−85 dBm/15 kHz) = 1 would indicate radiated performance identical to the device’s maximum SIMO throughput performance.(9)Evaluate MTE/MDTE/MTG/MDTG values for the DUT by where indicates the number of selected transmission modes.
MTE, MDTE, MTG, and MDTG range from 0 to 1. All MTE/MDTE/MTG/MDTG figures and final values can be referred to as the DUTs MIMO Isotropic Performance (MIP). There is no minimum performance value for the MIP parameter at this time.
3. MTE Examples with 3GPP LTE MIMO OTA Round Robin Data
A large number of measured data has become readily available by 3GPP LTE MIMO OTA round robin (RR) campaign. The data has been divided into three main methodologies: reverberation chamber, anechoic chamber, and two-stage methods. The analyses of the measured data revealed some difficulties for performing in-depth studies and obtaining conclusions. First, not all labs were able to test all available DUTs. Similarly, not all labs tested the DUTs in the diverse Pools for all the channel models. Some labs did not reach 100% throughput values, while others did for the same DUTs and channel models. Different relative throughput values (%) have been specified for the same base station emulator (BSE) settings by different test labs. The measured RS_EPRE range varied a lot between labs for the same devices. Some labs identified different 100% throughput reference for R.11 and R.xxx specified in the test plan [3], which could be attributable to BSE but also to specific subframe settings in the BSE. The fact that the test plan has been dynamically changed throughout the campaign has not helped solving these issues. As a conclusion, the lack of a clearly defined test environment and test equipment aspects has made the resulting data have large uncertainties for a good comparison.
Yet, the large amount of data allows for some statistical studies, and this is the first time it can be performed for LTE MIMO OTA. The statistical analyses performed by EMITE in this document are limited to MTE using 16QAM and open-loop transmission mode as these were the test parameters specified in the 3GPP LTE MIMO OTA test plan. While some labs have performed 64QAM and close-loop transmission mode tests, these do not share other settings and are, therefore, not suitable for comparison. Similarly, previous preliminary analyses [4] have outlined the need to compare tests with identical settings, and the statistical tests by EMITE have been performed over those data sets where similar DUT configurations and settings of the eNodeB have been identified. Finally, as averaged received channel power (RS EPRE) ranging varied slightly between labs, some interpolation has been used to have all test data ranging from −85 to −120 dBm. In the obtained results for MTE analyses, final throughput values higher than the maximum theoretical values/negative were not allowed and set to the maximum theoretical values/0.
3.1. RC-Based Tests
For the RC methodology, tests from EMITE, Bluetest, Azimuth, NTT DCM, and Ericsson have been performed. All these labs have employed three different mode-stirred reverberation chamber types: the E-Series by EMITE, the RTS-Series by Bluetest, and the MIC model by Azimuth. In [4], the coordinator for the mode-stirred reverberation chamber method analyses selected the figures, which showed at least two different labs with identical settings. In consequence, we have studied MTE for these figures. MTE() for Pools 1 to 4 is depicted in Figures 1, 2, 3, and 4.
(a)
(b)
(c)
(a)
(b)
(a)
(b)
Figures 5 and 6 illustrate MTE and the deviations of labs from averaged values. The figures are not complete as there are some DUTs for which there were no available data for comparison (at least two labs with identical settings).
The maximum MTE standard deviation (MTE STD) from group average for the RC-methodology falls below 0.08. It is interesting to observe that the deviations are very small for Pool 3, where three different mode-stirred reverberation chamber types are compared. Pool 3 was tested at the end of the campaign, and at that time many initial issues and settings doubts were solved and the test plan was updated accordingly. The small STD indicates that the RC methodology is a powerful and stable methodology for LTE Basic MIMO OTA device performance evaluation.
Averaged MTE deviations from group average for all Pools and those labs with more than one tested case compared have also been obtained by
If we consider that an MTE ≥ 0.7 represents a good MIMO device, from the RC-method MTE results it can be concluded that Pool1 and Pool 2 devices could be considered good MIMO devices, although Pool 2 devices generally performed slightly worse than Pool1 devices. Pool 3 devices could also be considered good MIMO devices, but they are much closer to the MTE = 0.7 threshold outlined before than any device in Pool 1 or Pool 2. Pool 4 devices could be considered bad MIMO devices (MTE < 0.7). The largest deviation between labs is obtained for the bad MIMO devices.
3.2. AC-Based Tests
For the AC methodology, tests from SATIMO, Nokia, NTT DCM, and ETS-Spirent have been performed. MTE for Pools 1 to 4 is depicted in Figures 7, 8, 9, and 10.
(a)
(b)
(a)
(b)
Figures 11 and 12 illustrate MTE and the deviations of labs from averaged values. The figures are not complete as there are some DUTs for which there were no available data for comparison (at least two labs with identical settings).
The maximum MTE standard deviation (MTE STD) from group average for the AC-methodology falls below 0.06. Averaged MTE deviations from group average for all Pools and those labs with more than one tested case compared have also been obtained by
From the AC-method MTE results, it can be concluded that Pool 1 and Pool 2 devices are found to be good MIMO devices. The AC-method also ranks Pool 3 DUT 1 as a good MIMO device. Pool 4 devices can be considered bad MIMO devices. The largest deviation between labs is obtained for the bad MIMO devices.
3.3. 2-Stage-Based Tests
The data from 2-stage tests (2S) was not entirely available for comparison. While data from Agilent was supplied in excel format, the data from Nokia was not supplied to the 3GPP reflector. Only some comments in [5] could be used (and reproduced here) to predict some conclusions, which would obviously need to be confirmed by MTE analyses. In particular, the ability to more precisely differentiate Pool 3 devices from Pool 1 or Pool 2 and the ability to identify Pool 4 device as bad MIMO devices using the 2S method should be clarified. The following sentences could be further studied.
R4-116104: For Pool3Dev1 and Pool3Dev2, the test results difference is quite large between the two-stage results and the reverberation chamber results. What makes it hard to understand is why the reverberation-chamber-based method has much higher throughput as compared with the test results of the two-stage method.
This statement above in [5] could mean that Pool 3 devices tested with the 2S method would lead to smaller MTE (smaller throughputs for the same RS EPRE), confirming the worse behavior of Pool 3 devices in comparison to Pool 1 and Pool 2 devices outlined by RC-based tests.
R4-116104: For Pool 4 devices, the test results difference is larger and the reverberation-chamber-based method has much worse performance as compared with two-stage method. However, the test results between the multiple-probe-antenna-based method and the two-stage method under this case have much less difference (2-3 dB) as compared to the difference between the two-stage method results and the reverberation-chamber-based method, which is a little bit less than 10 dB.
This statement above at [5] could mean that Pool4 devices may not be identified as bad MIMO devices by the 2S method, but rather as good MIMO devices, unlike what happened for both AC and RC methods.
When the analysis of MTE for 2-stage-based test data sets is finished, the above comments and some other issues could be further clarified.
4. Conclusions
Novel throughput-based figures of merit have been presented to respond to CTIA MOSG query for a global evaluation metric. 3GPP/CTIA LTE MIMO OTA Round Robin data has been used for validation of MIMO Throughput Effectiveness (MTE). MTE has demonstrated a great potential for solving the operators’ top priorities regarding MIMO OTA compliance testing and performance evaluation. RC-based and AC-based methods have also shown to be able to differentiate good from bad MIMO devices using MTE. Given the short testing times of RC-based methods in comparison to other methods, the mode-stirred reverberation chamber methodology has proven to be a strong candidate for LTE MIMO OTA Basic evaluation with very low MTE deviation from group average between labs. Future studies include MTE for 2S test data sets and obtaining MDTE, MTG, and MDTG figures.
Acknowledgment
Only this work was supported by EMITE Ing.