#### Abstract

The area of cooperative communications has recently attracted lots of research interest because of the potential benefit of increased spatial diversity. In this paper, we study the detector design for cooperative communications in intersymbol interference (ISI) channels. A novel system framework employing nonorthogonal amplify-and-forward half-duplex relays through ISI channels is introduced. We focus on detector design and first consider an optimal detector that consists of a whitening filter and a maximum-likelihood sequence estimator (MLSE). However, such an optimal detector embedded with Viterbi algorithm has practical issues of high complexity if the relay period is long. Consequently, we adopt a multitrellis Viterbi algorithm (MVA) that reduces the complexity significantly but still achieves near-optimal performance. Simulation results demonstrate the performance of both the optimal and near-optimal detector designs.

#### 1. Introduction

Multipath fading is one of the major impairments to the performance of wideband wireless communication systems [1]. Historically, such fading has been combated by using time and frequency diversity techniques. In the last fifteen years, results in information theory have shown that spatial diversity can yield significant gains in the spectral efficiency and power efficiency of point-to-point multiple-antenna communication (MIMO) systems [2, 3]. The transition from theory to practice has largely taken place with MIMO technology, as many modern consumer wireless standards exploit MIMO technology. To realize such gains, however, it is necessary that each of the paths between transmit and receive antennas is uncorrelated. For such an assumption to be valid, it is typically required that the antenna elements are spaced at least a half carrier wavelength apart [4], and perhaps even more in environments with minimal scattering. In many scenarios, however, it may not be practical for size-constrained nodes to have even two antennas with sufficient spacing between them. Furthermore, for each antenna that is added to the node, a complete RF front end must be added. There may be cost and power constraints that preclude the inclusion of multiple antennas, as the RF portion of a communications system often accounts for the majority of the cost and power.

More recently, cooperative diversity [5, 6] and relay networks [7] have attracted a lot of attention for their ability to exploit increased spatial diversity available at distributed antennas on other nodes in the system. By intelligent cooperation among nodes in the network which may only have a single antenna, a virtual multiple antenna system can be formed. Indeed, information theoretic results demonstrate that some of the loss associated with using only a single antenna can be recuperated by using intelligent cooperation among distributed nodes [8, 9].

While communication via cooperative relays has seen a lot of active research interest in recent years, most of the existing work has largely come from the information theory and coding communities. While there are few exceptions, for example [10, 11], little research has yet been conducted into the implementation issues of relaying and cooperation. As such, the majority of works in the field of cooperative diversity assume that receivers employ optimal detectors.

In this paper, we set out to investigate detector design for half-duplex relays in frequency-selective fading channels encountered in practice. While a variety of forwarding protocols have been previously proposed, we will consider amplify-and-forward (AF) for its simplicity and reduced implementation costs. Frequency selective fading channels are an inevitable impairment in wideband communication systems [1], and such channels cause the receiver to observe the superposition of multiple delayed reflections of the transmitted signal, resulting in intersymbol interference (ISI). Even in channels which do not exhibit significant time dispersion, the nonorthogonal AF relay itself effectively introduces ISI since the destination observes a superposition of the source and relayed signals. As our focus is on the complexity of the detector itself, we do not treat the performance gains possible with relays as this has been demonstrated elsewhere [12]. Similarly, with our focus on the complexity of detector implementation, we do not address the problem of channel estimation and thus consider the optimistic case where the detector has perfect channel knowledge. After developing a system model for the case of AF relays in ISI channels, we will present an optimal detector realization based on the Viterbi algorithm (VA), and we will address its implementation complexity. Then we will show that the delay introduced by the half-duplex AF relay causes the effective channel impulse response to become sparse when the relay period is large. To exploit the inherent sparsity in the effective channel, a multitrellis Viterbi algorithm (MVA) [13] is adopted which results in much lower complexity but negligible performance loss. Finally, we conclude with numerical simulations demonstrating performance of the detector in slow standardized fading channels.

#### 2. Cooperative Communication System Model

A basic model of a three-node relay system model is shown in Figure 1. Both source and relay can be considered as mobile users, and each has only one antenna. The source is attempting to send a message to the destination. Due to the broadcast nature of wireless communications, however, the relay receives transmissions from the source that are intended for the destination; therefore, the relay can assist by forwarding additional copies of these transmissions to the destination. Since the channels from source and relay to destination are statistically independent, the three-node cooperative communication scheme effectively forms spatial diversity. We consider a system where a source transmits a continuous stream of data to a destination, and a simplistic AF relay assists the source by amplifying and forwarding the data to the destination. We do not assume the relay has performed any synchronization with the destination, and so the relay forwards information to the destination in an open-loop fashion.

Relays have mainly two types: full-duplex relays that can transmit and receive simultaneously and half-duplex relays that can either transmit or receive in any time slot. Since full duplex relays are difficult to implement due to self-interference which occurs when both transmit and receive operations are in the same band, half-duplex is considered more practical for cooperative communication systems. In our system model, the half-duplex relay period is a parameter which defines the frame structure where the relay receives for symbol periods and then transmits for symbol periods. The relay repeats these two tasks alternately. The source and relay are assumed to transmit on the same channel, employing the so-called nonorthogonal amplify-forward protocol (NAF) [12].

The source sends the symbols at a symbol rate of , where is the number of transmitted symbols. We assume that a squared root raised cosine (SRRC) filter with the impulse response is applied as the transmitter filter. The received signal is processed by a matched filter with the impulse response and then sampled at the symbol rate of . The equivalent discrete-time channel impluse responses [14] which include the effect of pulse shaping are denoted by , , and for the source-destination, source-relay, and relay-destination channels, respectively, and they have corresponding channel lengths , and (e.g., ). The signals and are complex additive white Gaussian noise (AWGN) at the relay and the destination with variances and , respectively.

The destination receives the superposition of the two signals from the source and the relay, and the received signal can be expressed as where is the contribution from the source and is the contribution from the relay.

We first consider the source-destination link. The contribution from source to destination is written as where is the complex Tœplitz channel convolution matrix whose entries are defined by where and , that is,For the source-relay-destination link, the corresponding contribution is given by The Tœplitz channel matrices and are defined in the same way as , is the signal received by the relay, is the signal transmitted from the relay, and is a fixed matrix described below. Note that for the matrix dimensions to be compatible, we require that ; if this is not satisfied, we can append zeros to the appropriate matrix without loss of generality. The function of is to impose the half-duplex constraint by selecting groups of symbols from (receiving), scaling these symbols by a factor (amplifying), and then delaying the scaled symbols of for transmission in the next symbol block (forwarding). The value of is typically chosen to satisfy an average power constraint at the relay by choosing , where and are the source power and relay power, respectively. The constant matrix is given by where denotes Kronecker product. Here we implicitly require that be divisible by . As an example, when , the signals received and transmitted by the relay are shown in Figure 2, where the first eight time periods are considered, and the shadow indicates time period where the relay cannot receive because it is transmitting, or vice versa. For this example, such that as in Figure 2.

From (1), (2), and (5), the received signal at the destination is expressed as where .Note that is colored, not white, since the AWGN on the source-relay link is amplified-and-forwarded over the relay-destination ISI channel which colors the noise. Additionally, from (6) and (7), we see that the relay matrix has a repetitive structure with a period of . Accordingly, the channel matrix shows the same structure as . Consequently, can be interpreted as a periodically time-varying FIR channel which consists of sets of different channel coefficients. In summary, (8) allows us to describe the input-output behavior of the system with a linear equation. While the constituent channels themselves are not time-varying, the effective impulse response of the overall system is indeed time-varying due to the on/off behavior of the relay.

#### 3. Maximum-Likelihood Detector

Assuming that receivers can acquire perfect channel knowledge, maximum likelihood sequence estimation (MLSE) can be employed to combat the ISI by searching for the minimum Euclidean distance between observed signal and any given transmitted signals [15]. The Viterbi algorithm [16] is an efficient technique for solving the minimum distance problem, and its implementation has been investigated extensively [17–21]. The traditional Viterbi algorithm as proposed in [16] is directly applicable only to time-invariant channels. A modified Viterbi detector is proposed to address the periodically time-varying *effective* channel induced by the half-duplex relay. Furthermore, since the minimum Euclidean distance is not optimal in the presence of colored Gaussian noise, we employ a whitening filter before detection, which is optimal as shown in [15]. The block diagram of our design is given in Figure 3.

To whiten the noise, spectral factorization of the composite noise covariance must be performed. We factor the noise covariance matrix as. which can be accomplished by taking to be the Cholesky factorization of the covariance. We note that the Cholesky factorization is not the only such factorization , as the factorization in (9) is not unique. By filtering the received signal with (i.e., by forming ), the noise becomes whitened since the covariance of the filtered noise is given by We note that noise covariance matrix in (9) is positive definite, so the inverse of always exists. Ignoring end effects (or, equivalently, taking the block length ), follows the same repetitive structure as and thus also exhibits the periodically time-varying property.

After applying the whitening filter to the received signal (8), the whitening filter output becomes
where is now *white* Gaussian noise. Note that the effective whitened channel matrix maintains the periodically time-varying property due to the similar structures of and . The structure of the effective channel matrix is given aswhere we see that the matrix has a block Tœplitz structure with rows repeating every multiple of . It defines sets of effective channel coefficients as , where , , and is the effective channel length. The effective length may be significantly extended by the delay introduced by the relay, as well as the group delay introduced by the whitening filter. We can determine the lower bound of the effective channel length in terms of the constituent channel lengths and the relay period as
At time , the corresponding coefficients of the periodically time-varying effective channel are
where is the modulus. The ideal output of the whitening filter at time is then given by
which is simply the convolution of the source symbols with the periodically time-varying channel coefficients. Thus, the system model for relay-aided transmission through ISI channels reduces to a classical MLSE problem, with the additional twist that the effective channel is periodically time-varying. As the true output is of course corrupted by AWGN, the maximum-likelihood detector for estimating the source symbols from (11) can be accomplished most efficiently with the Viterbi algorithm.

For the branch metric unit (BMU) in Figure 3, the branch metrics along the trellis path are not only related to state transitions but also the current time instant. The branch metric calculation is modified as where is the signal from the whitening filter at the time instant , and are the estimated output signal and the corresponding branch metric the time instant , respectively.

The add-compare-select (ACS) unit in Figure 3 recursively computes path metrics and decision bits, where denotes the path metric at state at the time instant , and is the branch metric from state to state at instant . The path metrics for each state are updated for the next iteration, and the decision indicating the survivor path for state is recorded and retrieved from the survivor-path memory unit (SMU) in order to estimate the transmitted symbols along the final survivor path.

Similar to the traditional MLSE, the implementation cost of the ML detector for relay networks increases exponentially with respect to the effective channel length. Furthermore, the overhead of the proposed detector when compared with the traditional MLSE comprises the whitening filter, the multiplexers in the BMU, and additional control logic to account for the periodically time-varying effective channel [22]. As indicated in (13), the effective channel length increases with the relay period . In cooperative relay systems, the relay period is likely chosen to be very long, possibly spanning hundreds of symbols, so that the relay is not required to switch frequently between transmit and receive modes. When the relay period is large, however, an implementation of the Viterbi algorithm-based optimal detector becomes not practical. This problem will be addressed in Section 4.

#### 4. Near-Optimal Detector for Long Relay Periods

Consider the practical effect of the relay period in a simplified relay network with only AWGN channels and no ISI so that . When the relay is in the receiving period, the destination only receives the signal from the source. The channel impulse response at the destination is shown in Figure 4(a). After time instants, the relay begins to forward the copy of the signal to the destination. The effective channel impulse response as seen at the destination during relay transmission is depicted in Figure 4(b). We observe that the relay actually introduces ISI even in the non-ISI channels, and the effective channel impulse response alternates between that shown in Figures 4(a) and 4(b). Also, large values of will increase the number of zero coefficients in the channel impulse response, which make the effective channels become sparse.

**(a) During relay reception**

**(b) During relay transmission**

Viterbi algorithms for sparse channels have been investigated by several independent researchers. The parallel trellis Viterbi algorithm (PTVA) proposed in [23] reformulated the original single trellis into a set of independent trellises. These independent trellises operate in parallel and have less overall complexity than a single trellis. The PTVA requires that the sparse channel have equispaced coefficients, however, which usually cannot be satisfied in practice. Although a generalized PTVA is given to deal with general sparse channels, its performance loss is remarkable if the channel taps are not well-approximately by an equispaced structure.

A multitrellis Viterbi algorithm (MVA) was proposed in [13] for near-optimal detection in sparse ISI channels. For the MVA, the complexity does not depend on the channel impulse response length but only on the number of nonzero coefficients. In order to process the sparse time-varying channels for relay networks, the MVA is modified and incorporated in our MVA-based ML detector.

To illustrate the operation of the MVA, we begin by considering an example. Assume has only a few nonzero coefficients, for example, for , . The ideal (noiseless) output signal at the time is given by

When is to be estimated, we can see that is needed in , , . Next, in , a new symbol appears, and it is also needed in and . In this way, we record all the output signals and symbols related with in Table 1, and we use the notation of to indicate the dependency of outputs on input symbols and channels. Note that some output signals and symbols are not needed when is under detection; for example, if , there is no need to record , since , , in do not affect the estimation of . With the traceback length , for example, the estimation of depends on , , , , by a noninstantaneous relationship, assuming that , , are known.

We note that some related symbols appear only once in the first column, for example, , and do not need to be recorded. Its value can be determined by an instant decision given by where , and and are known for a given state.

When two or more symbols are determined by the instant decision, for example, and in , the estimation is given by where , and is known for a given state.

The definition of state depends only on the related time instant. By the list of related symbols, the state definition is derived and given in Table 2. Note that the state definition excludes the symbols assumed to be known, that is, , , , and the symbols that can be determined by the instant decision.

From Tables 1 and 2, it is observed that the corresponding trellis shrinks in two dimensions, which leads to a significant reduction in computational complexity. Furthermore, the process of traceback is faster, since, for some instant given the current state, the previous state can be obtained immediately without survivor path decisions. There are two categories for these instant tracebacks. First, the previous state definition is a subset of the current state definition. For example, the state at instant is defined as , while the previous state at the instant can be obtained from the current state without the help of the survivor path decision. Second, the state definition is the same for the current and previous state. For example, considering the instants , , , and , we can bypass the traceback from to . Once the start state at the instant is available, we can begin to traceback from the instant at the same state.

When subsequent symbols are under detection, the structure of the trellis remains the same, except that the branch metric calculation for the first several instants are slightly different, since the initial symbols (e.g., , , , for ) have been estimated. The available estimated symbols will be used in the calculation of output signal when needed.

Due to the reduced-size trellis, the detector can be realized by utilizing trellises working in parallel to increase the throughput. The received signals are filled in the trellises sequentially. At the instant , the first trellis is full, and is estimated. At the instant , the received signal is ready to fill in the first trellis, and, also, is available from the second trellis. Notice that these trellis are similar in structure; however, the channel coefficients used in the branch metric calculation are not the same at different instants. For example, in the first trellis, are used in sequence for each step, while in the second trellis, are used for each step. The corresponding channel coefficients for the received signals are .

For general Viterbi detectors in -ary modulation systems, complex multiplications dominate the computational cost. There are states, and each state corresponds to complex multiplications for branch metric calculation. Then the estimated total computational cost is . For the proposed MVA detector in the sparse channel with 3 nonzero coefficients, there are states at most, and the corresponding computational cost for each trellis is . Thus the estimated total computational cost for MVA detector is . Furthermore, survivor path decisions are recorded in the SMU which requires a significant amount of memory. The memory cost in bits for the general Viterbi detector is . For the proposed MVA detector, we do not need to record the survivor path decisions for all instants. Assuming that only instants are considered in each trellis, the memory size for each trellis is .

In general, the complexity of Viterbi detector is in , where is the channel length. In comparison, the complexity of the MVA detector does not depend on the channel length but on the number of nonzero coefficients. When instantaneous decisions are made appropriately so that the symbol dependency table simplifies [13], the computing and memory resource required for the MVA detector is in , where is the number of nonzero coefficients. Therefore, the MVA detector is a better solution for sparse channels.

#### 5. Simulation Results

We simulate the bit error rate (BER) performance of the proposed ML detector with the following parameters: the transmitted BPSK signal consists of i.i.d unit-power symbols , the relay transmitting power is . The SNR values and denote bit energy-to-noise ratio for the destination and the relay, respectively. Unless otherwise specified, we assumed , which represents a scenario where the source-relay link is better, on average, than the source-destination link.

We assume the SRRC pulse shaping filter is employed at both the transmitting and receiving ends. The SRRC filter is truncated to with a roll-off factor 0.5, where MHz is the symbol rate. We simulate over 200 fading realizations in the ITU-R 3G indoor office test environment [24] with 6 independent channel paths. The time delay relative to the first path is , and the average power relative to the strongest path is . As a practical matter, it is possible for the whitening filter to be quite long, depending on the noise covariance matrix. In these simulations, we truncate the whitening filter to have taps.

We consider the effect of the number of independent paths on system performance. While additional ISI is often viewed as an impairment to reliable communication, the additional fading paths result in increased diversity, and hence increased BER performance through the cooperative relay. This effect is observed in Figure 5, where we consider three cases. Specifically, we truncate the ITU-R 3G indoor office channel so that it only uses either the first two paths, the first three paths, or all the six paths. We note that the BER performance with respect to improves as the number of independent paths increases.

We next consider the effect of the relay cooperation period . Recall that the relay receives symbols and then amplifies and retransmits those symbols. As shown in Figure 6 for an uncoded system, better performance is obtained when choosing a smaller relay period. If the relay period is much larger than the channel delay spread, the diversity technique of using cooperative relays becomes less effective since there is almost no overlap between the symbols directly received from the source and forwarded by the relay.

The effect of the whitening filter is illustrated in Figure 7, where . As we expect, the BER performance with the whitening filter is better than the performance without it. It is also observed that if the noise on the source-relay link is small relative to the noise on the source-destination link so that , the whitening filter does little to help. The reason for this is that the noise looks approximately white when the noise on the source-destination link dominates since . Thus, in situations where the source-relay link is particularly good, it may be possible to reduce complexity without much performance penalty by removing the whitening filter.

To simulate the BER performance of the MVA-based detector, we set the channel length , so that each effective channel has 3 nonzero coefficients, and Table 1 can be applied directly. Without loss of generality, we assume that each coefficient is i.i.d. as . The performance is given in Figure 8(a) with and Figure 8(b) with . It is shown that performance loss from the MVA-based detector is negligible, as was claimed previously in [13].

(a) |

(b) |

#### 6. Conclusion

A system model is introduced for cooperative communication in ISI channels with amplify-and-forward relays. Based on the system model, we present a maximum-likelihood detector design based on the Viterbi algorithm. For long relay periods, the equivalent channel was shown to become sparse. Moreover, the channel length becomes much larger which makes the ML detector hardly practical. Therefore, we proposed use of the MVA algorithm in which the complexity is determined by the number of nonzero taps in the effective channel impulse response. The simulation results show that the MVA algorithm provides near-optimal BER performance but at significantly lower hardware complexity.