Edgeworth Expansion Based Model for the Convolutional Noise pdf

Abstract

Recently, the Edgeworth expansion up to order 4 was used to represent the convolutional noise probability density function (pdf) in the conditional expectation calculations where the source pdf was modeled with the maximum entropy density approximation technique. However, the applied Lagrange multipliers were not the appropriate ones for the chosen model for the convolutional noise pdf. In this paper we use the Edgeworth expansion up to order 4 and up to order 6 to model the convolutional noise pdf. We derive the appropriate Lagrange multipliers, thus obtaining new closed-form approximated expressions for the conditional expectation and mean square error (MSE) as a byproduct. Simulation results indicate hardly any equalization improvement with Edgeworth expansion up to order 4 when using optimal Lagrange multipliers over a nonoptimal set. In addition, there is no justification for using the Edgeworth expansion up to order 6 over the Edgeworth expansion up to order 4 for the 16QAM and easy channel case. However, Edgeworth expansion up to order 6 leads to improved equalization performance compared to the Edgeworth expansion up to order 4 for the 16QAM and hard channel case as well as for the case where the 64QAM is sent via an easy channel.

1. Introduction

In this work, we deal with the convolutional noise arising at the output from a blind deconvolutional process. A blind deconvolution process arises in many applications such as seismology, underwater acoustic, image restoration, and digital communication [1]. Consider the digital communication case. During transmission, a source signal undergoes a convolutive distortion between its symbols and the channel impulse response. This distortion is referred to as intersymbol interference (ISI) [2]. Thus, a blind adaptive filter is used to remove the convolutive effect of the system to produce the source signal [2]. This process is called blind deconvolution. Since the updated coefficients used in the blind adaptive filter are not the ideal values, a noise named convolutional noise occurs at the output of the deconvolution process in addition to the source signal. Blind deconvolution algorithms based on adaptive filtering techniques generate an estimate of the desired response by applying a nonlinear transformation to sequences involved in the adaptation process [1, 3]. The Bussgang algorithm is one of the three important families of blind equalization algorithms, where the nonlinearity is in the output of the adaptive equalization filter [1, 3]. According to [4, 5], we may find among the traditional Bussgang-type methods the Sato’s [6], Godard’s [7], Benveniste et al. [8], Besnveniste-Goursat’s [9], and the Stop-and-Go [10] algorithm. For Bussgang-type methods, the nonlinearity is designed to minimize a cost function based on high-order statistics (HOS) according to one approach [6, 7], or calculated directly according to the Bayes rules [11–15]. According to [1, 3], the main difference between the Bussgang type algorithms lies in the choice of the memoryless nonlinearity. Obviously, the performance of this kind of blind equalizer depends substantially on the memoryless nonlinearity [1]. Chapter eight in [5] deals with the question whether the chosen equalizer leads to perfect equalization performance from the MSE point of view. From the work in [1] we know that if the MSE tends to zero, then also the residual ISI tends to zero (perfect equalization). Furthermore, chapter eight in [5] distinguishes between the MSE obtained from the cost function approach and the MSE obtained from the Bayesian approach. According to chapter eight in [5], the MSE related to the Bayesian approach valid in the convergence state, where the convolutional noise is very small, is approximately given by the convolutional noise power (variance of the convolutional noise) multiplied by a constant. To show this outcome the approximated MSE obtained in [1] was recalled in this chapter [5]. Also the approximated MSE obtained in [16] could have been used for showing that the MSE related to the Bayesian approach valid in the convergence state, where the convolutional noise is very small, is approximately given by the convolutional noise power multiplied by a constant. The only difference between the approximated expression for the MSE obtained in [16] and that in [1] is the value of the constant multiplying the convolutional noise power. In [1] this constant is approximately equal to one regardless of the constellation input, while in [16] this constant is constellation input dependent and is approximately equal to two for the 16QAM input. This paper also adheres to the Bayesian approach where the approximated MSE is developed as a byproduct. We will show in this paper that the approximated MSE valid in the convergence state, where the convolutional noise is very small, is approximately given by the convolutional noise power multiplied by a constant. This constant is constellation input independent, and is approximately equal to one in addition to higher order statistics of the convolutional noise such as the kurtosis. For the case where the convolutional noise pdf is assumed to be Gaussian, our new approximated MSE tends to the expression obtained in [1].

Next we turn to the approximated MSE related to the cost function approach where the derivation of the cost function with respect to the equalized output signal is a polynomial function of the equalized output signal as is in Godard’s case [7], for example. For that case, the approximated MSE is approximately equal to a constant in addition to another constant multiplied with the convolutional noise power [5]. Since those constants are constellation input dependent, the approximated MSE may not tend to zero when the convolutional noise power tends to zero unlike in the Bayesian approach. For example, let us consider the 16QAM input and Godard’s [7] algorithm (cost function approach). The residual MSE for that case is not zero even when the convolutional noise power tends to zero while this is not the case for the Bayesian approach [1].

In the literature, we find many blind adaptive algorithms based on the cost function approach (e.g [7, 17–20]), while only a few are based on Bayes rules. The reason may lie in the fact that for the latter case (algorithm based on Bayes rules), the conditional expectation (the expectation of the source signal given the equalized output signal) has to be calculated. But, this task might be difficult because: (i) the source pdf might be non-Gaussian, and might not be known. In addition, the expression for the conditional expectation should hold for a wide range of source pdfs and not only for a specific source. (ii) No model is available for the convolutional noise pdf that is valid for the whole deconvolution process. Thus, without having the source and convolutional noise pdfs the derivation of the conditional expectation seems to be impossible. According to [1] the conditional expectation was derived for non-Gaussian sources by Bellini [11, 12], Fiori [13, 14] and Haykin [15]. However, all the mentioned expressions for the conditional expectation [11–15] are suitable only for uniformly distributed source signals. Thus they cannot cope with a source having a general pdf shape [1]. In addition, the works in [11–15] modeled the convolutional noise pdf as Gaussian. Recently ([1, 16]), two closed-form approximated expressions were obtained for the conditional expectation where the source pdf was approximated with the maximum entropy density estimation technique and Edgeworth Expansion series, respectively. Those expressions for the conditional expectation ([1, 16]) do not impose any restrictions (except of even symmetric) on the pdf of the unobserved input sequence. Hence they are suitable for a wider range of source pdf compared with Bellini’s [11, 12], Fiori’s [13, 14], or Haykin’s [15] expression. In [1, 16], the Laplace integral method was needed for approximating the integrals involving in the conditional expectation calculations. In addition, in both cases ([1, 16]), the convolutional noise pdf was assumed to be Gaussian for the whole deconvolution process as was assumed in [11–15]. But, according to Haykin [15], in the early stages of the iterative deconvolution process, the ISI is typically large with the result that the data sequence and the convolutional noise are strongly correlated. Furthermore, the convolutional noise sequence is more uniform than Gaussian [21]. The maximum entropy algorithm [1], where the conditional expectation was calculated with the maximum entropy density approximation technique and Gaussian model for the source signal and convolutional noise pdf, respectively, has shown to have improved equalization performance compared with the classical methods ([22] (RCA algorithm), [7, 9, 13, 17, 18, 23, 24]). However, probably a more appropriate model for the convolutional noise pdf than the choice of the Gaussian one might lead to improved equalization performance compared to the results presented in [1]. The first attempt to characterize the convolutional noise pdf different than the Gaussian model was introduced in [5]. Here the conditional expectation was approximately obtained by approximating the source and convolutional noise pdf with the maximum entropy density approximation technique and Edgeworth expansion series up to order four, respectively. An unknown pdf may be approximated with three types of orthogonal expansions using the Hermite polynomials, namely, the Gram-Charlier of type A, Gauss-Hermite, and Edgeworth expansions [4, 25]. According to [4, 25–27], the Edgeworth expansion is much more useful in many applications, since it is directly connected to the moments and cumulants of a pdf (the property which is lost in the Gauss-Hermite series). It is also a true asymptotic expansion, so that the error of the approximation is controlled (a property which is not found in the Gram-Charlier of type A) [4, 25]. Thus, Edgeworth expansion up to order 6 will describe the unknown pdf better than Edgeworth Expansion up to order four. The idea of approximating a non-Gaussian signal is not new in the literature. As a matter of fact, we may find several works [28–32] dealing with pdfs applicable for the non-Gaussian case but encompass also the Gaussian model. However, the idea of modeling the convolutional noise pdf, which is changing in time, different than the Gaussian pdf is quite new and was introduced at the first time in [5]. Here the convolutional noise pdf was modeled with the Edgeworth expansion series up to order four and the source signal was approximated with the maximum entropy density approximation technique.

The maximum entropy density approximation technique involves Lagrange multipliers which have to be defined; otherwise, this approximation is not applicable. However, according to [1], finding the Lagrange multipliers is not an easy task. In some cases, an analytical solution for the Lagrange multipliers does not exist (please refer to [1] for more details). In order to overcome the problem, the approximated MSE was derived in [1] and the required Lagrange multipliers were those Lagrange multipliers that lead the approximated MSE to minimum. Note that those Lagrange multipliers [1] were obtained assuming a Gaussian model for the convolutional noise pdf, but were also applied in [5] where the conditional expectation was calculated using the Maximum entropy density approximation technique and Edgeworth expansion series up to order four for the source signal and convolutional noise pdf, respectively even though those Lagrange multipliers are not the optimal set for that case. Although those Lagrange multipliers were not the appropriate set for [5], improved equalization performance was obtained with the new approximated expression for the conditional expectation introduced in [5] compared to the conditional expectation obtained in [1]. Note that the only difference between those expressions ([1, 5]) was the chosen model for the convolutional noise pdf that was taken in the conditional expectation calculations. Thus, it is probable that even more improved equalization performance may be obtained, if the appropriate Lagrange multipliers are applied in [5] and the Edgeworth expansion series up to order six are used instead of order four for the convolutional noise pdf. But to apply the appropriate Lagrange multiplier in [5], we must first derive the MSE related to the conditional expectation that uses the maximum entropy density approximation technique for the source signal pdf and the Edgeworth expansion series up to order four and six for the convolutional noise pdf. Hitherto, this derivation has not been done. Furthermore, the approximated conditional expectation has not been derived where the source signal and the convolutional noise pdf are approximated with the maximum entropy density approximation technique and Edgeworth expansion series up to order six, respectively.

In this paper we derive the appropriate set of Lagrange multipliers where the source signal and convolutional noise pdf are approximated with the maximum entropy density approximation technique and Edgeworth expansion series up to order four and six, respectively. We derive new approximated expressions for the conditional expectation and MSE where the source signal and convolutional noise pdf are approximated with the maximum entropy density approximation technique and Edgeworth expansion series up to order six, respectively. In this paper we test the effect of the use of the optimal set of Lagrange multipliers to the equalization performance. Furthermore, we examine whether increasing the series order in the Edgeworth Expansion can supply even more improved equalization performance compared to those already presented in [5] where we saw significant equalization performance improvement compared to the maximum entropy algorithm [1]. We also assess whether the increase in the series order is justified given the increase in the complexity of the algorithm.

The paper is organized as follows. In Section 2 we describe the system under consideration, and in Section 3 we present the new model for the convolutional noise pdf, closed-form approximated expressions for the conditional expectation, MSE, and Lagrange multipliers. Simulation results are given in Section 4, and Section 5 is our conclusion.

2. System Description

The system is illustrated in Figure 1, where we make the same assumptions as in [1]:(1)the input sequence 𝑥[𝑛] has an even symmetric probability distribution function with zero mean. 𝑥[𝑛] may be a real or a complex variables (where real and imaginary parts of 𝑥[𝑛] are independent);(2)the unknown channel ℎ[𝑛] can be a nonminimum phase linear time-invariant filter with no deep zeros (the zeros lie sufficiently far from the unit circle);(3)𝑐[𝑛] is a tap-delay equalizer;(4)𝑤[𝑛] is an additive Gaussian white noise;(5)𝑇[] is a memoryless nonlinear function.

Figure 1 

System description.

The input sequence 𝑥[𝑛] is transmitted through the channel ℎ[𝑛] and is corrupted with noise 𝑤[𝑛]. Therefore, the equalizer’s input sequence 𝑦[𝑛] may be written as 𝑦[𝑛]=𝑥[𝑛]∗ℎ[𝑛]+𝑤[𝑛], (1) where “∗” denotes the convolution operation.

From [1], the equalized output signal may be defined as 𝑧[𝑛]=𝑥[𝑛]+𝑝[𝑛]+𝑤[𝑛], (2) where 𝑝[𝑛] is the convolution noise arising from the difference between the ideal value of 𝑐[𝑛] and the initial guess of 𝑐[𝑛], 𝑐_𝑔[𝑛] and 𝑤[𝑛] represents the convolution between 𝑤[𝑛] and 𝑐_𝑔[𝑛].

The ISI is often used as a measure of performance in equalizer applications, as defined by Pinchas and Bobrovsky [1]: ΣISI=^𝑛𝑠̃_𝑛²−|𝑠|̃²_max|𝑠|̃²_max, (3) where |𝑠|̃_max is the maximum absolute value of the convolution between ℎ[𝑛] and 𝑐_𝑔[𝑛](𝑠_𝑛̃=𝑐_𝑔[𝑛]∗ℎ[𝑛]).

Next we consider the adaptation mechanism of the equalizer. According to Figure 1, we define 𝑑[𝑛] as an estimator of 𝑥[𝑛], which is produced by the function 𝑇[𝑧[𝑛]].

Thus the error signal is 𝑒̃[𝑛]=𝑇[𝑧[𝑛]]−𝑧[𝑛]. (4) The adaptive mechanism uses this error to update the equalizers taps [33]: 𝑐_eq[𝑛+1]=𝑐_eq[𝑛]+𝜇⋅𝑒̃[𝑛]𝑦^∗[𝑛], (5) where (⋅)^∗ is the conjugate operation on (⋅), 𝜇 is the step-size parameter, and 𝑐_eq[𝑛] is the equalizer vector where the input vector is 𝑦[𝑛]=[𝑦[𝑛]⋯𝑦[𝑛−𝑁+1]]^𝑇 and 𝑁 is the equalizer’s tap length. The operator (⋅)^𝑇 denotes the transpose of the function [⋅]. The conditional expectation [𝐸[𝑥[𝑛]∣𝑧[𝑛]], where 𝐸[⋅] stands for the expectation operation, is considered as a good estimate of 𝑇[𝑧[𝑛]] [3]. In the following, 𝑇[𝑧[𝑛]] is represented by the conditional expectation as was done in [1].

3. New Model for the Convolutional Noise pdf

In this section we describe the proposed model for the convolutional noise pdf which is based on the Edgeworth expansion. It differs from the Gaussian model but can turn under some conditions back to the Gaussian model as will be shown in this section. In Section 3.1 we derive the conditional expectation based on the new proposed model for the convolutional noise pdf. In Section 3.2 we present the new Lagrange multipliers via the MSE expression. It should be pointed out that we are not trying to find or propose a new expression for the conditional expectation but rather trying to show via the conditional expectation whether the new proposed model for the convolutional noise pdf (Edgeworth expansion of orders four and six) can lead to improved equalization performance with optimal Lagrange multipliers. The convolutional noise pdf for the real valued case may be defined as 𝑓_𝑝𝑝≅exp−𝑝²/2𝜎²_𝑝√2𝜋𝜎_𝑝1+^𝑟_𝑟=3𝐾_𝑟𝑝/𝜎_𝑝𝐻_𝑟𝑝/𝜎_𝑝,𝑟! (6) where ! represents factorial, the convolution noise 𝑝[𝑛] is defined as 𝑝, and 𝜎²_𝑝 is the variance of the convolutional noise. 𝐻_𝑟(𝑝/𝜎_𝑝) are normalized Hermite Polynomials of order 𝑟 described in [34] and presented in Appendix A as 𝐻_𝑟(𝜁). 𝐾_𝑟(𝑝/𝜎_𝑝) are the one-dimensional central cumulants in terms of central moments described in [34, Table 3], and are also presented in Appendix A.

Thus, (6) up to order 6 becomes 𝑓_𝑝𝑝≅exp−𝑝²/2𝜎²_𝑝√2𝜋𝜎_𝑝×𝐾1+⁴𝑝/𝜎_𝑝𝐻₄𝑝/𝜎_𝑝+𝐾4!₆𝑝/𝜎_𝑝𝐻₆𝑝/𝜎_𝑝,6! (7) where according to Appendices A and B we have 𝐾₄𝑝/𝜎_𝑝𝐻₄𝑝/𝜎_𝑝=𝐸𝑝4!₄−3𝜎²_𝑝²4!𝜎²_𝑝²𝑝⁴𝜎²_𝑝²−6𝑝²𝜎²_𝑝,𝐾+3₆𝑝/𝜎_𝑝𝐻₆𝑝/𝜎_𝑝=𝐸𝑝6!₆−15𝜎²_𝑝𝐸𝑝⁴+30𝜎²_𝑝³6!𝜎²_𝑝³×𝑝⁶𝜎²_𝑝³−15𝑝⁴𝜎²_𝑝²+45𝑝²𝜎²_𝑝−15. (8)

So, with the help of (8), (7) becomes 𝑓_𝑝𝑝≅exp−𝑝²/2𝜎²_𝑝√2𝜋𝜎_𝑝×𝐸𝑝1+⁴−3𝜎²_𝑝²4!𝜎²_𝑝²𝑝⁴𝜎²_𝑝²−6𝑝²𝜎²_𝑝+𝐸𝑝+3₆−15𝜎²_𝑝𝐸𝑝⁴+30𝜎²_𝑝³6!𝜎²_𝑝³×𝑝⁶𝜎²_𝑝³−15𝑝⁴𝜎²_𝑝²+45𝑝²𝜎²_𝑝−15. (9) According to (7) and (8) the convolutional noise pdf based on the Edgeworth expansion of order 4 can be obtained (by resetting the last product in (7)) as 𝑓_𝑝𝑝≅exp−𝑝²/2𝜎²_𝑝√2𝜋𝜎_𝑝×𝐸𝑝1+⁴−3𝜎²_𝑝²4!𝜎²_𝑝²𝑝⁴𝜎²_𝑝²−6𝑝²𝜎²_𝑝+3. (10) According to Haykin [15] a Gaussian convolutional noise pdf model is only applicable in the latter stages of the deconvolution process when the process is close to optimal. This is reflected in our model (6) when the expressions in the parentheses are close to one thus getting back to the expression that describes a Gaussian distribution: 𝑓_𝑝𝑝≅exp−𝑝²/2𝜎²_𝑝√2𝜋𝜎_𝑝. (11)

3.1. The Approximated Expression for the Conditional Expectation

In this subsection we use our new model for the convolutional noise pdf to calculate the conditional expectation.

Theorem 1. For the following assumptions:(1)the source signal 𝑥[𝑛] is an independent signal with known variance and higher moments. In the following we denote 𝑥[𝑛] as 𝑥,(2)no noise is added,(3)the convolutional noise 𝑝[𝑛] and the source signal 𝑥[𝑛] are independent; thus, 𝜎²_𝑧=𝜎²_𝑥+𝜎²_𝑝 where 𝜎²_𝑧 represents the variance of the equalizer’s output,(4)the convolutional noise 𝑝[𝑛] has zero mean and variance 𝜎²_𝑝=𝐸[𝑝[𝑛]𝑝^∗[𝑛]],the conditional expectation is given by 𝐸[𝑥∣𝑧]≅𝑧₁+𝑔₁^′′𝑧₁2𝑔𝑧₁𝜎²_𝑧1−𝜎²_𝑥1+𝑔₁^′′′′𝑧₁8𝑔𝑧₁𝜎²_𝑧1−𝜎²_𝑥1²×𝑔1+^′′𝑧₁2𝑔𝑧₁𝜎²_𝑧1−𝜎²_𝑥1+𝑔^′′′′𝑧₁8𝑔𝑧₁𝜎²_𝑧1−𝜎²_𝑥1²⁻¹+𝑗𝑧₂+𝑔₂^′′𝑧₂2𝑔𝑧₂𝜎²_𝑧2−𝜎²_𝑥2+𝑔₂^′′′′𝑧₂8𝑔𝑧₂𝜎²_𝑧2−𝜎²_𝑥2²×𝑔1+^′′𝑧₂2𝑔𝑧₂𝜎²_𝑧2−𝜎²_𝑥2+𝑔^′′′′𝑧₂8𝑔𝑧₂𝜎²_𝑧2−𝜎²_𝑥2²⁻¹, (12) where 𝑧=𝑧₁+𝑗𝑧₂;𝑥=𝑥₁+𝑗𝑥₂𝑠=1,2;𝑔𝑧_𝑠=exp^𝐾_𝑘=0𝜆_𝑘𝑧^𝑘_𝑠×𝐾1+⁴𝑝/𝜎_𝑝𝐻₄𝑝/𝜎_𝑝+𝐾4!₆𝑝/𝜎_𝑝𝐻₆𝑝/𝜎_𝑝,𝑔6!_′′𝑧_𝑠=𝑑²𝑑𝑥²_𝑠exp^𝐾_𝑘=0𝜆_𝑘𝑥^𝑘_𝑠×𝐾1+⁴𝑝/𝜎_𝑝𝐻₄𝑝/𝜎_𝑝+𝐾4!₆𝑝/𝜎_𝑝𝐻₆𝑝/𝜎_𝑝6!_{𝑥𝑠=𝑧𝑠},𝑔^′′′′𝑧_𝑠=𝑑⁴𝑑𝑥⁴_𝑠exp^𝐾_𝑘=0𝜆_𝑘𝑥^𝑘_𝑠×𝐾1+⁴𝑝/𝜎_𝑝𝐻₄𝑝/𝜎_𝑝+𝐾4!₆𝑝/𝜎_𝑝𝐻₆𝑝/𝜎_𝑝6!_{𝑥𝑠=𝑧𝑠},𝑔₁^′′𝑧_𝑠=𝑑²𝑑𝑥²_𝑠𝑥_𝑠exp^𝐾_𝑘=0𝜆_𝑘𝑥^𝑘_𝑠×𝐾1+⁴𝑝/𝜎_𝑝𝐻₄𝑝/𝜎_𝑝+𝐾4!₆𝑝/𝜎_𝑝𝐻₆𝑝/𝜎_𝑝6!_{𝑥𝑠=𝑧𝑠},𝑔₁^′′′′𝑧_𝑠=𝑑⁴𝑑𝑥⁴_𝑠𝑥_𝑠exp^𝐾_𝑘=0𝜆_𝑘𝑥^𝑘_𝑠×𝐾1+⁴𝑝/𝜎_𝑝𝐻₄𝑝/𝜎_𝑝+𝐾4!₆𝑝/𝜎_𝑝𝐻₆𝑝/𝜎_𝑝6!_{𝑥𝑠=𝑧𝑠}, (13) where ()^′′ and ()^′′′′ denote the second and fourth derivative of (), respectively, 𝑧=𝑧[𝑛], 𝑧₁, and 𝑥₁ are the real parts of 𝑧 and 𝑥, respectively. 𝑧₂ and 𝑥₂ are the imaginary parts of 𝑧 and 𝑥, respectively. 𝜎²_𝑥1, 𝜎²_𝑥2 are the variances of the real and imaginary parts of the source signal, respectively, and 𝜎²_𝑧1, 𝜎²_𝑧2 are the variances of the real and imaginary parts of the equalized output signal, respectively. The Lagrange multipliers 𝜆_𝑘 are given in Section 3.2. For more details concerning 𝑔^′′(𝑧), 𝑔^′′′′(𝑧), 𝑔₁^′′(𝑧), and 𝑔₁^′′′′(𝑧), please refer to Appendix B.

Comments. Please note that (12) is similar to the expression for the conditional expectation derived in [1]. However, 𝑔(𝑧), 𝑔^′′(𝑧), 𝑔^′′′′(𝑧), 𝑔₁(𝑧), 𝑔₁^′′(𝑧), and 𝑔₁^′′′′(𝑧) are very different from those obtained in [1].

Proof. We start our derivation for the real valued case and then we extend them to the two independent quadrature carrier one. By using Bayes’ rule we obtain 𝐸[]=∫𝑥∣𝑧^∞−∞𝑥𝑓_𝑧/𝑥(𝑧/𝑥)𝑓_𝑥(𝑥)𝑑𝑥∫^∞_−∞𝑓_𝑧/𝑥(𝑧/𝑥)𝑓_𝑥(𝑥),𝑑𝑥 (14) where in our case 𝑓_𝑧/𝑥(𝑧/𝑥) is the convolutional noise pdf and the unknown source pdf, 𝑓_𝑥(𝑥), will be estimated with the maximum entropy density approximation technique as was done in [1] and defined as 𝑓_𝑥(𝑥)𝑓_𝑥(𝑥)≅exp^𝐾_𝑘=0𝜆_𝑘𝑥^𝑘, (15) where 𝜆_𝑘(𝑘=0,1,2,…,𝐾) are the Lagrange multipliers given in Section 3.2.
Substituting (6) and (15) into (14) and by using (2) we obtain 𝐸[𝑥∣𝑧]=^∞_−∞𝑥1√2𝜋𝜎_𝑝−()exp𝑧−𝑥²2𝜎²_𝑝×𝐾1+⁴(𝑧−𝑥)/𝜎_𝑝𝐻₄(𝑧−𝑥)/𝜎_𝑝+𝐾4!₆(𝑧−𝑥)/𝜎_𝑝𝐻₆(𝑧−𝑥)/𝜎_𝑝6!⋅exp^𝐾_𝑘=0𝜆_𝑘𝑥^𝑘×𝑑𝑥_∞−∞1√2𝜋𝜎_𝑝−()exp𝑧−𝑥²2𝜎²_𝑝×𝐾1+⁴(𝑧−𝑥)/𝜎_𝑝𝐻₄(𝑧−𝑥)/𝜎_𝑝+𝐾4!₆(𝑧−𝑥)/𝜎_𝑝𝐻₆(𝑧−𝑥)/𝜎_𝑝6!⋅exp^𝐾_𝑘=0𝜆_𝑘𝑥^𝑘𝑑𝑥⁻¹. (16) Next we define 𝑔(𝑥)=exp^𝐾_𝑘=0𝜆_𝑘𝑥^𝑘×𝐾1+⁴(𝑧−𝑥)/𝜎_𝑝𝐻₄(𝑧−𝑥)/𝜎_𝑝+𝐾4!₆(𝑧−𝑥)/𝜎_𝑝𝐻₆(𝑧−𝑥)/𝜎_𝑝,6!𝑔₁(𝑥)=𝑥𝑔(𝑥),𝜌=2𝜎²_𝑝;𝜓(𝑥)=(𝑧−𝑥)²;𝜎²_𝑝=𝜎²_𝑧−𝜎²_𝑥, (17)(18)(19) so (16) becomes 𝐸[]=∫𝑥∣𝑧^∞−∞𝑔₁(𝑥)⋅exp−𝜓(𝑥)/𝜌𝑑𝑥∫^∞_−∞𝑔(𝑥)(𝑥).⋅exp−𝜓/𝜌𝑑𝑥 (20) This integral (20) may be approximated with the Laplace integral method according to [1, 16, 35]. Thus, we may write ^∞_−∞𝑔₁(𝑥)(𝑥)⋅exp−𝜓𝜌(𝑥)𝑑𝑥≅exp−𝜓𝜌2𝜋𝜌𝜓^′′𝑥₀𝑔₁𝑥₀+𝑔₁^′′𝑥₀2𝜌𝜓^′′𝑥₀+𝑔₁^′′′′𝑥₀8𝜌𝜓^′′𝑥₀²𝜌+𝑂𝜓^′′𝑥₀³_∞−∞𝑔(𝑥)(𝑥)⋅exp−𝜓𝜌(𝑥)𝑑𝑥≅exp−𝜓𝜌2𝜋𝜌𝜓^′′𝑥₀𝑔𝑥₀+𝑔^′′𝑥₀2𝜌𝜓^′′𝑥₀+𝑔^′′′′𝑥₀8𝜌𝜓^′′𝑥₀²𝜌+𝑂𝜓^′′𝑥₀³, (21) where 𝑂(𝑥) is defined as lim_𝑥→0(𝑂(𝑥)/𝑥)=𝑐, and 𝑐 is a constant and 𝜓^′(𝑥)=−2(𝑧−𝑥);𝜓^′′(𝑥)𝜓=2;_′(𝑥)=0⟹𝑥₀=𝑧. (22) By substituting (21) into (20) and dividing the numerator and denominator by 𝑔(𝑥₀), using (22), we obtain 𝐸[]≅𝑔𝑥∣𝑧𝑧+^1′′(𝑧)(𝑧)𝜎2𝑔^2𝑧−𝜎²_𝑥+𝑔₁^′′′′(𝑧)(𝑧)𝜎8𝑔^2𝑧−𝜎²_𝑥²×𝑔1+^′′(𝑧)(𝑧)𝜎2𝑔^2𝑧−𝜎²_𝑥+𝑔^′′′′(𝑧)(𝑧)𝜎8𝑔^2𝑧−𝜎²_𝑥²⁻¹. (23) Note that (23) is very similar to the approximated expression for the conditional expectation derived in [1]; however, 𝑔(𝑧) and its derivatives are very different. According to [11], the conditional mean estimate of the complex datum 𝑥(𝑥=𝑥₁+𝑗𝑥₂) given the complex observation 𝑧(𝑧=𝑧₁+𝑗𝑧₂) can be written as 𝐸[𝑥∣𝑧]=𝐸𝑥₁∣𝑧₁+𝑗𝐸𝑥₂∣𝑧₂. (24) This completes our Proof.

3.2. MSE and Lagrange Multipliers

Since we deal with the real valued or two independent quadrature carrier case, it is enough to derive the MSE as well as the expression for the Lagrange multipliers for the real valued case only. Thus, we consider in the following only the real valued case.

In this section we search for those Lagrange multipliers that bring the MSE to a minimum as was done in [1]: 𝑑𝑑𝜆_𝑘𝐸[𝑥−𝑥]²=0, (25) where 𝑥 is the conditional expectation given in (23).

The approximated MSE according to [1] is given by 𝐸[𝑥−𝑥]²≅𝜎²_𝑝1+𝜎²_𝑝𝐸𝑔^′′(𝑥)/2𝑔(𝑥)², (26) where 𝑥 is the conditional expectation given in [1].

Although 𝑔(𝑧), 𝑔^′′(𝑧), 𝑔^′′′′(𝑧), 𝑔₁(𝑧), 𝑔₁^′′(𝑧), and 𝑔₁^′′′′(𝑧) are very different from those obtained in [1], (23) looks quite similar to the approximated expression for the conditional expectation derived in [1]. Thus, it make sense to use in this paper the same technique for deriving the MSE as was carried out in [1]. The convolutional noise pdf (9) is for 𝑝=0 the pdf used in [1] (for 𝑝=0) multiplied by a constant parameter (1+3𝐴−15𝐵), where 𝐴 and 𝐵 are given in Appendix B in (B.6) and (B.7), respectively. Therefore, based on (94), (95), and (96) in [1], the MSE for our case is the same MSE given in (26) multiplied by (1+3𝐴−15𝐵) with our expression for 𝐸[(𝑔^′′(𝑥)/2𝑔(𝑥))_𝑝=0]. Thus, we obtain 𝐸[𝑥−𝑥]²≅𝜎²_𝑝(1+3𝐴−15𝐵)1+𝜎²_𝑝𝐸𝑔^′′(𝑥)/2𝑔(𝑥)_𝑝=0². (27) According to [36], for small values of 𝜎²_𝑝 such that |𝜎²_𝑝(𝐸[𝑔^′′(𝑥)/2𝑔(𝑥)]_𝑝=0)|<1 we obtain 𝐸[𝑥−𝑥]²≅()×𝜎1+3𝐴−15𝐵_2𝑝1+𝜎²_𝑝𝐸𝑔^′′(𝑥)/2𝑔(𝑥)_𝑝=0²≅()×𝜎1+3𝐴−15𝐵_2𝑝1−𝜎²_𝑝𝑔2𝐸^′′(𝑥)2𝑔(𝑥)_𝑝=0. (28) Next we minimize (28) with respect to the Lagrange multipliers, 𝜆_𝑘, 𝑑𝑑𝜆_𝑘(1+3𝐴−15𝐵)𝜎²_𝑝1−𝜎²_𝑝𝑔2𝐸^′′(𝑥)2𝑔(𝑥)_𝑝=0⟹𝑑=0𝑑𝜆_𝑘𝐸𝑔^′′(𝑥)2𝑔(𝑥)_𝑝=0=0. (29) In Appendix E we derive (29) and obtain 𝑚_𝑘−2𝑘()+2𝑘−1()×1+3𝐴−15𝐵_{𝐾𝑙=2,𝑙≠𝑘}𝜆_𝑙𝑘𝑙𝑚_{𝑘+𝑙−2}+2𝜆_𝑘𝑘²𝑚_2𝑘−21+3𝐴−15𝐵=0, (30) where 𝑚_𝑘=𝐸[𝑥^𝑘].

Please refer to Appendices C and D for explicit expressions for the Lagrange multipliers valid for the 16QAM and 64QAM input constellation case, respectively. It should be pointed out that (30) is the same expression obtained in [1] for 𝐴=0 and 𝐵=0.

4. Simulation

In this section we test and compare the equalization performance obtained for the 16QAM and 64QAM input case with the convolutional noise pdf (6) to the case where the convolutional noise pdf is modeled as Gaussian. In addition, we examine whether the optimal Lagrange multipliers have a significant effect on the quality of the equalization performance and determine whether the transition from order four to order six in the Edgeworth Expansion series is worthwhile despite the increase in complexity in the algorithm.

According to [1], where the convolutional noise pdf was modeled as a Gaussian pdf, the equalizer’s taps were updated as 𝑐_𝑙[𝑛+1]=𝑐_𝑙[𝑛]+𝜇𝑊𝑦^∗[]𝐸𝑥𝑛−𝑙with𝑊=₁∣𝑧₁𝑧₁(𝑛)𝐸𝑥₁∣𝑧₁𝑧₁²_𝑛+𝑗𝐸𝑥₂∣𝑧₂×𝑧₂(𝑛)𝐸𝑥₂∣𝑧₂𝑧₂²_𝑛−𝑧(𝑛) (31) with (12) and (30) for 𝐴=0, 𝐵=0, and 𝑧²_𝑠𝑛=1−𝛽𝑧²_𝑠𝑛−1+𝛽𝑧_𝑠²_𝑛, (32) where ⟨⟩ stands for the estimated expectation operator, 𝛽 and 𝜇 are positive step-size parameters, ⟨𝑧²_𝑠⟩₀>0, and 𝑙 stands for 𝑙th tap of the equalizer and 𝑠=1,2. In the following, this equalizer [1] will be denoted as “MaxEnt.” To illustrate the equalization performance with the convolutional noise pdf (6) we used the equalizer’s update mechanism given in (31) and (32) with the conditional expectation presented in (12) for the following settings.

For 𝐴≠0, 𝐵=0 the equalizer will be called “Edgeworth order 4.”

For 𝐴≠0,𝐵≠0 the equalizer will be called “Edgeworth order 6.”

For “Edgeworth order 4” we need (B.6) and for “Edgeworth order 6” we also need (B.7). In this work we used for 𝐸[𝑝⁴] and 𝐸[𝑝⁶] the following settings: 𝐸𝑝⁴=𝑎𝜎²_𝑝²,𝐸𝑝⁶=𝑏𝜎²_𝑝³, (33) where 𝑎 and 𝑏 were chosen for fast convergence speed and low steady state ISI.

The parameters 𝑎 and 𝑏 are denoted in the following as 𝑎_EE4 for “Edgeworth order 4” (𝑏=0 for “Edgeworth order 4”) and 𝑎_EE6,𝑏_EE6 for “Edgeworth order 6.”

The step-size parameter 𝜇 is denoted in the following as 𝜇_MaxEnt, 𝜇_EE4, and 𝜇_EE6 for “MaxEnt,” “Edgeworth order 4,” and “Edgeworth order 6,” respectively. In the following, the 𝛽 parameter is denoted as the 𝛽_MaxEnt, 𝛽_EE4, and 𝛽_EE6 for “MaxEnt,” “Edgeworth order 4,” and “Edgeworth order 6,” respectively. According to [1] the denominator of (23) cannot be zero. Therefore, during our simulations, the equalizer’s taps were updated only if the denominator was greater than 𝜀(𝜀>0). For the equalization performance comparison we also used the algorithm defined by Godard [7]. The equalizer’s taps for Godard’s algorithm [7] were updated according to 𝑐_𝑙[𝑛+1]=𝑐_𝑙[𝑛]−𝜇_𝐺|𝑧[𝑛]|²−𝐸|𝑥[𝑛]|⁴𝐸|𝑥[𝑛]|²×𝑧[𝑛]𝑦^∗[𝑛−𝑙], (34) where 𝜇_𝐺 is the step-size and || is the absolute operator.

Two different sources were considered: 16QAM and 64QAM sources (modulations using ±{1,3}, ±{1,3,5,7} levels respectively, for in-phase and quadrature components).

Two different channels were considered: channel case 1 (initial ISI = 0.44), where the channel parameters were taken according to Shalvi and Weinstein [17]: ℎ_𝑛={0for𝑛<0;−0.4for𝑛=0;0.84⋅0.4^𝑛−1for𝑛>0}, channel case 2 (initial ISI = 1.402), where the channel parameters were taken according to [18]: ℎ_𝑛={0.2258,0.5161,0.6452,0.5161}.

For channel case 1 and channel case 2 we used an equalizer with 13 and 21 taps, respectively. The equalizers where initialized by setting the center tap equal to one and all others to zero.

The step-size parameters, 𝜇_𝐺, 𝜇_MaxEnt, 𝛽_MaxEnt, 𝜇_EE4, 𝛽_EE4, 𝜇_EE6, and 𝛽_EE6 were chosen for fast convergence speed and low steady state ISI.

Figures 2–4 show the simulated equalization performance (ISI as a function of iteration number) of three equalization methods: “Godard,” “MaxEnt,” “Edgeworth order 4,” and “Edgeworth order 6” for channel case 1, 16QAM, and SNR = 30 dB.

Figure 2 

Performance comparison between equalization algorithms simulated for a 16QAM input source going through channel case 1. The averaged results were obtained in 100 Monte Carlo trials for SNR of 30 dB. The step-size parameters were set to 𝜇𝐺=1×10−4, 𝜇MaxEnt=4×10−4, 𝛽MaxEnt=2×10−4. For Edgeworth order 4 with nonoptimal Lagrange multipliers we used 𝑎EE4=4.5, 𝜇EE4=4.3×10−4, 𝛽EE4=4.5×10−4, and for Edgeworth order 4 with optimal Lagrange multipliers we used 𝑎EE4=4.5, 𝜇EE4=4.3×10−4, and 𝛽EE4=4.5×10−4. In addition we set for MaxEnt and Edgeworth order 4 with nonoptimal Lagrange multipliers 𝜀=0. For Edgeworth order 4 with optimal Lagrange multipliers we set 𝜀=0.5.

Figure 3 

Performance comparison between equalization algorithms simulated for a 16QAM input source going through channel case 1. The averaged results were obtained in 100 Monte Carlo trials for SNR of 30 dB. The step-size parameters were set to: 𝜇𝐺=1×10−4, 𝜇MaxEnt=4×10−4, 𝛽MaxEnt=2×10−4. For Edgeworth order 4 with non-optimal Lagrange multipliers we used 𝑎EE4=4.5, 𝜇EE4=4.3×10−4, 𝛽EE4=4.5×10−4, and for Edgeworth order 6 with optimal Lagrange multipliers we used 𝑎EE6=4.5, 𝑏EE6=38, 𝜇EE6=5.2×10−4, and 𝛽EE6=4.5×10−4. In addition we set for MaxEnt and Edgeworth order 4 with non-optimal Lagrange multipliers 𝜀=0. For Edgeworth order 6 with optimal Lagrange multipliers we set 𝜀=0.5.

Figure 4 

Performance comparison between equalization algorithms simulated for a 16QAM input source going through channel case 1. The averaged results were obtained in 100 Monte Carlo trials for SNR of 30 dB. The step-size parameters were set to: 𝜇𝐺=1×10−4, 𝜇MaxEnt=4×10−4, 𝛽MaxEnt=2×10−4. For Edgeworth order 4 with optimal Lagrange multipliers we used 𝑎EE4=4.5, 𝜇EE4=4.3×10−4, and𝛽EE4=4.5×10−4 and for Edgeworth order 6 with optimal Lagrange multipliers we used 𝑎EE6=4.5, 𝑏EE6=38, 𝜇EE6=5.2×10−4, and 𝛽EE6=4.5×10−4. In addition we set for MaxEnt 𝜀=0, and for Edgeworth order 4 and order 6 𝜀=0.5.

Table 1 lists the various equalization methods and the corresponding model for the convolutional noise pdf used in the algorithm.

Table 1 

Two cases were considered for “Edgeworth order 4”:

Case a. Nonoptimal Lagrange multipliers (Lagrange multipliers similar to those used in “MaxEnt” [1]).

Case b. Optimal Lagrange multipliers (according to Appendix C).

Based on the simulation results (Figures 2–4), “Edgeworth order 4” and “Edgeworth order 6” achieve better equalization performance in terms of convergence speed and lower residual ISI compared to the other methods.

According to Figure 2, “Edgeworth order 4” with optimal Lagrange multipliers has a slightly faster convergence speed and a slightly lower residual ISI compared to the non-optimal Lagrange multipliers case. According to Figure 3, “Edgeworth order 6” with optimal Lagrange multipliers leads to a faster convergence speed compared with “Edgeworth order 4” with nonoptimal Lagrange multipliers but leads the system to a slightly higher residual ISI. Figure 4 shows that “Edgeworth order 4” and “Edgeworth order 6” with optimal Lagrange multipliers have both approximately the same convergence speed while “Edgeworth order 4” leads the system to a lower residual ISI compared to “Edgeworth order 6”. Figures 2–4 indicate that there is no justification using “Edgeworth order 6” over “Edgeworth order 4” and “Edgeworth order 4” with optimal Lagrange multipliers over “Edgeworth order 4” with nonoptimal Lagrange multipliers for channel case 1.

Figures 5, 6, and 7 indicate that our previously made conclusion based on Figures 2–4 also holds when the SNR is decreased.

Figure 5 

Performance comparison between equalization algorithms simulated for a 16QAM input source going through channel case 1. The averaged results were obtained in 100 Monte Carlo trials for SNR of 26 dB. The step-size parameters were set to: 𝜇𝐺=1×10−4, 𝜇MaxEnt=4×10−4, 𝛽MaxEnt=2×10−4. For Edgeworth order 4 with non-optimal Lagrange multipliers we used 𝑎EE4=4.5, 𝜇EE4=3.7×10−4, and 𝛽EE4=4×10−4 and for Edgeworth order 4 with optimal Lagrange multipliers we used 𝑎EE6=4.5, 𝑏EE6=32,𝜇EE6=3.7×10−4, and 𝛽EE6=4×10−4. In addition we set for MaxEnt and Edgeworth order 4 with optimal and nonoptimal Lagrange multipliers 𝜀=0.5.

Figure 6 

Performance comparison between equalization algorithms simulated for a 16QAM input source going through channel case 1. The averaged results were obtained in 100 Monte Carlo trials for SNR of 26 dB. The step-size parameters were set to: 𝜇𝐺=1×10−4, 𝜇MaxEnt=4×10−4, 𝛽MaxEnt=2×10−4. For Edgeworth order 4 with non-optimal Lagrange multipliers we used 𝑎EE4=4.5, 𝜇EE4=3.7×10−4, and𝛽EE4=4×10−4 and for Edgeworth order 6 with optimal Lagrange multipliers we used 𝑎EE6=4.5, 𝑏EE4=32, 𝜇EE6=3.7×10−4, and 𝛽EE6=4×10−4. In addition we set for MaxEnt, Edgeworth order 4, and Edgeworth order 6 𝜀=0.5.

Figure 7 

Performance comparison between equalization algorithms simulated for a 16QAM input source going through channel case 1. The averaged results were obtained in 100 Monte Carlo trials for SNR of 26 dB. The step-size parameters were set to: 𝜇𝐺=1×10−5, 𝜇MaxEnt=4×10−4, 𝛽MaxEnt=2×10−6. For Edgeworth order 4 with non-optimal Lagrange multipliers we used 𝑎EE4=4.5,𝜇EE4=3.7×10−4, and 𝛽EE4=4×10−4 and for Edgeworth order 6 with optimal Lagrange multipliers we used 𝑎EE6=4.5, 𝑏EE6=32, 𝜇EE6=3.7×10−4, and 𝛽EE6=4×10−4. In addition we set for MaxEnt, Edgeworth order 4, and Edgeworth order 6 𝜀=0.5.

Figures 8–10 show the simulated equalization performance (ISI as a function of iteration number) of four equalization methods: “Godard,” “MaxEnt,” “Edgeworth order 4,” and “Edgeworth order 6” for channel case 2, 16QAM, and SNR = 30 dB. Based on Figures 8–10, “Edgeworth order 4” and “Edgeworth order 6” have better equalization performance in terms of convergence speed compared to “MaxEnt” but lead the system to a slightly higher residual ISI compared to the “MaxEnt” method.

Figure 8 

Performance comparison between equalization algorithms simulated for a 16QAM input source going through channel case 2. The averaged results were obtained in 100 Monte Carlo trials for SNR of 30 dB. The step-size parameters were set to 𝜇𝐺=1×10−5, 𝜇MaxEnt=2×10−4, and 𝛽MaxEnt=2×10−6. For Edgeworth order 4 with nonoptimal Lagrange multipliers we used 𝑎EE4=5, 𝜇EE4=3×10−4, and 𝛽EE4=2×10−6 and for Edgeworth order 4 with optimal Lagrange multipliers we used 𝑎EE4=4,𝜇EE4=3×10−4, and 𝛽EE4=2×10−6. In addition we set for MaxEnt, Edgeworth order 4 with optimal and nonoptimal Lagrange multipliers 𝜀=0.5.

Figure 9 

Performance comparison between equalization algorithms simulated for a 16QAM input source going through channel case 2. The averaged results were obtained in 100 Monte Carlo trials for SNR of 30 dB. The step-size parameters were set to 𝜇𝐺=1×10−5, 𝜇MaxEnt=2×10−4, and 𝛽MaxEnt=2×10−6. For Edgeworth order 4 with nonoptimal Lagrange multipliers we used 𝑎EE4=5, 𝜇EE4=3×10−4, and 𝛽EE4=2×10−6 and for Edgeworth order 6 with optimal Lagrange multipliers we used 𝑎EE6=4.1, 𝑏EE6=25, 𝜇EE6=3.3×10−4, and 𝛽EE6=2.1×10−6. In addition we set for MaxEnt, Edgeworth order 4, and Edgeworth order 6 𝜀=0.5.

Figure 10 

Performance comparison between equalization algorithms simulated for a 16QAM input source going through channel case 2. The averaged results were obtained in 100 Monte Carlo trials for SNR of 30 dB. The step-size parameters were set to 𝜇𝐺=1×10−5, 𝜇MaxEnt=2×10−4, and 𝛽MaxEnt=2×10−6. For Edgeworth order 4 with optimal Lagrange multipliers we used 𝑎EE4=4, 𝜇EE4=3×10−4, and 𝛽EE4=2×10−6 and for Edgeworth order 6 with optimal Lagrange multipliers we used 𝑎EE6=4.1, 𝑏EE6=25, 𝜇EE6=3.3×10−4, and 𝛽EE6=2.1×10−6. In addition we set for Edgeworth orders 4 and 6 𝜀=0.5.

Figure 8 indicates that “Edgeworth order 4” with optimal Lagrange multipliers leads to a slightly faster convergence speed compared to the nonoptimal Lagrange multipliers case while leaving the system with approximately the same residual ISI.

Figures 9 and 10 show that “Edgeworth order 6” with optimal Lagrange multipliers leads to a much faster convergence speed compared to the optimal and nonoptimal Lagrange multiplier case with “Edgeworth order 4,” while having approximately the same residual ISI for both cases.

Figure 11 shows the simulated equalization performance (ISI as a function of iteration number) of three equalization methods: “MaxEnt,” “Edgeworth order 4,” and “Edgeworth order 6” for channel case 1, 64QAM constellation input and for the noiseless case.

Figure 11 

Performance comparison between equalization algorithms simulated for a 64QAM input source going through channel case 1. The averaged results were obtained in 100 Monte Carlo trials for the noiseless case. The step-size parameters were set to: 𝜇MaxEnt=5.5×10−5, 𝛽MaxEnt=6×10−5. For Edgeworth order 4 with optimal Lagrange multipliers we used 𝑎EE4=2.4, 𝜇EE4=6.2×10−5, and 𝛽EE4=6×10−5 and for Edgeworth order 6 with optimal Lagrange multipliers we used 𝑎EE6=2.4, 𝑏EE6=4, 𝜇EE6=6.2×10−5, and 𝛽EE6=6×10−5. In addition we set for all methods 𝜀=0.7.

Three optimal Lagrange multipliers were used for “Edgeworth order 4” and for “Edgeworth order 6” (please refer to Appendix D).

Based on the simulation results (Figure 11), a much faster convergence speed is obtained by using “Edgeworth order 6” with optimal Lagrange multipliers compared to the case where “Edgeworth order 4” with optimal Lagrange multipliers or the “MaxEnt” method is used.

5. Conclusion

In this paper we used the Edgeworth expansion up to order 4 and up to order 6 to model the convolutional noise pdf in the conditional expectation calculations where the source pdf was approximated according to the maximum entropy density approximation technique. We derived new Lagrange multipliers and obtained new closed-form approximated expressions for the conditional expectation and MSE as byproducts. According to simulation results for the 16QAM constellation input, there is no justification for using “Edgeworth order 4” with optimal Lagrange multipliers over “Edgeworth order 4” with nonoptimal Lagrange multipliers when dealing with easy or hard channels. In addition, hardly any equalization performance improvement was observed with “Edgeworth order 6” (with optimal Lagrange multipliers) compared to “Edgeworth order 4” with optimal and nonoptimal Lagrange multipliers for the easy channel case (channel case 1). However, a much faster convergence speed was observed with “Edgeworth order 6” with optimal Lagrange multipliers for the hard channel case (channel case 2) compared to “Edgeworth order 4” with optimal and nonoptimal Lagrange multipliers. Furthermore, a much faster convergence speed was obtained for the 64QAM input and easy channel case with “Edgeworth order 6” compared to “Edgeworth order 4” both with optimal Lagrange multipliers.

Appendices

A. Hermite Polynomials and the One Dimensional Central Cumulants

A closed-form expression for the Hermite Polynomials can be defined as in [37] 𝐻_𝑟(𝜁)=^𝑟/2_𝑘=0(−1)^𝑘𝑟!()()𝑘!𝑟−2𝑘2𝜁^𝑟−2𝑘. (A.1)

Specifically, the first Hermite Polynomials up to order 6 are 𝐻₀(𝜁)𝐻=1,₁(𝜁)𝐻=2𝜁,₂(𝜁)=4𝜁²𝐻−2,₃(𝜁)=𝜁³𝐻−3𝜁,₄(𝜁)=𝜁⁴−6𝜁²𝐻+3,₅(𝜁)=𝜁⁵−10𝜁³𝐻+15𝜁,₆(𝜁)=𝜁⁶−15𝜁⁴+45𝜁²−15. (A.2) According to [34, Table 3], the one-dimensional central cumulants in terms of central moments are 𝐾₃(𝜁)=𝐸𝜁³,𝐾₄(𝜁)=𝐸𝜁⁴−3𝐸𝜁²²,𝐾₅(𝜁)=𝐸𝜁⁵−10𝐸𝜁²𝐸𝜁³,𝐾₆(𝜁)=𝐸𝜁⁶−15𝐸𝜁²𝐸𝜁⁴+30𝐸𝜁²³−10𝐸𝜁³². (A.3)

B. Expressions for 𝑔^′′(𝑧), 𝑔^′′′′(𝑧), 𝑔₁^′′(𝑧), 𝑔₁^′′′′(𝑧), 𝐴, 𝐵, 𝐾₄(𝑝/𝜎_𝑝), 𝐾₆(𝑝/𝜎_𝑝), 𝐻₄(𝑝/𝜎_𝑝), and 𝐻₆(𝑝/𝜎_𝑝)

In order to facilitate the derivation process of (17) and (18) we divide (17) into two parts: 𝑔(𝑥)=ℎ(𝑥)𝑡(𝑥), (B.1) where 𝑡(𝑥)=exp^𝐾_𝑘=0𝜆_𝑘𝑥^𝑘, (B.2) and ℎ(𝑥)=𝐾1+⁴𝑝/𝜎_𝑝𝐻₄𝑝/𝜎_𝑝+𝐾4!₆𝑝/𝜎_𝑝𝐻₆𝑝/𝜎_𝑝=𝐾6!1+⁴(𝑧−𝑥)/𝜎_𝑝𝐻₄(𝑧−𝑥)/𝜎_𝑝+𝐾4!₆(𝑧−𝑥)/𝜎_𝑝𝐻₆(𝑧−𝑥)/𝜎_𝑝.6! (B.3)

According to (A.3), the values for 𝐻₄(𝑝/𝜎_𝑝) and 𝐻₆(𝑝/𝜎_𝑝) are 𝐻₄𝑝𝜎_𝑝=(𝑧−𝑥)⁴𝜎²_𝑝²−6(𝑧−𝑥)²𝜎²_𝑝;𝐻+3₄𝑝𝜎_{𝑝𝑥=𝑧}𝐻=3,₆𝑝𝜎_𝑝=(𝑧−𝑥)⁶𝜎²_𝑝³−15(𝑧−𝑥)⁴𝜎²_𝑝²+45(𝑧−𝑥)²𝜎²_𝑝;𝐻−15₆𝑝𝜎_{𝑝𝑥=𝑧}=−15. (B.4)

Their derivatives may be expressed as 𝑑²𝑑𝑥²𝐻₄𝑝𝜎_{𝑝𝑥=𝑧}=−12𝜎²_𝑝;𝑑⁴𝑑𝑥⁴𝐻₄𝑝𝜎_{𝑝𝑥=𝑧}=24𝜎⁴_𝑝𝑑²𝑑𝑥²𝐻₆𝑝𝜎_{𝑝𝑥=𝑧}=90𝜎²_𝑝;𝑑⁴𝑑𝑥⁴𝐻₆𝑝𝜎_{𝑝𝑥=𝑧}=−360𝜎⁴_𝑝. (B.5)

For 𝐾₄(𝑝/𝜎_𝑝) and 𝐾₆(𝑝/𝜎_𝑝) as described in (A.3) we define the following constants (results after normalization): 𝐾𝐴=⁴𝑝/𝜎_𝑝=𝐸𝑝4!⁴−3𝜎²_𝑝²4!𝜎²_𝑝²,𝐾𝐵=⁶𝑝/𝜎_𝑝=𝐸𝑝6!⁶−15𝜎²_𝑝𝐸𝑝⁴+30𝜎²_𝑝³6!𝜎²_𝑝³. (B.6)(B.7)

We set 𝑔(𝑧) as follows: 𝑔(𝑧)=𝑔(𝑥)_𝑥=𝑧=exp^𝐾_𝑘=0𝜆_𝑘𝑧^𝑘(1+3𝐴−15𝐵). (B.8)

The second and fourth derivatives of 𝑔(𝑧) are as follows: 𝑔^′′(𝑧)=ℎ^′′𝑝𝑡(𝑥)+2ℎ^′𝑝𝑡^′(𝑥)+ℎ𝑝𝑡^′′(𝑥)_𝑥=𝑧,𝑔^′′′′(𝑧)=ℎ^′′′′𝑝𝑡(𝑥)+4ℎ^′′′𝑝𝑡^′(𝑥)+6ℎ^′′𝑝𝑡^′′(𝑥)+4ℎ^′(𝑝)𝑡^′′′(𝑥)+ℎ(𝑝)𝑡^′′′′(𝑥)_𝑥=𝑧=ℎ^′′′′𝑝𝑡(𝑥)+4ℎ^′′′𝑝𝑡^′(𝑥)+6ℎ^′′𝑝𝑡^′′(𝑥)+4ℎ^′𝑝𝑡^′′′(𝑥)+ℎ𝑝𝑡^′′′′(𝑥)_𝑥=𝑧. (B.9) The definitions for 𝑔₁(𝑥) and its derivatives are 𝑔₁(𝑥)=𝑥𝑔(𝑥)⟹𝑔₁(𝑧)(𝑧),𝑔=𝑧𝑔_1′′(𝑧)=2𝑔^′(𝑧)+𝑧𝑔^′′(𝑧),𝑔₁^′′′′(𝑧)=4𝑔^′′′(𝑧)+𝑧𝑔^′′′′(𝑧). (B.10)

By substituting (B.9), (B.10) in (23) the closed-form approximated expression for the conditional expectation is obtained.

C. Lagrange Multipliers Equations for 16QAM

For the 16QAM case we use two Lagrange multipliers (𝜆₂,𝜆₄); thus, we have 𝐾=4.

In order to get the required Lagrange multipliers we recall (30) 𝑚_𝑘−2()2𝑘−1𝑘+()×1+3𝐴−15𝐵_{𝐾𝑙=2,𝑙≠𝑘}𝜆_𝑙𝑘𝑙𝑚_{𝑘+𝑙−2}+2𝜆_𝑘𝑘²𝑚_2𝑘−21+3𝐴−15𝐵=0, (C.1) where we use 𝐴 and 𝐵 from (B.6) and (B.7), respectively. For 𝜆₂ we have 𝑘=2; thus, when substituting 𝑘=2 and 𝐾=4 into (30) we get 𝑚₀(2−1)2+2𝜆₂𝑚₄₋₂2²+1+3𝐴−15𝐵2𝜆₄𝑚₂₊₄₋₂⋅2⋅4⇓1+3𝐴−15𝐵=02+8𝜆₂𝑚₂1+3𝐴−15𝐵+16𝜆₄𝑚₄1+3𝐴−15𝐵=0. (C.2)

For 𝜆₄ we have 𝑘=4; thus, when substituting 𝑘=4 and 𝐾=4 into (30) we get 𝑚₄₋₂(4−1)4+2𝜆₄𝑚₆4²+1+3𝐴−15𝐵2𝜆₂𝑚₄₊₂₋₂⋅4⋅2⇓1+3𝐴−15𝐵=02𝑚₂+32𝜆₄𝑚₆1+3𝐴−15𝐵+16𝜆₂𝑚₄1+3𝐴−15𝐵=0. (C.3)

From (C.2) and (C.3) we get a linear system with two equations: 8𝜆₂𝑚₂1+3𝐴−15𝐵+16𝜆₄𝑚₄1+3𝐴−15𝐵+2=0,16𝜆₂𝑚₄1+3𝐴−15𝐵+32𝜆₄𝑚₆1+3𝐴−15𝐵+12𝑚₂=0, (C.4)

which can be written by 8𝑚₂𝑚1+3𝐴−15𝐵16⁴𝑚1+3𝐴−15𝐵16⁴𝑚1+3𝐴−15𝐵32⁶𝜆1+3𝐴−15𝐵²𝜆₄=−2−12𝑚₂, (C.5) where the solution is−14−15𝑚₆𝐵−3𝑚₄𝑚₂−9𝑚₄𝑚₂𝐴+45𝑚₄𝑚₂𝐵+𝑚₆+3𝑚₆𝐴𝑚₂𝑚₆−𝑚²₄18−15𝑚₄𝐵−3𝑚²₂−9𝑚²₂𝐴+45𝑚²₂𝐵+𝑚₄+3𝑚₄𝐴𝑚₂𝑚₆−𝑚²₄. (C.6)

Next, we substitute the known moments 𝑚₂=5, 𝑚₄=41, and 𝑚₆=365 into (C.5) and obtain𝜆₂𝜆₄=−14−15𝑚₆𝐵−3𝑚₄𝑚₂−9𝑚₄𝑚₂𝐴+45𝑚₄𝑚₂𝐵+𝑚₆+3𝑚₆𝐴𝑚₂𝑚₆−𝑚²₄18−15𝑚₄𝐵−3𝑚²₂−9𝑚²₂𝐴+45𝑚²₂𝐵+𝑚₄+3𝑚₄𝐴𝑚₂𝑚₆−𝑚²₄=−62596𝐵+125+288125𝐴9685192𝐵−17−57617𝐴192 (C.7)which can be written as: 𝜆₂=−62596𝐵+125+288125𝜆96𝐴,₄=85192𝐵−17−57617192𝐴. (C.8)

By substituting 𝐴=0 and 𝐵=0 into (C.8), we get the Lagrange multipliers for the 16QAM input as appeared in [1].

D. Lagrange Multipliers Equations for 64QAM

For the 64QAM case we use three Lagrange multipliers (𝜆₂,𝜆₄,𝜆₆); thus, we have 𝐾=6.

In order to get the required Lagrange multipliers we recall (30) 𝑚_𝑘−2()2𝑘−1𝑘+()×1+3𝐴−15𝐵_{𝐾𝑙=2,𝑙≠𝑘}𝜆_𝑙𝑘𝑙𝑚_{𝑘+𝑙−2}+2𝜆_𝑘𝑘²𝑚_2𝑘−21+3𝐴−15𝐵=0, (D.1) where we use 𝐴 and 𝐵 from (B.6) and (B.7), respectively.

For 𝜆₂ we have 𝑘=2; thus, when substituting 𝑘=2 and 𝐾=6 into (30) we get 𝑚₀(2−1)2+2𝜆₂𝑚₄₋₂2²+1+3𝐴−15𝐵2𝜆₄𝑚₂₊₄₋₂⋅2⋅4+1+3𝐴−15𝐵2𝜆₆𝑚₂₊₆₋₂⋅2⋅6⇓1+3𝐴−15𝐵=02𝑚₀+8𝜆₂𝑚₂1+3𝐴−15𝐵+16𝜆₄𝑚₄1+3𝐴−15𝐵+24𝜆₆𝑚₆1+3𝐴−15𝐵=0. (D.2)

For 𝜆₄ we have 𝑘=4; thus, when substituting 𝑘=4 and 𝐾=6 into (30) we get 𝑚₄₋₂(4−1)2+2𝜆₄𝑚₈₋₂4²+1+3𝐴−15𝐵2𝜆₂𝑚₄₊₂₋₂⋅4⋅2+1+3𝐴−15𝐵2𝜆₆𝑚₄₊₆₋₂⋅4⋅6⇓1+3𝐴−15𝐵=06𝑚₂+32𝜆₄𝑚₆1+3𝐴−15𝐵+16𝜆₂𝑚₄1+3𝐴−15𝐵+48𝜆₆𝑚₈1+3𝐴−15𝐵=0. (D.3)

For 𝜆₆ we have 𝑘=6; thus, when substituting 𝑘=6 and 𝐾=6 into (30) we get 𝑚₆₋₂(6−1)6+2𝜆₆𝑚_2⋅6−26²+1+3𝐴−15𝐵2𝜆₂𝑚₆₊₂₋₂⋅6⋅2+1+3𝐴−15𝐵2𝜆₄𝑚₆₊₄₋₂⋅6⋅4⇓1+3𝐴−15𝐵=030𝑚₄+72𝜆₆𝑚₁₀1+3𝐴−15𝐵+24𝜆₂𝑚₆1+3𝐴−15𝐵+48𝜆₄𝑚₈1+3𝐴−15𝐵=0. (D.4)

From (D.2), (D.3), and (D.4) we get a linear system with three equations: 8𝜆₂𝑚₂1+3𝐴−15𝐵+16𝜆₄𝑚₄1+3𝐴−15𝐵+24𝜆₆𝑚₆1+3𝐴−15𝐵=−2𝑚₀,16𝜆₂𝑚₄1+3𝐴−15𝐵+32𝜆₆𝑚₆1+3𝐴−15𝐵+48𝜆₈𝑚₈1+3𝐴−15𝐵=6𝑚₂,24𝜆₂𝑚₆1+3𝐴−15𝐵+48𝜆₄𝑚₈1+3𝐴−15𝐵+72𝜆₆𝑚₁₀1+3𝐴−15𝐵=30𝑚₄. (D.5)

By substituting the known moments, 𝑚₂=21,𝑚₄=777,𝑚₆𝑚=33501,₈=1540497,𝑚₁₀=73074981, (D.6) into (D.5) we obtain 𝜆₂=−14174092745600𝐴−1417409+82368001417409𝜆549120𝐵,₄=13573+164736013573549120𝐴−13573𝜆109824𝐵,₆=−211−2246400211748800𝐴+211149760𝐵. (D.7)

E. Derivation of (𝑑/𝑑𝜆_𝑘)𝐸[(𝑔^′′(𝑥)/2𝑔(𝑥))_𝑝=0]=0

By using (B.8) and (B.9) we can write 𝐸𝑔^′′(𝑥)2𝑔(𝑥)_𝑝=0=𝐸−12𝐴/𝜎²_𝑝−90𝐵/𝜎²_𝑝exp∑^𝐾_𝑘=0𝜆_𝑘𝑥^𝑘2(1+3𝐴−15𝐵)exp∑^𝐾_𝑘=0𝜆_𝑘𝑥^𝑘[]×+𝐸1+3𝐴−15𝐵_𝐾𝑘=0𝜆_𝑘𝑥^𝑘−2𝑘²−𝜆_𝑘𝑥^𝑘−2𝑘exp^𝐾_𝑘=0𝜆_𝑘𝑥^𝑘×2(1+3𝐴−15𝐵)exp^𝐾_𝑘=0𝜆_𝑘𝑥^𝑘−1∑+𝐸^𝐾𝑘=0𝜆_𝑘𝑥^𝑘−1𝑘²exp∑^𝐾_𝑘=0𝜆_𝑘𝑥^𝑘2(1+3𝐴−15𝐵)exp∑^𝐾_𝑘=0𝜆_𝑘𝑥^𝑘=𝐸−12𝐴/𝜎²_𝑝−90𝐵/𝜎²_𝑝2()∑1+3𝐴−15𝐵+𝐸^𝐾𝑘=0𝜆_𝑘𝑥^𝑘−2𝑘²−𝜆_𝑘𝑥^𝑘−2𝑘2∑+𝐸^𝐾𝑘=0𝜆_𝑘𝑥^𝑘−1𝑘²2().1+3𝐴−15𝐵 (E.1)

Thus we have 𝑑𝑑𝜆_𝑘𝐸𝑔^′′(𝑥)2𝑔(𝑥)_𝑝=0=𝑑𝑑𝜆_𝑘𝐸∑^𝐾_𝑘=0𝜆_𝑘𝑥^𝑘−2𝑘²−𝜆_𝑘𝑥^𝑘−2𝑘2+𝑑𝑑𝜆_𝑘𝐸∑^𝐾_𝑘=0𝜆_𝑘𝑥^𝑘−1𝑘²2().1+3𝐴−15𝐵 (E.2)

Next we set (E.2) to zero and obtain 𝑚_𝑘−2𝑘(𝑘−1)2+12(1+3𝐴−15𝐵)^𝐾_{𝑙=2,𝑙≠𝑘}𝜆_𝑙𝑘𝑙𝑚_{𝑘+𝑙−2}+𝜆_𝑘𝑘²𝑚_2𝑘−22(1+3𝐴−15𝐵)=0, (E.3) or 𝑚_𝑘−2𝑘()+2𝑘−1()×1+3𝐴−15𝐵_{𝐾𝑙=2,𝑙≠𝑘}𝜆_𝑙𝑘𝑙𝑚_{𝑘+𝑙−2}+2𝜆_𝑘𝑘²𝑚_2𝑘−21+3𝐴−15𝐵=0. (E.4)

Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.

Acknowledgment

The authors would like to thank the reviewers for their helpful comments.