Research Letters in Signal Processing
Volume 2008 (2008), Article ID 790607, 5 pages
doi:10.1155/2008/790607
Research Letter

Generalized Cumulative Residual Entropy for Distributions with Unrestricted Supports

Noomane Drissi, Thierry Chonavel, and Jean Marc Boucher

Lab-STICC (CNRS FRE 3167), Institut Telecom, Telecom Bretagne, Technopole Brest Iroise, CS 83818, 29238 Brest Cédex, France

Received 6 April 2008; Accepted 19 June 2008

Academic Editor: Andreas Jakobsson

Copyright © 2008 Noomane Drissi et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

We consider the cumulative residual entropy (CRE) a recently introduced measure of entropy. While in previous works distributions with positive support are considered, we generalize the definition of CRE to the case of distributions with general support. We show that several interesting properties of the earlier CRE remain valid and supply further properties and insight to problems such as maximum CRE power moment problems. In addition, we show that this generalized CRE can be used as an alternative to differential entropy to derive information-based optimization criteria for system identification purpose.

1. Introduction

The concept of entropy is important for studies in many areas of engineering such as thermodynamics, mechanics, or digital communications. An early definition of a measure of the entropy is the Shannon entropy [1, 2]. In Shannon's approach, discrete values and absolutely continuous distributions are treated in a somewhat different way through entropy and differential entropy, respectively. Considering the complementary cumulative distribution function (CCDF) instead of the probability density function in the definition of differential entropy leads to a new entropy measure named cumulative residual entropy (CRE) [3, 4]. In [3, 4], CRE is defined as

ℰ ( 𝑋 ) = − ℝ 𝑛 + 𝑃 ( | 𝑋 | > 𝑢 ) l o g 𝑃 ( | 𝑋 | > 𝑢 ) 𝑑 𝑢 , ( 1 ) where 𝑛 is the dimension of the random vector 𝑋 . Clearly, this formula is valid both for a discrete or an absolutely continuous random variable (RV), or with both a discrete and an absolutely continuous part, because it resorts to the CCDF of | 𝑋 | . In addition, unlike Shannon differential entropy it is always positive, while preserving many interesting properties of Shannon entropy. The concept of CRE has found nice interpretations and applications in the fields of reliability (see [5] where the concept of dynamic CRE is introduced) and images alignment [3].

Shannon entropy can be seen as a particular case of exponential entropy, when entropy order tends to 1. Thus, following the work in [4], a modified version of the exponential entropy, where PDF is replaced by CCDF, has been introduced in [6], leading to new entropy-type measures, called survival entropies.

However, both Rao et al.'s CRE and its exponential entropy generalization by Zografos and Nadarajah lead to entropy-type definitions that assume either positive valued RVs or apply to | 𝑋 | otherwise. Although the positive case is of great interest for many applications, CRE and exponential entropies entail difficulties when working with RVs with supports that are not restricted to positive values.

In this paper, we show that for an RV 𝑋 , (1) remains a valid expression when 𝑃 ( | 𝑋 | > 𝑢 ) is replaced by 𝑃 ( 𝑋 > 𝑢 ) and integration is performed over ℝ 𝑛 , without further hypothesis than in [4]. In addition, some desirable properties are enabled by this CRE definition extension. We also complete the power moment constrained maximum CRE distributions problem that was adressed in [7], for classes of distributions that have lower-unbounded supports. Finally, we illustrate the potential superiority of the proposed generalized CRE (GCRE) against differential entropy in mutual information-based estimation problems.

The paper is organized as follows. Section 2 introduces the GCRE definition. Some properties of GCRE are discussed in Section 3. In Section 4, we introduce cumulative entropy rate and mutual information rate. Section 5 deals with maximum GCRE distributions. With a view to illustrate the potentiality of GCRE, in Section 6, we show on a simple example a possible benefit of GCRE for systems identification.

2. Generalized Cumulative Residual Entropy (GCRE)

We will denote by 𝐹 𝑐 𝑋 ( 𝑥 ) the complementary cumulative distribution function (survival function) of a multivariate RV 𝑋 = [ 𝑋 1 , … , 𝑋 𝑛 ] 𝑇 of dimension 𝑛 : 𝐹 𝑐 𝑋 ( 𝑋 ) = 𝑃 ( 𝑋 > 𝑥 ) = 𝑃 ( 𝑋 𝑖 > 𝑥 𝑖 , 𝑖 = 1 , … , 𝑛 ) . We denote by 𝐻 𝐶 ( 𝑋 ) the GCRE of 𝑋 that we define by

𝐻 𝐶 ( 𝑋 ) = − ℝ 𝑛 𝐹 𝑐 𝑋 ( 𝑢 ) l o g 𝐹 𝑐 𝑋 ( 𝑢 ) 𝑑 𝑢 . ( 2 ) Clearly, like the CRE, the GCRE is a positive and concave function of 𝐹 𝑐 𝑋 . In addition, existence of GCRE can be established without further assumption upon distribution than those assumed for the CRE in [4].

Theorem 1. 𝐻 𝐶 ( 𝑋 ) < ∞ if for some 𝑝 > 𝑛 , 𝐸 [ | 𝑋 | 𝑝 ] < ∞ .

Proof. First let us remark that from the proof of the existence of CRE in [4], it is sufficient to prove the result when 𝑋 is a scalar RV, that is 𝑛 = 1 , and for 𝑝 > 1 . Then, letting 𝑝 − 1 < 𝛼 < 1 , we use the following inequality:
− 𝐹 𝑐 𝑋 ( 𝑥 ) l o g 𝐹 𝑐 𝑋 𝑒 ( 𝑥 ) ≤ − 1 𝐹 1 − 𝛼 𝑐 𝑋 ( 𝑥 ) 𝛼 1 I ] 0 , ∞ [ + ( 𝑥 ) 1 − 𝐹 𝑐 𝑋 ( 𝑥 ) 1 I ] − ∞ , 0 ] ( 𝑥 ) , ( 3 ) where 1 I 𝐴 ( 𝑥 ) = 1 if 𝑥 ∈ 𝐴 and 1 I 𝐴 ( 𝑥 ) = 0 otherwise. The existence of ∫ ℝ ( 𝑒 − 1 / ( 1 − 𝛼 ) ) [ 𝐹 𝑐 𝑋 ( 𝑥 ) ] 𝛼 1 I [ 0 , ∞ [ ( 𝑥 ) 𝑑 𝑥 can be proven just as in [4]. Now, letting 𝑢 = − 𝑥 , we have
1 − 𝐹 𝑐 𝑋 ( 𝑥 ) 1 I ] − ∞ , 0 ] ( 𝑥 ) = 𝐹 𝑋 ( 𝑥 ) 1 I ] − ∞ , 0 ] ( 𝑥 ) ≤ 1 I [ − 1 , 0 [ ( 𝑡 ) + 𝐹 𝑋 ( 𝑥 ) 1 I ] − ∞ , − 1 [ ( 𝑥 ) ≤ 1 I ] 0 , 1 ] ( 𝑢 ) + 𝐹 𝑋 ( − 𝑢 ) 1 I ] 1 , ∞ ] ( 𝑢 ) ≤ 1 I ] 0 , 1 ] ( 𝑢 ) + 𝐹 − 𝑋 ( 𝑢 ) 1 I ] 1 , ∞ ] ( 𝑢 ) ≤ 1 I ] 0 , 1 ] ( 𝑢 ) + 𝐹 𝑐 | | − 𝑋 | | ( 𝑢 ) 1 I ] 1 , ∞ ] ( 𝑢 ) ≤ 1 I ] 0 , 1 ] ( 𝑢 ) + 𝑢 − 𝑝 𝐸 | | 𝑋 | | 𝑝 1 I ] 1 , ∞ ] ( 𝑢 ) . ( 4 ) Thus,
∫ ℝ 1 − 𝐹 𝑐 𝑋 ( 𝑥 ) 1 I ] − ∞ , 0 ] ≤ ∫ ( 𝑥 ) 𝑑 𝑥 ℝ 1 I ] 0 , 1 ] ( 𝑢 ) + 𝑢 − 𝑝 𝐸 | | 𝑋 | | 𝑝 1 I ] 1 , ∞ ] ∫ ( 𝑢 ) 𝑑 𝑢 ≤ 1 + ∞ 1 𝑢 − 𝑝 𝐸 | | 𝑋 | | 𝑝 𝑑 𝑢 < ∞ . ( 5 ) Finally, putting all pieces together one finally proves convergence of right-hand side of (2).

3. A Few Properties of GCRE

Let us now exhibit a few more interesting properties of the GCRE. First, it is easy to check that like Shannon entropy the GCRE remains constant with respect to variable translation: ∀ 𝑎 ∈ ℝ 𝑛 , 𝐻 𝐶 ( 𝑋 + 𝑎 ) = 𝐻 𝐶 ( 𝑋 ) . ( 6 ) In the same way, it is clear that ∀ 𝑎 ∈ ℝ + , 𝐻 𝐶 ( 𝑎 𝑋 ) = 𝑎 𝐻 𝐶 ( 𝑋 ) . ( 7 ) When 𝑎 < 0 , we do not have such a nice property. However, let us consider the important particular case where the distribution of 𝑋 has a symmetry of the form

∃ 𝜇 , ∀ 𝑥 , 𝐹 𝑐 𝑋 ( 𝜇 − 𝑥 ) = 1 − 𝐹 𝑐 𝑋 ( 𝜇 + 𝑥 ) . ( 8 ) In this case, we get the following result.

Theorem 2. For an RV 𝑋 that satisfies symmetry property (8), one has ∀ 𝑎 ∈ ℝ , 𝐻 𝐶 | | 𝑎 | | 𝐻 ( 𝑎 𝑋 ) = 𝐶 ( 𝑋 ) . ( 9 )

Proof. Since it is clear that for all 𝑎 ∈ ℝ + , 𝐻 𝐶 ( 𝑎 𝑋 ) = 𝑎 𝐻 𝐶 ( 𝑋 ) , we just have to check that 𝐻 𝐶 ( − 𝑋 ) = 𝐻 𝐶 ( 𝑋 ) , which can be established as follows: − 𝐻 𝐶 ( − 𝑋 ) = ℝ 𝐹 𝑐 − 𝑋 ( 𝑥 ) l o g 𝐹 𝑐 − 𝑋 = ( 𝑥 ) 𝑑 𝑥 ℝ 𝐹 𝑐 − 𝑋 ( 𝑥 − 𝜇 ) l o g 𝐹 𝑐 − 𝑋 = ( 𝑥 − 𝜇 ) 𝑑 𝑥 ℝ 𝐹 𝑋 ( − 𝑥 + 𝜇 ) l o g 𝐹 𝑋 = ( − 𝑥 + 𝜇 ) 𝑑 𝑥 ℝ 1 − 𝐹 𝑋 ( 𝑥 + 𝜇 ) l o g 1 − 𝐹 𝑋 = ( 𝑥 + 𝜇 ) 𝑑 𝑥 ℝ 𝐹 𝑐 𝑋 ( 𝑥 + 𝜇 ) l o g 𝐹 𝑐 𝑋 = ( 𝑥 + 𝜇 ) 𝑑 𝑥 ℝ 𝐹 𝑐 𝑋 ( 𝑥 ) l o g 𝐹 𝑐 𝑋 ( 𝑥 ) 𝑑 𝑥 = − 𝐻 𝐶 ( 𝑋 ) . ( 1 0 )

When the entries of vector 𝑋 are independent, it has been shown in [4] that if the 𝑋 𝑖 are nonnegative, then

𝐻 𝐶 ( 𝑋 ) = 𝑖 Π 𝑗 ≠ 𝑖 𝔼 𝑋 𝑗 𝐻 𝐶 𝑋 𝑖 . ( 1 1 ) However, this formula does not extend to RVs with distributions carried by ℝ 𝑛 because 𝐹 𝐶 ( 𝑋 ) can be integrated over ℝ 𝑛 + in general but never over ℝ 𝑛 . However, if the 𝑋 𝑖 s are independent and have lower bounded supports with respective lower bounds 𝑚 1 , … , 𝑚 𝑛 ,

𝐻 𝐶 ( 𝑋 ) = Π 𝑖 ] 𝑚 𝑖 , ∞ ] − 𝐹 𝑐 𝑋 ( 𝑥 ) l o g 𝐹 𝑐 𝑋 = ( 𝑥 ) 𝑑 𝑥 𝑖 Π 𝑗 ≠ 𝑖 𝔼 𝑋 𝑗 − 𝑚 𝑗 𝐻 𝐶 𝑋 𝑖 , ( 1 2 ) because ∞ 𝑚 𝑖 𝐹 𝑐 𝑋 𝑖 ( 𝑢 ) 𝑑 𝑢 = 𝑢 𝐹 𝑐 𝑋 𝑖 ( 𝑢 ) ∞ 𝑚 𝑖 + ∞ 𝑚 𝑖 𝑢 𝑃 𝑋 𝑖 ( 𝑑 𝑢 ) = − 𝑚 𝑖 𝑋 + 𝔼 𝑖 . ( 1 3 )

Conditional GCRE definition is a direct extension of the definition of conditional CRE: the conditional GCRE of 𝑋 knowing that 𝑌 is equal to 𝑦 is defined by

𝐻 𝐶 | | = ∫ ( 𝑋 𝑌 = 𝑦 ) ℝ 𝑛 | | | | − 𝑃 ( 𝑋 > 𝑥 𝑌 = 𝑦 ) l o g 𝑃 ( 𝑋 > 𝑥 𝑌 = 𝑦 ) 𝑑 𝑥 . ( 1 4 ) We recall an important result from [4] that states that conditioning reduces the entropy.

Theorem 3. For any 𝑋 and 𝑌 , 𝐻 𝐶 | | ( 𝑋 𝑌 ) ≤ 𝐻 𝐶 ( 𝑋 ) ( 1 5 ) equality holds if and only if 𝑋 is independent of 𝑌 .

As a consequence, if 𝑋 → 𝑌 → 𝑍 is a Markov chain, we have the data processing inequality for GCRE: 𝐻 𝐶 | | ( 𝑍 𝑋 , 𝑌 ) ≤ 𝐻 𝐶 | | ( 𝑍 𝑋 ) . ( 1 6 )

4. Entropy and Mutual Information Rates

4.1. Entropy Rate

The GCRE of a stochastic process { 𝑋 𝑖 } is defined by 𝐻 𝐶 ( 𝑋 ) = l i m 𝑛 → ∞ 𝐻 𝐶 𝑋 𝑛 | | | 𝑋 𝑛 − 1 , 𝑋 𝑛 − 2 , … , 𝑋 1 ( 1 7 ) when the limit exists.

Theorem 4. For stationary processes, the limit exists.

Proof. Consider 𝐻 𝐶 𝑋 𝑛 + 1 | | | 𝑋 𝑛 , … , 𝑋 1 ≤ 𝐻 𝐶 𝑋 𝑛 + 1 | | | 𝑋 𝑛 , … , 𝑋 2 ≤ 𝐻 𝐶 𝑋 𝑛 | | | 𝑋 𝑛 − 1 , … , 𝑋 1 . ( 1 8 ) The first line follows from the fact that conditioning reduces entropy and the second follows from the stationarity (see [2] for the equivalent proof in the case of Shannon entropy).

4.2. Mutual Information

Let 𝑋 and 𝑌 be two RVs. We define the cumulative mutual information between 𝑋 and 𝑌 as follows: 𝐼 𝐶 ( 𝑋 ; 𝑌 ) = 𝐻 𝐶 ( 𝑋 ) − 𝐻 𝐶 | | ( 𝑋 𝑌 ) . ( 1 9 )

Theorem 5. 𝐼 𝐶 is nonnegative and it vanishes if and only if 𝑋 and 𝑌 are independent.

Proof. It is clear that 𝐼 𝐶 is nonnegative because of Theorem 3.

For a random vector 𝐗 = ( 𝑋 1 , 𝑋 2 , … , 𝑋 𝑛 ) of size 𝑛 , mutual information is defined by 𝐼 𝐶 ( 𝐗 ) = 𝑛 𝑖 = 1 𝐻 𝐶 𝑋 𝑖 − 𝐻 𝐶 𝑋 𝑛 | | | 𝑋 𝑛 − 1 , … , 𝑋 1 . ( 2 0 ) In the case of stochastic processes { 𝑋 𝑖 } , we have 𝐻 𝐶 ( 𝑋 ) = l i m 𝑛 → ∞ 𝐻 𝐶 ( 𝑋 𝑛 | 𝑋 𝑛 − 1 , … , 𝑋 1 ) and the limit exists for stationary processes. Then the mutual information rate for { 𝑋 𝑖 } is defined as 𝐼 𝐶 ( 𝑋 ) = l i m 𝑇 → ∞ 1 𝑇 𝑇 𝑡 = 1 𝐻 𝐶 𝑋 𝑡 − 𝐻 𝐶 ( 𝑋 ) , ( 2 1 ) where 𝐻 𝐶 ( 𝑋 𝑡 ) is the marginal GCRE of the process 𝑋 .

5. Maximum GCRE Distributions

In this section, we only consider the case of one-dimensional RVs ( 𝑛 = 1). Maximum entropy principle is useful in many scientific areas and most important distributions can be derived from it [8]. The maximum CRE distribution has been studied in [7]. For an RV 𝑋 with a symmetric CCDF in the sense of (8), we are looking for the maximum GCRE distribution, that is, the CCDF that solves the following moment problem:

m a x 𝐹 𝑐 𝐻 𝐶 𝐹 𝑐 ∫ ℝ 𝑟 𝑖 ( 𝑥 ) 𝑝 ( 𝑥 ) 𝑑 𝑥 = 𝑐 𝑖 , 𝑖 = 1 , … , 𝑚 , ( 2 2 ) where 𝑝 ( 𝑥 ) = − ( 𝑑 / 𝑑 𝑥 ) 𝐹 𝑐 ( 𝑥 ) , ( 𝑟 𝑖 ) 𝑖 = 1 , 𝑚 , and ( 𝑐 𝑖 ) 𝑖 = 1 , 𝑚 are fixed 𝐶 1 real valued functions and real coefficients, respectively. The solution of this problem is supplied by the following result.

Theorem 6. When the symmetry property (8) holds, the solution of problem (22), when it can be reached, is of the form 𝐹 𝑐 1 ( 𝑥 ) = ∑ 1 + e x p 𝑚 1 𝜆 𝑖 𝑟 𝑖 ( 𝑥 − 𝜇 ) ( 𝑥 ≥ 𝜇 ) . ( 2 3 )

Proof. − 𝐻 𝐶 ( 𝑋 ) = 𝜇 − ∞ 𝐹 𝑐 𝐹 ( 𝑥 ) l o g 𝑐 + ( 𝑥 ) 𝑑 𝑥 ∞ 𝜇 𝐹 𝑐 𝐹 ( 𝑥 ) l o g 𝑐 = ( 𝑥 ) 𝑑 𝑥 ℝ − 𝐹 𝑐 𝐹 ( 𝑥 + 𝜇 ) l o g 𝑐 + ( 𝑥 + 𝜇 ) 𝑑 𝑥 ℝ + 𝐹 𝑐 𝐹 ( 𝑥 + 𝜇 ) l o g 𝑐 = ( 𝑥 + 𝜇 ) 𝑑 𝑥 ℝ − 1 − 𝐹 𝑐 ( 𝜇 − 𝑥 ) l o g 1 − 𝐹 𝑐 + ( 𝜇 − 𝑥 ) 𝑑 𝑥 ℝ + 𝐹 𝑐 𝐹 ( 𝑥 + 𝜇 ) l o g 𝑐 = ( 𝑥 + 𝜇 ) 𝑑 𝑥 ℝ + 1 − 𝐹 𝑐 ( 𝜇 + 𝑥 ) l o g 1 − 𝐹 𝑐 ( 𝜇 + 𝑥 ) + 𝐹 𝑐 𝐹 ( 𝜇 + 𝑥 ) l o g 𝑐 ( 𝜇 + 𝑥 ) 𝑑 𝑥 . ( 2 4 ) Let us define 𝑓 by 𝑓 𝐹 𝑐 ( 𝑥 ) , − 𝑝 = − 1 − 𝐹 𝑐 ( 𝜇 + 𝑥 ) l o g 1 − 𝐹 𝑐 ( 𝜇 + 𝑥 ) − 𝐹 𝑐 𝐹 ( 𝜇 + 𝑥 ) l o g 𝑐 + ( 𝜇 + 𝑥 ) 𝑚 ∑ 1 𝜆 𝑖 𝑟 − 𝑝 ( 𝑥 ) 𝑖 ( 𝑥 ) . ( 2 5 ) Then, since ( 𝐹 𝑐 ( 𝑥 ) ) = − 𝑝 ( 𝑥 ) , the Euler-Lagrange equation [9] states that the solution 𝐹 𝑐 of problem (22) is a solution of equation 𝑑 𝑓 𝑑 𝑥 − 𝑝 𝑓 ( 𝑥 ) = 𝐹 𝑐 ( 𝑥 ) , ( 2 6 ) where 𝑓 𝑢 is the partial derivative of 𝑓 with respect to component 𝑢 . From (25), we get 𝑑 𝑓 𝑑 𝑥 − 𝑝 ( 𝑥 ) = 𝑚 ∑ 1 𝜆 𝑖 𝑟 𝑖 𝑓 ( 𝑥 ) , 𝐹 𝑐 ( 𝑥 ) = l o g 1 − 𝐹 𝑐 ( 𝜇 + 𝑥 ) − l o g 𝐹 𝑐 ( 𝜇 + 𝑥 ) . ( 2 7 ) Then, l o g 1 − 𝐹 𝑐 ( 𝜇 + 𝑥 ) 𝐹 𝑐 = ( 𝜇 + 𝑥 ) 𝑚 ∑ 1 𝜆 𝑖 𝑟 𝑖 𝐹 ( 𝑥 ) , 𝑐 1 ( 𝑥 ) = ∑ 1 + e x p 𝑚 1 𝜆 𝑖 𝑟 𝑖 , ( 𝑥 − 𝜇 ) ( 2 8 ) for 𝑥 ∈ [ 𝜇 , ∞ [ .

5.1. Example

We set the constraints 𝔼 [ 𝑋 ] = 𝜇 and 𝔼 [ 𝑋 2 ] = 𝜎 2 . Then the maximum GCRE symmetric solution for the CCDF of X is given by 𝐹 𝑐 𝑋 1 ( 𝑥 ) = 𝜆 1 + e x p 1 + 2 𝜆 2 𝑥 , ( 2 9 ) for 𝑥 > 0 , which is the CCDF of a logistic distribution. The moment constraints lead to 𝜆 1 + 2 𝜆 2 √ 𝑥 = ( 𝜎 3 / 𝜋 ) ( 𝑥 − 𝜇 ) . The corresponding PDF is defined on ℝ by 𝑝 𝑋 𝜎 √ ( 𝑥 ) = 3 𝜋 √ e x p − ( 𝜎 3 / 𝜋 ) ( 𝑥 − 𝜇 ) √ 1 + e x p − ( 𝜎 3 / 𝜋 ) ( 𝑥 − 𝜇 ) 2 . ( 3 0 )

5.2. Positive Random Variables

It has been shown in [7] that the maximum CRE (i.e., the maximum GCRE under additional nonnegative constraint) distribution has CCDF in the form 𝐹 𝐶 ( 𝑥 ) = e x p ( − 𝑚 𝑖 = 1 𝜆 𝑖 𝑟 𝑖 ( 𝑥 ) ) , ( 3 1 ) for 𝑥 ∈ [ 0 , ∞ [ . In [7], this result is derived from the log-sum inequality, but of course it can also be derived from the Euler-Lagrange equation along the same lines as in the proof of Theorem 6.

With a positive support constraint and under first and second moment constraints, it comes that the optimum CCDF is of the form 𝐹 𝑐 ( 𝑥 ) = e x p ( − 𝜆 1 − 2 𝜆 2 𝑥 ) , for 𝑥 > 0 . Thus the solution, if it exists, is an exponential distribution. In fact, the first and second power moment constraints must be such that 𝔼 [ 𝑋 2 ] = 2 ( 𝔼 [ 𝑋 ] ) 2 , otherwise the problem has no exact solution.

6. Simulation Results

With a view to emphasize the potential practical interest of GCRE, we consider a simple system identification problem. Here, we consider an 𝑀 𝐴 ( 1 ) process, denoted by 𝑌 = ( 𝑌 𝑛 ) 𝑛 ∈ ℤ , generated by a white noise 𝑋 = ( 𝑋 𝑛 ) 𝑛 ∈ ℤ and corrupted by a white noise 𝑁 : 𝑌 𝑛 = 𝑋 𝑛 − 𝑎 𝑋 𝑛 − 1 + 𝑁 𝑛 . ( 3 2 ) The model input 𝑋 and output 𝑌 are observed and the system model ( 𝑀 𝐴 ( 1 ) ) is assumed to be known. We want to estimate the coefficient 𝑎 without prior knowledge upon the distributions of 𝑋 and 𝑌 . Thus, we resort to mutual information (MI) to estimate 𝑎 as the coefficient 𝛼 such that RVs 𝑌 𝛼 𝑛 = 𝑋 𝑛 − 𝛼 𝑋 𝑛 − 1 and 𝑌 𝑛 show the highest dependence. Shannon MI between 𝑌 𝛼 𝑛 and 𝑌 𝑛 is given by 𝑓 𝑆 ( 𝛼 ) = 𝐼 𝑆 ( 𝑌 𝛼 𝑛 , 𝑌 𝑛 ) = 𝐻 𝑆 ( 𝑌 𝛼 𝑛 ) − 𝐻 𝑆 ( 𝑌 𝑛 | 𝑌 𝑛 ) , where 𝐻 𝑆 is Shannon differential entropy. Similarly, for GCRE, MI will be defined as 𝑓 𝐶 ( 𝛼 ) = 𝐼 𝐶 ( 𝑌 𝛼 𝑛 , 𝑌 ) = 𝐻 𝐶 ( 𝑌 𝛼 𝑛 ) − 𝐻 𝐶 ( 𝑌 𝛼 𝑛 | 𝑌 𝑛 ) . We compare estimation performance for 𝑎 by maximizing both 𝑓 𝑆 ( 𝛼 ) and 𝑓 𝐶 ( 𝛼 ) . Since true values of 𝑓 𝑆 ( 𝛼 ) and 𝑓 𝐶 ( 𝛼 ) are not available, they are estimated from empirical distributions of ( 𝑌 𝑛 , 𝑌 𝛼 𝑛 ) .

For simulations, we have chosen 𝑋 Gaussian and 𝑁 with a Laplace distribution: 𝑝 𝑁 ( 𝑥 ) = ( 𝜆 / 2 ) e x p ( − 𝜆 | 𝑥 | ) . We consider an experiment with 𝑎 = 0 . 5 and noise variance equal to 0.2. Estimation is carried out from observation of ( 𝑋 𝑛 , 𝑌 𝑛 ) 𝑛 = 1 , 4 0 0 . Here, optimization of MIs is realized on a fixed regular grid of 200 points over interval [0,1]. Estimation performance is calculated from 200 successive experiments. Estimation of 𝑎 from Shannon MI leads to bias and standard deviation that are equal to 0.032 and 0.18, respectively, while they are equal to 0.004 and 0.06, respectively, for GCRE MI.

More important, we see on Figure 1(a) that Shannon MI estimates are much more irregular than GCRE MI (Figure 1(b)) estimates because of smoothing brought by density integration in the calculation of CCDF. This difference is important since the use of an iterative local optimization technique would have failed in general to find Shannon's estimated MI global optimum, because of its many local maxima.

fig1
Figure 1: Ten estimates of (a) 𝑓 𝑆 ( 𝛼 ) and (b) 𝑓 𝐶 ( 𝛼 ) . Dotted line: mean estimate averaged from 200 realizations.

Of course, this drawback can be partly solved by kernel smoothing of the empirical distribution of ( 𝑌 𝑛 , 𝑌 𝛼 𝑛 ) , for instance by using the method proposed in [10]. However, we have checked that, for the above example, very strong smoothing is necessary and then bias and variance performance remain worse than with GCRE MI estimator.

7. Conclusion

We have shown that the concept of cumulative residual entropy (CRE) introduced in [3, 4] can be extended to distributions with general supports. Generalized CRE (GCRE) shares many nice features of CRE. We also pointed out specific properties of GCRE such as its maximum, moment constrained, and distribution and we have illustrated practical interest of GCRE by showing how it can be used in system identification procedures.

References

  1. C. E. Shannon, “A mathematical theory of communication,” Bell System Technical Journal, vol. 27, pp. 379–423, 1948.
  2. J. M. Cover and J. H. Thomas, Elements of Information Theory, Wiley Interscience, New York, NY, USA, 2006.
  3. F. Wang, B. C. Vemuri, M. Rao, and Y. Chen, “A new & robust information theoretic measure and its application to image alignment,” in Proceedings of the 18th International Conference on Information Processing in Medical Imaging (IPMI '03), vol. 2732 of Lecture Notes in Computer Science, pp. 388–400, Springer, Ambleside, UK, July 2003.
  4. M. Rao, Y. Chen, B. C. Vemuri, and F. Wang, “Cumulative residual entropy: a new measure of information,” IEEE Transactions on Information Theory, vol. 50, no. 6, pp. 1220–1228, 2004.
  5. M. Asadi and Y. Zohrevand, “On the dynamic cumulative residual entropy,” Journal of Statistical Planning and Inference, vol. 137, no. 6, pp. 1931–1941, 2007.
  6. K. Zografos and S. Nadarajah, “Survival exponential entropies,” IEEE Transactions on Information Theory, vol. 51, no. 3, pp. 1239–1246, 2005.
  7. M. Rao, “More on a new concept of entropy and information,” Journal of Theoretical Probability, vol. 18, no. 4, pp. 967–981, 2005.
  8. J. N. Kapur, Maximum Entropy Methods in Science and Engineering, John Wiley & Sons, New York, NY, USA, 1989.
  9. J. L. Troutman, Variational Calculus and Optimal Control, Optimization with Elementary Convexity, Springer, New York, NY, USA, 2nd edition, 1996.
  10. D. T. Pham, “Fast algorithm for estimating mutual information, entropies and score functions,” in Proceedings of the 4th International Symposium on Independent Component Analysis and Blind Signal Separation (ICA '03), pp. 17–22, Nara, Japan, April 2003.