A Strong Limit Theorem for Functions of Continuous Random Variables and an Extension of the Shannon-McMillan Theorem

Abstract

By means of the notion of likelihood ratio, the limit properties of the sequences of arbitrary-dependent continuous random variables are studied, and a kind of strong limit theorems represented by inequalities with random bounds for functions of continuous random variables is established. The Shannon-McMillan theorem is extended to the case of arbitrary continuous information sources. In the proof, an analytic technique, the tools of Laplace transform, and moment generating functions to study the strong limit theorems are applied.

1. Introduction

Let {𝑋_𝑛,𝑛≥1} be a sequence of arbitrary continuous real random variables on the probability space (Ω,ℱ,𝑃) with the joint density function𝑓_𝑛𝑥₁,…,𝑥_𝑛>0,𝑛=1,2,…, (1.1)where 𝑥_𝑖∈(−∞,∞), 1≤𝑖≤𝑛. Let 𝑄 be another probability measure on ℱ, and {𝑋_𝑛,𝑛≥1} is a sequence of independent random variables on the probability space (Ω,ℱ,𝑄) with the marginal density functions 𝑔_𝑘(𝑥_𝑘) (1≤𝑘≤𝑛), and let𝜋_𝑛𝑥₁,…,𝑥_𝑛=^𝑛_𝑘=1𝑔_𝑘𝑥_𝑘. (1.2) In order to indicate the deviation between {𝑋_𝑛,𝑛≥1} on the probability measure 𝑃 and 𝑄, we first introduce the following definitions.Definition 1.1. Let {𝑋_𝑛,𝑛≥1} be a sequence of random variables with joint distribution (1.1), and let 𝑔_𝑘(𝑥_𝑘) (𝑘=1,2,…,𝑛) be defined by (1.2). Let 𝑍_𝑛𝑓(𝜔)=^𝑛𝑋₁,…,𝑋_𝑛𝜋_𝑛𝑋₁,…,𝑋_𝑛. (1.3)In statistical terms, 𝑍_𝑛(𝜔) is called the likelihood ratio, which is of fundamental importance in the theory of testing the statistical hypotheses (cf. [1, page 388]; [2, page 483]).

The random variable𝑟(𝜔)=limsup_𝑛→∞1𝑛ln𝑍_𝑛(𝜔)=−liminf_𝑛→∞1𝑛ln^𝑛_𝑘=1𝑔_𝑘𝑋_𝑘/𝑓_𝑛𝑋₁,…,𝑋_𝑛 (1.4)is called asymptotic logarithmic likelihood ratio, relative to the product of marginal distribution of (1.2), of 𝑋_𝑛, 𝑛≥1, where ln is the natural logarithm, 𝜔 is the sample point. For the sake of brevity, we denote 𝑋_𝑘(𝜔) by 𝑋_𝑘.

Although 𝑟(𝜔) is not a proper metric between probability measures, we nevertheless think of it as a measure of “dissimilarity” between their joint distribution 𝑓_𝑛(𝑥₁,…,𝑥_𝑛) and the product 𝜋_𝑛(𝑥₁,…,𝑥_𝑛) of their marginals.

Obviously, 𝑟(𝜔)=0, a.s. if and only if {𝑋_𝑛,𝑛≥1} are independent.

A stochastic process of fundamental importance in the theory of testing hypotheses is the sequence of likelihood ratio. In view of the above discussion of the asymptotic logarithmic likelihood ratio, it is natural to think of 𝑟(𝜔) as a measure how far (the random deviation of) 𝑋_𝑛 is from being independent, how dependent they are. The smaller 𝑟(𝜔) is, the smaller the deviation is (cf. [3–5]).

In [3], the strong deviation theorems for discrete random variables were discussed by using the generating function method. Later, the approach of Laplace transform to study the strong limit theorems was first proposed by Liu [4]. Yang [6] further studied the limit properties for Markov chains indexed by a homogeneous tree through the analytic technique. Many comprehensive works may be found in Liu [7]. The purpose of this paper is to establish a kind of strong deviation theorems represented by inequalities with random bounds for functions of arbitrary continuous random variables, by combining the analytic technique with the method of Laplace transform, and to extend the strong deviation theorems to the differential entropy for arbitrary-dependent continuous information sources in more general settings.Definition 1.2. Let {ℎ_𝑛(𝑥_𝑛),𝑛≥1} be a sequence of nonnegative ℬ orel measurable functions defined on ℛ, the Laplace transform of random variables ℎ_𝑛(𝑋_𝑛) on the probability space (Ω,ℱ,𝑄) is defined by𝑓_𝑛(𝑠)=^∞_−∞𝑒^{−𝑠ℎ𝑛(𝑥𝑛)}𝑔_𝑛𝑥_𝑛𝑑𝑥_𝑛=𝐸_𝑄𝑒^{−𝑠ℎ𝑛(𝑋𝑛)},𝑛=1,2,…, (1.5)where 𝐸_𝑄 denotes the expectation under 𝑄.

We have the following assumptions in this paper.(1) Assume that there exists 𝑠₀∈(0,∞) such that𝑓_𝑛(𝑠)<∞,𝑠∈−𝑠₀,𝑠₀,𝑛=1,2,…. (1.6)(2) Assume 𝑀>0 is a constant, satisfyingsup_𝑛𝐸_𝑄ℎ_𝑛𝑋_𝑛≤𝑀<∞. (1.7)

In order to prove our main results, we first give a lemma, and it will be shown that it plays a central role in the proofs.Lemma 1.3. Let 𝑓_𝑛(𝑥₁,…,𝑥_𝑛), 𝑔_𝑛(𝑥₁,…,𝑥_𝑛) be two probability functions on (Ω,ℱ,𝑃), let𝑇_𝑛𝑔(𝜔)=^𝑛𝑋₁,…,𝑋_𝑛𝑓_𝑛𝑋₁,…,𝑋_𝑛, (1.8)thenlimsup_𝑛→∞1𝑛ln𝑇_𝑛(𝜔)≤0a.s. (1.9)

Proof. By [8], {𝑇_𝑛,ℱ,𝑛≥1} is a nonnegative martingale and 𝐸𝑇_𝑛=1, we have by the Doob martingale convergence theorem, there exists an integral random variable 𝑇_∞(𝜔), such that 𝑇_𝑛→𝑇_∞, a.s. and (1.9) follows.

2. Main Results

Theorem 2.1. Let {𝑋_𝑛,𝑛≥1}, 𝑍_𝑛(𝜔), 𝑟(𝜔), 𝑓_𝑛(𝑠) be defined as before, and under the assumptions of (1) and (2), let limsup_𝑛→∞1𝑛^𝑛_𝑘=1𝐸_𝑄𝑒^{𝑠0ℎ𝑘(𝑋𝑘)}=limsup_𝑛→∞1𝑛^𝑛_𝑘=1𝑓_𝑘−𝑠₀=𝑀𝑠₀<∞. (2.1)Thenliminf_𝑛→∞1𝑛^𝑛_𝑘=1ℎ_𝑘𝑋_𝑘−𝐸_𝑄ℎ_𝑘𝑋_𝑘≥−𝛽𝑟(𝜔),a.s.limsup_𝑛→∞1𝑛^𝑛_𝑘=1ℎ_𝑘𝑋_𝑘−𝐸_𝑄ℎ_𝑘𝑋_𝑘≤𝛽𝑟(𝜔),a.s. (2.2)(2.3)where𝛽(𝑥)=inf𝜙(𝑠,𝑥),−𝑠₀<𝑠<0,𝑥≥0,𝜙(𝑠,𝑥)=−2𝑒⁻²𝑠𝑀𝑠₀𝑠₀−|𝑠|²−𝑥𝑠√,𝑥≥0,0≤𝛽(𝑥)≤𝑥1+√𝑥𝑠₀2𝑒⁻²𝑀𝑠₀+1,lim_𝑥→0+𝛽(𝑥)=𝛽(0)=0. (2.4)(2.5)(2.6)Remark 2.2. Let𝛼(𝑥)=sup𝜙(𝑠,𝑥),0<𝑠<𝑠₀,𝑥≥0, (2.7)then𝛼(𝑥)=−𝛽(𝑥). (2.8)

Proof. Let 𝑠 be an arbitrary real number in (−𝑠₀,𝑠₀), let𝑔_𝑛𝑠,𝑥_𝑛=𝑒^{−𝑠ℎ𝑛(𝑥𝑛)}𝑔_𝑛𝑥_𝑛/𝑓_𝑛(𝑠),𝑥_𝑛∈(−∞,∞),𝑛=1,2,…, (2.9)then ∫^∞_−∞𝑔_𝑛(𝑠,𝑥_𝑛)𝑑𝑥_𝑛=1, and let𝑞_𝑛𝑠;𝑥₁,…,𝑥_𝑛=^𝑛_𝑘=1𝑔_𝑘𝑠,𝑥_𝑘. (2.10)Therefore, 𝑞_𝑛(𝑠;𝑥₁,…,𝑥_𝑛) is an 𝑛 multivariate probability density function, let𝑡_𝑛𝑞(𝑠,𝜔)=^𝑛𝑠;𝑋₁,…,𝑋_𝑛𝑓_𝑛𝑋₁,…,𝑋_𝑛,𝑛=1,2,…. (2.11)By Lemma 1.3, there exists a set 𝐴(𝑠) such that 𝑃(𝐴(𝑠))=1, so we havelimsup_𝑛→∞1𝑛ln𝑡_𝑛(𝑠,𝜔)≤0,𝜔∈𝐴(𝑠). (2.12)By (1.3), (2.9), (2.11), and (2.12), we havelimsup_𝑛→∞1𝑛−𝑠^𝑛_𝑘=1ℎ_𝑘𝑋_𝑘−^𝑛_𝑘=1ln𝑓_𝑘(𝑠)−ln𝑍_𝑛(𝜔)≤0,𝜔∈𝐴(𝑠). (2.13)Therefore,𝑟(𝜔)≥0,𝜔∈𝐴(0). (2.14)By (2.13) and (1.4), the property of the superior limitlimsup_𝑛→∞𝑎_𝑛−𝑏_𝑛≤0⟹limsup_𝑛→∞𝑎_𝑛≤limsup_𝑛→∞𝑏_𝑛, (2.15)and the inequality ln𝑥≤𝑥−1 (𝑥>0), we havelimsup_𝑛→∞1𝑛(−𝑠)^𝑛_𝑘=1ℎ_𝑘𝑋_𝑘−𝐸_𝑄ℎ_𝑘𝑋_𝑘≤limsup_𝑛→∞1𝑛^𝑛_𝑘=1ln𝑓_𝑘(𝑠)+𝑠𝐸_𝑄ℎ_𝑘𝑋_𝑘+𝑟(𝜔)≤limsup_𝑛→∞1𝑛^𝑛_𝑘=1𝑓_𝑘(𝑠)−1+𝑠𝐸_𝑄ℎ_𝑘𝑋_𝑘+𝑟(𝜔)=limsup_𝑛→∞1𝑛^𝑛_𝑘=1𝑓_𝑘(𝑠)−1−(−𝑠)𝐸_𝑄ℎ_𝑘𝑋_𝑘+𝑟(𝜔),𝜔∈𝐴(𝑠). (2.16)By the inequality 0≤𝑒^𝑥−1−𝑥≤(1/2)𝑥²𝑒^|𝑥|, which can be found in [9], we have0≤𝑒^{−𝑠ℎ𝑘(𝑋𝑘)}−1−(−𝑠)ℎ_𝑘(𝑋_𝑘)≤(1/2)𝑠ℎ_𝑘𝑋_𝑘²𝑒^{|𝑠ℎ𝑘(𝑋𝑘)|}. (2.17)By (2.5) and (2.17), we havelimsup_𝑛→∞1𝑛(−𝑠)^𝑛_𝑘=1ℎ_𝑘𝑋_𝑘−𝐸_𝑄ℎ_𝑘𝑋_𝑘≤(1/2)𝑠²limsup_𝑛→∞1𝑛^𝑛_𝑘=1𝐸_𝑄ℎ_𝑘𝑋_𝑘²𝑒^{|𝑠|ℎ𝑘(𝑋𝑘)}+𝑟(𝜔),𝜔∈𝐴(𝑠). (2.18) It is easy to see that 𝜑(𝑥)=𝑡^𝑥𝑥² (𝑡>1) attains its largest value 𝜑(−2/ln𝑡)=4𝑒⁻²/(ln𝑡)² on the interval (−∞,0], and 𝜑(𝑥)=𝑡^𝑥𝑥² (0<𝑡<1) attains its largest value 𝜑(−2/ln𝑡)=4𝑒⁻²/(ln𝑡)² on the interval [0,∞), we havesup𝑒^{(𝑠−𝑠0)ℎ𝑘(𝑋𝑘)}ℎ_𝑘𝑋_𝑘²,𝑘≥1≤4𝑒⁻²𝑠−𝑠₀²=4𝑒⁻²𝑠₀−𝑠²,0<𝑠<𝑠₀,sup𝑒^{(𝑠0+𝑠)(−ℎ𝑘(𝑋𝑘))}−ℎ_𝑘𝑋_𝑘²,𝑘≥1≤4𝑒⁻²𝑠₀+𝑠²,−𝑠₀<𝑠<0. (2.19)(2.20)Let 0<𝑠<𝑠₀ in (2.18), by (2.19) and (2.1), we obtain(−𝑠)liminf_𝑛→∞1𝑛^𝑛_𝑘=1ℎ_𝑘𝑋_𝑘−𝐸_𝑄ℎ_𝑘𝑋_𝑘≤(1/2)𝑠²limsup_𝑛→∞1𝑛^𝑛_𝑘=1𝐸_𝑄ℎ_𝑘𝑋_𝑘²𝑒^{|𝑠|ℎ𝑘(𝑋𝑘)}+𝑟(𝜔)=(1/2)𝑠²limsup_𝑛→∞1𝑛^𝑛_𝑘=1𝐸_𝑄𝑒^{𝑠0ℎ𝑘(𝑋𝑘)}𝑒^{(𝑠−𝑠0)ℎ𝑘(𝑋𝑘)}ℎ_𝑘𝑋_𝑘²≤+𝑟(𝜔)2𝑒⁻²𝑠²𝑀𝑠₀𝑠₀−𝑠²+𝑟(𝜔),𝜔∈𝐴(𝑠). (2.21)Dividing the two sides of (2.21) by −𝑠, we obtainliminf_𝑛→∞1𝑛^𝑛_𝑘=1ℎ_𝑘𝑋_𝑘−𝐸_𝑄ℎ_𝑘𝑋_𝑘≥−2𝑒⁻²𝑠𝑀𝑠₀𝑠₀−𝑠²−𝑟(𝜔)𝑠=𝜙𝑠,𝑟(𝜔),0<𝑠<𝑠₀,𝜔∈𝐴(𝑠)∩𝐴(0). (2.22)By (2.14) and 0<𝑠<𝑠₀, obviously 𝜙(𝑠,𝑟(𝜔))≤0, hence 𝛼(𝑟(𝜔))≤0. Let 𝑄⁺ be the set of rational numbers in the interval (0,𝑠₀), and let 𝐴^∗=∩_𝑠∈𝑄⁺𝐴(𝑠), then 𝑃(𝐴^∗)=1. By (2.22), then we haveliminf_𝑛→∞1𝑛^𝑛_𝑘=1ℎ_𝑘𝑋_𝑘−𝐸_𝑄ℎ_𝑘𝑋_𝑘≥𝜙𝑠,𝑟(𝜔),𝜔∈𝐴^∗∩𝐴(0),∀𝑠∈𝑄⁺. (2.23) It is easy to see that 𝜙(𝑠,𝑥) is a continuous function with respect to 𝑠 on the interval (0,𝑠₀). For each 𝜔∈𝐴^∗∩𝐴(0) (0≤𝑟(𝜔)<∞), take 𝑠_𝑛(𝜔)∈𝑄⁺, 𝑛=1,2,…, such thatlim_𝑛→∞𝜙𝑠_𝑛(𝜔),𝑟(𝜔)=𝛼𝑟(𝜔). (2.24)By (2.23), (2.24), and (2.8), we haveliminf_𝑛→∞1𝑛^𝑛_𝑘=1ℎ_𝑘𝑋_𝑘−𝐸_𝑄ℎ_𝑘𝑋_𝑘≥−𝛽𝑟(𝜔),𝜔∈𝐴^∗∩𝐴(0). (2.25)Since 𝑃(𝐴^∗∩𝐴(0))=1, (2.2) follows from (2.25).
Let −𝑠₀<𝑠<0 in (2.18), by (2.20) and (2.1), we havelimsup_𝑛→∞1𝑛^𝑛_𝑘=1ℎ_𝑘𝑋_𝑘−𝐸_𝑄ℎ_𝑘𝑋_𝑘≤−(1/2)𝑠limsup_𝑛→∞1𝑛^𝑛_𝑘=1𝐸_𝑄ℎ_𝑘𝑋_𝑘²𝑒^{|𝑠|ℎ𝑘(𝑋𝑘)}−𝑟(𝜔)𝑠=−(1/2)𝑠limsup_𝑛→∞1𝑛^𝑛_𝑘=1𝐸_𝑄𝑒^{𝑠0ℎ𝑘(𝑋𝑘)}𝑒^{(𝑠0+𝑠)(−ℎ𝑘(𝑋𝑘))}−ℎ_𝑘𝑋_𝑘²−𝑟(𝜔)𝑠≤−2𝑒⁻²𝑠𝑀𝑠₀𝑠₀+𝑠²−𝑟(𝜔)𝑠=𝜙𝑠,𝑟(𝜔),−𝑠₀<𝑠<0,𝜔∈𝐴(𝑠)∩𝐴(0). (2.26)By (2.14) and −𝑠₀<𝑠<0, obviously 𝜙(𝑠,𝑟(𝜔))≥0, hence 𝛽(𝑟(𝜔))≥0. Let 𝑄⁻ be the set of rational numbers in the interval (−𝑠₀,0), and let 𝐴_∗=∩_𝑠∈𝑄⁻𝐴(𝑠), then 𝑃(𝐴_∗)=1. Then we have by (2.26)limsup_𝑛→∞1𝑛^𝑛_𝑘=1ℎ_𝑘𝑋_𝑘−𝐸_𝑄ℎ_𝑘𝑋_𝑘≤𝜙𝑠,𝑟(𝜔),𝜔∈𝐴_∗∩𝐴(0),∀𝑠∈𝑄⁻. (2.27)It is clear that 𝜙(𝑠,𝑥) is a continuous function with respect to 𝑠 on the interval (−𝑠₀,0). For each 𝜔∈𝐴_∗∩𝐴(0) (0≤𝑟(𝜔)<∞), take 𝜆_𝑛(𝜔)∈𝑄⁻, 𝑛=1,2,…, such thatlim_𝑛→∞𝜙𝜆_𝑛(𝜔),𝑟(𝜔)=𝛽𝑟(𝜔). (2.28)By (2.27) and (2.28), we havelimsup_𝑛→∞1𝑛^𝑛_𝑘=1ℎ_𝑘𝑋_𝑘−𝐸_𝑄ℎ_𝑘𝑋_𝑘≤𝛽𝑟(𝜔),𝜔∈𝐴_∗∩𝐴(0). (2.29)Since 𝑃(𝐴_∗∩𝐴(0))=1, (2.3) follows from (2.29).
By (2.4), (2.5), and (2.14), if 𝑥>0, we have−𝑠0≤𝛽(𝑥)≤𝜙⁰√𝑥1+√𝑥=√,𝑥𝑥1+√𝑥𝑠₀2𝑒⁻²𝑀𝑠₀+1. (2.30)If 𝑥=0, we have𝛽(0)≤𝜙−𝑛⁻¹,0=2𝑒⁻²𝑀𝑠₀𝑛𝑠₀−𝑛⁻¹²,𝑛≥1. (2.31)Noticing that 𝛽(𝑥)≥0, (𝑥≥0), (2.6) follows from (2.30) and (2.31).

Corollary 2.3. If 𝑃=𝑄, or {𝑋_𝑛,𝑛≥1} is a sequence of independent random variables, and under the assumptions of (1) and (2), thenlim_𝑛→∞1𝑛^𝑛_𝑘=1ℎ_𝑘𝑋_𝑘−𝐸ℎ_𝑘𝑋_𝑘=0a.s. (2.32)

Proof. In this case, 𝑓_𝑛(𝑥₁,…,𝑥_𝑛)=∏^𝑛_𝑘=1𝑔_𝑘(𝑥_𝑘), and 𝑟(𝜔)=0 a.s. Hence, (2.32) follows directly from (2.2) and (2.3).

3. An Extension of the Shannon-McMillan Theorem

In order to understand better, we first introduce some definitions in information theory in this section.

Let {𝑋_𝑛,𝑛≥1} be a sequence produced by an arbitrary continuous information source on the probability space (Ω,ℱ,𝑃) with the joint density function𝑓_𝑛𝑥₁,…,𝑥_𝑛>0,𝑥_𝑘∈(−∞,∞),1≤𝑘≤𝑛,𝑛=1,2,…. (3.1) For the sake of brevity, we denote 𝑓_𝑛>0, and 𝑋_𝑘 stands for 𝑋_𝑘(𝜔). Let𝑝_𝑛(𝜔)=−(1/𝑛)ln𝑓_𝑛𝑋₁,…,𝑋_𝑛, (3.2)where 𝜔 is the sample point, 𝑝_𝑛(𝜔) is called the sample entropy or the entropy density of {𝑋_𝑘,1≤𝑘≤𝑛}. Also let 𝑄 be another probability measure on ℱ with the density function𝑞_𝑛𝑥₁,…,𝑥_𝑛>0,𝑥_𝑘∈(−∞,∞),1≤𝑘≤𝑛,𝑛=1,2,…. (3.3)Let𝐿_𝑛(𝜔)=ln𝑓_𝑛𝑋₁,…,𝑋_𝑛/𝑞_𝑛𝑋₁,…,𝑋_𝑛,𝐿(𝜔)=limsup_𝑛→∞(1/𝑛)𝐿_𝑛(𝜔)=−liminf_𝑛→∞(1/𝑛)ln𝑞_𝑛𝑋₁,…,𝑋_𝑛/𝑓_𝑛𝑋₁,…,𝑋_𝑛,𝐷𝑓_𝑛‖𝑞_𝑛=𝐸_𝑃𝐿_𝑛=𝐸_𝑃ln𝑓_𝑛𝑋₁,…,𝑋_𝑛/𝑞_𝑛𝑋₁,…,𝑋_𝑛. (3.4)𝐿_𝑛(𝜔), 𝐿(𝜔), and 𝐷(𝑓_𝑛‖𝑞_𝑛) are called the sample relative entropy, the sample relative entropy rate, and the relative entropy, respectively, relative to the reference density function 𝑞_𝑛(𝑥₁,…,𝑥_𝑛). Indeed, they all are the measure of the deviation between the true joint distribution density function 𝑓_𝑛(𝑥₁,…,𝑥_𝑛) and the reference distribution density function 𝑞_𝑛(𝑥₁,…,𝑥_𝑛) (cf. [10, pages 12, 18]).

A question of importance in information theory is the study of the limit properties of the relative entropy density 𝑓_𝑛(𝜔). Since Shannon's initial work was published (cf. [11]), there has been a great deal of investigation about this question (e.g., cf. [12–20]).

In this paper, a class of small deviation theorems (i.e., the strong limit theorems represented by inequalities) is established by using the analytical technique, and an extension of the Shannon-McMillan theorem to the arbitrary-dependent continuous information sources is given. Especially, an approach of applying the tool of Laplace transform to the study of the strong deviation theorems on the differential entropy is proposed.

Let ℎ_𝑘(𝑥_𝑘)=−ln𝑔_𝑘(𝑥_𝑘) (1≤𝑘≤𝑛,𝑛=1,2,…) in (1.5), then we give the following definitions.Definition 3.1. The Laplace transform of −ln𝑔_𝑘(𝑥_𝑘) is defined by 𝑓_𝑘(𝑠)=𝐸𝑒^{−𝑠(−ln𝑔𝑘(𝑋𝑘))}=^∞_−∞𝑒^{−𝑠(−ln𝑔𝑘(𝑥𝑘))}𝑔_𝑘𝑥_𝑘𝑑𝑥_𝑘. (3.5)Definition 3.2. The differential entropy for continuous random variables 𝑋_𝑘 is defined by ℎ𝑋_𝑘=𝐸−ln𝑔_𝑘𝑋_𝑘=−^∞_−∞𝑔_𝑘𝑥_𝑘ln𝑔_𝑘𝑥_𝑘𝑑𝑥_𝑘. (3.6)

In the following theorem, let {𝑋_𝑛,𝑛≥1} be independent random variables with respect to 𝑄, then the reference density function 𝑞_𝑛(𝑥₁,…,𝑥_𝑛)=∏^𝑛_𝑘=1𝑔_𝑘(𝑥_𝑘), and let ℎ_𝑘(𝑋_𝑘)=−ln𝑔_𝑘(𝑋_𝑘) (1≤𝑘≤𝑛) in Theorem 2.1.

Theorem 3.3. Let {𝑋_𝑛,𝑛≥1}, 𝐿_𝑛(𝜔), 𝐿(𝜔), 𝑓_𝑛(𝑠) be given as above, and under the assumptions of (1) and (2), letlimsup_𝑛→∞1𝑛^𝑛_𝑘=1𝐸_𝑄𝑒^{𝑠0(−ln𝑔𝑘(𝑋𝑘))}=limsup_𝑛→∞1𝑛^𝑛_𝑘=1𝑓_𝑘−𝑠₀=𝑀𝑠₀<∞. (3.7)Thenliminf_𝑛→∞1𝑛^𝑛_𝑘∑₌₁−ln𝑔_𝑘𝑋_𝑘−ℎ𝑋_𝑘≥−𝛽𝐿(𝜔),a.s.limsup_𝑛→∞1𝑛^𝑛_𝑘∑₌₁−ln𝑔_𝑘𝑋_𝑘−ℎ𝑋_𝑘≤𝛽𝐿(𝜔),a.s. (3.8)where𝛽(𝑥)=inf𝜙(𝑠,𝑥),−𝑠₀<𝑠<0,𝑥≥0,𝜙(𝑠,𝑥)=−2𝑒⁻²𝑠𝑀𝑠₀𝑠₀−|𝑠|²−𝑥𝑠√,𝑥≥0,0≤𝛽(𝑥)≤𝑥1+√𝑥𝑠₀2𝑒⁻²𝑀𝑠₀+1,lim_𝑥→0+𝛽(𝑥)=𝛽(0)=0. (3.9)(3.10)(3.11)Remark 3.4. Let𝛼(𝑥)=sup𝜙(𝑠,𝑥),0<𝑠<𝑠₀,𝑥≥0, (3.12)then𝛼(𝑥)=−𝛽(𝑥). (3.13)Corollary 3.5. Let 𝑝_𝑛(𝜔) be defined by (3.2). Under the condition of Theorem 3.3, thenliminf_𝑛→∞𝑝_𝑛(𝜔)−(1/𝑛)ℎ𝑋₁,…,𝑋_𝑛≥𝛼𝐿(𝜔)−𝐿(𝜔)+𝐻_∗a.s.limsup_𝑛→∞𝑝_𝑛(𝜔)−(1/𝑛)ℎ𝑋₁,…,𝑋_𝑛≤𝛽𝐿(𝜔)+𝐻^∗a.s. (3.14)where ℎ(𝑋₁,…,𝑋_𝑛)=𝐸[−ln𝑓_𝑛(𝑋₁,…,𝑋_𝑛)] is the differential entropy for (𝑋₁,…,𝑋_𝑛), and𝐻_∗=liminf_𝑛→∞(1/𝑛)^𝑛_𝑘=1ℎ𝑋_𝑘−ℎ𝑋₁,…,𝑋_𝑛=liminf_𝑛→∞(1/𝑛)𝐷𝑓_𝑛‖𝑞_𝑛,𝐻^∗=limsup_𝑛→∞(1/𝑛)^𝑛_𝑘=1ℎ𝑋_𝑘−ℎ𝑋₁,…,𝑋_𝑛=limsup_𝑛→∞(1/𝑛)𝐷𝑓_𝑛‖𝑞_𝑛, (3.15)where 𝛼(𝐿(𝜔)), 𝛽(𝐿(𝜔)) are denoted by (3.9)–(3.13).Corollary 3.6. If 𝑃=𝑄, or {𝑋_𝑛,𝑛≥1} are independent random variables, and there exists 𝑠₀>0, such that (2.1) holds, thenlim_𝑛→∞𝑝_𝑛(𝜔)−(1/𝑛)ℎ𝑋₁,…,𝑋_𝑛=0a.s. (3.16)

Acknowledgments

This research is supported by the National Natural Science Foundation of China (Grants nos. 10671052 and 10571008), the Natural Science Foundation of Beijing (Grant no. 1072004), Funding Project for Academic Human Resources Development in Institutions of Higher Learning Under the Jurisdiction of Beijing Municipality, the Basic Research and Frontier Technology Foundation of Henan (Grant no. 072300410090), and the Natural Science Research Project of Henan (Grant no. 2008B110009). The authors would like to thank the editor and the referees for helpful comments, which helped to improve an earlier version of the paper.