Research Article | Open Access

# Subgeometric Ergodicity under Random-Time State-Dependent Drift Conditions

**Academic Editor:**Steve Su

#### Abstract

Motivated by possible applications of Lyapunov techniques in the stability of stochastic networks, subgeometric ergodicity of Markov chains is investigated. In a nutshell, in this study we take a look at -ergodic general Markov chains, subgeometrically ergodic at rate , when the random-time Foster-Lyapunov drift conditions on a set of stopping times are satisfied.

#### 1. Introduction

Ergodicity of Markov processes has, in recent years, received quite substantial attention. Of the three standard types of ergodicity, being ordinary, geometric (often referred to as exponential), and strong ergodicity, ordinary ergodicity stands out as the most well studied of the three, especially on discrete time Markov chains. However for subgeometric ergodicity, which can be referred to as a kind of convergence that is faster than ordinary ergodicity but slower than geometric ergodicity, much study is needed especially for continuous Markov processes [1].

Ergodicity in the context of state-dependent drift conditions on a deterministic time index has been studied by many authors, [2–4]. Connor and Fort [2] and Yüksel and Meyn [3] studied ergodicity with a drift condition taking the form for some deterministic function and a constant . According to Theorem 2.1(ii) of [5] a deterministic sequence of functions exists and satisfies a Foster-Lyapunov drift condition: for a petite set and a constant such that is bounded on .

It has been proved that this Foster-Lyapunov condition holds not only for every but also for a sequence of stopping times , for some discrete time Markov chain , [4]. The results of Zurkowski [4] relied heavily on the work of Connor and Fort [2], Meyn and Tweedie [6], and Tuominen and Tweedie [5]. The aim of this study is to investigate and refine these random-time state-dependent drift conditions results with our emphasis on the subgeometric ergodicity for a general Markov chain.

As observed by Yüksel and Meyn [3], there is a function on the state space taking positive values and an increasing sequence of stopping times , with , such that, for each , where the function is positive (bounded away from zero) outside a small set and denotes the filtration of events up to time . The reader needs to take note that although inequality (2) is important to this study, it will not be of much use because we will only use its refinement which is availed as (23) in Theorem 7.

Throughout the paper the sequence of stopping times is assumed to be nondecreasing, with . On state-dependent criteria for stability, traditionally it is assumed that the stopping times take the form where is a deterministic function of the state.

Although our work will mostly be confined to working with* petite* sets, we know that for -irreducible and aperiodic processes petite sets are in fact* small* sets [5]. Thus we need not worry that the techniques that rely on small sets in some of the previous studies may become unavailable in the random-time drift setting thinking that a small set for may not necessarily be small for . Small sets are analogous to compact sets in the stability theory of -irreducible Markov chains. Also in most applications of -irreducible Markov chains we find that any compact set is small.

The paper is organized as follows. In Section 2, we have the preliminaries which introduces basic notations, definitions, and theorems. The main results are availed in Section 3 which is mostly concerned with -ergodicity. In Section 4 we make the conclusions of this study.

#### 2. Preliminaries

Let , , and . Let (resp., ) denote a discrete time Markov chain (DTMC), (resp., a continuous time Markov chain (CTMC) on a countable space, as given in Liu et al. [7]). The state space for all chains is denoted . Unless otherwise specified -irreducibility, aperiodicity, and positive recurrence of all the chains we will be dealing with is assumed. The corresponding transition function of the DTMC process is denoted by , where , here and hereafter, is the indicator function of set and and , respectively, denote the probability and expectation of the chain under the condition that . Analogously we get for the CTMC.

##### 2.1. Subgeometric Rate Function

Let function , where is the family of measurable increasing functions satisfying as . Let denote the class of positive functions such that for some we have Then is referred to as the class of subgeometric rate functions [8]. Indeed (4) implies the equivalence of the class of functions with the class of functions . Example of functions in the class is the rate , , , which has been discussed only recently in literature. Without loss to generality we suppose that whenever .

The properties of which follow from (4) and are to be used frequently in this study are

##### 2.2. Subgeometric Rate Ergodicity

Let ; then, the ergodic chain is said to be subgeometric ergodic of order in the -norm or simply -ergodic if for all , then where is the unique invariant distribution of the process for a (signed) measure and and is a measurable function.

In the analysis of ergodicity, for any nonempty set the two first hitting times are given as , where denotes the first jump-time of the chain and . If is a singleton consisting of state , we write for . We also note that if ; however, , if . For hitting times of the continuous chain we need only replace with in the above notations and we are done. It is known that (7) holds if and only if the Foster-Lyapunov (or just Lyapunov) drift condition holds. It is for this reason that we cannot do without the Lyapunov drift conditions in our study. Let , for the case of .

##### 2.3. -Ergodicity

For such that , the chain is called -ergodic if for some finite nonempty .

##### 2.4. Small Set

A set is said to be* small* (or -*small*) if there exists some , , and a probability measure on such that

##### 2.5. Petite Set

A set is said to be* petite* if there exists some nontrivial measure on and some probability distribution such that
Petite sets generalize small sets. If -irreducibility and aperiodicity of the process are assumed (as is the case throughout this study) then set is a petite set if and only if (8) is satisfied. Thus petite sets in this case are in fact small sets.

Next we state condition and Theorems 1 and 2 which will be useful in this study especially in the proof of Theorem 8.

##### 2.6. Condition

There exist a function , a concave monotone function , a measurable set , and a constant such that

Theorem 1 (Theorem in [6]). *If condition holds for some petite set and there exists such that , then there exists a unique invariant distribution , and
**
where , for all .*

Theorem 2 (Proposition 2.5 of Douc et al. [9]). *Let be a -irreducible and aperiodic kernel. Assume that holds for function with , a petite set , and a function with . Then there exists an invariant probability measure , and for all in the full and absorbing set , we got
**
where , *

#### 3. Main Results

The core results in this Section are presented as Theorems 4, 7, and 8. Theorem 4 is a modified version of Theorem in [4], while Theorem 7 is a modified version of both Theorem 2.1 in Yüksel and Meyn [3] and Proposition in Zurkowski [4]. Then we have the refinement of Theorem 7 in the light of the work of [9] given as Theorem 8.

The skeleton chain method will be employed in the proof of Proposition 3, so next we define it. Let the chain be aperiodic and irreducible. Let ; for the skeleton chain we define, then denotes the first hitting time on of the skeleton chain . Proposition 3 will be employed in the proof of Theorem 4.

Proposition 3. *For some (and equivalently for any) finite nonempty set , some constant and any subgeometric rate function , we have
*

*Proof. *For any , when , there is the possibility that the skeleton chain can miss visits of the continuous time process to resulting in . Supposing that , we let be the th sojourn time at and be the interval length between the th exit from and the next visit to for the continuous time Markov process . We note the are independent and identically distributed, having an exponential distribution with parameter . Also we note that the are independent of each other and the .

We define
Then , so by the submultiplicative property (5) we have
We define . Also we have ; by hypothesis we get , and by property (6) we get ; therefore we can choose small enough such that . By (16) we get the following recursive formula:

Theorem 4. *Suppose that there exist a function and constants and such that for an increasing sequence of stopping times **
Then for any the chain is -ergodic.*

*Proof. *It is worth noting that is also an aperiodic and irreducible Markov chain that satisfies the univariate condition of (Lemma 14 of [10]); hence, is bounded on . For any and , with for all , we have
Owing to the equivalence of and we suppose that therefore rate satisfies the submultiplicative property (5). We have established by Proposition 3 that which satisfies (19) for some and so by Theorem 2.1 in [1] implies that . Then by Theorem 3.3 of Liu et al. [7], the set is -regular hence we conclude that the chain is -ergodic.

As a consequence of studying subgeometric ergodicity in the total variation norm (that is in the -norm, when we let ), it is worth noting that the results of Theorem 4 above are applicable to continuous time Markov chains, on a countable state space studied in [7], by letting . Mao [11] investigated -ergodicity when is restricted to . Liu et al. [7] extended the results to the case when . Under random-time state-dependent drift function, we state the following corollary which is an extension of Corollary 2.1 of Liu et al. [7].

Corollary 5. *Let with . Then the following are equivalent.*(i)*The chain is -ergodic, that is, for some finite nonempty set .*(ii)*For some nonempty finite set and an increasing sequence of stopping times , there exist finite nonnegative functions , and a constant such that and
**for , where and denote the integer-part function, that is, the largest integer which is smaller than or equal to .*

*Proof. *The proof is analogous to the proof of Corollary 2.1 of Liu et al [7] and follows naturally therefrom.

For stochastically monotone CTMCs, some computable bounds on exponential convergence based on first hitting times and the drift function were investigated in Corollary 2.2 in Liu et al. [7] which leads us to the next corollary. The following corollary is analogous to Corollary 2.2 of Liu et al. [7] for a chain of the form and also follows naturally consequent to the results of Theorem 4.

Corollary 6. *Let . Suppose that the regular and totally stable intensity matrix is monotone and is strictly increasing. If then is subgeometrically ergodic of order and for any and any we have
**
where . In the specific case of , such that and the drift criterion (ii) of Corollary 5 holds for the singleton set , , and ; then, for any and any **
where .*

*Proof. *The proof is analogous to the proof of Corollary 2.2 of Liu et al. [7] and follows naturally therefrom.

Theorem 7 which follows deals with subgeometric rate convergence of -ergodic chains, in the case when isn’t a total variation norm. In a nutshell we’ll be dealing with -regularity of the petite subset , or simply -ergodicity of the chain .

Theorem 7. *Suppose that is a -irreducible and aperiodic Markov chain. Suppose further that there are functions , , , a small set , constant and such that for an increasing sequence of stopping times **
then, is -ergodic.*

*Proof. *We define a sequence by
where by convention whenever ; hence, .

Then under the assumed drift conditions we have
which implies that is a supermartingale. For a measurable subset we denote the first hitting time for the sampled chain delayed by a constant as
then for any , and
because is bounded on .

It is assumed that
so that
which implies -ergodicity of the chain .

Theorem 8 below is similar to Proposition in [4] and Theorem 2.8 in [9]. Its importance is that it generalizes Theorem 7. By employing condition , Theorems 1 and 2 we are a able to get quite a wide range of functions of the forms and .

Theorem 8. *Let the chain be aperiodic and irreducible, that is, both -ergodic and -ergodic for some , and subgeometric rate function . Assume that condition holds for such that and a petite set such that . Further let be pairs of ultimately nondecreasing functions, which satisfy and , where (resp., ) whenever (resp., ). Then the chain is -ergodic.*

*Proof. *By Theorem 2 we know that the chain is -ergodic and also that for a large enough , it holds that . Then by Lemma 2.7 in [9] for some constant such that we have that , where . Hence the chain is -ergodic. Further Theorem 1 states that the chain is -ergodic; then, it follows that there exists a class of sets where the chain is -ergodic . Its worth noting that there is a balance between the* rate* of convergence and the* norm *; that is, the slower the former the larger the latter.

Because of the -irreducibility and aperiodicity of the chain , there is only one full absorbing set , which can be partitioned into two partitions say, of -regular sets and say, of -regular sets. The intersection of these two partitions yields the set of -regular sets, and by Proposition 2.6 in [9] we have
for any accessible set . Indeed (30) implies that the chain is -ergodic.

#### 4. Conclusion

The refinement of Theorem 2.1 in Yüksel and Meyn [3] and Proposition in Zurkowski [4] has been accomplished in this study and presented in the form of Theorem 7. Theorems 4 and 8 follow as a consequence of Theorem 7. This study like a multitude of previous studies before it may pave way for more research on random-time state-dependent Foster-Lyapunov drift conditions especially that interest on the same has developed from researchers on control and optimization theory, information theory, and so forth.

#### Conflict of Interests

The author declares that there is no conflict of interests regarding the publication of this paper.

#### References

- Z. Hou, Y. Liu, and H. Zhang, “Subgeometric rates of convergence for a class of continuous-time Markov process,”
*Journal of Applied Probability*, vol. 42, no. 3, pp. 698–712, 2005. View at: Publisher Site | Google Scholar | Zentralblatt MATH | MathSciNet - S. B. Connor and G. Fort, “State-dependent Foster-Lyapunov criteria for subgeometric convergence of Markov chains,”
*Stochastic Processes and their Applications*, vol. 119, no. 12, pp. 4176–4193, 2009. View at: Publisher Site | Google Scholar | Zentralblatt MATH | MathSciNet - S. Yüksel and S. P. Meyn, “Random-time, state-dependent stochastic drift for Markov chains and application to stochastic stabilization over erasure channels,”
*IEEE Transactions on Automatic Control*, vol. 58, no. 1, pp. 47–59, 2013. View at: Publisher Site | Google Scholar | MathSciNet - R. A. Zurkowski,
*Lyapunov analysis for rates of convergence in Markov chains and random-time state-dependent drift [M.S. thesis]*, Queen's University, Ontario, Canada, 2013. - P. Tuominen and R. L. Tweedie, “Subgeometric rates of convergence of $f$-ergodic Markov chains,”
*Advances in Applied Probability*, vol. 26, no. 3, pp. 775–798, 1994. View at: Publisher Site | Google Scholar | Zentralblatt MATH | MathSciNet - S. P. Meyn and R. L. Tweedie,
*Markov Chains and Stochastic Stability*, Springer, 1993. View at: Publisher Site | MathSciNet - Y. Liu, H. Zhang, and Y. Zhao, “Subgeometric ergodicity for continuous-time Markov chains,”
*Journal of Mathematical Analysis and Applications*, vol. 368, no. 1, pp. 178–189, 2010. View at: Publisher Site | Google Scholar | MathSciNet - E. Nummelin and P. Tuominen, “The rate of convergence in Orey's theorem for Harris recurrent Markov chains with applications to renewal theory,”
*Stochastic Processes and Their Applications*, vol. 15, no. 3, pp. 295–311, 1983. View at: Publisher Site | Google Scholar | MathSciNet - R. Douc, G. Fort, E. Moulines, and P. Soulier, “Practical drift conditions for subgeometric rates of convergence,”
*The Annals of Applied Probability*, vol. 14, no. 3, pp. 1353–1377, 2004. View at: Publisher Site | Google Scholar | MathSciNet - G. O. Roberts and J. S. Rosenthal, “General state space Markov chains and {MCMC} algorithms,”
*Probability Surveys*, vol. 1, pp. 20–71, 2004. View at: Publisher Site | Google Scholar | Zentralblatt MATH | MathSciNet - Y. H. Mao, “Ergodic degrees for continuous-time Markov chains,”
*Science in China A*, vol. 47, no. 2, pp. 161–174, 2004. View at: Publisher Site | Google Scholar | MathSciNet

#### Copyright

Copyright © 2014 Mokaedi V. Lekgari. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.