Subgeometric Ergodicity Analysis of Continuous-Time Markov Chains under Random-Time State-Dependent Lyapunov Drift Conditions

Abstract

We investigate random-time state-dependent Foster-Lyapunov analysis on subgeometric rate ergodicity of continuous-time Markov chains (CTMCs). We are mainly concerned with making use of the available results on deterministic state-dependent drift conditions for CTMCs and on random-time state-dependent drift conditions for discrete-time Markov chains and transferring them to CTMCs.

1. Introduction

The ergodic theory of Markov processes in recent years has received quite substantial attention. With our focus on subgeometric ergodicity, which, loosely speaking, is a kind of convergence that is faster than ordinary ergodicity but slower than geometric ergodicity, much study is needed especially for continuous Markov processes [1]. It is for this reason that interest in this study developed.

Meyn and Tweedie [2] pioneered the study of state-dependent drift for -irreducible Markov processes. Then Connor and Fort [3] found that state-dependent Foster-Lyapunov criteria can be employed not only to determine the ergodic properties of the Markov process but also to infer a greater range of convergence rates for . The study focused on the process subsampled at some deterministic times. In [3] the results of the study were used for the classification of tame chains. Ergodicity in the context of state-dependent drift conditions on a deterministic time index was then studied by many authors, amongst them [4, 5]. Yüksel and Meyn [4] applied their results to “stochastic stabilization over an erasure channel.” The motivation for Zurkowski’s [5] study was its possible applications in network control and event triggered control systems. Although we cannot ignore the practical applications of our study in areas such as stability of stochastic networks, presenting such applications is beyond the scope of our work.

Connor and Fort [3] and Yüksel and Meyn [4] studied ergodicity with a drift condition taking the form for some deterministic function and a constant . According to Theorem  2.1(ii) of [6] a deterministic sequence of functions exists, , and satisfies a Foster-Lyapunov drift condition: for a petite set and a constant such that is bounded on .

It has been proved that this Foster-Lyapunov condition holds not just for every but also for a sequence of stopping times , for some discrete time Markov chain   [5]. The results of Zurkowski [5] relied heavily on the work of Connor and Fort [3], Meyn and Tweedie [7], and Tuominen and Tweedie [6].

This study follows on the work of previous studies to investigate random-time state-dependent drift conditions results on the subgeometric ergodicity for CTMCs in an easy and reader-friendly manner. Amongst the previous authors Connor and Fort [3] studied state-dependent geometric Lyapunov drift condition for a general Markov chain subsampled at some deterministic times. Random-time state-dependent drift studies were then done by Yüksel and Meyn [4] and Zurkowski [5] for general Markov chains and discrete chains, respectively. The work in this study utilizes the results of the above mentioned studies with focus on subgeometric ergodicity for countable state space -processes.

The paper is organized as follows. In Section 2, we have the Preliminaries section which introduces basic notations, definitions, and theorems. The main results are availed in Section 3 which is mostly concerned with subgeometric ergodicity at rate , in the -norm. Section 4 is about the conclusions of this study.

2. Preliminaries

Let and let . We let denote a continuous-time Markov chain (CTMC) on a countable state space . The transition function of the continuous-time Markov process is denoted by , where here and hereafter is the indicator function of set and and , respectively, denote the probability and expectation of the chain under the condition that .

2.1. First Hitting Times

Given a subset we let be the first hitting time on delayed by a constant . In the case when we have . We also have as the first hitting time on the set after the first jump of the process. We also note that if . If is a singleton consisting only of state , then we write for and equivalently for .

2.2. -Ergodicity

For , such that , the moment is denoted by and . By definition the Markov process is said to be -ergodic (or of ergodic degree ) if for some (and then for all) finite nonempty . For more studies on ergodic degrees, -ergodicity, and related topics, the interested reader may consult Mao [8] and references therein.

2.3. Geometric Ergodicity

According to [3], is said to be geometrically ergodic if there exist a function and constants , , such that, for any , , which is equivalent to the existence of a scale function , a small set , and constants , , such that .

2.4. Subgeometric Rate Function

Let function , where is the family of measurable increasing functions satisfying as . Let denote the class of positive functions such that for some we have Then is referred to as the class of subgeometric rate functions [9]. Indeed (4) implies the equivalence of the class of functions with the class of functions . Example of the functions in the class is the rate , , , which has been discussed only recently in literature. Without loss of generality, we suppose that whenever .

The properties of which follow from (4) and are to be used frequently in this study are

2.5. Modulated Moments

Sufficient conditions for -ergodicity involve the modulated moments of the first hitting times. Having the subgeometric rate function given by (4), a function , a subset , and a constant , we define the modulated moment of by Analogous to the modulated moments of the first hitting times given as (7) above, the modulated moments of the first hitting times , namely, , are defined as follows. For a subgeometric rate function , a function , and a subset , then

2.6. Subgeometric Rate Ergodicity

Let ; then the ergodic chain is said to be subgeometric ergodic of order in the -norm or simply -ergodic if for all ; then where is the unique invariant distribution of the process for a (signed) measure , where and is a measurable function.

It is known that (9) holds if and only if the Foster-Lyapunov (or just Lyapunov) drift condition holds. It is for this reason that the Lyapunov drift conditions play a very crucial role in our study. The Lyapunov drift conditions provide bounds on the return time to accessible sets thereby availing some control on the Markov process dynamics by focusing on the hitting times on a particular set.

We are aware that techniques that rely on small sets in some of the previous studies may become unavailable in the random-time drift setting because a small set for may not necessarily be small for . However, we also know that, for a -irreducible process, all finite subsets of are petite and that an accessible closed petite set always exists [10], suggesting our work will be confined to working with petite sets. Then by Theorem  5.5.7 in Meyn and Tweedie [7] we know that if the Markov chain is -irreducible and aperiodic, then every petite set is small. Therefore all the petite sets in this study are assumed to be small because all chains are assumed to be -irreducible and aperiodic.

In light of the foregoing definitions and notations we are ready to state the following subgeometric rate ergodicity propositions about -ergodicity and -ergodicity.

Proposition 1 (see Proposition 2.11 in [11]). Let be an irreducible Markov process, and suppose that, for some and some function on , one has , for all and . If there exist a petite set and some constant such that then is -ergodic.

Proposition 2 (see Theorem  3.3 in Liu et al. [12]). An irreducible Markov chain is subgeometrically ergodic at rate in the total variation norm if and only if for some (and then for all) finite subset one has

Proposition 3 (Theorem  3.2 in Liu et al. [12]). For some finite nonempty set , if and only if for any finite set and any , then .

Throughout this study the nondecreasing sequence of stopping times with is assumed to be similar to the sequence in Assumption 2.1 of Spieksma [13] , which is restated as follows. Recursively we define and , if is not an absorbing state (i.e., ). We put if is an absorbing state; then, we have . In this sense the sequence is a nondecreasing sequence of stopping times representing successive jump times.

3. Main Results

The content presented in this section is on the subgeometric rate ergodicity of , where is assumed to be an irreducible and aperiodic CTMC. We have results presented as Theorems 4, 6, and 7. Theorem 4 is a modified version of Theorem  3.1 in [12], while Theorem 6 is a continuous counterpart of Proposition  5.1.2 in Zurkowski [5]. Finally we have Theorem 7 which follows from Theorem  2.16 of [11].

Theorem 4. Let be an irreducible and aperiodic chain. Further suppose that for some (and then for any) finite nonempty set there exist a function and a constant such that, for an increasing sequence of stopping times , then for any the chain is -ergodic.

Proof. It is worthwhile to realize that the chain is also aperiodic and irreducible and that is petite and hence small and that is bounded on . Let . We know that   , because the chain can miss some of the visits of the chain to the set , so that, for any , we have which implies that for any . We get (14) owing to the equivalence of and and supposing that and hence satisfies the submultiplicative property (5). By employing Proposition 3 we get that (14) implies for any . Then, by Proposition 2, the set is -regular; hence we conclude that the chain is -ergodic.

It is worth noting that, in practice, the drift condition in Theorem 4 above is not easy to verify because it involves the unknown information . The practically more favorable drift condition is stated in Corollary 5. It was Mao [8] who investigated -ergodicity when is restricted to ; then Liu et al. [12] extended the results to the case when . Under random-time state-dependent drift function, we state the following corollary which is a modified version of Corollary 2.1 of Liu et al. [12].

Corollary 5. Let with . Then the chain is -ergodic (i.e., ) if and only if for some finite nonempty set and an increasing sequence of stopping times there exist finite nonnegative functions , , and a constant such that and for , where and denotes the integer-part function, that is, the largest integer which is smaller than or equal to .

Proof. The proof of this corollary is analogous to the proof of Corollary 2.1 of Liu et al. [12] and follows naturally therefrom.

Remark. In a particular case of Corollary 5, the chain is said to be subgeometrically ergodic at rate , for some , in the total variation norm if and only if we have nonnegative functions on , with and such that , and for and any .

Also the Lyapunov drift in Theorem 4 and Corollary 5 which is of the where and , is used here and in the rest of this paper for convenience. We note that for the -matrix and the real-valued function on we have .

Theorem 6 which follows deals with subgeometric rate convergence of -ergodic chains, in the case when is not a total variation norm. In a nutshell we will be dealing with -regularity of the petite subset .

Theorem 6. Suppose that is a -irreducible and aperiodic Markov chain. Further suppose that there are functions and , where is bounded on the small set , constant , , and such that, for an increasing sequence of stopping times , then is -ergodic.

Proof. For all and for all stopping time , by Dynkin’s inequality, we get then for all we get because is bounded on . Again as in Theorem 4 we have that , for all , where is the sampled hitting times for the chain . Then which implies -ergodicity of the chain .

The equation is taken from formula (15) in [14, Section  5], with the obvious notational changes of course.

The following theorem generalizes the results of -ergodicity in Theorem 6 which can be established by combining the results of Proposition 1 (i.e., -ergodicity) with Theorem 4 (i.e., -ergodicity). This theorem is similar to Proposition  5.1.4 in [5] and Theorem  2.8 in [15].

Theorem 7 (borrowed from Theorem 2.16 in [11]). Let be an irreducible and aperiodic chain. Further suppose that for some (and then for any) finite nonempty set there exist a function and constants and such that, for an increasing sequence of stopping times , , for all and . Then, for some and a constant , Then is subgeometrically ergodic at a rate in the -norm, where the functions are a pair of inverse Young functions.

The drift condition for the chain in Theorem 7 is not easy to find but an easier way to verify drift is given in [11, Corollary  2.17]. It is restated as the following corollary.

Corollary 8. Let the chain be as in Theorem 7, for a fixed constant and positive constants such that, for an increasing sequence of stopping times , for every ; then the chain is subgeometrically ergodic at rate , in the -norm.

4. Conclusion

Through this study we have succeeded in the subgeometric rate ergodicity analysis of random-time state-dependent Foster-Lyapunov drift conditions for CTMCs. However there is still more work to be done which could pave way for more research especially on applications. The studies we relied on such as Connor and Fort [3] applied the results of their work to “tame” chains (which technically speaking are any chains with subgeometric drift ). Also the study on continuous-time controlled Markov chains on a countable state space by [11] was applied on discounted and average reward optimality criteria. Therefore our future research will focus on making any possible improvements to this study by refining our results and identifying applications of our results. Indeed interest on the same has developed from research fields such as “control and optimization theory,” “information theory,” and “queuing theory.”

Conflict of Interests

The author declares that there is no conflict of interests regarding the publication of this paper.