Policy Iteration for Continuous-Time Average Reward Markov Decision Processes in Polish Spaces

Abstract

We study the policy iteration algorithm (PIA) for continuous-time jump Markov decision processes in general state and action spaces. The corresponding transition rates are allowed to be unbounded, and the reward rates may have neither upper nor lower bounds. The criterion that we are concerned with is expected average reward. We propose a set of conditions under which we first establish the average reward optimality equation and present the PIA. Then under two slightly different sets of conditions we show that the PIA yields the optimal (maximum) reward, an average optimal stationary policy, and a solution to the average reward optimality equation.

1. Introduction

In this paper we study the average reward optimality problem for continuous-time jump Markov decision processes (MDPs) in general state and action spaces. The corresponding transition rates are allowed to be unbounded, and the reward rates may have neither upper nor lower bounds. Here, the approach to deal with this problem is by means of the well-known policy iteration algorithm (PIA)—also known as Howard's policy improvement algorithm.

As is well known, the PIA was originally introduced by Howard (1960) in [1] for finite MDPs (i.e., the state and action spaces are both finite). By using the monotonicity of the sequence of iterated average rewards, he showed that the PIA converged with a finite number of steps. But, when a state space is not finite, there are well-known counterexamples to show that the PIA does not converge even though the action space is compact (see [2–4], e.g.,). Thus, an interesting problem is to find conditions to ensure that the PIA converges. To do this, extensive literature has been presented; for instance, see [1, 5–14] and the references therein. However, most of those references above are concentrated on the case of discrete-time MDPs; for instance, see [1, 5, 11] for finite discrete-time MDPs, [10, 15] for discrete-time MDPs with a finite state space and a compact action set, [13] for denumerable discrete-time MDPs, and [8, 9, 12] for discrete-time MDPs in Borel spaces. For the case of continuous-time models, to the best of our knowledge, only Guo and Hernández-Lerma [6], Guo and Cao [7], and Zhu [14] have addressed this issue. In [6, 7, 14], the authors established the average reward optimality equation and the existence of average optimal stationary policies. However, the treatments in [6, 7] are restricted to only a denumerable state space. In [14] we used the policy iteration approach to study the average reward optimality problem for the case of continuous-time jump MDPs in general state and action spaces. One of the main contributions in [14] is to prove the existence of the average reward optimality equation and average optimal stationary policies. But the PIA is not stated explicitly in [14], and so the value of the average optimal reward value function and an average optimal stationary policy are also not be computed in [14]. In this paper we further study the average reward optimality problem for such a class of continuous-time jump MDPs in general state and action spaces. Our main objective is to use the PIA to compute or at least approximate (when the PIA takes infinitely many steps to converge) the value of the average optimal reward value function and an average optimal stationary policy. To do this, we first use the so-called “drift" condition, the standard continuity-compactness hypotheses, and the irreducible and uniform exponential ergodicity condition to establish the average reward optimality equation and present the PIA. Then under two differently extra conditions we show that the PIA yields the optimal (maximum) reward, an average optimal stationary policy, and a solution to the average reward optimality equation. A key feature of this paper is that the PIA provides an approach to compute or at least approximate (when the PIA takes infinitely many steps to converge) the value of the average optimal reward value function and an average optimal stationary policy.

The remainder of this paper is organized as follows. In Section 2, we introduce the control model and the optimal control problem that we are concerned with. After our optimality conditions and some technical preliminaries as well as the PIA stated in Section 3, we show that the PIA yields the optimal (maximum) reward, an average optimal stationary policy, and a solution to the average reward optimality equation in Section 4. Finally, we conclude in Section 5 with some general remarks.

Notation 1. If 𝑋 is a Polish space (i.e., a complete and separable metric space), we denote by ℬ(𝑋) the Borel 𝜎-algebra.

2. The Optimal Control Problem

The material in this section is quite standard (see [14, 16, 17] e.g.,), and we shall state it briefly. The control model that we are interested in is continuous-time jump MDPs with the following form:

{𝑆,(𝐴(𝑥)⊂𝐴,𝑥∈𝑆),𝑞(⋅∣𝑥,𝑎),𝑟(𝑥,𝑎)}, (2.1) where one has the following.

(i)𝑆 is a state space and it is supposed to be a Polish space.(ii)𝐴 is an action space, which is also supposed to be a Polish space, and 𝐴(𝑥) is a Borel set which denotes the set of available actions at state 𝑥∈𝑆. The set 𝐾∶={(𝑥,𝑎)∶𝑥∈𝑆,𝑎∈𝐴(𝑥)} is assumed to be a Borel subset of 𝑆×𝐴.(iii)𝑞(⋅∣𝑥,𝑎) denotes the transition rates, and they are supposed to satisfy the following properties: for each (𝑥,𝑎)∈𝐾 and 𝐷∈ℬ(S),(𝑄₁)𝐷↦𝑞(𝐷∣𝑥,𝑎) is a signed measure on ℬ(𝑆), and (𝑥,𝑎)↦𝑞(𝐷∣𝑥,𝑎) is Borel measurable on 𝐾;(𝑄₂)0≤𝑞(𝐷∣𝑥,𝑎)<∞, for all 𝑥∉𝐷∈ℬ(S);(𝑄₃)𝑞(𝑆∣𝑥,𝑎)=0,0≤−𝑞(𝑥∣𝑥,𝑎)<∞;(𝑄₄)𝑞(𝑥)∶=sup_{𝑎∈𝐴(𝑥)}(−𝑞(𝑥∣𝑥,𝑎))<∞, for all 𝑥∈𝑆.

It should be noted that the property (𝑄₃) shows that the model is conservative, and the property (𝑄₄) implies that the model is stable.

(iv)𝑟(𝑥,𝑎) denotes the reward rate and it is assumed to be measurable on 𝐾. (As 𝑟(𝑥,𝑎) is allowed to take positive and negative values; it can also be interpreted as a cost rate.)

To introduce the optimal control problem that we are interested in, we need to introduce the classes of admissible control policies.

Let Π_𝑚 be the family of function 𝜋_𝑡(𝐵∣𝑥) such that

(i)for each 𝑥∈𝑆 and 𝑡≥0, 𝐵→𝜋_𝑡(𝐵∣𝑥) is a probability measure on ℬ(𝐴(𝑥)),(ii)for each 𝑥∈𝑆 and 𝐵∈ℬ(𝐴(𝑥)), 𝑡→𝜋_𝑡(𝐵∣𝑥) is a Borel measurable function on [0,∞).

Definition 2.1. A family 𝜋=(𝜋_𝑡,𝑡≥0)∈Π_𝑚 is said to be a randomized Markov policy. In particular, if there exists a measurable function 𝑓 on 𝑆 with 𝑓(𝑥)∈𝐴(𝑥) for all 𝑥∈𝑆, such that 𝜋_𝑡({𝑓(𝑥)}∣𝑥)≡1 for all 𝑡≥0 and 𝑥∈𝑆, then 𝜋 is called a (deterministic) stationary policy and it is identified with 𝑓. The set of all stationary policies is denoted by 𝐹.

For each 𝜋=(𝜋_𝑡,𝑡≥0)∈Π_𝑚, we define the associated transition rates 𝑞(𝐷∣𝑥,𝜋_𝑡) and the reward rates 𝑟(𝑥,𝜋_𝑡), respectively, as follows.

For each 𝑥∈𝑆, 𝐷∈ℬ(𝑆) and 𝑡≥0,

𝑞(𝐷∣𝑥,𝜋_𝑡)∶=_𝐴(𝑥)𝑞(𝐷∣𝑥,𝑎)𝜋_𝑡(𝑑𝑎∣𝑥),𝑟(𝑥,𝜋_𝑡)∶=_𝐴(𝑥)𝑟(𝑥,𝑎)𝜋_𝑡(𝑑𝑎∣𝑥). (2.2) In particular, we will write 𝑞(𝐷∣𝑥,𝜋_𝑡) and 𝑟(𝑥,𝜋_𝑡) as 𝑞(𝐷∣𝑥,𝑓) and 𝑟(𝑥,𝑓), respectively, when 𝜋∶=𝑓∈𝐹.

Definition 2.2. A randomized Markov policy is said to be admissible if 𝑞(𝐷∣𝑥,𝜋_𝑡) is continuous in 𝑡≥0, for all 𝐷∈ℬ(𝑆) and 𝑥∈𝑆.

The family of all such policies is denoted by Π. Obviously, Π⊇𝐹 and so that Π is nonempty. Moreover, for each 𝜋∈Π, Lemma 2.1 in [16] ensures that there exists a 𝑄-process—that is, a possibly substochastic and nonhomogeneous transition function 𝑃^𝜋(𝑠,𝑥,𝑡,𝐷) with transition rates 𝑞(𝐷∣𝑥,𝜋_𝑡). As is well known, such a 𝑄-process is not necessarily regular; that is, we might have 𝑃^𝜋(𝑠,𝑥,𝑡,𝑆)<1 for some state 𝑥∈𝑆 and 𝑡≥𝑠≥0. To ensure the regularity of a 𝑄-process, we shall use the following so-called “drift" condition, which is taken from [14, 16–18].

Assumption A. There exist a (measurable) function 𝑤₁≥1 on 𝑆 and constants 𝑏₁≥0, 𝑐₁>0, 𝑀₁>0 and 𝑀>0 such that(1)∫_𝑆𝑤₁(𝑦)𝑞(𝑑𝑦∣𝑥,𝑎)≤−𝑐₁𝑤₁(𝑥)+𝑏₁ for all (𝑥,𝑎)∈𝐾;(2)𝑞(𝑥)≤𝑀₁𝑤₁(𝑥) for all 𝑥∈𝑆, with 𝑞(𝑥) as in (𝑄₄);(3)|𝑟(𝑥,𝑎)|≤𝑀𝑤₁(𝑥) for all (𝑥,𝑎)∈𝐾.

Remark 2.1 in [16] gives a discussion of Assumption A. In fact, Assumption A(1) is similar to conditions in the previous literature (see [19, equation (2.4)] e.g.,), and it is together with Assumption A(3) used to ensure the finiteness of the average expected reward criterion (2.5) below. In particular, Assumption A(2) is not required when the transition rate is uniformly bounded, that is, sup_𝑥∈𝑆𝑞(𝑥)<∞.

For each initial state 𝑥∈𝑆 at time 𝑠≥0 and 𝜋∈Π, we denote by 𝑃^𝜋_𝑠,𝑥 and 𝐸^𝜋_𝑠,𝑥 the probability measure determined by 𝑃^𝜋(𝑠,𝑥,𝑡,𝐷) and the corresponding expectation operator, respectively. Thus, for each 𝜋∈Π by [20, pages 107–109] there exists a Borel measure Markov process {𝑥^𝜋_𝑡} (we shall denote {𝑥^𝜋_𝑡} by {𝑥_𝑡} for simplicity when there is no risk of confusion) with value in 𝑆 and the transition function 𝑃^𝜋(𝑠,𝑥,𝑡,𝐷), which is completely determined by the transition rates 𝑞(𝐷∣𝑥,𝜋_𝑡). In particular, if 𝑠=0, we write 𝐸^𝜋_0,𝑥 and 𝑃^𝜋_0,𝑥 as 𝐸^𝜋_𝑥 and 𝑃^𝜋_𝑥, respectively.

If Assumption A holds, then from [17, Lemma 3.1] we have the following facts.

Lemma 2.3. Suppose that Assumption A holds. Then the following statements hold.(a)For each 𝑥∈𝑆, 𝜋∈Π and 𝑡≥0, 𝐸^𝜋_𝑥[𝑤₁(𝑥_𝑡)]≤𝑒^−𝑐1𝑡𝑤₁𝑏(𝑥)+¹𝑐₁, (2.3) where the function 𝑤₁ and constants 𝑏₁ and 𝑐₁ are as in Assumption A.(b)For each 𝑢∈𝐵_𝑤1(𝑆), 𝑥∈𝑆 and 𝜋∈Π, lim_𝑡→∞𝐸^𝜋_𝑥[𝑢(𝑥_𝑡)]𝑡=0. (2.4)

For each 𝑥∈𝑆 and 𝜋∈Π, the expected average reward  𝑉(𝑥,𝜋) as well as the corresponding optimal reward value functions 𝑉^∗(𝑥) are defined as

𝑉(𝑥,𝜋)∶=liminf_𝑇→∞∫^𝑇₀[𝐸^𝜋_𝑥𝑟(𝑥_𝑡,𝜋_𝑡)]𝑑𝑡𝑇,𝑉^∗(𝑥)∶=sup_𝜋∈Π𝑉(𝑥,𝜋). (2.5)

As a consequence of Assumption A(3) and Lemma 2.3(a), the expected average reward 𝑉(𝑥,𝜋) is well defined.

Definition 2.4. A policy 𝜋^∗∈Π is said to be average optimal if 𝑉(𝑥,𝜋^∗)=𝑉^∗(𝑥) for all 𝑥∈𝑆.

The main goal of this paper is to give conditions for ensuring that the policy iteration algorithm converges.

3. Optimality Conditions and Preliminaries

In this section we state conditions for ensuring that the policy iteration algorithm (PIA) converges and give some preliminary lemmas that are needed to prove our main results.

To guarantee that the PIA converges, we need to establish the average reward optimality equation. To do this, in addition to Assumption A, we also need two more assumptions. The first one is the following so-called standard continuity-compactness hypotheses, which is taken from [14, 16–18]. Moreover, it is similar to the version for discrete-time MDPs; see, for instance, [3, 8, 21–23] and their references. In particular, Assumption B(3) is not required when the transition rate is uniformly bounded, since it is only used to ensure the applying of the Dynkin formula.

Assumption B. For each 𝑥∈𝑆,(1)𝐴(𝑥) is compact;(2)𝑟(𝑥,𝑎) is continuous in 𝑎∈𝐴(𝑥), and the function ∫_𝑆𝑢(𝑦)𝑞(𝑑𝑦∣𝑥,𝑎) is continuous in 𝑎∈𝐴(𝑥) for each bounded measurable function 𝑢 on 𝑆, and also for 𝑢∶=𝑤₁ as in Assumption A;(3)there exist a nonnegative measurable function 𝑤₂ on 𝑆, and constants 𝑏₂≥0,𝑐₂>0 and 𝑀₂>0 such that 𝑞(𝑥)𝑤₁(𝑥)≤𝑀₂𝑤₂(𝑥),_𝑆𝑤₂(𝑦)𝑞(𝑑𝑦∣𝑥,𝑎)≤𝑐₂𝑤₂(𝑥)+𝑏₂ (3.1) for all (𝑥,𝑎)∈𝐾.

The second one is the irreducible and uniform exponential ergodicity condition. To state this condition, we need to introduce the concept of the weighted norm used in [8, 14, 22]. For the function 𝑤₁≥1 in Assumption A, we define the weighted supremum norm ‖⋅‖_𝑤1 for real-valued functions 𝑢 on 𝑆 by

‖𝑢‖_𝑤1∶=sup_𝑥∈𝑆[𝑤₁(𝑥)⁻¹|𝑢(𝑥)|] (3.2) and the Banach space

𝐵_𝑤1(𝑆)∶={𝑢∶‖𝑢‖_𝑤1<∞}. (3.3)

Definition 3.1. For each 𝑓∈𝐹, the Markov process {𝑥_𝑡}, with transition rates 𝑞(⋅∣𝑥,𝑓), is said to be uniform 𝑤₁-exponentially ergodic if there exists an invariant probability measure 𝜇_𝑓 on 𝑆 such that sup_𝑓∈𝐹|𝐸^𝑓_𝑥[𝑢(𝑥_𝑡)]−𝜇_𝑓(𝑢)|≤Re^−𝜌𝑡‖𝑢‖_𝑤1𝑤₁(𝑥) (3.4) for all 𝑥∈𝑆, 𝑢∈𝐵_𝑤1(𝑆) and 𝑡≥0, where the positive constants 𝑅 and 𝜌 do not depend on 𝑓, and where 𝜇_𝑓(𝑢)∶=∫_𝑆𝑢(𝑦)𝜇_𝑓(𝑑𝑦).

Assumption C. For each 𝑓∈𝐹, the Markov process {𝑥_𝑡}, with transition rates 𝑞(⋅∣𝑥,𝑓), is uniform 𝑤₁-exponentially ergodic and 𝜆-irreducible, where 𝜆 is a nontrivial 𝜎-finite measure on ℬ(𝑆) independent of 𝑓.

Remark 3.2. (a) Assumption C is taken from [14] and it is used to establish the average reward optimality equation. (b) Assumption C is similar to the uniform 𝑤₁-exponentially ergodic hypothesis for discrete-time MDPs; see [8, 22], for instance. (c) Some sufficient conditions as well as examples in [6, 16, 19] are given to verify Assumption C. (d) Under Assumptions A, B, and C, for each 𝑓∈𝐹, the Markov process {𝑥_𝑡}, with the transition rate 𝑞(⋅∣𝑥,𝑓), has a unique invariant probability measure 𝜇_𝑓 such that _𝑆𝜇_𝑓(𝑑𝑥)𝑞(𝐷∣𝑥,𝑓)=0foreach𝐷∈ℬ(𝑆). (3.5)  (e) As in [9], for any given stationary policy 𝑓∈𝐹, we shall also consider two functions in 𝐵_𝑤1(𝑆) to be equivalent and do not distinguish between equivalent functions, if they are equal 𝜇_𝑓-almost everywhere (a.e.). In particular, if 𝑢(𝑥)=0𝜇_𝑓-a.e. holds for all 𝑥∈𝑆, then the function 𝑢 will be taken to be identically zero.

Under Assumptions A, B, and C, we can obtain several lemmas, which are needed to prove our main results.

Lemma 3.3. Suppose that Assumptions A, B, and C hold, and let 𝑓∈𝐹 be any stationary policy. Then one has the following facts.(a)For each 𝑥∈𝑆, the function ℎ_𝑓(𝑥)∶=^∞₀[𝐸^𝑓_𝑥(𝑟(𝑥_𝑡,𝑓))−𝑔(𝑓)]𝑑𝑡 (3.6) belongs to 𝐵_𝑤1(𝑆), where 𝑔(𝑓)∶=∫_𝑆𝑟(𝑦,𝑓)𝜇_𝑓(𝑑𝑦) and 𝑤₁ is as in Assumption A.(b)(𝑔(𝑓),ℎ_𝑓) satisfies the Poisson equation 𝑔(𝑓)=𝑟(𝑥,𝑓)+_𝑆ℎ_𝑓(𝑦)𝑞(𝑑𝑦∣𝑥,𝑓)∀𝑥∈𝑆, (3.7) for which the 𝜇_𝑓-expectation of ℎ_𝑓 is zero, that is, 𝜇_𝑓(ℎ_𝑓)∶=_𝑆ℎ_𝑓(𝑦)𝜇_𝑓(𝑑𝑦)=0. (3.8)(c)For all 𝑥∈𝑆, |𝑉(𝑥,𝑓)|≤𝑀𝑏₁/𝑐₁.(d)For all 𝑥∈𝑆, |𝑔(𝑓)|=|𝑉(𝑥,𝑓)|≤𝑀𝑏₁/𝑐₁.

Proof. Obviously, the proofs of parts (a) and (b) are from [14, Lemma 3.2]. We now prove (c). In fact, from the definition of 𝑉(𝑥,𝑓) in (2.5), Assumption A(3), and Lemma 2.3(a) we have |𝑉(𝑥,𝑓)|≤liminf_𝑇→∞∫^𝑇₀𝑀[𝑒^−𝑐1𝑡𝑤₁(𝑥)+𝑏₁/𝑐₁]𝑑𝑡𝑇=𝑀𝑏₁𝑐₁, (3.9) which gives (c). Finally, we verify part (d). Obviously, by Assumption A(3) and Assumption C we can easily obtain 𝑔(𝑓)=𝑉(𝑥,𝑓) for all 𝑥∈𝑆, which together with part (c) yields the desired result.

The next result establishes the average reward optimality equation. For the proof, see [14, Theorem 4.1].

Theorem 3.4. Under Assumptions A, B, and C, the following statements hold.(a)There exist a unique constant 𝑔^∗, a function ℎ^∗∈𝐵_𝑤1(𝑆), and a stationary policy 𝑓^∗∈𝐹 satisfying the average reward optimality equation𝑔^∗=max_{𝑎∈𝐴(𝑥)}{𝑟(𝑥,𝑎)+_𝑆ℎ^∗(𝑦)𝑞(𝑑𝑦∣𝑥,𝑎)}=𝑟(𝑥,𝑓^∗)+_𝑆ℎ^∗(𝑦)𝑞(𝑑𝑦∣𝑥,𝑓^∗)∀𝑥∈𝑆. (3.10)(3.11)(b)𝑔^∗=sup_𝜋∈Π𝑉(𝑥,𝜋) for all 𝑥∈𝑆.(c)Any stationary policy 𝑓∈𝐹 realizing the maximum of (3.10) is average optimal, and so 𝑓^∗ in (3.11) is average optimal.

Then, under Assumptions A, B, and C we shall present the PIA that we are concerned with. To do this, we first give the following definition.

For any real-valued function 𝑢 on 𝑆, we define the dynamic programming operator 𝑇 as follows:

𝑇𝑢(𝑥)∶=max_{𝑎∈𝐴(𝑥)}{𝑟(𝑥,𝑎)+_𝑆𝑢(𝑦)𝑞(𝑑𝑦∣𝑥,𝑎)}∀𝑥∈𝑆. (3.12)

Algorithm A (policy iteration)Step 1 (initialization). Take 𝑛=0 and choose a stationary policy 𝑓_𝑛∈𝐹.Step 2 (policy evaluation). Find a constant 𝑔(𝑓_𝑛) and a real-valued function ℎ_𝑓𝑛 on 𝑆satisfying the Poisson equation (3.7), that is, 𝑔(𝑓_𝑛)=𝑟(𝑥,𝑓_𝑛)+_𝑆ℎ_𝑓𝑛(𝑦)𝑞(𝑑𝑦∣𝑥,𝑓_𝑛)∀𝑥∈𝑆. (3.13) Obviously, by (3.12) and (3.13) we have 𝑔(𝑓_𝑛)≤𝑇ℎ_𝑓𝑛(𝑥)=max_{𝑎∈𝐴(𝑥)}{𝑟(𝑥,𝑎)+_𝑆ℎ_𝑓𝑛(𝑦)𝑞(𝑑𝑦∣𝑥,𝑎)}∀𝑥∈𝑆. (3.14)Step 3 (policy improvement). Set 𝑓_𝑛+1(𝑥)∶=𝑓_𝑛(𝑥) for all 𝑥∈𝑆 for which 𝑟(𝑥,𝑓_𝑛)+_𝑆ℎ_𝑓𝑛(𝑦)𝑞(𝑑𝑦∣𝑥,𝑓_𝑛)=𝑇ℎ_𝑓𝑛(𝑥); (3.15) otherwise (i.e., when (3.15) does not hold), choose 𝑓_𝑛+1(𝑥)∈𝐴(𝑥) such that 𝑟(𝑥,𝑓_𝑛+1)+_𝑆ℎ_𝑓𝑛(𝑦)𝑞(𝑑𝑦∣𝑥,𝑓_𝑛+1)=𝑇ℎ_𝑓𝑛(𝑥). (3.16)Step 4. If 𝑓_𝑛+1 satisfies (3.15) for all 𝑥∈𝑆, then stop (because, from Proposition 4.1 below, 𝑓_𝑛+1 is average optimal); otherwise, replace 𝑓_𝑛 with 𝑓_𝑛+1 and go back to Step 2.

Definition 3.5. The policy iteration Algorithm A is said to converge if the sequence {𝑔(𝑓_𝑛)} converges to the average optimal reward value function in (2.5), that is, lim_𝑛→∞𝑔(𝑓_𝑛)=𝑉^∗(𝑥)=𝑔^∗∀𝑥∈𝑆, (3.17) where 𝑔^∗ is as in Theorem 3.4.

Obviously, under Assumptions A, B, and C from Proposition 4.1 we see that the sequence {𝑔(𝑓_𝑛)} is nondecreasing; that is, 𝑔(𝑓_𝑛)≤𝑔(𝑓_𝑛+1) holds for all 𝑛≥1. On the other hand, by Lemma 3.3(d) we see that {𝑔(𝑓_𝑛)} is bounded. Therefore, there exists a constant 𝑔 such that

lim_𝑛→∞𝑔(𝑓_𝑛)=𝑔. (3.18) Noting that, in general, we have 𝑔≤𝑔^∗. In order to ensure that the policy iteration Algorithm A converges, that is, 𝑔=𝑔^∗, in addition to Assumptions A, B, and C, we need an additional condition (Assumption D (or D^′) below).

Assumption D. There exist a subsequence {ℎ_𝑓𝑚} of {ℎ_𝑓𝑛} and a measurable function ℎ on 𝑆 such that lim_𝑚→∞ℎ_𝑓𝑚(𝑥)=ℎ(𝑥)∀𝑥∈𝑆. (3.19)

Remark 3.6. (a) Assumption D is the same as the hypothesis H1 in [9], and Remark 4.6 in [9] gives a detailed discussion of Assumption D. (b) In particular, Assumption D trivially holds when the state space 𝑆 is a countable set (with the discrete topology). (c) When the state space 𝑆 is not countable, if the sequence {ℎ_𝑓𝑛} is equicontinuous, Assumption D also holds.

Assumption D’. There exists a stationary policy 𝑓^∗∈𝐹 such that lim_𝑛→∞𝑓_𝑛(𝑥)=𝑓^∗(𝑥)∀𝑥∈𝑆. (3.20)

Remark 3.7. Assumption D^′ is the same as the hypothesis H2 in [9]. Obviously, Assumption D^′ trivially holds when the state space 𝑆 is a countable set (with the discrete topology) and 𝐴(𝑥) is compact for all 𝑥∈𝑆.

Finally, we present a lemma (Lemma 3.8) to conclude this section, which is needed to prove our Theorem 4.2. For a proof, see [24, Proposition 12.2], for instance.

Lemma 3.8. Suppose that 𝐴(𝑥) is compact for all 𝑥∈𝑆, and let {𝑓_𝑛} be a stationary policy sequence in 𝐹. Then there exists a stationary policy 𝑓∈𝐹 such that 𝑓(𝑥)∈𝐴(𝑥) is an accumulation point of {𝑓_𝑛(𝑥)} for each 𝑥∈𝑆.

4. Main Results

In this section we will present our main results, Theorems 4.2-4.3. Before stating them, we first give the following proposition, which is needed to prove our main results.

Proposition 4.1. Suppose that Assumptions A, B, and C hold, and let 𝑓∈𝐹 be an arbitrary stationary policy. If any policy 𝑓∈𝐹 such that 𝑇ℎ_𝑓(𝑥)=𝑟(𝑥,𝑓)+_𝑆ℎ_𝑓(𝑦)𝑞(𝑑𝑦∣𝑥,𝑓)∀𝑥∈𝑆, (4.1) then (a) 𝑔(𝑓)≤𝑔(𝑓); (4.2)(b)if 𝑔(𝑓)=𝑔(𝑓), then ℎ_𝑓(⋅)=ℎ_𝑓(⋅)+𝑘forsomeconstant𝑘; (4.3)(c)if 𝑓 is average optimal, then ℎ_𝑓(⋅)=ℎ^∗(⋅)+𝑘^′forsomeconstant𝑘^′, (4.4) where ℎ^∗ is as in Theorem 3.4;(d)if 𝑔(𝑓)=𝑔(𝑓), then (𝑔(𝑓),ℎ_𝑓) satisfies the average reward optimality equation (3.10), and so 𝑓 is average optimal.

Proof. (a) Combining (3.7) and (4.1) we have 𝑔(𝑓)≤𝑟(𝑥,𝑓)+_𝑆ℎ_𝑓(𝑦)𝑞(𝑑𝑦∣𝑥,𝑓)∀𝑥∈𝑆. (4.5) Obviously, taking the integration on both sides of (4.5) with respect to 𝜇_𝑓 and by Remark 3.2(d) we obtain the desired result.
(b) If 𝑔(𝑓)=𝑔(𝑓), we may rewrite the Poisson equation for 𝑓 as
𝑔(𝑓)=𝑟(𝑥,𝑓)+_𝑆ℎ_𝑓(𝑦)𝑞(𝑑𝑦∣𝑥,𝑓)∀𝑥∈𝑆. (4.6) Then, combining (4.5) and (4.6) we obtain _𝑆[ℎ_𝑓(𝑦)−ℎ_𝑓(𝑦)]𝑞(𝑑𝑦∣𝑥,𝑓)≥0∀𝑥∈𝑆. (4.7) Thus, from (4.7) and using the Dynkin formula we get 𝐸^𝑓_𝑥[ℎ_𝑓(𝑥_𝑡)−ℎ_𝑓(𝑥_𝑡)]≥ℎ_𝑓(𝑥)−ℎ_𝑓(𝑥)∀𝑥∈𝑆. (4.8) Letting 𝑡→∞ in (4.8) and by Assumption C we have 𝜇_𝑓(ℎ_𝑓−ℎ_𝑓)≥ℎ_𝑓(𝑥)−ℎ_𝑓(𝑥)∀𝑥∈𝑆. (4.9) Now take 𝑘∶=sup_𝑥∈𝑆[ℎ_𝑓(𝑥)−ℎ_𝑓(𝑥)]. Then take the supremum over 𝑥∈𝑆 in (4.9) to obtain 𝑘≥𝜇_𝑓(ℎ_𝑓−ℎ_𝑓)≥𝑘, (4.10) and so 𝜇_𝑓(ℎ_𝑓−ℎ_𝑓)=𝑘, (4.11) which implies ℎ_𝑓(⋅)=ℎ_𝑓(⋅)+𝑘𝜇_𝑓-a.e. (4.12) Hence, from Remark 3.2(e) and (4.12) we obtain (4.3).
(c) Since 𝑓 is average optimal, by Definition 2.4 and Theorem 3.4(b) we have
𝑔(𝑓)=𝑔^∗=sup_𝜋∈Π𝑉(𝑥,𝜋)∀𝑥∈𝑆. (4.13) Hence, the Poisson equation (3.7) for 𝑓 becomes 𝑔^∗=𝑟(𝑥,𝑓)+_𝑆ℎ_𝑓(𝑦)𝑞(𝑑𝑦∣𝑥,𝑓)∀𝑥∈𝑆. (4.14) On the other hand, by (3.10) we obtain 𝑔^∗=max_{𝑎∈𝐴(𝑥)}{𝑟(𝑥,𝑎)+_𝑆ℎ^∗(𝑦)𝑞(𝑑𝑦∣𝑥,𝑎)}≥𝑟(𝑥,𝑓)+_𝑆ℎ^∗(𝑦)𝑞(𝑑𝑦∣𝑥,𝑓)∀𝑥∈𝑆, (4.15) which together with (4.14) gives _𝑆[ℎ_𝑓(𝑦)−ℎ^∗(𝑦)]𝑞(𝑑𝑦∣𝑥,𝑓)≥0∀𝑥∈𝑆. (4.16) Thus, as in the proof of part (b), from (4.16) we see that (4.4) holds with 𝑘^′∶=sup_𝑥∈𝑆[ℎ_𝑓(𝑥)−ℎ^∗(𝑥)].
(d) By (3.7), (4.1), (4.3), and 𝑄₃ we have
𝑔(𝑓)≤𝑇ℎ_𝑓(𝑥)=𝑟(𝑥,𝑓)+_𝑆[ℎ_𝑓(𝑦)+𝑘]𝑞(𝑑𝑦∣𝑥,𝑓)=𝑟(𝑥,𝑓)+_𝑆ℎ_𝑓(𝑦)𝑞(𝑑𝑦∣𝑥,𝑓)=𝑔(𝑓)=𝑔(𝑓)∀𝑥∈𝑆, (4.17) which gives 𝑔(𝑓)=𝑇ℎ_𝑓(𝑥)∀𝑥∈𝑆, (4.18) that is, 𝑔(𝑓)=max_{𝑎∈𝐴(𝑥)}{𝑟(𝑥,𝑎)+_𝑆ℎ_𝑓(𝑦)𝑞(𝑑𝑦∣𝑥,𝑎)}∀𝑥∈𝑆. (4.19) Thus, as in the proof of Theorem 4.1 in [14], from Lemma 2.3(b), (3.7), and (4.19) we show that 𝑓 is average optimal, that is, 𝑔(𝑓)=𝑔^∗. Hence, we may rewrite (4.19) as 𝑔^∗=max_{𝑎∈𝐴(𝑥)}{𝑟(𝑥,𝑎)+_𝑆ℎ_𝑓(𝑦)𝑞(𝑑𝑦∣𝑥,𝑎)}∀𝑥∈𝑆. (4.20) Thus, from (4.20) and part (c) we obtain the desired conclusion.

Theorem 4.2. Suppose that Assumptions A, B, C, and D hold, then the policy iteration Algorithm A converges.

Proof. From Lemma 3.3(a) we see that the function ℎ_𝑓𝑛 in (3.13) belongs to 𝐵_𝑤1(𝑆), and so the function ℎ in (3.19) also belongs to 𝐵_𝑤1(𝑆). Now let {ℎ_𝑓𝑛} be as in Assumption D, and let {ℎ_𝑓𝑚} be the corresponding subsequence of {ℎ_𝑓𝑛}. Then by Assumption D we have lim_𝑚→∞ℎ_𝑓𝑚(𝑥)=ℎ(𝑥)∀𝑥∈𝑆. (4.21) Moreover, from Lemma 3.8 there is a stationary policy 𝑓∈𝐹 such that 𝑓(𝑥)∈𝐴(𝑥) is an accumulation point of {𝑓_𝑚(𝑥)} for each 𝑥∈𝑆; that is, for each 𝑥∈𝑆 there exists a subsequence {𝑚_𝑖} (depending on the state 𝑥) such that lim_𝑖→∞𝑓_𝑚𝑖(𝑥)=𝑓(𝑥)∀𝑥∈𝑆. (4.22) Also, by (3.13) we get 𝑔(𝑓_𝑚𝑖)=𝑟(𝑥,𝑓_𝑚𝑖)+_𝑆ℎ_𝑓𝑚𝑖(𝑦)𝑞(𝑑𝑦∣𝑥,𝑓_𝑚𝑖)∀𝑥∈𝑆. (4.23) On the other hand, take any real-valued measurable function 𝑚 on 𝑆 such that 𝑚(𝑥)>𝑞(𝑥)≥0 for all 𝑥∈𝑆. Then, for each 𝑥∈𝑆 and 𝑎∈𝐴(𝑥), by the properties (𝑄₁)-(𝑄₃) we can define 𝑃(⋅∣𝑥,𝑎) as follows: 𝑃(𝐷∣𝑥,𝑎)∶=𝑞(𝐷∣𝑥,𝑎)𝑚(𝑥)+𝐼_𝐷(𝑥)∀𝐷∈ℬ(𝑆). (4.24) Obviously, 𝑃(⋅∣𝑥,𝑎) is a probability measure on 𝑆. Thus, combining (4.23) and (4.24) we have 𝑔(𝑓_𝑚𝑖)=𝑟(𝑥,𝑓_𝑚𝑖)+_𝑆ℎ_𝑓𝑚𝑖(𝑦)𝑃(𝑑𝑦∣𝑥,𝑓_𝑚𝑖)∀𝑥∈𝑆. (4.25) Letting 𝑖→∞ in (4.25), then by (3.18), (4.21), and (4.22) as well as the “extension of Fatou's lemma 8.3.7" in [8] we obtain 𝑔=𝑟(𝑥,𝑓)+_𝑆ℎ(𝑦)𝑞(𝑑𝑦∣𝑥,𝑓)∀𝑥∈𝑆. (4.26) To complete the proof of Theorem 4.2, by Proposition 4.1(d) we only need to prove that 𝑔,ℎ, and 𝑓 satisfy the average reward optimality equation (3.10) and (3.11), that is, 𝑔=𝑇ℎ(𝑥)=𝑟(𝑥,𝑓)+_𝑆ℎ(𝑦)𝑞(𝑑𝑦∣𝑥,𝑓)∀𝑥∈𝑆. (4.27) Obviously, from (4.26), and the definition of 𝑇 in (3.12) we obtain 𝑔≤𝑇ℎ(𝑥)∀𝑥∈𝑆. (4.28) The rest is to prove the reverse inequality, that is, 𝑔≥𝑇ℎ(𝑥)∀𝑥∈𝑆. (4.29) Obviously, by (3.19) we have lim_𝑖→∞[ℎ_𝑓𝑚𝑖(𝑥)−ℎ_{𝑓𝑚𝑖−1}(𝑥)]=0∀𝑥∈𝑆. (4.30) Moreover, from Lemma 3.3(a) again we see that there exists a constant 𝑘 such that ‖ℎ_𝑓𝑛‖_𝑤1≤𝑘∀𝑛≥1, (4.31) which gives ‖ℎ_𝑓𝑚𝑖−ℎ_{𝑓𝑚𝑖−1}‖_𝑤1≤‖ℎ_𝑓𝑚𝑖‖_𝑤1+‖ℎ_{𝑓𝑚𝑖−1}‖_𝑤1≤2𝑘. (4.32) Thus, by (4.24), (4.31), (4.32) and the “extension of Fatou's lemma 8.3.7" in [8] we obtain lim_{𝑖→∞𝑆}[ℎ_𝑓𝑚𝑖(𝑦)−ℎ_{𝑓𝑚𝑖−1}(𝑦)]𝑃(𝑑𝑦∣𝑥,𝑓_𝑚𝑖)=0∀𝑥∈𝑆, (4.33) which implies lim_{𝑖→∞𝑆}[ℎ_𝑓𝑚𝑖(𝑦)−ℎ_{𝑓𝑚𝑖−1}(𝑦)]𝑞(𝑑𝑦∣𝑥,𝑓_𝑚𝑖)=0∀𝑥∈𝑆. (4.34) Also, from (3.7), (3.16), and the definition of 𝑇 in (3.12) we get 𝑔(𝑓_𝑚𝑖)=𝑟(𝑥,𝑓_𝑚𝑖)+_𝑆ℎ_𝑓𝑚𝑖(𝑦)𝑞(𝑑𝑦∣𝑥,𝑓_𝑚𝑖)=𝑇ℎ_{𝑓𝑚𝑖−1}(𝑥)+_𝑆[ℎ_𝑓𝑚𝑖(𝑦)−ℎ_{𝑓𝑚𝑖−1}(𝑦)]𝑞(𝑑𝑦∣𝑥,𝑓_𝑚𝑖)≥𝑟(𝑥,𝑎)+_𝑆ℎ_{𝑓𝑚𝑖−1}+(𝑦)𝑞(𝑑𝑦∣𝑥,𝑎)_𝑆[ℎ_𝑓𝑚𝑖(𝑦)−ℎ_{𝑓𝑚𝑖−1}(𝑦)]𝑞(𝑑𝑦∣𝑥,𝑓_𝑚𝑖)∀𝑥∈𝑆,𝑎∈𝐴(𝑥). (4.35) Letting 𝑖→∞ in (4.35), then by (3.18), (4.21), (4.22), (4.34), and the “extension of Fatou's lemma 8.3.7" in [8] we obtain 𝑔≥𝑟(𝑥,𝑎)+_𝑆ℎ(𝑦)𝑞(𝑑𝑦∣𝑥,𝑎)∀𝑥∈𝑆,𝑎∈𝐴(𝑥), (4.36) which gives 𝑔≥max_{𝑎∈𝐴(𝑥)}{𝑟(𝑥,𝑎)+_𝑆ℎ(𝑦)𝑞(𝑑𝑦∣𝑥,𝑎)}=𝑇ℎ(𝑥)∀𝑥∈𝑆. (4.37) This completes the proof of Theorem 4.2.

Theorem 4.3. Suppose that Assumptions A, B, C, and D^′ hold, then the policy iteration Algorithm A converges.

Proof. To prove Theorem 4.3, from the proof of Theorem 4.2 we only need to verify that (4.26) and (4.27) hold true for 𝑓^∗ as in Assumption D^′ and some function ℎ in 𝐵_𝑤1(𝑆). To do this, we first define two functions ℎ,ℎ^′ in 𝐵_𝑤1(𝑆) as follows: ℎ(𝑥)∶=limsup_𝑛→∞ℎ_𝑓𝑛(𝑥),ℎ^′(𝑥)∶=liminf_𝑛→∞ℎ_𝑓𝑛(𝑥)∀𝑥∈𝑆. (4.38) Then by (3.7) we get 𝑔(𝑓_𝑛)=𝑟(𝑥,𝑓_𝑛)+_𝑆ℎ_𝑓𝑛(𝑦)𝑞(𝑑𝑦∣𝑥,𝑓_𝑛)∀𝑥∈𝑆, (4.39) which together with (4.24) yields 𝑔(𝑓_𝑛)=𝑟(𝑥,𝑓_𝑛)+_𝑆ℎ_𝑓𝑛(𝑦)𝑃(𝑑𝑦∣𝑥,𝑓_𝑛)∀𝑥∈𝑆. (4.40) Applying the “extension of Fatou's Lemma" 8.3.7 in [8] and letting 𝑛→∞ in (4.40), then by (3.18), (4.38) and Assumption D^′ we obtain 𝑔≤𝑟(𝑥,𝑓^∗)+_𝑆ℎ(𝑦)𝑃(𝑑𝑦𝑥,𝑓^∗)∀𝑥∈𝑆,𝑔≥𝑟(𝑥,𝑓^∗)+_𝑆ℎ^′(𝑦)𝑃(𝑑𝑦∣𝑥,𝑓^∗)∀𝑥∈𝑆, (4.41) which implies 𝑔≤𝑟(𝑥,𝑓^∗)+_𝑆ℎ(𝑦)𝑞(𝑑𝑦∣𝑥,𝑓^∗)∀𝑥∈𝑆,𝑔≥𝑟(𝑥,𝑓^∗)+_𝑆ℎ^′(𝑦)𝑞(𝑑𝑦∣𝑥,𝑓^∗)∀𝑥∈𝑆. (4.42)(4.43) Thus, combining (4.42) and (4.43) we get _𝑆[ℎ(𝑦)−ℎ^′(𝑦)]𝑞(𝑑𝑦∣𝑥,𝑓^∗)≥0∀𝑥∈𝑆. (4.44) Then, from the proof of Proposition 4.1(b) and (4.44) we have ℎ(⋅)=ℎ^′(⋅)+𝑘^′′forsomeconstant𝑘^′′, (4.45) which together with (4.42), (4.43), and the definition of 𝑇 in (3.12) gives 𝑔=𝑟(𝑥,𝑓^∗)+_𝑆ℎ(𝑦)𝑞(𝑑𝑦∣𝑥,𝑓^∗)≤𝑇ℎ(𝑥)∀𝑥∈𝑆. (4.46) The remainder is to prove the reverse inequality, that is, 𝑔≥𝑇ℎ(𝑥)=max_{𝑎∈𝐴(𝑥)}{𝑟(𝑥,𝑎)+_𝑆ℎ(𝑦)𝑞(𝑑𝑦∣𝑥,𝑎)}∀𝑥∈𝑆. (4.47) Obviously, by (3.16) and (4.24) we get 𝑟(𝑥,𝑓_𝑛+1)+𝑚(𝑥)_𝑆ℎ_𝑓𝑛(𝑦)𝑃(𝑑𝑦∣𝑥,𝑓_𝑛+1)≥𝑟(𝑥,𝑎)+𝑚(𝑥)_𝑆ℎ_𝑓𝑛(𝑦)𝑃(𝑑𝑦∣𝑥,𝑎)∀𝑥∈𝑆,𝑎∈𝐴(𝑥). (4.48) Then, letting 𝑛→∞ in (4.48), by (4.38), Assumption D^′, and the “extension of Fatou's Lemma" 8.3.7 in [8], we obtain 𝑟(𝑥,𝑓^∗)+𝑚(𝑥)_𝑆ℎ(𝑦)𝑃(𝑑𝑦∣𝑥,𝑓^∗)≥𝑟(𝑥,𝑎)+𝑚(𝑥)_𝑆ℎ(𝑦)𝑃(𝑑𝑦∣𝑥,𝑎)∀𝑥∈𝑆,𝑎∈𝐴(𝑥), (4.49) which implies 𝑟(𝑥,𝑓^∗)+_𝑆ℎ(𝑦)𝑞(𝑑𝑦∣𝑥,𝑓^∗)≥𝑟(𝑥,𝑎)+_𝑆ℎ(𝑦)𝑞(𝑑𝑦∣𝑥,𝑎)∀𝑥∈𝑆,𝑎∈𝐴(𝑥), (4.50) and so 𝑟(𝑥,𝑓^∗)+_𝑆ℎ(𝑦)𝑞(𝑑𝑦∣𝑥,𝑓^∗)≥max_{𝑎∈𝐴(𝑥)}{𝑟(𝑥,𝑎)+_𝑆ℎ(𝑦)𝑞(𝑑𝑦∣𝑥,𝑎)}∀𝑥∈𝑆. (4.51) Thus, combining (4.46) and (4.51) we see that (4.47) holds. And so Theorem 4.3 follows.

5. Concluding Remarks

In the previous sections we have studied the policy iteration algorithm (PIA) for average reward continuous-time jump MDPs in Polish spaces. Under two slightly different sets of conditions we have shown that the PIA yields the optimal (maximum) reward, an average optimal stationary policy, and a solution to the average reward optimality equation. It should be mentioned that the approach presented here is different from the policy iteration approach used in [14] because the PIA in this paper provides an approach to compute or at least approximate (when the PIA takes infinitely many steps to converge) the value of the average optimal reward value function and an average optimal stationary policy.

Acknowledgments

The author would like to thank the editor and anonymous referees for their good comments and valuable suggestions, which have helped us to improve the paper. This work was jointly supported by the National Natural Science Foundation of China (10801056), the Natural Science Foundation of Ningbo (201001A6011005) the Scientific Research Fund of Zhejiang Provincial Education Department, K.C. Wong Magna Fund in Ningbo University, the Natural Science Foundation of Yunnan Provincial Education Department (07Y10085), the Natural Science Foundation of Yunnan Provincial (2008CD186), the Foundation of Chinese Society for Electrical Engineering (2008).