A Version of the Euler Equation in Discounted Markov Decision Processes

Abstract

This paper deals with Markov decision processes (MDPs) on Euclidean spaces with an infinite horizon. An approach to study this kind of MDPs is using the dynamic programming technique (DP). Then the optimal value function is characterized through the value iteration functions. The paper provides conditions that guarantee the convergence of maximizers of the value iteration functions to the optimal policy. Then, using the Euler equation and an envelope formula, the optimal solution of the optimal control problem is obtained. Finally, this theory is applied to a linear-quadratic control problem in order to find its optimal policy.

1. Introduction

This paper deals with the optimal control problem in discrete time and with an infinite horizon. This problem is presented with the help of the Markov decision processes (MDPs) theory. To describe the MDPs, it is necessary to provide a Markov control model. The components of the Markov control model are used to describe the dynamic of the system. In this way at each time 𝑡 (𝑡=0,1,…) the state of the system is affected by an admissible action. This sequence of actions is called a policy. The optimal control problem consists in determining an optimal policy, which is characterized through a performance criterion. In this paper, the infinite-horizon expected total discounted reward is considered.

An approach for solving the optimal control problem is through the dynamic programming technique (DP) (see [1–4]). DP characterizes the optimal solution of the optimal control problem using a functional equation, known as the dynamic programming equation (see [1–4]). In the literature there exists conditions that guarantee the value iteration (VI) functions procedure, which is used to approximate the optimal value function of the optimal control problem. However, this technique has problems when the reward and/or dynamic have a complicated functional form (see [5, page 93]).

An alternative for solving this problem is using the Euler Equation (EE), which is well known in the applications of MDPs to economic models. This equation is established and solved in this context (in some cases empirically) (see [6–13]).

An iterative method for deterministic MDPs is presented in [14]. In this case, the EE is obtained in terms of the VI functions. Retaking this idea, this article presents an iterative method of finding the solution of EE in terms of the VI functions in stochastic MDPs.

In this paper, the Euler equation is obtained using an envelope formula (see [15–17]) under interiority conditions of the VI functions. The envelope formula characterizes the performance criterion derivative with respect to the initial state of the system. The performance criterion derivative is important in analyzing the behavior of the Markov control process. Also, in [18], a general study about the performance sensitivities in the policy space is presented. In this context, two performance sensitivity formulas are studied: one for performance derivatives at any policy in the policy space and the other one for performance differences between any two policies in the policy space.

The technique proposed in this paper is used as follows. Firstly, EE is applied to obtain the VI functions. Secondly, applying the envelope formula, the maximizers of the VI functions are obtained. Then using the maximizers convergence to the optimal policy, the optimal control problem is solved. This procedure is exemplified by a linear quadratic problem.

The paper is organized as follows: in Section 2, the theory of MDPs necessary for subsequent sections is presented. In Section 3, some conditions on the Markov Control Model are presented to ensure both the differentiability of the VI functions and the optimal value function. These conditions guarantee the validity of a version of the EE for the VI. Finally, in Section 4, a linear quadratic problem is presented to illustrate the theory.

2. Markov Decision Process

A discrete-time markov control model is a quintuple (𝑋,𝐴,{𝐴(𝑥)∣𝑥∈𝑋},𝑄,𝑟), where 𝑋 is the state space, 𝐴 is the action space, 𝐴(𝑥) is the set of feasible actions in the state 𝑥∈𝑋, 𝑄 is a transition law and 𝑟∶𝕂→ℝ is the one-step reward function (see [3]). 𝑋 and 𝐴 are (nonempty) Borel spaces with the Borel 𝜎-algebras ℬ(𝑋) and ℬ(𝐴), respectively. 𝑄(⋅∣⋅) is a stochastic kernel on 𝑋 given 𝕂, where 𝕂∶={(𝑥,𝑎)∣𝑥∈𝑋,𝑎∈𝐴(𝑥)}, and 𝑟 is a measurable function.

Consider a Markov control model and, for each 𝑡=0,1,…, define the space ℍ_𝑡 of admissible histories up to time 𝑡 as ℍ₀=𝑋, and ℍ_𝑡=𝕂×ℍ_𝑡−1, for 𝑡=1,2,….

A policy is a sequence 𝜋={𝜋_𝑡} of stochastic kernels 𝜋_𝑡 on the action space 𝐴 given ℍ_𝑡. The set of policies will be denoted by Π.

Let 𝔽 be the set of decision functions or measurable selectors, that is, the set of all measurable functions 𝑓∶𝑋→𝐴 such that 𝑓(𝑥)∈𝐴(𝑥) for all 𝑥∈𝑋.

A sequence {𝑓_𝑡} of functions 𝑓_𝑡∈𝔽 is called a Markov policy. A stationary policy is a Markov policy 𝜋={𝑓_𝑡} such that 𝑓_𝑡=𝑓 for all 𝑡=0,1,2,…, with 𝑓∈𝔽, and it will be denoted by 𝑓 (see [3]).

Given the initial state 𝑥₀=𝑥∈𝑋, and any policy 𝜋∈Π, there is a probability measure 𝑃^𝜋_𝑥 on the space (Ω,ℱ), with Ω∶=(𝑋×𝐴)^∞ and ℱ, the product 𝜎-algebra (see [3]). The corresponding expectation operator will be denoted by 𝐸^𝜋_𝑥. The stochastic process ((Ω,ℱ,𝑃^𝜋_𝑥),{𝑥_𝑡}) is called a discrete-time Markov decision process.

The total expected discounted reward is defined as 𝑣(𝜋,𝑥)∶=𝐸^𝜋_𝑥^∞_𝑡=0𝛼^𝑡𝑟𝑥_𝑡,𝑎_𝑡, (2.1)𝜋∈Π and 𝑥∈𝑋, where 𝛼∈(0,1) is called the discount factor.

Definition 2.1. A policy 𝜋^∗∈Π is optimal, if for each 𝑥∈𝑋, 𝑣𝜋^∗,𝑥=sup_𝜋∈Π𝑣(𝜋,𝑥). (2.2) The function defined by 𝑉(𝑥)=sup_𝜋∈Π𝑣(𝜋,𝑥), (2.3)𝑥∈𝑋, will be called the optimal value function.

The optimal control problem consists in determining an optimal policy.

2.1. Dynamic Programming

Definition 2.2. A measurable function 𝜆∶𝑋→ℝ is said to be a solution to the optimal equation (OE) if it satisfies

𝜆(𝑥)=sup_{(𝑥)𝑎∈𝐴}𝑟(𝑥,𝑎)+𝛼_𝑋𝜆𝑦𝑄𝑑𝑦∣𝑥,𝑎, (2.4)

𝑥∈𝑋.

Assumption 2.3. (a) The one-step reward function 𝑟 is nonpositive, upper semicontinuous (u.s.c), and sup-compact on 𝕂. (𝑟 is a sup-compact function if the set {𝑎∈𝐴(𝑥)∣𝑟(𝑥,𝑎)≥𝛾} is compact for every 𝑥∈𝑋 and 𝛾∈ℝ.)
(b) The transition law 𝑄 is strongly continuous.
(c) There exists a policy 𝜋 such that 𝑣(𝜋,𝑥)>−∞, for each 𝑥∈𝑋.

Definition 2.4. The value iteration (VI) functions are defined as follows: 𝑣_𝑛(𝑥)=sup_{(𝑥)𝑎∈𝐴}𝑟(𝑥,𝑎)+𝛼_𝑋𝑣_𝑛−1𝑦𝑄𝑑𝑦∣𝑥,𝑎, (2.5) for all 𝑥∈𝑋 and 𝑛=1,2,…, with 𝑣₀(𝑥)=0.

The following theorem is well-known in the literature of MDPs (see, [1–4]). The proof can be consulted in (see [3, page 46]).

Theorem 2.5. Suppose that Assumption 2.3 holds. Then
(a) The optimal value function 𝑉 is a solution of the OE (see Definition 2.2).
(b) There exists 𝑓∈𝔽 such that 𝑉(𝑥)=𝑟𝑥,𝑓(𝑥)+𝛼_𝑋𝑉𝑦𝑄𝑑𝑦∣𝑥,𝑓(𝑥). (2.6)
(c) For every 𝑥∈𝑋, 𝑣_𝑛(𝑥)→𝑉(𝑥), when 𝑛→∞.

Remark 2.6. Under Assumption 2.3, it is possible to demonstrate that for each 𝑛=1,2,…, there exists a stationary policy 𝑓_𝑛∈𝔽 such that 𝑣_𝑛(𝑥)=𝑟𝑥,𝑓_𝑛(𝑥)+𝛼_𝑋𝑣_𝑛−1𝑦𝑄𝑑𝑦∣𝑥,𝑓_𝑛(𝑥), (2.7)𝑥∈𝑋 (see [3, page. 27, 28]).

3. Differentiability in MDPs

3.1. Notation and Preliminaries

Let 𝑋 and 𝑌 be Euclidean spaces and consider the following notation: 𝐶²(𝑋,𝑌) denotes the set of functions 𝑙∶𝑋→𝑌 with a continuous second derivative (when 𝑋=𝑌, 𝐶²(𝑋,𝑌) will be denoted by 𝐶²(𝑋) and in some cases it will be written only as 𝐶²). Let Γ∶𝑋×𝑌→ℝ be a measurable function such that Γ∈𝐶²(𝑋×𝑌,ℝ). Γ_𝑥, and Γ_𝑦 denote the partial derivative of Γ for 𝑥 and 𝑦, respectively. The notations for the second partial derivatives of Γ are Γ_𝑥𝑥, Γ_𝑥𝑦, Γ_𝑦𝑥 and Γ_𝑦𝑦.

For any set 𝐶⊂𝑋, a point 𝑥∈𝐶 is called an interior point of 𝐶 if there exists an open set 𝑈 such that 𝑥∈𝑈⊂𝐶. The interior of 𝐶 is the set of all interior points of 𝐶 denoted by int(𝐶).

The set-value mapping Θ from 𝑋 to 𝑌 is said to be(a) nondecreasing, if 𝑥,𝑧∈𝑋 with 𝑥<𝑧 then Θ(𝑥)⊆Θ(𝑧),(b)convex, if 𝑥,𝑧∈𝑋 and 𝛽∈[0,1], then 𝛽𝑎+(1−𝛽)𝑎∈Θ(𝛽𝑥+(1−𝛽)𝑧), with 𝑎∈Θ(𝑥) and 𝑎∈Θ(𝑧).

Let Υ∶𝕂→ℝ be a measurable function. Define 𝜐∶𝑋→ℝ by 𝜐(𝑥)∶=sup_{(𝑥)𝑎∈Θ}Υ𝑥,𝑦, (3.1)𝑥∈𝑋.

The proof of the following lemma is similar to the proof of Theorem 1 in [16].

Lemma 3.1. Suppose that(a)Υ∈𝐶²(int(𝕂);ℝ), furthermore Υ_𝑦𝑦(𝑥,⋅) is negative definite, for every 𝑥∈𝑋;(b) for each 𝑥∈𝑋, argmaxΥ(𝑥,𝑦)⊆int(Θ(𝑥)).
Then there exists a function 𝑙∶𝑋→𝐴 such that 𝜐(𝑥)=Υ(𝑥,𝑙(𝑥)), for every 𝑥∈𝑋. Moreover 𝑙∈𝐶¹(int(𝑋);𝑌) and Υ∈𝐶²(int(𝑋);ℝ).

Remark 3.2. Observe that (a) implies that Λ(𝑥,⋅) is a strictly concave function, for each 𝑥∈𝑋. Then the maximizer 𝑙∶𝑋→𝐴 is unique.
The proof of the following lemma can be consulted in [19], Theorem 25.7, page 248.

Lemma 3.3. Let 𝐶⊂ℝ^𝑛 be an open and convex set. Let 𝑔∶𝐶→ℝ be a concave and differentiable function, and {𝑔_𝑛} be a sequence of differentiable, concave and real-valued functions on 𝐶, such that 𝑔_𝑛(𝑥)→𝑔(𝑥), when 𝑛→∞, for all 𝑥∈𝐶. Then lim_𝑛→∞𝑔^′_𝑛(𝑥)=𝑔^′(𝑥). (3.2)

3.2. An Envelope Formula in MDPs

Let (𝑋,𝐴,{𝐴(𝑥)∣𝑥∈𝑋},𝑄,𝑟) be a fixed Markov control model. Throughout this section it is assumed that Assumption 2.3 holds. Also, it is supposed that 𝑋⊆ℝ^𝑛 and 𝐴⊆ℝ^𝑚 are convex sets with nonempty interiors and 𝑋 is partially ordered. It is considered that the set-valued mapping 𝑥→𝐴(𝑥) is nondecreasing and convex, and 𝐴(𝑥) has nonempty interior, for each 𝑥∈𝑋. Also, it is assumed that the transition law 𝑄 is given by a difference equation: 𝑥_𝑡+1=𝐿𝐹𝑥_𝑡,𝑎_𝑡,𝜉_𝑡, (3.3)𝑡=0,1,…, with a given initial state 𝑥₀=𝑥∈𝑋 fixed, where {𝜉_𝑡} is a sequence of independent and identically distributed (iid) random variables, independent of 𝑥₀=𝑥∈𝑋 and taking values in a Borel space 𝑆⊂ℝ^𝑘. Let 𝜉 be a generic element of the sequence {𝜉_𝑡}. The density of 𝜉 is designated by Δ; 𝐿∶𝑋^′×𝑆→𝑋 is a measurable function, with 𝑋^′⊂ℝ^𝑚, and 𝐹∶𝕂→𝑋^′, is a measurable function too.

Since Assumption 2.3 is assumed, then Theorem 2.5 yields. Therefore, the optimal value function (see Definition 2.1) satisfies 𝑉(𝑥)=sup_{(𝑥)𝑎∈𝐴}𝑟(𝑥,𝑎)+𝛼𝐸𝑉(𝐿(𝐹(𝑥,𝑎),𝜉)), (3.4) and the VI functions (see Definition 2.4) satisfy 𝑣_𝑛(𝑥)=sup_{(𝑥)𝑎∈𝐴}𝑟(𝑥,𝑎)+𝛼𝐸𝑣_𝑛−1(𝐿(𝐹(𝑥,𝑎),𝜉)), (3.5) for each 𝑛=1,2,…, with 𝑣₀(𝑥)∶=0. In addition, by Theorem 2.5, there exists the optimal policy, which will be denoted by 𝑓. Furthermore, there exists the maximizer 𝑓_𝑛 of 𝑣_𝑛 for each 𝑛=1,2,… (see Remark 2.6).

Let 𝐺∶𝕂→ℝ be a function defined as 𝐺(𝑥,𝑎)∶=𝑟(𝑥,𝑎)+𝛼𝐻(𝑥,𝑎), (3.6)(𝑥,𝑎)∈𝕂, where 𝐻(𝑥,𝑎)∶=𝐸𝑉(𝐿(𝐹(𝑥,𝑎),𝜉)). (3.7)

Define 𝐺^𝑛∶𝕂→ℝ by 𝐺^𝑛(𝑥,𝑎)∶=𝑟(𝑥,𝑎)+𝛼𝐸𝑣_𝑛−1(𝐿(𝐹(𝑥,𝑎),𝜉)), (3.8) for each 𝑛=1,2,…, with 𝑣₀(𝑥)∶=0 and (𝑥,𝑎)∈𝕂.

Assumption 3.4. (a) 𝑟 is a strictly concave function and 𝑟(⋅,𝑎) is an increasing function on 𝑋 for each 𝑎∈𝐴 fixed;
(b) 𝐿(⋅,𝑠) is a concave and increasing function, for each 𝑠∈𝑆; 𝐹 is a concave function, 𝐹(⋅,𝑎) is an increasing function on 𝑋, for each 𝑎∈𝐴.

Lemma 3.5. Under Assumption 3.4, it results that 𝑣_𝑛 is a strictly concave function and 𝑓_𝑛 is unique, for all 𝑛=1,2,…. Also, 𝑉 is a strictly concave function and 𝑓 is unique.

Proof. By Assumption 3.4(a), it suffices to prove Condition C1 (see [20, Lemma 6.2]), which guarantees the result. Let Ψ∶𝕂×𝑆→𝑋 be defined by Ψ(𝑥,𝑎,𝑠)∶=𝐿(𝐹(𝑥,𝑎),𝑠). (3.9)
Then for each 𝑠∈𝑆, the function Ψ(⋅,⋅,𝑠) is concave in 𝕂 by Assumption 3.4(b). Indeed, since 𝐹 is a concave function, then 𝐹𝛽(𝑥,𝑎)+1−𝛽𝑦,𝑏≥𝛽𝐹(𝑥,𝑎)+1−𝛽𝐹𝑦,𝑏, (3.10)(𝑥,𝑎),(𝑦,𝑏)∈𝕂 and 𝛽∈[0,1]. Furthermore, it is known that 𝐿(⋅,𝑠) is a concave and increasing function, for each 𝑠∈𝑆, then 𝐿𝐹𝛽(𝑥,𝑎)+1−𝛽𝑦,𝑏,𝑠≥𝛽𝐿(𝐹(𝑥,𝑎),𝑠)+1−𝛽𝐿𝐹𝑦,𝑏,𝑠. (3.11)
From similar arguments, it can be shown that if 𝑥<𝑦, then Ψ(𝑥,𝑎,𝑠)≤Ψ(𝑦,𝑎,𝑠), for each 𝑠∈𝑆 and 𝑎∈𝐴(𝑦). Then the result follows.

Assumption 3.6. (a) 𝑟∈𝐶²(int(𝕂);ℝ) and 𝑟_𝑎𝑎(𝑥,⋅) is negative definite for each 𝑥∈𝑋;
(b) 𝐹∈𝐶²(int(𝕂);𝑋^′) and 𝐹_𝑎(𝑥,⋅) is invertible, for each 𝑎∈𝐴;
(c) 𝐿(⋅,𝑠)∈𝐶²(int(𝑋^′);𝑋), for each 𝑠∈𝑆. Besides, 𝐿(⋅,⋅) has an inverse in the second variable 𝑅, such that 𝑅(⋅,𝑠)∈𝐶²(int(𝑋^′×𝑋);𝑆), and |det𝑅_𝑠(⋅,𝑠)|∈𝐶²(int(𝑋^′);ℝ), for all 𝑠∈𝑆, where, in this case, 𝑅_𝑠 denotes the derivative of 𝑅 with respect to the second variable, and the determinant of 𝑅_𝑠 is denoted as det𝑅_𝑠;
(d) Δ∈𝐶²(int(𝑆);𝑅) and the interchange between derivatives and integrals is valid (see Remark 3.8).

Lemma 3.7. By Assumption 3.6 it results that 𝐻∈𝐶²(int(𝕂);𝑋^′), with 𝐻 defined in (3.7).

Proof. The proof is similar to the proof of Lemma 5 in [16]. Assumption 3.6 allows to express the stochastic kernel (see (3.3)) in the following form: for each measurable subset 𝐵 of 𝑋 and (𝑥,𝑎)∈𝕂, 𝑄((((𝐹()((𝐹()=𝐵∣𝑥,𝑎))=Pr𝑠∈𝑆∣𝐿𝑥,𝑎,𝑠))=Pr(𝑠∈𝑆∣𝑠𝑠∈𝑅𝑥,𝑎,𝐵))_{𝑅(𝐹(𝑥,𝑎),𝐵)}Δ(𝑢)𝑑𝑢. (3.12)(3.13) Then for the change of variable theorem, it results that 𝑄(𝐵∣(𝑥,𝑎))=_{𝑅(𝐹(𝑥,𝑎),𝐵)}Δ(𝑅(𝐹(𝑥,𝑎),𝑢))det𝑅_𝑠(𝐹(𝑥,𝑎),𝑢)𝑑𝑢. (3.14) It follows from (3.13) that 𝐻 can be expressed as 𝐻(𝑥,𝑎)=_{𝑅(𝐹(𝑥,𝑎),𝐵)}𝑉(𝑢)Δ(𝑅(𝐹(𝑥,𝑎),𝑢))det𝑅_𝑠(𝐹(𝑥,𝑎),𝑢)𝑑𝑢. (3.15)
Now, using Assumption 3.6, the result follows.

Remark 3.8. In Lemma 3.7, Assumption 3.6(d) was used to guarantee the differentiability of the second order of the integral ∫𝐾(𝑥,𝑎,𝑢)𝑑𝑢, with respect to 𝑥 or 𝑎, where 𝐾(𝑥,𝑎,𝑢)∶=𝑉(𝑢)Δ(𝑅(𝐹(𝑥,𝑎),𝑢))det𝑅_𝑠(𝐹(𝑥,𝑎),𝑢), (3.16)(𝐹(𝑥,𝑎),𝑢)∈int(𝑋^′×𝑋). This condition can be verified in practice when the derivatives of 𝐾 can be bounded in the following sense: for (𝐹(𝑥,𝑎),𝑢)∈int(𝑋^′×𝑋), |𝐾_𝑥(𝑥,𝑎,𝑢)|≤𝑘₁(𝑎,𝑢),|𝐾_𝑎(𝑥,𝑎,𝑢)|≤𝑘₂(𝑥,𝑢), |𝐾_𝑥𝑥(𝑥,𝑎,𝑢)|≤𝑘₃(𝑎,𝑢), |𝐾_𝑎𝑎(𝑥,𝑎,𝑢)|≤𝑘₄(𝑎,𝑢), |𝐾_𝑥𝑎(𝑥,𝑎,𝑢)|≤𝑘₁(𝑎,𝑢), for some functions 𝑔_𝑖 integrable with respect to 𝑢, for 𝑖=1,…,5 (see Remark 10 in [16]).

Assumption 3.9. (a) The optimal policy 𝑓 satisfies that 𝑓(𝑥)∈int(𝐴(𝑥)), for each 𝑥∈𝑋;
(b) The sequence {𝑓_𝑛} of the maximizers of the VI functions satisfies that 𝑓_𝑛(𝑥)∈int(𝐴(𝑥)), for each 𝑥∈𝑋 and 𝑛=1,2,….
Define 𝑊 by 𝑊(𝑥,𝑎)∶=𝑟_𝑥−𝑟_𝑎𝐹_𝑎⁻¹𝐹_𝑥(𝑥,𝑎), (3.17)(𝑥,𝑎)∈𝕂.

Remark 3.10. Assumption 3.9 evidently holds; if 𝐴(𝑥) is open for every 𝑥∈𝑋, then 𝑓(𝑥) (𝑓_𝑛(𝑥)) belongs to the interior of 𝐴(𝑥), 𝑥∈𝑋. Also, in some particular cases (see [8, 16]), the interiority of 𝑓(𝑥) (𝑓_𝑛(𝑥)) is guaranteed by the mean value theorem.

Theorem 3.11. Under Assumptions 3.4, 3.6, and 3.9(a), it results that 𝑓∈𝐶¹(int(𝑋);𝐴), 𝑉∈𝐶²(int(𝑋);ℝ) and for each 𝑥∈int(𝑋), 𝑉^′(𝑥)=𝑊𝑥,𝑓(𝑥), (3.18) where 𝑊 is defined in (3.17).

Proof. Let 𝑥∈int(𝑋) fixed. Note that Assumptions 3.4 and 3.6 imply that 𝐺∈𝐶²(int(𝕂);ℝ) where 𝐺 is defined in (3.6). Indeed, since Assumptions 3.4(a) and 3.6(a) hold, it is known that 𝑟∈𝐶²(int(𝕂);ℝ) and 𝑟_𝑎𝑎(𝑥,⋅) is negative definite. Moreover, Lemma 3.5 implies that 𝐻(𝑥,𝑎)=𝐸[𝑉(𝐿(𝐹(𝑥,𝑎),𝜉))] is a concave function, and by Lemma 3.7, it follows that 𝐻∈𝐶²(int(𝕂);ℝ), obtaining that 𝐻_𝑎𝑎(𝑥,⋅) is negative semidefinite (see [21, page 260]). Furthermore, by Assumption 3.9(a) and applying Lemma 3.1, it concludes that 𝑓∈𝐶¹(int(𝑋);𝐴) and 𝑉∈𝐶²(int(𝑋);ℝ).
On the other hand, it is obtained that 𝐺_𝑎(𝑥,𝑎)=𝑟_𝑎(𝑥,𝑎)+𝛼𝐸𝑉^′(𝐿(𝐹(𝑥,𝑎),𝜉))𝐿^′(𝐹(𝑥,𝑎),𝜉)𝐹_𝑎(𝑥,𝑎), (3.19) for each 𝑎∈int(𝐴(𝑥)). Then, the first order condition and the invertibility of 𝐹_𝑎 (see Assumption 3.6(b)) imply that 𝐺_𝑎(𝑥,𝑓(𝑥))=0, that is, −𝑟_𝑎𝐹_𝑎⁻¹𝑥,𝑓(𝑥)=𝛼𝐸𝑉^′𝐿𝐹𝑥,𝑓(𝑥),𝜉𝐿^′𝐹𝑥,𝑓(𝑥),𝜉. (3.20) Moreover, since 𝑉 satisfies (2.4) and 𝑓∈𝔽 is the optimal policy, then 𝑉(𝑥)=𝐺𝑥,𝑓(𝑥). (3.21)
Using the fact that 𝐺_𝑎(𝑥,𝑓(𝑥))=0, it is possible to obtain the following envelope formula: 𝑉^′(𝑥)=𝐺_𝑥𝑥,𝑓(𝑥)+𝐺_𝑎𝑥,𝑓(𝑥)𝑓^′(𝑥),=𝐺_𝑥𝑥,𝑓(𝑥). (3.22) Equivalently, 𝑉^′(𝑥)=𝑟_𝑥𝑥,𝑓(𝑥)+𝛼𝐸𝑉^′𝐿𝐹𝑥,𝑓(𝑥),𝜉𝐿^′𝐹𝑥,𝑓(𝑥),𝜉𝐹_𝑥𝑥,𝑓(𝑥). (3.23) Finally, substituting (3.20) in (3.23), it follows that 𝑉^′(𝑥)=𝑊𝑥,𝑓(𝑥). (3.24)

Theorem 3.12. Under Assumptions 3.4, 3.6 and 3.9(b), it results that 𝑓_𝑛∈𝐶¹(int(𝑋);𝐴), 𝑣_𝑛∈𝐶²(int(𝑋);ℝ), for each 𝑛=1,2,…. Furthermore, 𝑣^′_𝑛(𝑥)=𝑊𝑥,𝑓_𝑛(𝑥), (3.25) for each 𝑥∈int(𝑋) and 𝑛=2,3,….

Proof. The proof will be made by induction. Let 𝑥∈int(𝑋) be fixed. Since 𝑣₁(𝑥)=max_{(𝑥)𝑎∈𝐴}𝐺¹(𝑥,𝑎), (3.26) where 𝐺¹ is defined in (3.8) and by Assumptions 3.4(a) and 3.6(a), it follows that 𝐺¹∈𝐶²(int(𝕂);ℝ) and 𝐺¹_𝑎𝑎(𝑥,𝑎) is negative definite. By Assumption 3.9(b), it yields that 𝑓₁(𝑥)∈int(𝐴(𝑥)), and applying Lemma 3.1, it follows that 𝑓₁∈𝐶¹(int(𝑋);𝐴), 𝑣₁∈𝐶²(int(𝑋);ℝ). Straightforward computations allow to obtain that 𝑣^′₁(𝑥)=𝐺¹_𝑥𝑥,𝑓₁(𝑥). (3.27) Moreover, by Lemma 3.5 it is known that 𝑣₁ is strictly concave, then it is negative semidefinite (see [21, page 260]).
Let 𝑛=2, then 𝑣₂(𝑥)=max_{(𝑥)𝑎∈𝐴}𝐺²(𝑥,𝑎), (3.28) where 𝐺² is defined in (3.8).
Since 𝑟, 𝐿, 𝐹, 𝑣₁∈𝐶², then 𝐺²∈𝐶² too. Moreover, Lemmas 3.5 and 3.7 imply that 𝐻²(𝑥,𝑎)∶=𝐸𝑣₁(𝐿(𝐹(𝑥,𝑎),𝜉)) (3.29) is a concave function and 𝐻²∈𝐶². It follows that 𝐻²_𝑎𝑎(𝑥,⋅) is negative semidefinite.
Consequently, 𝐺²_𝑎𝑎 is negative definite. Now, since 𝑓₂(𝑥)∈int(𝐴(𝑥)) (see Assumption 3.9(b)), applying again Lemma 3.1, it follows that 𝑓₂∈𝐶¹(int(𝑋);𝐴), 𝑣₂∈𝐶²(int(𝑋);ℝ). Furthermore, the first order condition implies that 𝐺²_𝑎(𝑥,𝑓₂(𝑥))=0. By the invertibility of 𝐹_𝑎 (see Assumption 3.6(b)), it follows that −𝑟_𝑎𝐹_𝑎⁻¹𝑥,𝑓₂(𝑥)=𝛼𝐸𝑣^′₁𝐿𝐹𝑥,𝑓₂(𝑥),𝜉𝐿^′𝐹𝑥,𝑓₂(𝑥),𝜉. (3.30)
On the other hand, 𝑣^′₂(𝑥)=𝑟_𝑥𝑥,𝑓₂(𝑥)+𝛼𝐸𝑣^′₁𝐿𝐹𝑥,𝑓₂(𝑥),𝜉𝐿^′𝐹𝑥,𝑓₂(𝑥),𝜉𝐹_𝑥𝑥,𝑓₂(𝑥), (3.31) and substituting (3.30) in (3.31), it is obtained that 𝑣^′₂(𝑥)=𝑊𝑥,𝑓₂(𝑥), (3.32) where 𝑊 is defined in (3.17).
Now, suppose that 𝑣_𝑛−1∈𝐶²(int(𝑋);ℝ) with 𝑛>2. Using arguments similar to the case 𝑛=2, it is possible to demonstrate that 𝑓_𝑛∈𝐶¹(int(𝑋);𝐴), 𝑣_𝑛∈𝐶²(int(𝑋);ℝ) and 𝑣^′_𝑛(𝑥)=𝑊𝑥,𝑓_𝑛(𝑥). (3.33)

Assumption 3.13. For each 𝑥∈𝑋, the function 𝑊(𝑥,⋅) has a continuous inverse function, denoted by 𝑤.

Theorem 3.14. Under Assumptions 3.4, 3.6, 3.9 and 3.13, it follows that 𝑓_𝑛(𝑥)→𝑓(𝑥), (3.34) when 𝑛→∞, for each 𝑥∈int(𝑋).

Proof. Let 𝑥∈int(𝑋) fixed. It is known by Lemma 3.5 and Theorem 3.11 that the optimal value function 𝑉 is concave and differentiable on int(𝑋). In addition, it is known that for each 𝑛∈ℕ, 𝑣_𝑛 is a concave and differentiable function on int(𝑋). Then from Lemma 3.3 it follows that 𝑣^′_𝑛(𝑥)→𝑉^′(𝑥), (3.35) when 𝑛 goes to ∞.
Now by Assumption 3.13, it concludes that for 𝑛=2,3,…, 𝑓_𝑛(𝑥)=𝑤𝑥,𝑣^′_𝑛(𝑥),𝑓(𝑥)=𝑤𝑥,𝑉^′(𝑥), (3.36) where 𝑓_𝑛 is a stationary policy of 𝑣_𝑛 and 𝑓 is the optimal policy. Finally, the convergence is guaranteed by the continuity of 𝑤 (see Assumption 3.13).

3.3. Euler Equation

Theorem 3.15. Under Assumptions 3.4, 3.6, 3.9, and 3.13 it follows that 𝑣^′_𝑛(𝑥)=𝑟_𝑥𝑥,𝑤𝑥,𝑣^′_𝑛(𝑥)+𝛼𝐸𝑣^′_𝑛−1𝐿𝐹𝑥,𝑤𝑥,𝑣^′_𝑛(𝑥),𝜉𝐿^′𝐹𝑥,𝑤𝑥,𝑣^′_𝑛(𝑥),𝜉𝐹_𝑥𝑥,𝑤𝑥,𝑣^′_𝑛(𝑥), (3.37) for each 𝑥∈int(𝑋) and 𝑛∈ℕ, where 𝑤 is the function given in Assumption 3.13.

Proof. Let 𝑥∈int(𝑋) be fixed. By Lemma 3.5 and Theorem 3.12, it is known that 𝑣_𝑛∈𝐶²(int(𝑋);ℝ) and it is a concave function. Now, from the first order condition and the invertibility of 𝐹_𝑎 (see Assumption 3.6(b)), it follows that −𝑟_𝑎𝐹_𝑎⁻¹𝑥,𝑓_𝑛(𝑥)=𝛼𝐸𝑣^′_𝑛−1𝐿𝐹𝑥,𝑓_𝑛(𝑥),𝜉𝐿^′𝐹𝑥,𝑓_𝑛(𝑥),𝜉. (3.38) Since 𝑣^′_𝑛(𝑥)=𝑊𝑥,𝑓_𝑛(𝑥), (3.39) and using the invertibility of 𝑊(𝑥,⋅) (see Assumption 3.13), it follows that 𝑓_𝑛(𝑥)=𝑤𝑥,𝑣^′_𝑛(𝑥). (3.40) Finally, substituting (3.40) in (3.38), (3.37) is obtained.

Corollary 3.16. The optimal value function satisfies 𝑉^′(𝑥)=𝑟_𝑥𝑥,𝑤𝑥,𝑉^′(𝑥)+𝛼𝐸𝑉^′𝐿𝐹𝑥,𝑤𝑥,𝑉^′(𝑥),𝜉𝐿^′𝐹𝑥,𝑤𝑥,𝑉^′(𝑥),𝜉𝐹_𝑥𝑥,𝑤𝑥,𝑉^′(𝑥), (3.41) for each 𝑥∈int(𝑋).

Proof. Let 𝑥∈int(𝑋) be fixed. It is known that the VI functions satisfy the Euler equation (3.37), so applying Lemma 3.3, it is obtained that 𝑣^′_𝑛(𝑥)→𝑉^′(𝑥), (3.42) when 𝑛→∞. Also from Assumption 3.13, 𝑤 is a continuous function. Then, when 𝑛 goes to infinite in (3.37), it follows that the optimal value function satisfies (3.41).

4. A Linear-Quadratic Model

Consider that ℝ^𝑛=𝑋=𝐴=𝐴(𝑥), for each 𝑥∈𝑋. The dynamic of the system is given by 𝑥_𝑡+1=𝐵𝑥_𝑡+𝐶𝑎_𝑡+𝜉_𝑡, (4.1)𝑡=1,2,…, with 𝑥₀=𝑥∈𝑋 given. 𝐵 and 𝐶 are invertible matrices of size 𝑛×𝑛, {𝜉_𝑡} is a sequence of iid column random vectors with values in ℝ^𝑛. Let 𝜉 be a generic element of the sequence {𝜉_𝑡}, assume that 𝜉 has a density Δ with Δ∈𝐶², and 𝐸(𝜉) equals vector zero. Furthermore, it is assumed that if 𝑃 is a symmetric negative definite matrix of size 𝑛×𝑛, then 𝐸[𝜉^𝑇𝑃𝜉] is finite. In addition, it is assumed that the interchange between derivatives and integrals is valid (see Remark 3.8). A particular case of this assumption can be found in [16, page 315].

The reward function is given by 𝑟(𝑥,𝑎)=𝑥^𝑇𝑄𝑥+𝑎^𝑇𝑅𝑎, (4.2) where 𝑥^𝑇 and 𝑎^𝑇 denote the transpose of vectors 𝑥 and 𝑎; 𝑄 and 𝑅 are symmetric matrices of size 𝑛×𝑛, and both of them are negative definite.

Lemma 4.1. The linear quadratic model satisfies Assumption 2.3.

Proof. Note that 𝑂^𝑥_𝛾∶=𝑎∈𝐴(𝑥)∣𝑥^𝑇𝑄𝑥+𝑎^𝑇𝑅𝑎≥𝛾 (4.3) is a compact set, for each 𝑥∈𝑋 and 𝛾∈ℝ. Indeed, let 𝑥∈𝑋 and 𝛾∈ℝ. If any sequence {𝑎_𝑛} of 𝑂^𝑥_𝛾 satisfies 𝑥^𝑇𝑄𝑥+𝑎^𝑇_𝑛𝑅𝑎_𝑛→−∞, then there is a contradiction. Therefore 𝑂^𝑥_𝛾 is a set bounded below. Moreover, since 𝑄 and 𝑅 are negative definite, then 𝑂^𝑥_𝛾 is a set bounded above. In addition, if {𝑎_𝑛}⊂𝑂^𝑥_𝛾 so that 𝑎_𝑛→𝑎, then by the continuity of 𝑟, it follows that 𝑟(𝑥,𝑎)≥𝛾, implying that 𝑂^𝑥_𝛾 is a closed set. Therefore, the reward function 𝑟 is sup-compact. Finally, note that 𝑟 is a nonpositive and continuous function on 𝕂. So Assumption 2.3(a) holds.
On the other hand, let 𝑈∈ℬ(𝑋), then 𝑄𝑈∣𝑥_𝑡=𝑥,𝑎_𝑡=𝑎=Pr𝑥_𝑡+1∈𝑈∣𝑥_𝑡=𝑥,𝑎_𝑡=𝑎=𝐼_𝑈(𝐵𝑥+𝐶𝑎)Δ(𝑠)𝑑𝑠, (4.4) where 𝐼_𝑈 denotes the indicator function of 𝑈. Since the density Δ is continuous, it is obtained that the transition law 𝑄 is weakly continuous, that is, Assumption 2.3(b) holds.
Finally, let ℎ∈𝔽 be defined as ℎ(𝑥)=−𝐶⁻¹𝐵𝑥, (4.5) then the dynamic of the system is given by 𝑥_𝑡=𝐵𝑥_𝑡−1+𝐶ℎ𝑥_𝑡−1+𝜉_𝑡−1=𝜉_𝑡−1, (4.6) for 𝑡=1,2,…, with 𝑥₀=𝑥∈𝑋.
It follows that 𝐸^ℎ_𝑥𝑥^𝑇_𝑡𝑄𝑥_𝑡+ℎ𝑥_𝑡^𝑇𝑅ℎ𝑥_𝑡=𝐸𝜉^𝑇_𝑡𝑃𝜉_𝑡=∶𝜃, (4.7) where 𝑃∶=(𝑄+(𝐶⁻¹𝐵)^𝑇𝑅(𝐶⁻¹𝐵)).
Since {𝜉_𝑡} is i.i.d, then 𝑣()=𝜃ℎ,𝑥1−𝛼<∞, (4.8) where 𝑣 is given by (2.1). Therefore, Assumption 2.3(c) holds.

Lemma 4.2. The linear quadratic problem satisfies Assumptions 3.4, 3.6, 3.9, and 3.13.

Proof. It is easy to obtain that 𝑟 is a concave function and 𝑟∈𝐶², implying Assumptions 3.4(a) and 3.6(a). Assumption 3.4(b) is satisfied using Condition C2 in [20]. Furthermore, observe that 𝐹(𝑥,𝑎)=𝐵𝑥_𝑡+𝐶𝑎_𝑡, (4.9)(𝑥,𝑎)∈𝕂. Since 𝐶 is an invertible matrix, then Assumption 3.6(b) holds.
In addition, note that 𝐿𝑦,𝑠=𝑦+𝑠, (4.10)(𝑦,𝑠)∈𝑋^′×𝑆. Then it follows that 𝐿 has an inverse 𝑅 in the second variable, which is 𝑅𝑦,𝑢=𝑢−𝑦. (4.11) Therefore, Assumption 3.6(c) yields. Furthermore, since Δ∈𝐶²,it results that Assumption 3.6(d) holds. On the other hand, Assumption 3.9 is satisfied since 𝐴(𝑥)=ℝ^𝑛for each𝑥∈𝑋.Finally, it is obtained 𝑊(𝑥,𝑎)=2𝑄𝑥−𝑅𝐶⁻¹𝐵^𝑇−1𝑎, (4.12) where 𝑊 is defined in (3.17), implying that the inverse of 𝑊(𝑥,⋅) is 𝑤(𝑥,𝑧)=𝑅𝐶⁻¹𝐵^𝑇−11𝑄𝑥−2𝑧, (4.13) which is a continuous function. Therefore, Assumption 3.13 is satisfied.

Lemma 4.3. VI functions for the linear quadratic problem satisfy 𝑣^′_𝑛(𝑥)=2𝐾_𝑛𝑥, (4.14) for each 𝑛=1,2,…, where 𝐾_𝑛=𝑄+𝛼𝐵^𝑇𝐾_𝑛−1−𝐾_𝑛−1𝐶𝑅+𝐶^𝑇𝐾_𝑛−1𝐶⁻¹𝐶^𝑇𝐾_𝑛−1𝐵, (4.15) with 𝐾₁=𝑄.

Proof. Observe that the validity of Theorem 3.15 is guaranteed by Lemmas 4.1 and 4.2. Now, since 𝑄 and 𝑅 are negative definite, then 𝑣₁(𝑥)=𝑥^𝑇𝑄𝑥. (4.16)
By Theorem 3.15, it is known that 𝑣₂(𝑥) satisfies the Euler equation (3.37), and by (4.13), it follows that 𝑣^′₂(𝑥)=2𝑄𝑥+𝛼𝐵^𝑇𝐸𝑣^′₁𝐵+𝐶𝑅𝐶⁻¹𝐵^𝑇−1𝑄1𝑥−2𝐶𝑅𝐶⁻¹𝐵^𝑇−1𝑣^′₂(𝑥)+𝜉. (4.17) Since 𝑣^′₁(𝑥)=2𝑄𝑥 and 𝐸(𝜉) is equal to zero vector, then 𝑣^′₂(𝑥)=2𝑄+𝛼𝐵^𝑇𝑄𝐵+𝐶𝑅𝐶⁻¹𝐵^𝑇−1𝑄𝑥−𝛼𝐵^𝑇𝑄𝐶𝑅𝐶⁻¹𝐵^𝑇−1𝑣^′₂(𝑥), (4.18) and by direct calculations, it is obtained that 𝑣^′₂(𝑥)=2𝐾₂𝑥, (4.19) where 𝐾₂=𝑄+𝛼𝐵^𝑇𝑄−𝑄𝐶𝑅+𝐶^𝑇𝑄𝐶⁻¹𝐶^𝑇𝑄𝐵. (4.20)
Now, suppose that 𝑣^′_𝑛(𝑥)=2𝐾_𝑛𝑥 for 𝑛>2, with 𝐾_𝑛 defined in (4.15). Then, by Theorem 3.15 and (4.13), it is known that 𝑣^′_𝑛+1(𝑥)=2𝑄𝑥+𝛼𝐵^𝑇𝐸𝑣^′_𝑛𝐵+𝐶𝑅𝐶⁻¹𝐵^𝑇−1𝑄1𝑥−2𝐶𝑅𝐶⁻¹𝐵^𝑇−1𝑣^′_𝑛+1(𝑥)+𝜉. (4.21)
Then 𝑣^′_𝑛+1(𝑥)=2𝑄+𝛼𝐵^𝑇𝐾_𝑛𝐵+𝐶𝑅𝐶⁻¹𝐵^𝑇−1𝑄𝑥−𝛼𝐵^𝑇𝐾_𝑛𝐶𝑅𝐶⁻¹𝐵^𝑇−1𝑣^′_𝑛+1(𝑥), (4.22) and using matrix algebra, it yields that 𝑣^′_𝑛+1(𝑥)=2𝐾_𝑛+1𝑥, (4.23) where 𝐾_𝑛+1 satisfies (4.15).

Lemma 4.4. The optimal policy for the linear quadratic problem is 𝑓(𝑥)=−𝛼𝑅+𝐶^𝑇𝐾𝐶⁻¹𝐶^𝑇𝐾𝐵𝑥, (4.24) where 𝐾 satisfies 𝐾=𝑄+𝛼𝐵^𝑇𝐾−𝐾𝐶𝑅+𝐶^𝑇𝐾𝐶⁻¹𝐶^𝑇𝐾𝐵. (4.25)

Proof. Lemma 4.3 and (4.13) allow to obtain 𝑓_𝑛(𝑥)=𝑅𝐶⁻¹𝐵^𝑇−1𝑄−𝐾_𝑛𝑥, (4.26) for each 𝑛=1,2,… and 𝑥∈𝑋. Moreover, the validity of Theorem 3.14 is guaranteed by Lemma 4.2, that is, 𝑓_𝑛(𝑥)→𝑓(𝑥), implying the convergence of the sequence {𝐾_𝑛} which, according to its definition in (4.15), guarantees that its limit, denoted by 𝐾, must satisfy (4.25). Finally using matrix algebra (4.24) is obtained.

5. Conclusion

In this paper a method to solve the optimal control problem is presented. This method is based on the use of the Euler equation. The procedure proposed to solve the optimal control problem is by means of an envelope formula and the use of the convergence of the maximizers of values iteration functions to a stationary optimal policy. Future work aims to study possible error bounds for approximating the maximizers toward the optimal policy.