Study on Indefinite Stochastic Linear Quadratic Optimal Control with Inequality Constraint

Abstract

This paper studies the indefinite stochastic linear quadratic (LQ) optimal control problem with an inequality constraint for the terminal state. Firstly, we prove a generalized Karush-Kuhn-Tucker (KKT) theorem under hybrid constraints. Secondly, a new type of generalized Riccati equations is obtained, based on which a necessary condition (it is also a sufficient condition under stronger assumptions) for the existence of an optimal linear state feedback control is given by means of KKT theorem. Finally, we design a dynamic programming algorithm to solve the constrained indefinite stochastic LQ issue.

1. Introduction

The study on LQ control problems can be traced back to the pioneering work of Kalman [1] and Wonham [2] several decades ago. The LQ control theory is elegantly established and developed, and the main work can be seen in [3–11]. In particular, it is found [6] that a stochastic LQ problem with indefinite control weighting matrices may still be well-posed. This discovery evokes a series of subsequent researches, and many important achievements are obtained [5, 12–16]. Up to now, most work deals with the indefinite stochastic LQ problems without constraints. However, as a practical optimization problem, the indefinite stochastic LQ problem unavoidably has various constraints on the state or control; in particulary the inequality constraints often appear.

For the constrained indefinite stochastic LQ problems, [17] studied the equally constrained stochastic LQ optimization for Itô systems. In this paper, we will study the stochastic LQ problem with inequality constraint.

Firstly, we present and prove the generalized KKT theorem under hybrid constraints. Secondly, a necessary condition for the existence of an optimal linear state feedback control is given by means of the generalized KKT theorem. Thirdly, if we strengthen the condition, we can obtain a necessary and sufficient condition for the existence of the optimal linear feedback control to indefinite stochastic LQ optimal control problem with inequality constraint. Finally, we give a dynamic programming algorithm to solve the stochastic LQ problem with the inequality constraint. We provide an example to demonstrate the effectiveness of our main theoretical results.

The outline of this paper is organized as follows. In Section 2, we present a generalized KKT theorem under hybrid constraints. Section 3 proposes a KKT condition for the existence of an optimal linear state feedback control. In Section 4, we provide a necessary and sufficient condition and a dynamic programming algorithm for the stochastic LQ problem with inequality constraint. Section 5 concludes the paper.

For convenience, throughout the paper, we adopt the following notations: 𝐴^𝑇 denotes the transpose of a matrix 𝐴. 𝐴>0(𝐴≥0): 𝐴 is a positive definite (positive semidefinite) symmetric matrix. tr(𝐴): trace of a square matrix. 𝑅^𝑚×𝑛: the space of all 𝑚×𝑛 real matrices. 𝑆^𝑛: a  𝑛×𝑛 symmetric matrix space.

2. Preliminaries

Consider the following indefinite stochastic LQ control.

Problem 1. Considermin𝐽𝑥₀,𝑢=𝐸^𝑇₀𝑥^𝑇(𝑡)𝑄(𝑡)𝑥(𝑡)+𝑢^𝑇(𝑡)𝑅(𝑡)𝑢(𝑡(𝑡)=[𝐴(𝑡)𝑥(𝑡)(𝑡)𝑢(𝑡+[𝐶(𝑡)𝑥(𝑡)(𝑡)𝑢(𝑡(𝑡),𝑥(0))𝑑𝑡,(1a)s.t.𝑑𝑥+𝐵)]𝑑𝑡+𝐷)]𝑑𝑤(1b)=𝑥₀,𝐸𝑥(𝑇(1c)‖)‖²=𝐸𝑥^𝑇(𝑇)𝑥(𝑇)≤𝑐,(1d)where 𝑥(𝑡)=[𝑥₁(𝑡),𝑥₂(𝑡),…,𝑥_𝑛(𝑡)]^𝑇 is an 𝑛-dimensional state variable, 𝑢(𝑡)∈𝑅^𝑚 is a control input, 𝑤(⋅) is a one-dimensional standard Brownian motion defined on a filtered probability space (Ω,𝐹,𝐹_𝑡,𝑃). We denote the information flow ℱ_𝑡=𝜎[𝑤(𝑠)∶0≤𝑠≤𝑡≤𝑇]. 𝑢(⋅) belongs to ℒ²_ℱ(𝑅^𝑚), where ℒ²_ℱ(𝑅^𝑚) is a space of all 𝑅^𝑚-valued, ℱ_𝑡-adapted measurable processes satisfying 𝐸∫^𝑇₀‖𝑢(𝑡)‖²𝑑𝑡<+∞. For each admissible control, the corresponding trajectory satisfies the constraint (1c). 𝑐 in constraint (1c) is a given nonnegative constant. 𝐴(𝑡), 𝐵(𝑡), 𝐶(𝑡), 𝐷(𝑡), 𝑄(𝑡), and 𝑅(𝑡) are time-varying matrices of suitable dimensions. 𝑄(𝑡) and 𝑅(𝑡) in objective functional are symmetric matrices. To study the issue, we first put forward the following Assumption H₁.
Assumption 𝐻₁.  𝐴(𝑡),𝐶(𝑡)∈ℒ^∞(0,𝑇,𝑅^𝑛×𝑛),  𝑄(𝑡)∈ℒ^∞(0,𝑇,𝑆^𝑛), 𝐵(𝑡),𝐷(𝑡)∈ℒ^∞(0,𝑇,𝑅^𝑛×𝑚), and 𝑅(𝑡)∈ℒ^∞(0,𝑇,𝑆^𝑚), where 𝐿^∞(0,𝑇,𝑋)∶={𝑓(𝑡)∶𝑋-valued essential bounded measurable function and ess sup_{𝑡∈[0,𝑇]}‖𝑓(𝑡)‖<+∞}.
In this paper, the weighting matrices in the objective functional are not required to be definite. Therefore Problem 1 is an indefinite stochastic LQ optimal control problem. For later use, we recall KKT theorem for this type of mathematical programming (MP) problems: (𝑥)(𝑥)𝐡(𝑥)min𝑓s.t.𝐠≤𝟎,=𝟎,(2) where 𝐠(𝑥)=(𝑔₁(𝑥),…,𝑔_𝑝(𝑥)), and 𝐡(𝑥)=(ℎ₁(𝑥),…,ℎ_𝑞(𝑥)).
The KKT conditions [18–20], which are also known as the Kuhn-Tucker (KT) conditions, are the first-order necessary conditions for a solution in nonlinear programming to be optimal, provided that some regularity conditions are satisfied. The Lagrange multiplier method, which allows only equality constraints, can be viewed as a special case of KKT conditions.
Regularity Condition (or Constraint Qualification). In MP above, let 𝐼^∗={𝑖∣𝑔_𝑖(𝑥^∗)=0}(𝑔_𝑖(𝑥^∗)  𝑖∈𝐼^∗ be active constraints at 𝑥^∗). The gradient vectors ∇𝑔_𝑖(𝑥^∗),𝑖∈𝐼^∗, and ∇ℎ_𝑗(𝑥^∗),𝑗=1,…,𝑞, are linearly independent, is known as a linear independent constraint qualification (LICQ).
Regular Point. In MP above, 𝑥^∗ is said to be a regular point of the constraints if the gradient vectors ∇𝑔_𝑖(𝑥^∗),∇ℎ_𝑗(𝑥^∗),𝑖∈𝐼^∗,𝑗=1,…,𝑞, are linearly independent.
KKT Theorem. In MP above, we assume that the functions 𝑓,𝐠=(𝑔₁,…,𝑔_𝑝)^𝑇,𝐡=(ℎ₁,…,ℎ_𝑞)^𝑇 are twice continuously differentiable and we assume that all the constraints satisfy the regularity condition LICQ. Let 𝑥^∗ be a point satisfying all the constraints and let 𝑥^∗ be a regular point of the above constraints. Now suppose that this regular point 𝑥^∗ is also a relative minimum point for the original MP. Then it is shown that there exist a vector 𝜆≥0∈𝑅 and a vector 𝜇∈𝑅, such that ∇_𝑥𝐿𝑥^∗,𝜆^∗,𝜇^∗𝜆=𝟎,_∗𝑖𝑔_𝑖𝑥^∗=0,𝑖=1,…,𝑝,(3) where 𝐿(𝑥,𝜆,𝜇)=𝑓(𝑥)+𝜆^𝑇𝐠(𝑥)+𝜇^𝑇𝐡(𝑥) is the Lagrangian function and 𝜆^∗_𝑖𝑔_𝑖(𝑥^∗)=0,𝑖=1,…,𝑝, are complementary clackness condition.
It is particularly important to check the regularity condition before we apply the conclusion of KKT theorem. If it is not so, the conclusion of KKT theorem would not be valid, just as the following example shows.

Example 2. Consider min𝑓𝑥₁,𝑥₂=−2𝑥₁−3𝑥₂,s.t.𝐠𝑥₁,𝑥₂=𝑥²₁+𝑥²₂=0.(4) Obviously, the minimum point is 𝑥^∗(0,0)^𝑇. According to KKT theorem, we obtain 2−𝜆𝑥^∗₁=0,3−𝜆𝑥^∗₂𝑥=0,_∗12+𝑥^∗₂²=0.(5) The conclusion of KKT theorem does not hold at point 𝑥^∗(0,0)^𝑇, because ∇𝐠(0,0)=(0,0)^𝑇 is not linearly independent. It does not satisfy the LICQ regularity condition.
In order for a minimum point 𝑥^∗ to satisfy the above KKT conditions, it should satisfy some regularity conditions. Except for LICQ regularity condition, the most used ones are listed below.
Constant Rank Constraint Qualification. For each subset of the gradients of the active inequality constraints and the gradients of the equality constraints the rank at a vicinity of 𝑥^∗ is constant.
Mangasarian-Fromovitz Constraint Qualification. The gradients of the active inequality constraints and the gradients of the equality constraints are linear independent at 𝑥^∗.
Constant Positive Linear Dependence Constraint Qualification. For each subset of the gradients of the active inequality constraints and the gradients of the equality constraints, if it is positive-linear dependent at 𝑥^∗, then it is positive-linear dependent at a vicinity of 𝑥^∗.
The Slater condition for a convex MP is also a common regularity condition.

Remark 3. In this paper, for convenience, when we use the KKT theorem, we always assume that the local optimal 𝑥^∗ meets the LICQ regularity condition. The same goes for other regularity conditions.

Definition 4 (see [21]). Let 𝑋 be a vector space, 𝑌 a normed space, and 𝑇 a transformation from 𝑋 to 𝑌. If the limit 𝛿𝑇(𝑥;ℎ)=lim_𝛼→0[𝑇(𝑥+𝛼ℎ)−𝑇(𝑥)]𝛼(6) exists, it is called the Gateaux differential of 𝑇 at 𝑥 with increment ℎ. If the limit exists for each ℎ∈𝑋, the transformation 𝑇 is said to be Gateaux differentiable at 𝑥.

Definition 5 (see [21]). Let 𝑋 be a vector space and 𝑍 a Banach space with a positive cone 𝑃 having nonempty interior. Let 𝐺 be a mapping from 𝑋 to 𝑍 which has a Gateaux differential that is linear in its increment. A point 𝑥₀∈𝑋 is said to be a regular point of the inequality 𝐺(𝑥)≤0, if 𝐺(𝑥₀)≤0 and there is an ℎ∈𝑋 such that 𝐺(𝑥₀)+𝛿𝐺(𝑥₀;ℎ)<0.

Definition 6 (see [21]). Let 𝑋 be a vector space and 𝑍 a Banach space. Let 𝐻(𝑥)=[ℎ₁(𝑥),…,ℎ₁(𝑥)]^𝑇 be a mapping from 𝑋 to 𝑍 which has a Gateaux differential that is linear in its increment. A point 𝑥₀∈𝑋 is said to be a regular point of the equality 𝐻(𝑥)=0, if 𝛿ℎ₁(𝑥₀;ℎ),…,𝛿ℎ_𝑛(𝑥₀;ℎ) are linearly independent.
On the basis of the definitions above, let us discuss the KKT theorem in Banach space, where the objective function and the constraint functions in MP are functionals.
Let us consider (𝑥)(𝑥)𝐻(𝑥)()min𝑓s.t.𝐺≤0=0.MPI As a special case, (𝑥)(𝑥)()min𝑓s.t.𝐺≤0MPII has the local necessary condition as follows.

Lemma 7 (see [21] (generalized KKT theorem)). Let 𝑋 be a vector space and 𝑍 a Banach space having positive cone 𝑃. Assume that 𝑃 contains an interior point. Let 𝑓 be a Gateaux differentiable functional on 𝑋 and 𝐺 a Gateaux differentiable mapping from 𝑋 to 𝑍. Assume that the Gateaux differentials are linear in their increments. Suppose that 𝑥₀ minimizes 𝑓 subject to 𝐺(𝑥)≤0 and that 𝑥₀ is a regular point of the inequality 𝐺(𝑥)≤0. Then there is a 𝑧^∗₀≥0 in 𝑍, such that the Lagrangian function 𝑓(𝑥)+𝑧^∗₀𝐺(𝑥) is stationary at 𝑥₀. Furthermore 𝑧^∗₀𝐺(𝑥₀)=0.
The following theorem is the local necessary condition of (MPI).

Theorem 8. Let 𝑋 be a vector space and 𝑍 a Banach space having positive cone 𝑃. Assume that 𝑃 contains an interior point. Let 𝑓 be a Gateaux differentiable functional on 𝑋. Let 𝐺 and 𝐻 be Gateaux differentiable mappings from 𝑋 to 𝑍. Assume that the Gateaux differentials are linear in their increments. Suppose that 𝑥₀ minimizes 𝑓 subject to 𝐺(𝑥)≤0, 𝐻(𝑥)=0 and that 𝑥₀ is a regular point of 𝐺(𝑥)≤0, 𝐻(𝑥)=0. Then there is a 𝜆^∗≥0 in 𝑍, 𝜇^∗∈𝑍, such that the Lagrangian function 𝑓(𝑥)+𝜆^∗𝐺(𝑥)+𝜇^∗𝐻(𝑥) is stationary at 𝑥₀. Namely, 𝛿𝑓(𝑥₀;ℎ)+𝜆^∗𝛿𝐺(𝑥₀;ℎ)+𝜇^∗𝛿𝐻(𝑥₀;ℎ)=0. Furthermore, 𝑧^∗₀𝐺(𝑥₀)=0.

Proof. 𝐻(𝑥)=0 is equivalent to 𝐻(𝑥)≤0 and −𝐻(𝑥)≤0. If 𝑥₀ is a regular point of 𝐻(𝑥)=0, then 𝛿ℎ₁(𝑥₀;ℎ),…,𝛿ℎ_𝑛(𝑥₀;ℎ) are linearly independent. So 𝛿ℎ₁(𝑥₀;ℎ),…,𝛿ℎ_𝑛(𝑥₀;ℎ) are all nonzero, because the Gateaux differentials are linear in their increments. Using Definition 5, it is easy to verify that 𝑥₀ is a regular point of both −𝐻(𝑥)≤0 and 𝐻(𝑥)≤0. According to Lemma 7, we know that the multiplier 𝜇^∗ of equality 𝐻(𝑥)=0 has no nonnegative requirement.

Definition 9 (see [22]). Suppose that 𝑓(𝑋) is a scalar-valued function of the elements 𝑥_𝑖𝑗 of 𝑋. Then the gradient matrix of 𝑓(𝑋) is defined as ∇𝑓(𝑋=(𝑋))𝜕𝑓𝜕(𝑋)(7) with 𝜕𝑓(𝑋)𝜕(𝑋)_𝑖𝑗=𝜕𝑓(𝑋)𝜕𝑥_𝑖𝑗.(8) Based on Definition 9, we can easily extend KKT theorem from Banach space to matrix space. Because 𝑋(𝑡) can be treated as a vector [𝑋₁(𝑡),…,𝑋_𝑛(𝑡)]^𝑇, one can work out ∇[𝑋_𝑖(𝑡)] and the KKT theorem holds.
When we apply the matrix KKT theorem, we need to give the partial list of gradient matrices [22] that we will use in this paper. In the following formulas, 𝑋 is an 𝑛×𝑚 matrix. The formulas are not valid if the elements 𝑥_𝑖𝑗 of 𝑋 are not independent. 𝐴,𝐵 are assumed to have appropriate dimensions determined from context.
Consider the following: 𝜕(𝑋)𝜕𝜕𝑋tr=𝐼,𝜕𝑋tr𝑋𝑋^𝑇𝜕=2𝑋,𝜕𝑋tr𝐴𝑋^𝑇𝜕=𝐴,()𝜕𝑋tr𝐴𝑋=𝐴^𝑇,𝜕𝜕𝑋tr𝐴𝑋^𝑇𝐵𝜕=𝐵𝐴,()𝜕𝑋tr𝐴𝑋𝐵=𝐴^𝑇𝐵^𝑇,𝜕𝜕𝑋tr𝐴𝑋𝐵𝑋^𝑇=𝐴^𝑇𝑋𝐵^𝑇𝜕+𝐴𝑋𝐵,()𝜕𝑋tr𝐴𝑋𝐵𝑋=𝐴^𝑇𝑋^𝑇𝐵^𝑇+𝐵^𝑇𝑋^𝑇𝐴^𝑇.(9)

3. KKT Conditions and a New Type of GDREs

Definition 10. Problem 1 is well-posed, if for any 𝑥₀∈𝑅^𝑛, −∞<𝑉(𝑥₀)=inf_{𝑢(⋅)∈𝑈ad}{𝐽(𝑥₀,𝑢(⋅))}. 𝑢_∗ is called an optimal control, if 𝑉(𝑥₀)=𝐽(𝑥₀,𝑢_∗), and 𝑥_∗ denotes the corresponding optimal trajectory.
Let the control law be 𝑢=𝐾𝑥(𝐾∈𝐶^𝑚×𝑛[0,𝑇]), and 𝑋(𝑡)=𝐸[𝑥(𝑡)𝑥^𝑇(𝑡)]. By substituting 𝑢=𝐾𝑥 into (1a) of Problem 1, we obtain the new objective functional: 𝐽(𝑋,𝐾)=tr^𝑇₀𝑄𝑋+𝐾^𝑇𝑅𝐾𝑋𝑑𝑡,(10) where 𝐶^𝑛×𝑛[0,𝑇] is the space of 𝑛-order square matrix whose elements are continuous functions. By substituting 𝑢=𝐾𝑥 into (1b) of Problem 1, we obtain a closed-loop system: ()+()(0)𝑑𝑥=𝐴+𝐵𝐾𝑥𝑑𝑡𝐶+𝐷𝐾𝑥𝑑𝑤,𝑥=𝑥₀.(11) By applying Itô’s formula to 𝑋(𝑡), we obtain 𝑋̇=(𝐴+𝐵𝐾)𝑋+𝑋(𝐴+𝐵𝐾)^𝑇+(𝐶+𝐷𝐾)𝑋(𝐶+𝐷𝐾)^𝑇,𝑋(0)=𝑋₀=𝑥₀𝑥^𝑇₀.(12) Define the transformation 𝐻(𝑋,𝐾) from 𝐶^𝑛×𝑛[0,𝑇]×𝐶^𝑚×𝑛[0,𝑇] to 𝐶^𝑛×𝑛[0,𝑇]: 𝐻()(𝑡)(0)−𝑋,𝐾∶=𝑋−𝑋_𝑡0(𝐴+𝐵𝐾)𝑋+𝑋(𝐴+𝐵𝐾)^𝑇+(𝐶+𝐷𝐾)𝑋(𝐶+𝐷𝐾)^𝑇𝑑𝑡.(13) By substituting 𝑢=𝐾𝑥 into (1c) of Problem 1, we obtain tr𝑋(𝑇)−𝑐≤0.(14) Define the transformation 𝐺(𝑋,𝐾) from 𝐶^𝑛×𝑛[0,𝑇] to 𝑅¹: 𝐺(𝑋(𝑇))∶=tr𝑋(𝑇)−𝑐.(15) So the original stochastic Problem 1 can be transformed into the deterministic Problem 11 as follows.

Problem 11. Consider the following:

min𝐽(𝑋,𝐾)=tr^𝑇₀𝑄𝑋+𝐾^𝑇()[],𝐺(𝑋(𝑇𝑅𝐾𝑋𝑑𝑡,(16a)s.t.𝐻𝑋,𝐾=0,∀𝑡∈0,𝑇(16b)))≤0.(16c)

Lemma 12. 𝐽(𝑋,𝐾), 𝐻(𝑋,𝐾), and 𝐺(𝑋(𝑇)) have continuous Gateaux derivative as follows: 𝛿𝐽_𝑋(𝑋,𝐾;Δ𝑋)=tr^𝑇₀𝑄+𝐾^𝑇𝑅𝐾Δ𝑋𝑑𝑡,𝛿𝐽_𝐾(𝑋,𝐾;Δ𝑋)=tr^𝑇₀Δ𝐾^𝑇𝑅𝐾𝑋+𝐾^𝑇𝑅Δ𝐾𝑋𝑑𝑡,𝛿𝐻_𝑋(𝑡)(𝑡)𝑋,𝐾;Δ𝑋)(=Δ𝑋−tr^𝑡₀(𝐴+𝐵𝐾)Δ𝑋+Δ𝑋(𝐴+𝐵𝐾)^𝑇+(𝐶+𝐷𝐾)Δ𝑋(𝐶+𝐷𝐾)^𝑇𝑑𝑡,𝛿𝐻_𝐾(𝑋,𝐾;Δ𝐾)(𝑡)=−tr^𝑡₀𝐵Δ𝐾𝑋+𝑋Δ𝐾^𝑇𝐵^𝑇+(𝐶+𝐷𝐾)𝑋(𝐷Δ𝐾)^𝑇+(𝐷Δ𝐾)𝑋(𝐶+𝐷𝐾)^𝑇𝑑𝑡,𝛿𝐺_𝑋(𝑋;Δ𝑋(𝑇))=trΔ𝑋(𝑇).(17)

Proof. We prove only the most complicated one. The rest can be verified in the same way. From Definition 4, 𝛿𝐻_𝑋(𝑋,𝐾;Δ𝑋)=lim_𝛼→0[𝐻(𝑋+𝛼Δ𝑋,𝐾)−𝐻(𝑋,𝐾)]𝛼,𝐻()(𝑇)(0)−𝑋,𝐾∶=𝑋−𝑋^𝑡₀(𝐴+𝐵𝐾)𝑋+𝑋(𝐴+𝐵𝐾)^𝑇+(𝐶+𝐷𝐾)𝑋(𝐶+𝐷𝐾)^𝑇𝑑𝑡.(18) Replace the 𝑋 in 𝐻(𝑋,𝐾) with (𝑋+𝛼Δ𝑋) and then let 𝛼→0 showing the conclusion.

Lemma 13 (see [21]). If 𝛼(𝑡) and 𝛽(𝑡) are continuous in [𝑡₁,𝑡₂] and ∫^𝑡2_𝑡1[𝛼(𝑡)ℎ(𝑡)+𝛽(𝑡)ℎ(𝑡)]𝑑𝑡=0 for every continuously differentiable ℎ(𝑡) with ℎ(𝑡₁)=ℎ(𝑡₁)=0, then 𝛽 is differentiable and 𝛼(𝑡)≡𝛽(𝑡) in [𝑡₁,𝑡₂].

Lemma 14 (see [21]). If 𝛼(𝑡) is continuous in [𝑡₁,𝑡₂] and ∫^𝑡2_𝑡1[𝛼(𝑡)ℎ(𝑡)]𝑑𝑡=0 for every continuously differentiable ℎ(𝑡) with ℎ(𝑡₁)=ℎ(𝑡₁)=0, then 𝛼(𝑡)≡0 on [𝑡₁,𝑡₂].

Theorem 15. Assume that 𝐾_∗ is the optimal solution of Problem 1, and then there exist a symmetric matrix 𝑃∈𝑁𝐵𝑉^𝑛×𝑛[0,𝑇] and a nonnegative 𝜆∈𝑅¹, such that −𝑃̇=𝑄+𝐾^𝑇_∗𝑅𝐾_∗+𝑃𝐴+𝐵𝐾_∗+𝐴+𝐵𝐾_∗^𝑇𝑃+𝐶+𝐷𝐾_∗𝑃𝐶+𝐷𝐾_∗^𝑇,𝑃(𝑇)(19a)=𝜆𝐼_𝑛,𝐾(19b)_𝑇∗𝑅+𝑃𝐵+𝐶+𝐷𝐾_∗^𝑇𝜆[(𝑇)]𝑃𝐷=0,(19c)tr𝑋−𝑐=0,(19d)where 𝑁𝐵𝑉^𝑛×𝑛[0,𝑇] is a matrix space whose elements are bounded functions in [0,𝑇] with 0 value at point 0 and right continuous at (0,𝑇).

Proof. 𝐾_∗ is also the optimal solution of Problem 11. Problem 11 is the type of (MPI). Assume that the optimal solution to Problem 11 is (𝑋_∗,𝐾_∗). Using Theorem 8 and Lemma 12, there exist a symmetric matrix 𝑃∈𝑁𝐵𝑉^𝑛×𝑛[0,𝑇] and a nonnegative 𝜆∈𝑅¹, such that tr^𝑇₀𝑄+𝐾^𝑇_∗𝑅𝐾_∗Δ𝑋𝑑𝑡+tr^𝑇₀(𝑡)−Δ𝑋_𝑡0𝐴+𝐵𝐾_∗Δ𝑋+Δ𝑋𝐴+𝐵𝐾_∗^𝑇+𝐶+𝐷𝐾_∗Δ𝑋𝐶+𝐷𝐾_∗^𝑇[(𝑇𝑑𝑡𝑑𝑃+𝜆trΔ𝑋)]=0,(20)tr^𝑇₀Δ𝐾^𝑇𝑅𝐾_∗𝑋_∗+𝐾^𝑇_∗𝑅Δ𝐾𝐾_∗𝑑𝑡−tr^𝑇₀^𝑡₀𝐵Δ𝐾𝑋_∗+𝑋_∗Δ𝐾^𝑇𝐵^𝑇+𝐶+𝐷𝐾_∗𝑋_∗(𝐷Δ𝐾)^𝑇+(𝐷Δ𝐾)𝑋_∗×𝐶+𝐷𝐾_∗^𝑇𝜆[(𝑇)]𝑑𝑡𝑑𝑃=0,(21)tr𝑋−𝑐=0.(22) For all (Δ𝑋,Δ𝐾)∈𝐶^𝑛×𝑛[0,𝑇]×𝐶^𝑚×𝑛[0,𝑇], (20)-(21) are established. According to Riesz representation theorem, we obtain the second item of (20) and the same of (21).
Without loss of generality, let 𝑝(𝑇)=0. Integrate (20) by parts yielding tr^𝑇₀𝑄+𝐾^𝑇_∗𝑅𝐾_∗Δ𝑋𝑑𝑡+tr^𝑇₀Δ𝑋𝑑𝑃+tr^𝑇₀𝑃𝐴+𝐵𝐾_∗Δ𝑋+Δ𝑋𝐴+𝐵𝐾_∗^𝑇+𝐶+𝐷𝐾_∗^𝑇Δ𝑋𝐶+𝐷𝐾_∗^𝑇[(𝑇𝑑𝑡+𝜆trΔ𝑋)]=0.(23) Clearly, 𝑃 has no jump on [0,𝑇). But 𝑃 has a jump at 𝑇, and the value is −𝜆𝐼_𝑛. Because the above results are established for all continuous Δ𝑋, then ^𝑇₀Δ𝑋𝑑𝑃=𝑃Δ𝑋|^𝑇₀−^𝑇₀𝑃Δ𝑋̇𝑑𝑡=−^𝑇₀𝑃Δ𝑋̇𝑑𝑡.(24) Thus, (20) becomes tr^𝑇₀𝑄+𝐾^𝑇_∗𝑅𝐾_∗Δ𝑋+𝑃𝐴+𝐵𝐾_∗+Δ𝑋𝐴+𝐵𝐾_∗^𝑇𝑃Δ𝑋+𝐶+𝐷𝐾_∗^𝑇𝑃𝐶+𝐷𝐾_∗^𝑇𝑋̇Δ𝑋−𝑃Δ𝑑𝑡=0.(25) From Lemma 13, 𝑃 is differential in [0,𝑇) and (19a) is obtained.
In the same way, integrating (21) by parts, we obtain tr^𝑇₀𝐾^𝑇_∗𝑅Δ𝐾𝑋_∗+𝑃𝐵Δ𝐾𝑋_∗+𝐶+𝐷𝐾_∗^𝑇𝑃𝐷Δ𝐾𝑋_∗𝑑𝑡=0.(26) From Lemma 14, (19c) is obtained.
To ensure the continuity of 𝑃, replace 𝑃(𝑇)=0 with 𝑃(𝑇)=𝜆𝐼_𝑛 (i.e., (19b)).
Equation (19d) is called complementary slackness conditions.

Remark 16. Equations (16b)-(16c) of Problem 11 and (19a)–(19c) of Theorem 15 are 2𝑛-dimensional, first-order differential equations including 2𝑛 terminal conditions and 𝑚 algebraic equations. Equation (19d) is called a complementary slackness condition. By using these conditions, 𝑋_∗, 𝐾_∗, 𝑃, and 𝜆 are obtained.

Remark 17. As for the complementary slackness condition, if the inequality constraint of Problem 11 is strict, then 𝜆=0, and the problem becomes easier. If the inequality constraint of Problem 11 is an equality constraint, it simplifies Theorem 15 as Lagrange multiplier method.

Definition 18. 𝑀∈𝑅^𝑛×𝑛 is a given matrix. One calls 𝑀⁺ the Moore-Penrose generalized inverse of 𝑀, if 𝑀𝑀^†𝑀=𝑀,𝑀^†𝑀𝑀^†=𝑀^†,𝑀𝑀^†𝑇=𝑀𝑀^†,𝑀^†𝑀^𝑇=𝑀^†𝑀.(27) Based on Definition 18, we can rewrite Theorem 15 by expressing 𝐾_∗ in terms of 𝑃.

Lemma 19 (see [13]). Let matrices 𝐿, 𝑀, and 𝑁 be given with appropriate sizes. Then the matrix equation 𝐿𝑋𝑀=𝑁(28) has a solution 𝑋 if and only if 𝐿𝐿^†𝑁𝑀^†𝑀=𝑁.(29) Moreover, any solution to 𝐿𝑋𝑀=𝑁 is represented by 𝑋=𝐿^†𝑁𝑀^†+𝑆−𝐿^†𝐿𝑆𝑀𝑀^†,(30) where 𝑆 is a matrix with an appropriate size.

Theorem 20. If 𝐾_∗ is optimal solution of Problem 1, then there exist a unique 𝑃∈𝑁𝐵𝑉^𝑛×𝑛[0,𝑇] and a nonnegative 𝜆∈𝑅¹, such that −𝑃̇=𝑄+𝐾^𝑇_∗𝑅𝐾_∗+𝑃𝐴+𝐵𝐾_∗+𝐴+𝐵𝐾_∗^𝑇𝑃+𝐶+𝐷𝐾_∗𝑃𝐶+𝐷𝐾_∗^𝑇,𝑃(𝑇)=𝜆𝐼_𝑛,𝐾^𝑇_∗𝑅+𝑃𝐵+𝐶+𝐷𝐾_∗^𝑇𝜆[(𝑇)]𝐾𝑃𝐷=0,tr𝑋−𝑐=0,_∗=−𝑅+𝐷^𝑇𝑃𝐷^†𝐵^𝑇𝑃+𝐷^𝑇𝑃𝐶+𝑌−𝑅+𝐷^𝑇𝑃𝐷^†𝑅+𝐷^𝑇𝑃𝐷𝑌,(31) where 𝑌∈𝐿²(0,𝑇;𝑅^𝑚×𝑛).

Proof. Form (19c) in Theorem 15, we obtain 𝑅+𝐷^𝑇𝑃𝐷𝐾_∗=−𝐵^𝑇𝑃+𝐷^𝑇𝑃𝐶.(32) According to Lemma 19, 𝐾_∗𝑅+𝐷^𝑇𝑃𝐷⁺𝐵^𝑇−𝑃+𝐷𝑇𝑃𝐶+𝑌𝑅+𝐷^𝑇𝑃𝐷⁺𝐵^𝑇𝑃+𝐷^𝑇𝑃𝐶𝑌,(33)
where 𝑌∈𝐿²(0,𝑇;𝑅^𝑚×𝑛).

As a special case, let us consider the following discrete stochastic LQ control problem without inequality constraint.

Problem 21. Consider the following. min𝐽(𝑋,𝐾)=tr^𝑇₀𝑄𝑋(𝑡)+𝐾(𝑡)^𝑇(𝑡)𝑋(𝑡̇()()𝑅𝐾)𝑑𝑡s.t.𝑋=𝐴+𝐵𝐾𝑋+𝑋𝐴+𝐵𝐾^𝑇+(𝐶+𝐷𝐾)𝑋(𝐶+𝐷𝐾)^𝑇𝑋(0)=𝑋₀.(34)

Corollary 22. If 𝐾_∗ is optimal solution of Problem 21, then there exists a unique 𝑃∈𝑁𝐵𝑉^𝑛×𝑛[0,𝑇] to the following constrained GDRE [13] −𝑃̇=𝑃𝐴+𝐴𝑃^𝑇+𝐶^𝑇−𝑃𝐶+𝑄𝑃𝐵+𝐶^𝑇𝑃𝐷𝑅+𝐷^𝑇𝑃𝐷𝑃𝐵+𝐶^𝑇𝑃𝐷^𝑇,𝑃(𝑇)=0,𝑅+𝐷^𝑇𝑃𝐷𝑅+𝐷^𝑇𝑃𝐷⁺𝐵^𝑇𝑃+𝐷^𝑇𝑃𝐶=𝐵^𝑇𝑃+𝐷^𝑇𝑃𝐶,𝑅+𝐷^𝑇𝐾𝑃𝐷≥0,_∗=−𝑅+𝐷^𝑇𝑃𝐷^†𝐵^𝑇𝑃+𝐷^𝑇𝑃𝐶+𝑌−𝑅+𝐷^𝑇𝑃𝐷^†𝑅+𝐷^𝑇𝑃𝐷𝑌,(35) where 𝑌∈𝐿²(0,𝑇;𝑅^𝑚×𝑛).

Proof. Because of Problem 21 without the equality constraint, 𝜆=0, therefore 𝑃(𝑇)=0. Theorem 20 yields this corollary directly.

4. Application

4.1. A Necessary and Sufficient Condition

In Theorem 15, to ensure the uniqueness of 𝜆 and 𝑃, let us strengthen the condition (𝑅+𝐷^𝑇𝑃𝐷)≥0 as (𝑅+𝐷^𝑇𝑃𝐷)>0.

Theorem 23. If 𝐾_∗ is optimal solution of Problem 1, then 𝑃∈𝑁𝐵𝑉^𝑛×𝑛[0,𝑇] and 𝜆∈𝑅¹>0, such that −𝑃̇=𝑄+𝐾^𝑇_∗𝑅𝐾_∗+𝑃𝐴+𝐵𝐾_∗+𝐴+𝐵𝐾_∗^𝑇𝑃+𝐶+𝐷𝐾_∗𝑃𝐶+𝐷𝐾_∗^𝑇,𝑃(𝑇)=𝜆𝐼_𝑛,𝐾^𝑇_∗𝑅+𝑃𝐵+𝐶+𝐷𝐾_∗^𝑇𝜆[(𝑇)]𝑃𝐷=0,tr𝑋−𝑐=0,(36) and if (𝑅+𝐷^𝑇𝑃𝐷)>0 hold, then Problem 1 is well posed. The optimal solutions of Problem 1 are 𝑈_∗=−𝑅+𝐷^𝑇𝑃𝐷⁻¹𝐵^𝑇𝑃+𝐷^𝑇𝐽𝑃𝐶𝑥,_∗=𝑥^𝑇₀𝑃(0)𝑥₀−𝜆𝑐.(37)

Proof. In Remark 3, in order for a minimum point 𝑥^∗ of a mathematical programming to satisfy the KKT conditions, we have assumed that 𝑥^∗ satisfies LICQ regularity condition. From (19c) of Theorem 15, we obtain 𝐾_∗𝑅+𝐷^𝑇𝑃𝐷=−𝐵^𝑇𝑃+𝐷^𝑇𝑃𝐶.(38) Because (𝑅+𝐷^𝑇𝑃𝐷)>0, Problem 11 has a unique solution; namely, 𝐾_∗=−𝑅+𝐷^𝑇𝑃𝐷⁻¹𝐵^𝑇𝑃+𝐷^𝑇𝑃𝐶.(39) From the relationship between Problems 1 and 11 and by using Itô’s formula, we have 𝐽_∗=𝑉𝑥₀=𝑥^𝑇₀𝑃(0)𝑥₀−𝜆𝑐.(40)

4.2. A Dynamic Programming Algorithm

A method for solving the indefinite stochastic linear quadratic (LQ) optimal control problem with unequal terminal state constraint is proposed as follows.

Reconsider the following LQ Problem 1′(𝑡)=[𝐴(𝑡)𝑥(𝑡)(𝑡)𝑢(𝑡+[𝐶(𝑡)𝑥(𝑡)(𝑡)𝑢(𝑡(𝑡),𝑥(𝑠)𝐸𝑥(𝑇𝑑𝑥+𝐵)]𝑑𝑡+𝐷)]𝑑𝑤=𝑦,‖)‖²=𝐸𝑥^𝑇(𝑇)𝑥(𝑇𝐽𝑥)≤𝑐,₀,𝑢=𝐸^𝑇₀𝑥^𝑇(𝑡)𝑄(𝑡)𝑥(𝑡)+𝑢^𝑇(𝑡)𝑅(𝑡)𝑢(𝑡𝑉)𝑑𝑡,𝑠,𝑦=inf_{𝑢(⋅)∈𝑈ad}𝐽𝑠,𝑦;𝑢(⋅).(41)

Then the backward dynamic programming optimality principle [8] yields the following theorem.

Theorem 24. Let 𝑥_∗(⋅) and 𝑢_∗(⋅) be an optimal solution of Problem 1′, then 𝑥_∗(⋅), and 𝑢_∗(⋅) satisfy the following Bellman equation: 𝑉𝑠,𝑦=min_{𝑢(⋅)∈𝑈𝑎𝑑}𝐸_𝑠^𝑠+ℎ𝑥^𝑇_∗(𝑡)𝑄(𝑡)𝑥_∗(𝑡)+𝑢^𝑇_∗(𝑡)𝑅(𝑡)𝑢_∗(𝑡)𝑑𝑡+𝑉𝑠+ℎ,𝑥_∗()𝑉𝑠+ℎ∀ℎ>0,𝑇−ℎ,𝑦=min_{𝑢(⋅)∈𝑈𝑎𝑑}𝐸^𝑇_𝑇−ℎ𝑥^𝑇_∗(𝑡)𝑄(𝑡)𝑥_∗(𝑡)+𝑢^𝑇_∗(𝑡)𝑅(𝑡)𝑢_∗(𝑡𝐸𝑥(𝑇)𝑑𝑡,‖)‖²=𝐸𝑥^𝑇(𝑇)𝑥(𝑇)≤𝑐.(42)

Proof. By applying the backward dynamic programming optimality principle, we obtain recursive relationships between 𝑉(𝑠,𝑥_∗(𝑠)) and 𝑉(𝑠+ℎ,𝑥_∗(𝑠+ℎ)) as follows: 𝑉𝑠,𝑦=min_{𝑢(⋅)∈𝑈ad}𝐸_𝑠^𝑠+ℎ𝑥^𝑇_∗(𝑡)𝑄(𝑡)𝑥_∗(𝑡)+𝑢^𝑇_∗(𝑡)𝑅(𝑡)𝑢_∗(𝑡)𝑑𝑡+𝑉𝑠+ℎ,𝑥_∗(𝑠+ℎ),(43) where 𝑉(𝑠,𝑥_∗(𝑠)) is the objective of [𝑠,𝑇] and 𝑉(𝑠+ℎ,𝑥_∗(𝑠+ℎ)) of [𝑠+ℎ,𝑇]. Let 𝑉(𝑠,𝑥_∗(𝑠)) be the objective of [𝑇−ℎ,𝑇] and then 𝑉𝑇−ℎ,𝑦=min_{𝑢(⋅)∈𝑈ad}𝐸^𝑇_𝑇−ℎ𝑥^𝑇_∗(𝑡)𝑄(𝑡)𝑥_∗(𝑡)+𝑢^𝑇_∗(𝑡)𝑅(𝑡)𝑢_∗(𝑡)𝑑𝑡.(44) Combine the terminal state constraint 𝐸‖𝑥(𝑇)‖²=𝐸𝑥^𝑇(𝑇)𝑥(𝑇𝑢(𝑡)(𝑡);)≤𝑐,=𝐾𝑥𝐾isgivenbyTheorem20(45) thus we can solve the above recursive relations. Furthermore, let ℎ→0 generating the dynamic programming algorithm.

Remark 25. The dynamic programming algorithm can be applied to solve the stochastic LQ problem with inequality constraint except for a heavy computation. Nonetheless, it makes more sense in theory.

Example 26. In Problem 1, let 𝐴=−1,𝐵=1,𝐶=−2,𝐷=0,𝑐=1+𝑒²²4𝑒²,𝑥₀=1,𝑄=0,𝑅=1,𝑇=1.(46) By Theorem 15, we have −𝑃̇=𝐾²_∗+2𝑃−1+𝐾_∗𝑃(1)𝐾+4𝑃,=𝜆,_∗𝜆𝑋(1)−+𝑃=0,1+𝑒²²4𝑒²=0,𝜆≥0.(47) The constraints of Problem 11 are −̇𝑋(0)𝑋(1)≤𝑋=2𝑋+2𝐾𝑋,=1,1+𝑒²²4𝑒^2𝑡.(48) Via a series of computations, we obtain 𝑋_∗=1+𝑒²²4𝑒^2𝑡2,𝑃=1+𝑒^2𝑡,2𝜆=1+𝑒²,𝐾_∗2=−1+𝑒^2𝑡.(49) In Problem 11, the inequality constraint is 𝐺𝑋_∗(1)=𝑋_∗(1)−𝑐≤0.(50) When 𝑇=1, 𝑋_∗(1)=(1+𝑒²)²4𝑒²=𝑐.(51) Hence 𝐺𝑋_∗(1)=𝑋_∗(1)−𝑐=0.(52) This shows that the optimal solution of Problem 11 satisfies the inequality constraint. In this case, the inequality constraint is a nonactive constraint. Because (𝑅+𝐷^𝑇𝑃𝐷)=1>0, using Theorem 23, the optimal control of Problem 1 is 𝑢_∗(𝑡)2=−1+𝑒^2𝑡𝑥_∗(𝑡),(53) and the optimal value is 𝑉𝑥₀=𝑥^𝑇₀𝑃(0)𝑥₀−𝜆𝑐=1−𝑒⁻²2.(54)

5. Conclusion

We have studied the indefinite stochastic LQ optimal control problem with unequal terminal state constraint, which can be transformed into a hybrid constrained mathematical programming problem. By applying KKT theorem, we have presented a necessary condition for the constrained indefinite stochastic LQ optimal control problem. By adding some conditions, we obtain a necessary and sufficient condition for indefinite stochastic LQ optimal control problem with inequality constraint. We demonstrate that the solvability of the generalized Riccati equation is sufficient for the well posedness of the indefinite LQ problem. Since this kind of LQ control problems can be transformed into a hybrid constrained mathematical programming, we have given a dynamic programming algorithm.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (61174078, 61170183), the Research Fund for the Taishan Scholar Project of Shandong Province of China, the SDUST Research Fund (no. 2011KYTD105), and the State Key Laboratory of Alternate Electrical Power System with Renewable Energy Sources (no. LAPS13018).