Study on Stochastic Linear Quadratic Optimal Control with Quadratic and Mixed Terminal State Constraints

Abstract

This paper studies the indefinite stochastic LQ control problem with quadratic and mixed terminal state equality constraints, which can be transformed into a mathematical programming problem. By means of the Lagrangian multiplier theorem and Riesz representation theorem, the main result given in this paper is the necessary condition for indefinite stochastic LQ control with quadratic and mixed terminal equality constraints. The result shows that the different terminal state constraints will cause the endpoint condition of the differential Riccati equation to be changed. It coincides with the indefinite stochastic LQ problem with linear terminal state constraint, so the result given in this paper can be viewed as the extension of the indefinite stochastic LQ problem with the linear terminal state equality constraint. In order to guarantee the existence and the uniqueness of the linear feedback control, a sufficient condition is also presented in the paper. A numerical example is presented at the end of the paper.

1. Introduction

Linear quadratic (LQ) control is an extremely important class of control problems in both theory and application. It is pioneered by Kalman [1] for deterministic systems and was extended to stochastic systems by Wonham [2]. In recent years, extensive research has been carried out in the so-called indefinite stochastic LQ control, in which the cost weighting matrices are allowed to be indefinite; refer to [3–6] for detailed accounts. A basic assumption in the LQ theory, both for deterministic and stochastic cases, is that the variable is unconstrained except for the differential equations constraint. As far as we know, very few results for constrained deterministic LQ can be found compared with the unconstrained one, not to mention the stochastic LQ control [7]. While in many real applications, constrained LQ control problem (such as nonnegativity and bound constraints for state and control variables) is a well-posed problem, constrained stochastic LQ control problem has a concrete application background, but the conventional LQ approach would collapse in the presence of any constraints. Study on the constrained stochastic LQ control will contribute to both theory and application a lot.

Huang and Zhang [8] studied the indefinite stochastic LQ control problem with linear terminal state equality constraints. Necessary and sufficient conditions for indefinite stochastic LQ control problems were investigated based on the Lagrangian multiplier theorem and Riesz representation theorem. The result showed that the linear feedback optimal control can be obtained by solving systems of algebraic and differential equations. The previous results on unconstrained indefinite stochastic LQ can be viewed as a specified case of the main theorem in that paper.

This paper studied the indefinite stochastic LQ control problem with quadratic terminal equality constraints and mixed constraints, which can be viewed as the extension of [8]. When the terminal state constraint is quadratic, the feasible region defined by the terminal constraint is nonconvex and multiple local minima abound, which makes the problem more complex to locate the optimum consistently. Developing a deeper understanding of the problems, as well as efficient algorithms for solving them, will have a big impact in many applications. Another reason for the study of this problem is that the methods used for solving this type of problem can be used to solve more general constrained optimal control problems. By means of the Lagrangian multiplier theorem and Riesz representation theorem, the main result in this paper is the necessary condition for indefinite stochastic LQ control with quadratic terminal constraints and mixed terminal constraints. The result showed that the difference of the terminal state constraints will cause the endpoint condition to be changed in the differential equations we obtained for the linear constraint control problem, which coincides with the reality. In order to guarantee the existence and the uniqueness of the linear feedback control, a sufficient condition is also presented in the paper. Numerical example is presented at the end of this paper.

For the convenience, we make use of the following basic notation in this paper. 𝑅^𝑛 is the 𝑛-dimensional real column vector, 𝑥(𝑡)=(𝑥₁(𝑡),…,𝑥_𝑛(𝑡))^′ is the 𝑛-dimensional statement column vector, 𝑢(𝑡)∈𝑅^𝑚 is the control input vector, 𝜔(⋅) is the one-dimensional Brownian motion defined on filtered probability (Ω,𝐹,𝑃;𝐹_𝑙) with a standard 𝐹_𝑙-adapted on [0,𝑇], and 𝐹_𝑙=𝜎(𝜔(𝑠)∶0≤𝑠≤𝑡≤𝑇) is an information fluid produced by Brownian motion. 𝑢(𝑡) belongs to the admissible control set  𝑈_ad={𝑢(𝑡)∈𝑅^𝑚,𝑢(𝑡)∈𝐿²_𝐹(0,𝑇;𝑅^𝑚)} is the adapted stochastic process which satisfies 𝐸∫^𝑇₀|𝑢²(𝑡)|𝑑𝑡<+∞ and the corresponding 𝑥(𝑡) satisfies (3), 𝜉_𝑖 is the given 𝐹_𝑇 measurable, integrable random variable; that is, 𝐸(𝜉_𝑖)<+∞,  and 𝑖=1,2,…,𝑙+𝑞 is a known number. 𝐴,𝐶∈𝐿^∞(0,𝑇;𝑆^𝑛×𝑛),𝑄∈𝐿^∞(0,𝑇;𝑆^𝑛×𝑛), 𝐵,𝐷∈𝐿^∞(0,𝑇;𝑅^𝑛×𝑚), 𝑅∈𝐿^∞(0,𝑇;𝑆^𝑚×𝑚), 𝑅^𝑚×𝑚 is 𝑚 row 𝑛 column matrix, and 𝑆^𝑛×𝑛 is an 𝑛×𝑛 symmetric matrix. 𝑀_𝑙∈𝑅^𝑙×𝑛 is the terminal state linear constraint coefficient matrix, and 𝑀_𝑞∈𝑅^𝑞×𝑛 is the terminal state quadratic constraint matrix. Two assumptions are given in this paper.𝐻₁: 𝑀_𝑙, 𝑀_𝑞 (the coefficient matrix for terminal linear and quadratic constraints respectively in Problem 1 to Problem 21) are full row rank and the set defined by the terminal state constraints is not empty;𝐻₂: 𝐿^∞(0,𝑇;𝑋)∶={𝑓(𝑡)∶𝑓(𝑡) is an 𝐹_𝑙-adapted, 𝑋-valued measurable process on [0,𝑇], and 𝐸∫^𝑇₀‖𝑓(𝑡)‖²_𝑋𝑑𝑡<+∞}.

The rest of this paper is organized as follows. Section 2 gives the problem statement and some preliminaries, Section 3 presents the main results of this paper. Numerical example is given in Section 4. Finally, Section 5 concludes this paper.

2. Problem Statement and Preliminaries

Problem 1. We study the following indefinite stochastic LQ control problem: inf𝐽𝑥₀,𝑢∶=𝐸^𝑇₀𝑥^′𝑄𝑥+𝑢^′𝑅𝑢𝑑𝑡 (1)

s.t. (𝑡)=(𝐴(𝑡)𝑥(𝑡)(𝑡)𝑢(𝑡+(𝐶(𝑡)𝑥(𝑡)(𝑡)𝑢(𝑡(𝑡),𝑥(0)𝑑𝑥+𝐵))+𝐷))𝑑𝜔=𝑥₀,𝑚₁₁𝑥²₁₁(𝑇)+𝑚₁₂𝑥²₁₂(𝑇)+⋯+𝑚_1𝑛𝑥²_1𝑛(𝑇)=𝜉₁,𝑚₂₁𝑥²₂₁(𝑇)+𝑚₂₂𝑥²₂₂(𝑇)+⋯+𝑚_2𝑛𝑥²_2𝑛(𝑇)=𝜉₂,⋮𝑚_𝑞1𝑥²_𝑞1(𝑇)+𝑚_𝑞2𝑥²_𝑞2(𝑇)+⋯+𝑚_𝑞𝑛𝑥²_𝑞𝑛(𝑇)=𝜉_𝑞. (2)(3)

Suppose that the feasible set defined by the quadratic constraint is not empty. We investigate the necessary condition under the linear feedback optimal control 𝑢(𝑡)=𝐾(𝑡)𝑥(𝑡), (4)

in which 𝐾(𝑡)∈𝐶^𝑚×𝑛[0,𝑇]. The cost weight matrix in Problem 1 is not necessarily positive, which usually causes Problem 1 to be called indefinite stochastic LQ control problem with terminal constraint. First, we present some definitions and lemmas that will be used and then transform the optimal control Problem 1 into a deterministic control problem.

Definition 2 (Gateaux differential). Let 𝑋 be a vector space, let 𝑌 be a normed space, and let 𝑇 be a (possibly nonlinear) transformation defined on a domain 𝐷⊆𝑋 with a range 𝑅⊆𝑌. Given 𝑥∈𝐷⊂𝑋, and ℎ is an arbitrary vector in 𝑋. If the limit 𝛿𝑇(𝑥;ℎ)=lim_𝛼→01𝛼[𝑇(𝑥+𝛼ℎ)−𝑇(𝑥)] (5) exists, then it is called Gateaux differential at 𝑥 with an increment ℎ. If the limit exists for arbitrary ℎ∈𝑋, the transformation 𝑇 is called Gateaux differentiable at 𝑥.

Definition 3 (Frechet differential). Let 𝑋 and 𝑌 be normed linear spaces and let 𝑇 be a transformation defined on an open domain 𝐷⊂𝑋 with a range 𝑅⊂𝑌. If, for a fixed 𝑥∈𝐷⊂𝑋, there exist 𝛿𝑇(𝑥;ℎ)∈𝑌, which is linear and continuous with respect to ℎ such that lim_{‖ℎ(𝑡)‖→0}‖𝑇(𝑥+ℎ)−𝑇(𝑥)−𝛿𝑇(𝑥;ℎ)‖‖ℎ(𝑡)‖=0, (6) then 𝑇 is said to be Frechet differentiable at 𝑥, and 𝛿𝑇(𝑥;ℎ) is said to be the Frechet differential of 𝑇 at 𝑥 with an increment ℎ.

Definition 4 (Frechet derivative and continuously Frechet differentiable). Suppose that a transformation 𝑇 defined on an open domain 𝐷⊂𝑋 is Frechet differentiable on 𝐷. The Frechet differential 𝛿𝑇(𝑥;ℎ)=𝑇^′(𝑥)ℎ for a fixed 𝑥∈𝐷, where 𝑇^′(𝑥) is a bounded linear operator from 𝑋 to 𝑌, one calls 𝑇^′(𝑥) the derivative of 𝑇(𝑥). If the derivative 𝑇^′(𝑥) is continuous on some open ball 𝑆, then 𝑇 is continuously Frechet differentiable on 𝑆.

Definition 5 (regular point of transformation). Let 𝑇(𝑥) be a transformation defined on a Banach space 𝑋 with a range 𝑌, which is also a Banach space. 𝑇(𝑥) is continuously Frechet differentiable if, for a given 𝑥₀∈𝑋,  𝛿𝑇(𝑥₀;ℎ) is an onto mapping from 𝑋 onto 𝑌; then 𝑥₀ is called a regular point.
Considering Problem 1 under the linear feedback optimal control, substituting 𝑢(𝑡) in Problem 1 with 𝑢(𝑡)=𝐾(𝑡)𝑥(𝑡). (7) And taking the place of 𝑢(𝑡) in (2), we have 𝑑𝑥=(𝐴+𝐵𝐾)𝑥𝑑𝑡+(𝐶+𝐷𝐾)𝑥𝑑𝜔. (8) According to the Itô integrals formula for 𝑥𝑥^′, in which 𝑥(𝑡) is the solution of (8), we have 𝑑𝑥𝑥^′=(𝐴+𝐵𝐾)𝑥𝑥^′+𝑥𝑥^′(𝐴+𝐵𝐾)^′+(𝐶+𝐷𝐾)𝑥𝑥^′(𝐶+𝐷𝐾)𝑑𝑡. (9) Define 𝐸(𝑥𝑥^′)=𝑋; it is obvious that 𝑋 is a symmetric matrix. Integrate both sides of (9) from 0 to 𝑡 with variable 𝑡, then compute the derivatives of both sides of (9) after taking the expectation, then get the following matrix differential equation with the initial condition: 𝑋̇=(𝐴+𝐵𝐾)𝑋+𝑋(𝐴+𝐵𝐾)^′+(𝐶+𝐷𝐾)𝑋(𝐶+𝐷𝐾)^′,𝑋(0)=𝑋₀=𝑥₀𝑥^′₀. (10) Substituting 𝑢(𝑡) with 𝑢(𝑡)=𝐾(𝑡)𝑥(𝑡) in (1) to inf𝐽(𝑋,𝐾)=tr^𝑇₀𝑄𝑋(𝑡)+𝐾^′(𝑡)𝑅𝐾(𝑡)𝑋(𝑡)𝑑𝑡, (11) (“tr” is the trace of a matrix, 𝐸(𝜉_𝑖)=𝑙_𝑖, 𝑖=1,2,…,𝑞), the quadratic constraints are tr𝑄₁𝑋(𝑇)=𝑙₁,tr𝑄₂𝑋(𝑇)=𝑙₂,…,tr𝑄_𝑞𝑋(𝑇)=𝑙_𝑞, (12) where 𝑄_𝑖=diag[𝑚_𝑖1,𝑚_𝑖2,…,𝑚_𝑖𝑛], 𝑖=1,2,…,𝑞, and 𝐿=(𝑙₁,𝑙₂,…,𝑙_𝑞)^′. Then Problem 1 can be transformed into the following deterministic optimal control Problem 6.

Problem 6. Consider inf𝐽(𝑋,𝐾)=tr^𝑇₀𝑄𝑋(𝑡)+𝐾^′(𝑡)𝑅𝐾(𝑡)𝑋(𝑡)𝑑𝑡 (13) s.t. 𝑋̇=(𝐴+𝐵𝐾)𝑋+𝑋(𝐴+𝐵𝐾)^′+(𝐶+𝐷𝐾)𝑋(𝐶+𝐷𝐾)^′,𝑋(0)=𝑋₀=𝑥₀𝑥₀,tr𝑄₁𝑋(𝑇)=𝑙₁,tr𝑄₂𝑋(𝑇)=𝑙₂,⋯tr𝑄_𝑞𝑋(𝑇)=𝑙_𝑞. (14)(15)(16)

Remark 7. It is obvious that if Problem 1 has the optimal linear feedback control 𝑢_∗(𝑡)=𝐾_∗(𝑡)𝑥(𝑡), then 𝐾_∗(𝑡) must be the solution of Problem 6, while the inverse does not hold.
Take the objective functional 𝐽(𝑋,𝐾) as a functional defined on the space 𝐶^𝑛×𝑛[0,𝑇]×𝐶^𝑛×𝑛[0,𝑇], in which 𝐶^𝑛×𝑛[0,𝑇] is an 𝑛×𝑛 square matrix space, the element in this space is the continuous function defined on [0,𝑇]: 𝐻(𝑡)(𝑡)(0)−𝑋,𝐾)(=𝑋−𝑋_𝑡0)()+()𝑋()(𝐴+𝐵𝐾𝑋+𝑋𝐴+𝐵𝐾𝐶+𝐷𝐾𝐶+𝐷𝐾^′𝑑𝑡, (17) and terminal state constraints (16) defines a transformation 𝐺(𝑋) from 𝐶^𝑛×𝑛[0,𝑇] to 𝑅^𝑞∶𝐺(𝑥(𝑇=𝑄))tr₁𝑋(𝑇)⋮tr𝑄_𝑞𝑋(𝑇). (18) By the virtue of (17)-(18), constraints (14)–(16) can be reformulated as 𝐻(𝑋,𝐾)(𝑡)=0,𝐺(𝑋(𝑇))=𝐿,∀𝑡∈[0,𝑇]. (19)

Lemma 8. Functional 𝐽(𝑋,𝐾), 𝐻(𝑋,𝐾), 𝐺(𝑋) are continuous Frechet differentiable functionals and have Frechet derivative 𝛿𝐽_𝑋(𝑋,𝐾;Δ𝑋)=tr^𝑇₀𝑄+𝐾^′𝑅𝐾Δ𝑋𝑑𝑡,𝛿𝐽_𝐾(𝑋,𝐾;Δ𝐾)=tr^𝑇₀Δ𝐾^′𝑅𝐾𝑋+𝐾^′𝑅Δ𝐾𝑋𝑑𝑡,𝛿𝐻_𝑋(𝑡)(𝑡)−𝑋,𝐾;Δ𝑋)(=Δ𝑋^𝑡₀)()+()()(𝐴+𝐵𝐾Δ𝑋+Δ𝑋𝐴+𝐵𝐾𝐶+𝐷𝐾Δ𝑋𝐶+𝐷𝐾^′𝑑𝑡,𝛿𝐻_𝐾(𝑋,𝐾;Δ𝐾)(𝑡)=−^𝑡₀𝐵Δ𝐾𝑋+𝑋Δ𝐾^′𝐵^′+()𝑋()()𝐶+𝐷𝐾𝐷Δ𝐾+𝐷Δ𝐾𝑋𝐶+𝐷𝐾^′𝑑𝑡,𝛿𝐺_𝑋((𝑇=𝑄𝑋;Δ𝑋))tr₁(𝑇)⋮𝑄Δ𝑋tr_𝑞Δ𝑋(𝑇). (20)(21)(22)(23)(24)

Proof. The proof is mainly based on Definition 2. Apply Definition 2 to the functional constraints (19); the proofs for the first three results are the same to those in [8], and we only need the proof of (24) here: Δ𝐺(𝑋;Δ𝑋)=lim_𝛼→0𝐺(𝑋;𝑋+𝛼Δ𝑋−𝐺(𝑋))𝛼,𝐺()(𝑋)=𝑄𝑋;𝑋+𝛼𝑋−𝐺tr₁𝑋(𝑇)(𝑇)⋮𝑄+𝛼Δ𝑋tr_𝑞𝑋(𝑇)(𝑇)−𝑄+𝛼Δ𝑋tr₁𝑋(𝑇)⋮tr𝑄_𝑞𝑋(𝑇),𝛿𝐺(𝑋,Δ𝑋)=lim_𝛼→0𝐺(𝑋;𝑋+Δ𝑋)−𝐺(𝑋)𝛼=tr𝑄₁⋮𝑄Δ𝑋tr_𝑞.Δ𝑋 (25)

Lemma 9. Constraint (19) satisfies the regular condition; that is, 𝛿𝐻(𝑋,𝐾;Δ𝑋,Δ𝐾),𝛿𝐺(𝑋;Δ𝑋(𝑇)) (26) are onto mapping when Δ𝑋×Δ𝐾∈𝐶^𝑛×𝑛×𝐶^𝑚×𝑛. (27)

Proof. The proof of the onto mapping for 𝛿𝐻(𝑋,𝐾;Δ𝑋,Δ𝐾) has been given in [8]; we only need to prove that 𝛿𝐺(𝑋;Δ𝑋) is a onto mapping when Δ𝑋 varies. For a given 𝑌∈𝑅^𝑞, the following equation has a solution: tr𝑄₁⋮𝑄Δ𝑋tr_𝑞Δ𝑋=𝑌. (28) Because the coefficient matrix for quadratic terminal state constraints is full row rank (Assumption 𝐻₁), equation 𝑚₁₁⋯𝑚_1𝑛𝑚⋮⋮⋮_𝑞1⋯𝑚_𝑞𝑛Δ𝑋₁₁⋮Δ𝑋_𝑞𝑛=𝑌, (29) exists a solution, which finishes the proof.

Definition 10 (the well-posedness). The LQ optimal control problem is to minimize the cost functional 𝐽(𝑥₀,𝑢) over 𝑢∈𝑈_ad. Define the optimal value function as 𝑉(𝑥₀)=inf_{𝑢∈𝑈ad}𝐽(𝑥₀,𝑢). The LQ problem is called well posed if −∞<𝑉𝑥₀<+∞,∀𝑥₀∈𝑅^𝑛. (30) A well-posed problem is called attainable (with respect to 𝑥₀) if there is a 𝑢_∗(⋅) control that achieves 𝑉(𝑥₀).

Theorem 11 (Lagrange multiplier theorem). If the continuously Frechet differentiable real functional 𝑓 defined on Banach space has a local extremum under the constraint 𝐻(𝑥)=0 at the regular point 𝑥₀, 𝐻(𝑥) is a mapping from space 𝑋 to Banach space 𝑍, and then there exists an element 𝑧^∗₀∈𝑍^∗ such that the Lagrangian functional 𝐿(𝑥)=𝑓(𝑥)+𝑧^∗₀𝐻(𝑥) (31) is stationary at 𝑥₀; that is, 𝛿𝑓𝑥₀;ℎ+𝑧^∗₀𝛿𝐻𝑥₀;ℎ=0 (32) for each ℎ∈𝑋.

Theorem 12 (Riesz representation theorem). Let 𝑓(𝑥) be a bounded linear functional on 𝑋=𝐶[𝑎,𝑏], and then there is a bounded variation function 𝜈 on [𝑎,𝑏] such that, for all 𝑥∈𝑋, 𝑓(𝑥)=^𝑏_𝑎𝑥(𝑡)𝑑𝜈(𝑡), (33) where the norm of 𝑓(𝑥) is the total variation on [𝑎,𝑏]. Conversely, every function of bounded variation on [𝑎,𝑏] defines a bounded linear function on 𝑋 in this way.

3. Main Results

Lemma 13 (see [9]). If 𝛼(𝑡) is continuous in [𝑡₁,𝑡₂] and ∫^𝑡2_𝑡1𝛼(𝑡)𝑑ℎ(𝑡)=0  for every ℎ∈𝐷[𝑡₁,𝑡₂] (set of all continuously differentiable functions on [𝑡₁,𝑡₂]) with ℎ(𝑡₁)=ℎ(𝑡₂)=0, then 𝛼(𝑡)=0 on [𝑡₁,𝑡₂].

Lemma 14 (see [9]). If 𝛼(𝑡) is continuous in [𝑡₁,𝑡₂] and ∫^𝑡2_𝑡1𝛼(𝑡)𝑑ℎ(𝑡)=0 for every ℎ∈𝐷[𝑡₁,𝑡₂] with ℎ(𝑡₁)=ℎ(𝑡₂)=0, then 𝛼(𝑡)=𝑐 in [𝑡₁,𝑡₂], where 𝑐 is a constant.

Lemma 15 (see [9]). If 𝛼(𝑡) and 𝛽(𝑡) are continuous in [𝑡₁,𝑡₂] and ^𝑡2_𝑡1𝛼(𝑡)ℎ(𝑡)+𝛽(𝑡)ℎ^′(𝑡)𝑑𝑡=0 (34) for every ℎ∈𝐷[𝑡₁,𝑡₂] with ℎ(𝑡₁)=ℎ(𝑡₂)=0, then 𝛽 is differentiable and 𝛽̇(𝑡)=𝛼(𝑡) in [𝑡₁,𝑡₂].

We make use of NBV^𝑛×𝑛[0,𝑇] (nonnegative bounded variation functional on [0,𝑇]) to express the matrix space with the element in [0,𝑇]. The space is a bounded variation right continuous function that function takes value 0 at the point at the point 𝑡=0. Based on the Lagragian theorem and the Riesz representation theorem, we obtained a necessary condition for Problem 1.

3.1. Necessary Condition

Theorem 16 (necessary condition for indefinite stochastic LQ with quadratic constraints). Suppose the set defined by the terminal state constraint is not empty and the optimal control Problem 6 exists an optimal feedback control matrix 𝐾_∗; then there must exist a symmetric matrix 𝑃∈𝑁𝐵𝑉^𝑛×𝑛[0,𝑇] and a vector 𝜆∈𝑅^𝑞 such that for all 𝑡∈[0,𝑇], 𝑃 and 𝜆 satisfy the following equations: −𝑃̇=𝑄+𝐾^′_∗𝑅𝐾_∗+𝑃𝐴+𝐵𝐾_∗+𝐴+𝐵𝐾_∗^′𝑃+𝐶+𝐷𝐾_∗^′𝑃𝐶+𝐷𝐾_∗,𝑃(𝑇)=^𝑞_𝑖=1𝜆_𝑖𝑄_𝑖,𝐾^′_∗𝑅+𝑃𝐵+𝐶+𝐷𝐾_∗^′𝑃𝐷=0. (35)(36)(37)

Proof. From Remark 7, we know that 𝐾_∗ is the solution of deterministic optimal control Problem 6; take 𝑋 and 𝐾 in Problem 6 as the variables. Suppose that (𝑋_∗,𝐾_∗) is the optimal solution for Problem 6; according to Lemmas 14 and 15, Problem 6 satisfies the conditions of Lagrange multiplier theorem, and then there exists a symmetric matrix 𝑃∈NBV^𝑛×𝑛[0,𝑇] and 𝜆∈𝑅^𝑞 such that tr^𝑇₀𝑄+𝐾_∗𝑅𝐾_∗Δ𝑋𝑑𝑡+tr^𝑇₀Δ𝑋−^𝑡₀𝐴+𝐵𝐾_∗Δ𝑋+Δ𝑋𝐴+𝐵𝐾_∗+𝐶+𝐷𝐾_∗Δ𝑋𝐶+𝐷𝐾_∗^′𝑑𝑡𝑑𝑃+𝜆^′tr𝑄₁𝑋⋮tr𝑄_𝑞𝑋=0,tr^𝑇₀Δ𝐾^′𝑅𝐾_∗𝑋_∗+𝐾^′_∗𝑅Δ𝐾𝑋_∗𝑑𝑡−tr^𝑇₀^𝑡₀𝐵Δ𝐾𝑋_∗+𝑋_∗Δ𝐾^′𝐵^′+𝐶+𝐷𝐾_∗𝑋_∗×𝐷Δ𝐾^′+(𝐷Δ𝐾)𝑋_∗𝐶+𝐷𝐾_∗𝑑𝑡𝑑𝑃=0. (38)(39) The second parts of (38) and (39) are from Riesz representation theorem. In general, we take that 𝑃(𝑇)=0, and then (38) becomes tr^𝑇₀𝑄+𝐾^′_∗𝑅𝐾_∗Δ𝑋𝑑𝑡+tr^𝑇₀Δ𝑋𝑑𝑃+tr^𝑇₀𝑃𝐴+𝐵𝐾_∗Δ𝑋+Δ𝑋𝐴+𝐵𝐾_∗+𝐶+𝐷𝐾_∗Δ𝑋𝐶+𝐷𝐾_∗^′+tr𝜆₁𝑄₁𝑋+⋯+𝜆_𝑞𝑄_𝑞𝑋=0. (40) It is obvious that there is no jump in interval [0,𝑇) for function 𝑃(𝑡), otherwise we can choose Δ𝑋 that makes tr∫^𝑇₀Δ𝑋𝑑𝑃 be far more than the other parts in the equality. But 𝑃(𝑡) has jump at the point 𝑡=𝑇, and the height is 𝜆₁,…,𝜆_𝑞tr𝑄₁𝑋(𝑇)⋮tr𝑄_𝑞𝑋(𝑇). (41) Because the previous equalities (38) and (39) hold for all continuous functions Δ𝑋, for specified function which has continuous derivative and Δ𝑋(𝑇)=0, all the previous equalities also hold, and then Δ𝑋𝑑𝑃=𝑃Δ𝑋|^𝑇₀^𝑇₀−^𝑇₀̇(𝑇)(𝑇)(0)(0)−𝑃Δ𝑋𝑑𝑡=𝑃Δ𝑋−𝑃Δ𝑋^𝑇₀̇(𝑇)(𝑇)−𝑃Δ𝑋𝑑𝑡=𝑃Δ𝑋^𝑇₀𝑃Δ𝑋̇𝑑𝑡. (42) Because 𝑃(𝑇)=0, then ^𝑇₀Δ𝑋𝑑𝑃=−^𝑇₀𝑃Δ𝑋̇𝑑𝑡, (43) and we have the following from (38) tr^𝑇₀𝑄+𝐾^′_∗𝑅𝐾_∗Δ𝑋+𝑃𝐴+𝐵𝐾_∗Δ𝑋+𝐴+𝐵𝐾_∗^′×𝑃Δ𝑋+𝐶+𝐷𝐾_∗^′𝑃𝐶+𝐷𝐾_∗Δ𝑋−𝑃Δ𝑋̇𝑑𝑡=0. (44) According to Lemma 15, we have −𝑃̇=𝑄+𝐾^′_∗𝑅𝐾_∗+𝑃𝐴+𝐵𝐾_∗+𝐴+𝐵𝐾^′_∗𝑃+𝐶+𝐷𝐾_∗^′𝑃𝐶+𝐷𝐾_∗. (45) We take the integral by parts of the second part of (39), and then tr^𝑇₀𝐾^′_∗𝑅(Δ𝐾)𝑋_∗+𝑃𝐵Δ𝐾𝑋_∗+𝐶+𝐷𝐾_∗^′𝑃𝐷Δ𝐾𝑋_∗𝑑𝑡=0. (46) Based on Lemma 13, we have 𝐾^′_∗𝑅+𝑃𝐵+𝐶+𝐷𝐾_∗^′𝑃𝐷=0. (47) Change the endpoint conditions 𝑃(𝑇)=0 with 𝑃(𝑇)=𝜆₁𝑄₁+⋯+𝜆_𝑞𝑄_𝑞; that is, (36) holds.

Remark 17. The necessary condition in Theorem 16 (35)–(37) and the constraints (14)–(16) in Problem 6 have 2𝑛 one order differential equations, 2𝑛 bound conditions, 𝑞 terminal state conditions, and 𝑚 algebra equations, and then we can determine 𝑋_∗(𝑡),  𝐾_∗(𝑡),  𝑃(𝑡), and 𝜆, respectively.

Remark 18. The result in Theorem 12 is the same as that in [8] except for the terminal conditions.

3.2. Sufficient Condition

We have pointed out that the necessary conditions (14)–(16) and (35)–(37) are not sufficient for the existence and the uniqueness of the solution in Problem 6. In order to guarantee the uniqueness of 𝜆 and 𝑃(𝑡), the conditions must be strengthened to 𝑃+𝐷^′𝑃𝐷>0; (48) that is, the matrix 𝑅+𝐷^′𝑃𝐷 must be positive.

Theorem 19 (sufficient condition for stochastic LQ problem with quadratic constraints). If (14)–(16), (35)–(37), and (39) exist solutions 𝑋_∗(𝑡), 𝐾_∗(𝑡), 𝑃(𝑡) , and 𝜆, then Problem 6 is well posed and the optimal feedback control is 𝑢_∗(𝑡)=−𝑅+𝐷^′𝑃𝐷⁻¹𝐵^′𝑃+𝐷^′𝑃𝐶𝑥(𝑡); (49) the optimal cost value is 𝑉𝑥₀=inf_{𝑢(⋅)∈𝑈𝑎𝑑}𝐽𝑥₀,𝑢(𝑡)=𝑥(0)^′𝑃(0)𝑥(0)−𝜆^′𝐿. (50)

Proof. Suppose that 𝐾_∗(𝑡), 𝑃(𝑡), and 𝜆 are the solutions that satisfy (14)–(16), (35)–(37), and (48), and from (37), we have 𝐾_∗(𝑡)=−𝑅+𝐷^′𝑃𝐷⁻¹𝐵^′𝑃+𝐷^′𝑃𝐶. (51) Substitute 𝐾_∗(𝑡) in (35), and then 𝑃̇+𝑃𝐴+𝐴^′𝑃+𝐶^′−𝑃𝐶+𝑄𝑃𝐵+𝐶^′𝑃𝐷𝑅+𝐷^′𝑃𝐷⁻¹𝑃𝐵+𝐶^′𝐽𝑥𝑃𝐷=0,₀,𝑢(⋅)=𝐸^𝑇₀𝑥^′𝑄𝑥+𝑢^′𝑅𝑢𝑑𝑡+𝑑𝑥^′𝑃𝑥−𝑑𝑥^′𝑃𝑥. (52) Apply Itô’s formula again, so =𝑥^′(0)𝑃(0)𝑥(0)−𝐸𝑥^′(𝑇)𝑃(𝑇)𝑥(𝑇)+𝐸^𝑇₀𝑥^′𝑃̇+𝑃𝐴+𝐴^′𝑃+𝐶^′𝑃𝐶+𝑄𝑥+2𝑢^′𝐵^′𝑃+𝐷^′𝑃𝐶𝑥+𝑢^′𝑅+𝐷^′𝑃𝐷𝑢𝑑𝑡=𝑥^′(0)𝑃(0)𝑥(0)−𝜆^′tr𝑄₁𝑋(𝑇)⋮tr𝑄_𝑞𝑋(𝑇)+𝐸^𝑇₀𝑥^′𝑃̇+𝑃𝐴+𝐴^′𝑃+𝐶^′𝑃𝐶+𝑄−𝑃𝐵+𝐶^′×𝑃𝐷𝑅+𝐷^′𝑃𝐷⁻¹𝑃𝐵+𝐶^′𝑃𝐷^′𝑥𝑑𝑡+𝐸^𝑇₀𝑢+𝑅+𝐷^′𝑃𝐷⁻¹𝐵^′𝑃+𝐷^′𝑃𝐶𝑥^′×𝑅+𝐷^′𝑃𝐷𝑢+𝑅+𝐷^′𝑃𝐷⁻¹×𝐵^′𝑃+𝐷^′𝑃𝐶𝑥𝑑𝑡=𝑥^′(0)𝑃(0)𝑥(0)−𝜆^′tr𝑄₁𝑋(𝑇)⋮tr𝑄_𝑞𝑋(𝑇)+𝐸^𝑇₀𝑢+𝑅+𝐷^′𝑃𝐷⁻¹𝐵^′𝑃+𝐷^′𝑃𝐶𝑥^′×𝑅+𝐷^′𝑃𝐷𝑢+𝑅+𝐷^′𝑃𝐷⁻¹×𝐵^′𝑃+𝐷^′𝑃𝐶𝑥𝑑𝑡, (53) where the last equality is from (52). It is obvious that linear feedback control (49) has the minimum cost 𝐽(𝑥₀,𝑢(𝑡)), and the cost value is 𝑥^′(0)𝑃(0)𝑥(0)−𝜆^′tr𝑄₁𝑋(𝑇)⋯tr𝑄_𝑞𝑋(𝑇). (54)

3.3. Necessary Conditions for Stochastic LQ with Mixed Terminal State Constraint

We study the following indefinite stochastic LQ control problem with mixed constraints.

Problem 20. Consider inf𝐽𝑥₀,𝑢∶=𝐸^𝑇₀𝑥^′𝑄𝑥+𝑢^′𝑅𝑢𝑑𝑡 (55) s.t. (𝑡)=(𝐴(𝑡)𝑥(𝑡)(𝑡)𝑢(𝑡+(𝐶(𝑡)𝑥(𝑡)(𝑡)𝑢(𝑡(𝑡),𝑥(0)𝑑𝑥+𝐵))+𝐷))𝑑𝜔=𝑥₀,𝑚_1,1𝑥_1,1(𝑇)+𝑚_1,2𝑥_1,2(𝑇)+⋯+𝑚_1,𝑛𝑥_1,𝑛(𝑇)=𝜉₁,𝑚_2,1𝑥_2,1(𝑇)+𝑚_2,2𝑥_2,2(𝑇)+⋯+𝑚_2,𝑛𝑥_2,𝑛(𝑇)=𝜉₂,⋯𝑚_𝑙,1𝑥_𝑙,1(𝑇)+𝑚_𝑙,2𝑥_𝑙,2(𝑇)+⋯+𝑚_𝑙,𝑛𝑥_𝑙,𝑛(𝑇)=𝜉_𝑙,𝑚_𝑙+1,1𝑥_𝑙+1,12(𝑇)+𝑚_𝑙+1,2𝑥²_𝑙+1,2(𝑇)+⋯+𝑚_𝑙+1,𝑛𝑥²_𝑙+1,𝑛(𝑇)=𝜉_𝑙+1,𝑚_𝑙+2,1𝑥_𝑙+2,12(𝑇)+𝑚_𝑙+2,2𝑥²_𝑙+2,2(𝑇)+⋯+𝑚_𝑙+2,𝑛𝑥²_𝑙+2,𝑛(𝑇)=𝜉_𝑙+2,⋯𝑚_𝑙+𝑞,1𝑥_𝑙+𝑞,12(𝑇)+𝑚_𝑙+𝑞,2𝑥²_𝑙+𝑞,2(𝑇)+⋯+𝑚_{𝑙+𝑞,𝑛}𝑥²_{𝑙+𝑞,𝑛}(𝑇)=𝜉_𝑙+𝑞. (56) Using the same method as the above, the stochastic LQ control problem with linear and quadratic terminal constraints can be transformed into the following Problem 21.

Problem 21. Consider inf𝐽(𝑋,𝐾)=tr^𝑇₀𝑄𝑋(𝑡)+𝐾^′(𝑡)𝑅𝐾(𝑡)𝑋(𝑡)𝑑𝑡 (57) s.t. 𝑋̇=(𝐴+𝐵𝐾)𝑋+𝑋(𝐴+𝐵𝐾)^′+(𝐶+𝐷𝐾)𝑋(𝐶+𝐷𝐾)^′,𝑋(0)=𝑋₀=𝑥₀𝑥₀,𝑀_𝑙𝑋(𝑇)𝑀^′_𝑙𝑄=𝑁,tr₁𝑋(𝑇)=𝑙₁,tr𝑄₂𝑋(𝑇)=𝑙₂,⋯tr𝑄_𝑞𝑋(𝑇)=𝑙_𝑞. (58)

Theorem 22 (necessary and sufficient conditions for stochastic LQ problem with mixed constraint). Suppose that the terminal constraint set defined by the linear and quadratic state constraints is not empty and the optimal control Problem 6 exists the optimal feedback control matrix 𝐾_∗. Then there must exist a symmetric matrix 𝑃∈𝑁𝐵𝑉^𝑛×𝑛[0,𝑇] and a vector 𝜆∈𝑅^𝑞, 𝜇∈𝑅^𝑙×𝑙, for all 𝑡∈[0,𝑇], satisfing the following equalities: −𝑃̇=𝑄+𝐾^′_∗𝑅𝐾_∗+𝑃𝐴+𝐵𝐾_∗+𝐴+𝐵𝐾_∗^′𝑃+𝐶+𝐷𝐾_∗^′𝑃𝐶+𝐷𝐾_∗,𝑃(𝑇)=^𝑞_𝑖=1𝜆_𝑖𝑄_𝑖+𝑀^′_𝑙𝜇𝑀_𝑙,𝐾^′_∗𝑅+𝑃𝐵+𝐶+𝐷𝐾_∗^′𝑃𝐷=0. (59)

If (14)–(16), (35)–(37), and (39) exist solutions 𝑋_∗(𝑡),  𝐾_∗(𝑡), 𝑃(𝑡), and 𝜆, then Problem 6 is well posed and the optimal feedback control is 𝑢_∗(𝑡)=−𝑅+𝐷^′𝑃𝐷⁻¹𝐵^′𝑃+𝐷^′𝑃𝐶𝑥(𝑡). (60) The optimal value is 𝑉𝑥₀=inf_{𝑢(⋅)∈𝑈ad}𝐽𝑥₀,𝑢(𝑡)=𝑥^′₀𝑃(0)𝑥(0)−𝜆^′𝐿_𝑞−tr𝜇𝑁. (61)

The proof of the theorem is the same as that of Theorem 16 except for the terminal condition.

4. Numerical Examples

Let 𝐴=−1, 𝐵=1,  𝐶=−2, 𝐷=0,  𝑀_𝑞=1,  𝐿=𝑒², 𝑋₀=2, 𝑄=0, 𝑅=1, and 𝑇=1 in Problem 6, according to (35)–(37) in Theorem 16 and the constraint (14)–(16) in Problem 1, and then −𝑃̇=𝐾²_∗+2𝑃𝐾_∗𝑃(1)𝐾−1+4𝑃,=𝜆,_∗𝑋+𝑃=0,_∗=2𝐾_∗−1𝑋_∗+4𝑋_∗,𝑋(0)𝑋=2,_2∗(1)=𝑒². (62) Solve the system equations, and then 𝑋_∗(𝑡)=𝑒+𝑒−1𝑒−2𝑒𝑒−1^2𝑡,𝐾_∗(𝑡)=2(2/𝑒−1)𝑒^2𝑡,𝑃(𝑡)=2−11−(2/𝑒−1)𝑒^2𝑡2,𝜆=𝑒².−2𝑒+1 (63) The optimal control for Problem 6 is 𝑢_∗(𝑡)=2𝑒(𝑒−2)𝑒^2𝑡/((2−𝑒)(𝑒−1)𝑒^2𝑡−𝑒(𝑒−1)), and the optimal index cost value is 𝐽𝑥₀,𝑢_∗(𝑡)=2𝑒²−4𝑒(𝑒−1)². (64)

5. Conclusion

This paper studied a class of indefinite stochastic LQ control problems with quadratic terminal state constraints and mixed terminal state constraints. By means of the Lagrange multiplier theorem and Riesz representation theorem, this paper presented a necessary condition for indefinite stochastic LQ control problems with quadratic terminal state constraints and mixed terminal state constraints. The result shows that the necessary condition for quadratic terminal constraints is the same as for the linear terminal state constraints that is presented in [8] except for the terminal condition. This coincides with the reality. A sufficient condition also was presented for the existence and uniqueness of the optimal linear feedback control. Numerical example verified the main theorem in this paper.

Acknowledgments

The author would like to thank Professor Kristian Ranestad, who gave the invitation to visit Mathematics Department of Oslo University. Further, the author is grateful to RCN, who financed the visit Grant 218993/F11/mm. This work is supported by the National Natural Science Foundation of China (11241005).