`Journal of Applied MathematicsVolume 2013 (2013), Article ID 674327, 8 pageshttp://dx.doi.org/10.1155/2013/674327`
Research Article

## Study on Stochastic Linear Quadratic Optimal Control with Quadratic and Mixed Terminal State Constraints

Information and Electrical Engineering College, Shandong University of Science and Technology, Qingdao, Shandong 266590, China

Received 5 April 2013; Accepted 14 August 2013

Copyright © 2013 Yang Hongli. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

#### Abstract

This paper studies the indefinite stochastic LQ control problem with quadratic and mixed terminal state equality constraints, which can be transformed into a mathematical programming problem. By means of the Lagrangian multiplier theorem and Riesz representation theorem, the main result given in this paper is the necessary condition for indefinite stochastic LQ control with quadratic and mixed terminal equality constraints. The result shows that the different terminal state constraints will cause the endpoint condition of the differential Riccati equation to be changed. It coincides with the indefinite stochastic LQ problem with linear terminal state constraint, so the result given in this paper can be viewed as the extension of the indefinite stochastic LQ problem with the linear terminal state equality constraint. In order to guarantee the existence and the uniqueness of the linear feedback control, a sufficient condition is also presented in the paper. A numerical example is presented at the end of the paper.

#### 1. Introduction

Linear quadratic (LQ) control is an extremely important class of control problems in both theory and application. It is pioneered by Kalman [1] for deterministic systems and was extended to stochastic systems by Wonham [2]. In recent years, extensive research has been carried out in the so-called indefinite stochastic LQ control, in which the cost weighting matrices are allowed to be indefinite; refer to [36] for detailed accounts. A basic assumption in the LQ theory, both for deterministic and stochastic cases, is that the variable is unconstrained except for the differential equations constraint. As far as we know, very few results for constrained deterministic LQ can be found compared with the unconstrained one, not to mention the stochastic LQ control [7]. While in many real applications, constrained LQ control problem (such as nonnegativity and bound constraints for state and control variables) is a well-posed problem, constrained stochastic LQ control problem has a concrete application background, but the conventional LQ approach would collapse in the presence of any constraints. Study on the constrained stochastic LQ control will contribute to both theory and application a lot.

Huang and Zhang [8] studied the indefinite stochastic LQ control problem with linear terminal state equality constraints. Necessary and sufficient conditions for indefinite stochastic LQ control problems were investigated based on the Lagrangian multiplier theorem and Riesz representation theorem. The result showed that the linear feedback optimal control can be obtained by solving systems of algebraic and differential equations. The previous results on unconstrained indefinite stochastic LQ can be viewed as a specified case of the main theorem in that paper.

This paper studied the indefinite stochastic LQ control problem with quadratic terminal equality constraints and mixed constraints, which can be viewed as the extension of [8]. When the terminal state constraint is quadratic, the feasible region defined by the terminal constraint is nonconvex and multiple local minima abound, which makes the problem more complex to locate the optimum consistently. Developing a deeper understanding of the problems, as well as efficient algorithms for solving them, will have a big impact in many applications. Another reason for the study of this problem is that the methods used for solving this type of problem can be used to solve more general constrained optimal control problems. By means of the Lagrangian multiplier theorem and Riesz representation theorem, the main result in this paper is the necessary condition for indefinite stochastic LQ control with quadratic terminal constraints and mixed terminal constraints. The result showed that the difference of the terminal state constraints will cause the endpoint condition to be changed in the differential equations we obtained for the linear constraint control problem, which coincides with the reality. In order to guarantee the existence and the uniqueness of the linear feedback control, a sufficient condition is also presented in the paper. Numerical example is presented at the end of this paper.

For the convenience, we make use of the following basic notation in this paper. is the -dimensional real column vector, is the -dimensional statement column vector, is the control input vector, is the one-dimensional Brownian motion defined on filtered probability with a standard -adapted on , and is an information fluid produced by Brownian motion. belongs to the admissible control set   is the adapted stochastic process which satisfies and the corresponding satisfies (3), is the given measurable, integrable random variable; that is, ,  and is a known number. ,, , , is row column matrix, and is an symmetric matrix. is the terminal state linear constraint coefficient matrix, and is the terminal state quadratic constraint matrix. Two assumptions are given in this paper.:, (the coefficient matrix for terminal linear and quadratic constraints respectively in Problem 1 to Problem 21) are full row rank and the set defined by the terminal state constraints is not empty;: is an -adapted, -valued measurable process on , and .

The rest of this paper is organized as follows. Section 2 gives the problem statement and some preliminaries, Section 3 presents the main results of this paper. Numerical example is given in Section 4. Finally, Section 5 concludes this paper.

#### 2. Problem Statement and Preliminaries

Problem 1. We study the following indefinite stochastic LQ control problem:

s.t.

Suppose that the feasible set defined by the quadratic constraint is not empty. We investigate the necessary condition under the linear feedback optimal control

in which . The cost weight matrix in Problem 1 is not necessarily positive, which usually causes Problem 1 to be called indefinite stochastic LQ control problem with terminal constraint. First, we present some definitions and lemmas that will be used and then transform the optimal control Problem 1 into a deterministic control problem.

Definition 2 (Gateaux differential). Let be a vector space, let be a normed space, and let be a (possibly nonlinear) transformation defined on a domain with a range . Given , and is an arbitrary vector in . If the limit exists, then it is called Gateaux differential at with an increment . If the limit exists for arbitrary , the transformation is called Gateaux differentiable at .

Definition 3 (Frechet differential). Let and be normed linear spaces and let be a transformation defined on an open domain with a range . If, for a fixed , there exist , which is linear and continuous with respect to such that then is said to be Frechet differentiable at , and is said to be the Frechet differential of at with an increment .

Definition 4 (Frechet derivative and continuously Frechet differentiable). Suppose that a transformation defined on an open domain is Frechet differentiable on . The Frechet differential for a fixed , where is a bounded linear operator from to , one calls the derivative of . If the derivative is continuous on some open ball , then is continuously Frechet differentiable on .

Definition 5 (regular point of transformation). Let be a transformation defined on a Banach space with a range , which is also a Banach space. is continuously Frechet differentiable if, for a given ,   is an onto mapping from onto ; then is called a regular point.
Considering Problem 1 under the linear feedback optimal control, substituting in Problem 1 with And taking the place of in (2), we have According to the Itô integrals formula for , in which is the solution of (8), we have Define ; it is obvious that is a symmetric matrix. Integrate both sides of (9) from 0 to with variable , then compute the derivatives of both sides of (9) after taking the expectation, then get the following matrix differential equation with the initial condition: Substituting with in (1) to (“” is the trace of a matrix, , ), the quadratic constraints are where , , and . Then Problem 1 can be transformed into the following deterministic optimal control Problem 6.

Problem 6. Consider s.t.

Remark 7. It is obvious that if Problem 1 has the optimal linear feedback control , then must be the solution of Problem 6, while the inverse does not hold.
Take the objective functional as a functional defined on the space , in which is an square matrix space, the element in this space is the continuous function defined on : and terminal state constraints (16) defines a transformation from to By the virtue of (17)-(18), constraints (14)–(16) can be reformulated as

Lemma 8. Functional , , are continuous Frechet differentiable functionals and have Frechet derivative

Proof. The proof is mainly based on Definition 2. Apply Definition 2 to the functional constraints (19); the proofs for the first three results are the same to those in [8], and we only need the proof of (24) here:

Lemma 9. Constraint (19) satisfies the regular condition; that is, are onto mapping when

Proof. The proof of the onto mapping for has been given in [8]; we only need to prove that is a onto mapping when varies. For a given , the following equation has a solution: Because the coefficient matrix for quadratic terminal state constraints is full row rank (Assumption ), equation exists a solution, which finishes the proof.

Definition 10 (the well-posedness). The LQ optimal control problem is to minimize the cost functional over . Define the optimal value function as . The LQ problem is called well posed if A well-posed problem is called attainable (with respect to ) if there is a control that achieves .

Theorem 11 (Lagrange multiplier theorem). If the continuously Frechet differentiable real functional defined on Banach space has a local extremum under the constraint at the regular point , is a mapping from space to Banach space , and then there exists an element such that the Lagrangian functional is stationary at ; that is, for each .

Theorem 12 (Riesz representation theorem). Let be a bounded linear functional on , and then there is a bounded variation function on such that, for all , where the norm of is the total variation on . Conversely, every function of bounded variation on defines a bounded linear function on in this way.

#### 3. Main Results

Lemma 13 (see [9]). If is continuous in and   for every (set of all continuously differentiable functions on with , then on .

Lemma 14 (see [9]). If is continuous in and for every with , then in , where is a constant.

Lemma 15 (see [9]). If and are continuous in and for every with , then is differentiable and in .

We make use of (nonnegative bounded variation functional on ) to express the matrix space with the element in . The space is a bounded variation right continuous function that function takes value 0 at the point at the point . Based on the Lagragian theorem and the Riesz representation theorem, we obtained a necessary condition for Problem 1.

##### 3.1. Necessary Condition

Theorem 16 (necessary condition for indefinite stochastic LQ with quadratic constraints). Suppose the set defined by the terminal state constraint is not empty and the optimal control Problem 6 exists an optimal feedback control matrix ; then there must exist a symmetric matrix and a vector such that for all , and satisfy the following equations:

Proof. From Remark 7, we know that is the solution of deterministic optimal control Problem 6; take and in Problem 6 as the variables. Suppose that is the optimal solution for Problem 6; according to Lemmas 14 and 15, Problem 6 satisfies the conditions of Lagrange multiplier theorem, and then there exists a symmetric matrix and such that The second parts of (38) and (39) are from Riesz representation theorem. In general, we take that , and then (38) becomes It is obvious that there is no jump in interval for function , otherwise we can choose that makes be far more than the other parts in the equality. But has jump at the point , and the height is Because the previous equalities (38) and (39) hold for all continuous functions , for specified function which has continuous derivative and , all the previous equalities also hold, and then Because , then and we have the following from (38) According to Lemma 15, we have We take the integral by parts of the second part of (39), and then Based on Lemma 13, we have Change the endpoint conditions with ; that is, (36) holds.

Remark 17. The necessary condition in Theorem 16 (35)–(37) and the constraints (14)–(16) in Problem 6 have one order differential equations, bound conditions, terminal state conditions, and algebra equations, and then we can determine ,  ,  , and , respectively.

Remark 18. The result in Theorem 12 is the same as that in [8] except for the terminal conditions.

##### 3.2. Sufficient Condition

We have pointed out that the necessary conditions (14)–(16) and (35)–(37) are not sufficient for the existence and the uniqueness of the solution in Problem 6. In order to guarantee the uniqueness of and , the conditions must be strengthened to that is, the matrix must be positive.

Theorem 19 (sufficient condition for stochastic LQ problem with quadratic constraints). If (14)–(16), (35)–(37), and (39) exist solutions , ,  , and , then Problem 6 is well posed and the optimal feedback control is the optimal cost value is

Proof. Suppose that , , and are the solutions that satisfy (14)–(16), (35)–(37), and (48), and from (37), we have Substitute in (35), and then Apply Itô’s formula again, so where the last equality is from (52). It is obvious that linear feedback control (49) has the minimum cost , and the cost value is

##### 3.3. Necessary Conditions for Stochastic LQ with Mixed Terminal State Constraint

We study the following indefinite stochastic LQ control problem with mixed constraints.

Problem 20. Consider s.t. Using the same method as the above, the stochastic LQ control problem with linear and quadratic terminal constraints can be transformed into the following Problem 21.

Problem 21. Consider s.t.

Theorem 22 (necessary and sufficient conditions for stochastic LQ problem with mixed constraint). Suppose that the terminal constraint set defined by the linear and quadratic state constraints is not empty and the optimal control Problem 6 exists the optimal feedback control matrix . Then there must exist a symmetric matrix and a vector , , for all , satisfing the following equalities:

If (14)–(16), (35)–(37), and (39) exist solutions ,  , , and , then Problem 6 is well posed and the optimal feedback control is The optimal value is

The proof of the theorem is the same as that of Theorem 16 except for the terminal condition.

#### 4. Numerical Examples

Let , ,  , ,  ,  , , , , and in Problem 6, according to (35)–(37) in Theorem 16 and the constraint (14)–(16) in Problem 1, and then Solve the system equations, and then The optimal control for Problem 6 is , and the optimal index cost value is

#### 5. Conclusion

This paper studied a class of indefinite stochastic LQ control problems with quadratic terminal state constraints and mixed terminal state constraints. By means of the Lagrange multiplier theorem and Riesz representation theorem, this paper presented a necessary condition for indefinite stochastic LQ control problems with quadratic terminal state constraints and mixed terminal state constraints. The result shows that the necessary condition for quadratic terminal constraints is the same as for the linear terminal state constraints that is presented in [8] except for the terminal condition. This coincides with the reality. A sufficient condition also was presented for the existence and uniqueness of the optimal linear feedback control. Numerical example verified the main theorem in this paper.

#### Acknowledgments

The author would like to thank Professor Kristian Ranestad, who gave the invitation to visit Mathematics Department of Oslo University. Further, the author is grateful to RCN, who financed the visit Grant 218993/F11/mm. This work is supported by the National Natural Science Foundation of China (11241005).

#### References

1. R. E. Kalman, “Contributions to the theory of optimal control,” Boletin de la Sociedad Matematica Mexicana, vol. 5, pp. 102–119, 1960.
2. W. M. Wonham, “On a matrix Riccati equation of stochastic control,” SIAM Journal on Control and Optimization, vol. 6, no. 2, pp. 312–326, 1968.
3. M. A. Rami and X. Y. Zhou, “Linear matrix inequalities, Riccati equations, and indefinite stochastic linear quadratic controls,” IEEE Transactions on Automatic Control, vol. 45, no. 6, pp. 1131–1143, 2000.
4. D. D. Yao, S. Zhang, and X. Y. Zhou, “Stochastic linear-quadratic control via semidefinite programming,” SIAM Journal on Control and Optimization, vol. 40, no. 3, pp. 801–823, 2001.
5. S. Chen, X. Li, and X. Y. Zhou, “Stochastic linear quadratic regulators with indefinite control weight costs,” SIAM Journal on Control and Optimization, vol. 36, no. 5, pp. 1685–1702, 1998.
6. S. Chen and J. Yong, “Stochastic linear quadratic optimal control problems,” Applied Mathematics and Optimization, vol. 43, no. 1, pp. 21–45, 2001.
7. A. E. B. Lim and X. Y. Zhou, “Stochastic optimal LQR control with integral quadratic constraints and indefinite control weights,” IEEE Transactions on Automatic Control, vol. 44, no. 7, pp. 1359–1369, 1999.
8. Y. L. Huang and W. H. Zhang, “Study on stochastic linear quadratic optimal control with constraint,” Acta Automatica Sinica, vol. 32, no. 2, pp. 246–254, 2006.
9. D. G. Luenberger, Optimization by Vector Space Methods, John Wiley & Sons, New York, NY, USA, 1968.