Journal of Applied Mathematics

Volume 2013 (2013), Article ID 805829, 9 pages

http://dx.doi.org/10.1155/2013/805829

## Study on Indefinite Stochastic Linear Quadratic Optimal Control with Inequality Constraint

^{1}College of Information Science and Engineering, Shandong University of Science and Technology, Qingdao 266591, China^{2}College of Information and Electrical Engineering, Shandong University of Science and Technology, Qingdao 266591, China

Received 1 April 2013; Revised 12 September 2013; Accepted 2 October 2013

Academic Editor: Lotfollah Najjar

Copyright © 2013 Guiling Li and Weihai Zhang. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

#### Abstract

This paper studies the indefinite stochastic linear quadratic (LQ) optimal control problem with an inequality constraint for the terminal state. Firstly, we prove a generalized Karush-Kuhn-Tucker (KKT) theorem under hybrid constraints. Secondly, a new type of generalized Riccati equations is obtained, based on which a necessary condition (it is also a sufficient condition under stronger assumptions) for the existence of an optimal linear state feedback control is given by means of KKT theorem. Finally, we design a dynamic programming algorithm to solve the constrained indefinite stochastic LQ issue.

#### 1. Introduction

The study on LQ control problems can be traced back to the pioneering work of Kalman [1] and Wonham [2] several decades ago. The LQ control theory is elegantly established and developed, and the main work can be seen in [3–11]. In particular, it is found [6] that a stochastic LQ problem with indefinite control weighting matrices may still be well-posed. This discovery evokes a series of subsequent researches, and many important achievements are obtained [5, 12–16]. Up to now, most work deals with the indefinite stochastic LQ problems without constraints. However, as a practical optimization problem, the indefinite stochastic LQ problem unavoidably has various constraints on the state or control; in particulary the inequality constraints often appear.

For the constrained indefinite stochastic LQ problems, [17] studied the equally constrained stochastic LQ optimization for Itô systems. In this paper, we will study the stochastic LQ problem with inequality constraint.

Firstly, we present and prove the generalized KKT theorem under hybrid constraints. Secondly, a necessary condition for the existence of an optimal linear state feedback control is given by means of the generalized KKT theorem. Thirdly, if we strengthen the condition, we can obtain a necessary and sufficient condition for the existence of the optimal linear feedback control to indefinite stochastic LQ optimal control problem with inequality constraint. Finally, we give a dynamic programming algorithm to solve the stochastic LQ problem with the inequality constraint. We provide an example to demonstrate the effectiveness of our main theoretical results.

The outline of this paper is organized as follows. In Section 2, we present a generalized KKT theorem under hybrid constraints. Section 3 proposes a KKT condition for the existence of an optimal linear state feedback control. In Section 4, we provide a necessary and sufficient condition and a dynamic programming algorithm for the stochastic LQ problem with inequality constraint. Section 5 concludes the paper.

For convenience, throughout the paper, we adopt the following notations: denotes the transpose of a matrix . : is a positive definite (positive semidefinite) symmetric matrix. : trace of a square matrix. : the space of all real matrices. : a symmetric matrix space.

#### 2. Preliminaries

Consider the following indefinite stochastic LQ control.

*Problem 1. *Considerwhere is an -dimensional state variable, is a control input, is a one-dimensional standard Brownian motion defined on a filtered probability space (). We denote the information flow . belongs to , where is a space of all -valued, -adapted measurable processes satisfying . For each admissible control, the corresponding trajectory satisfies the constraint (1c). in constraint (1c) is a given nonnegative constant. , , , , , and are time-varying matrices of suitable dimensions. and in objective functional are symmetric matrices. To study the issue, we first put forward the following Assumption .*Assumption *. *, *, , and , where -valued essential bounded measurable function and ess .

In this paper, the weighting matrices in the objective functional are not required to be definite. Therefore Problem 1 is an indefinite stochastic LQ optimal control problem. For later use, we recall KKT theorem for this type of mathematical programming (MP) problems:
where , and .

The KKT conditions [18–20], which are also known as the Kuhn-Tucker (KT) conditions, are the first-order necessary conditions for a solution in nonlinear programming to be optimal, provided that some regularity conditions are satisfied. The Lagrange multiplier method, which allows only equality constraints, can be viewed as a special case of KKT conditions.*Regularity Condition (or Constraint Qualification).* In MP above, let be active constraints at . The gradient vectors , and , are linearly independent, is known as a linear independent constraint qualification (LICQ). *Regular Point.* In MP above, is said to be a regular point of the constraints if the gradient vectors , are linearly independent.*KKT Theorem.* In MP above, we assume that the functions are twice continuously differentiable and we assume that all the constraints satisfy the regularity condition LICQ. Let be a point satisfying all the constraints and let be a regular point of the above constraints. Now suppose that this regular point is also a relative minimum point for the original MP. Then it is shown that there exist a vector and a vector , such that
where is the Lagrangian function and , are complementary clackness condition.

It is particularly important to check the regularity condition before we apply the conclusion of KKT theorem. If it is not so, the conclusion of KKT theorem would not be valid, just as the following example shows.

*Example 2. *Consider
Obviously, the minimum point is . According to KKT theorem, we obtain
The conclusion of KKT theorem does not hold at point , because is not linearly independent. It does not satisfy the LICQ regularity condition.

In order for a minimum point to satisfy the above KKT conditions, it should satisfy some regularity conditions. Except for LICQ regularity condition, the most used ones are listed below.*Constant Rank Constraint Qualification.* For each subset of the gradients of the active inequality constraints and the gradients of the equality constraints the rank at a vicinity of is constant.*Mangasarian-Fromovitz Constraint Qualification.* The gradients of the active inequality constraints and the gradients of the equality constraints are linear independent at .*Constant Positive Linear Dependence Constraint Qualification.* For each subset of the gradients of the active inequality constraints and the gradients of the equality constraints, if it is positive-linear dependent at , then it is positive-linear dependent at a vicinity of .

The Slater condition for a convex MP is also a common regularity condition.

*Remark 3. *In this paper, for convenience, when we use the KKT theorem, we always assume that the local optimal meets the LICQ regularity condition. The same goes for other regularity conditions.

*Definition 4 (see [21]). *Let be a vector space, a normed space, and a transformation from to . If the limit
exists, it is called the Gateaux differential of at with increment . If the limit exists for each , the transformation is said to be Gateaux differentiable at .

*Definition 5 (see [21]). *Let be a vector space and a Banach space with a positive cone having nonempty interior. Let be a mapping from to which has a Gateaux differential that is linear in its increment. A point is said to be a regular point of the inequality , if and there is an such that .

*Definition 6 (see [21]). *Let be a vector space and a Banach space. Let be a mapping from to which has a Gateaux differential that is linear in its increment. A point is said to be a regular point of the equality , if are linearly independent.

On the basis of the definitions above, let us discuss the KKT theorem in Banach space, where the objective function and the constraint functions in MP are functionals.

Let us consider
As a special case,
has the local necessary condition as follows.

Lemma 7 (see [21] (generalized KKT theorem)). *Let be a vector space and a Banach space having positive cone . Assume that contains an interior point. Let be a Gateaux differentiable functional on and a Gateaux differentiable mapping from to . Assume that the Gateaux differentials are linear in their increments. Suppose that minimizes subject to and that is a regular point of the inequality . Then there is a in , such that the Lagrangian function is stationary at . Furthermore .**The following theorem is the local necessary condition of .*

Theorem 8. *Let be a vector space and a Banach space having positive cone . Assume that contains an interior point. Let be a Gateaux differentiable functional on . Let and be Gateaux differentiable mappings from to . Assume that the Gateaux differentials are linear in their increments. Suppose that minimizes subject to , and that is a regular point of , . Then there is a in , , such that the Lagrangian function is stationary at . Namely, . Furthermore, .*

*Proof. * is equivalent to and . If is a regular point of , then are linearly independent. So are all nonzero, because the Gateaux differentials are linear in their increments. Using Definition 5, it is easy to verify that is a regular point of both and . According to Lemma 7, we know that the multiplier of equality has no nonnegative requirement.

*Definition 9 (see [22]). *Suppose that is a scalar-valued function of the elements of . Then the gradient matrix of is defined as
with
Based on Definition 9, we can easily extend KKT theorem from Banach space to matrix space. Because can be treated as a vector , one can work out and the KKT theorem holds.

When we apply the matrix KKT theorem, we need to give the partial list of gradient matrices [22] that we will use in this paper. In the following formulas, is an matrix. The formulas are not valid if the elements of are not independent. are assumed to have appropriate dimensions determined from context.

Consider the following:

#### 3. KKT Conditions and a New Type of GDREs

*Definition 10. *Problem 1 is well-posed, if for any , . is called an optimal control, if , and denotes the corresponding optimal trajectory.

Let the control law be , and . By substituting into (1a) of Problem 1, we obtain the new objective functional:
where is the space of -order square matrix whose elements are continuous functions. By substituting into (1b) of Problem 1, we obtain a closed-loop system:
By applying Itô’s formula to , we obtain
Define the transformation from to :
By substituting into (1c) of Problem 1, we obtain
Define the transformation from to :
So the original stochastic Problem 1 can be transformed into the deterministic Problem 11 as follows.

*Problem 11. *Consider the following:

Lemma 12. *, , and have continuous Gateaux derivative as follows:
*

*Proof. *We prove only the most complicated one. The rest can be verified in the same way. From Definition 4,
Replace the in with () and then let showing the conclusion.

Lemma 13 (see [21]). *If and are continuous in and for every continuously differentiable with , then is differentiable and in .*

Lemma 14 (see [21]). *If is continuous in and for every continuously differentiable with , then on .*

Theorem 15. *Assume that is the optimal solution of Problem 1, and then there exist a symmetric matrix and a nonnegative , such that **where is a matrix space whose elements are bounded functions in with value at point 0 and right continuous at .*

*Proof. * is also the optimal solution of Problem 11. Problem 11 is the type of . Assume that the optimal solution to Problem 11 is . Using Theorem 8 and Lemma 12, there exist a symmetric matrix and a nonnegative , such that
For all , (20)-(21) are established. According to Riesz representation theorem, we obtain the second item of (20) and the same of (21).

Without loss of generality, let . Integrate (20) by parts yielding
Clearly, has no jump on . But has a jump at , and the value is . Because the above results are established for all continuous , then
Thus, (20) becomes
From Lemma 13, is differential in and (19a) is obtained.

In the same way, integrating (21) by parts, we obtain
From Lemma 14, (19c) is obtained.

To ensure the continuity of , replace with (i.e., (19b)).

Equation (19d) is called complementary slackness conditions.

*Remark 16. *Equations (16b)-(16c) of Problem 11 and (19a)–(19c) of Theorem 15 are -dimensional, first-order differential equations including terminal conditions and algebraic equations. Equation (19d) is called a complementary slackness condition. By using these conditions, , , , and are obtained.

*Remark 17. *As for the complementary slackness condition, if the inequality constraint of Problem 11 is strict, then , and the problem becomes easier. If the inequality constraint of Problem 11 is an equality constraint, it simplifies Theorem 15 as Lagrange multiplier method.

*Definition 18. * is a given matrix. One calls the Moore-Penrose generalized inverse of , if
Based on Definition 18, we can rewrite Theorem 15 by expressing in terms of .

Lemma 19 (see [13]). *Let matrices , , and be given with appropriate sizes. Then the matrix equation
**
has a solution if and only if
**
Moreover, any solution to is represented by
**
where is a matrix with an appropriate size.*

Theorem 20. *If is optimal solution of Problem 1, then there exist a unique and a nonnegative , such that
**
where .*

*Proof. *Form (19c) in Theorem 15, we obtain
According to Lemma 19,

where .

As a special case, let us consider the following discrete stochastic LQ control problem without inequality constraint.

*Problem 21. *Consider the following.

Corollary 22. *If is optimal solution of Problem 21, then there exists a unique to the following constrained GDRE [13]
**
where .*

*Proof. *Because of Problem 21 without the equality constraint, , therefore . Theorem 20 yields this corollary directly.

#### 4. Application

##### 4.1. A Necessary and Sufficient Condition

In Theorem 15, to ensure the uniqueness of and , let us strengthen the condition as .

Theorem 23. *If is optimal solution of Problem 1, then and , such that
**
and if hold, then Problem 1 is well posed. The optimal solutions of Problem 1 are
*

*Proof. *In Remark 3, in order for a minimum point of a mathematical programming to satisfy the KKT conditions, we have assumed that satisfies LICQ regularity condition. From (19c) of Theorem 15, we obtain
Because , Problem 11 has a unique solution; namely,
From the relationship between Problems 1 and 11 and by using Itô’s formula, we have

##### 4.2. A Dynamic Programming Algorithm

A method for solving the indefinite stochastic linear quadratic (LQ) optimal control problem with unequal terminal state constraint is proposed as follows.

Reconsider the following LQ Problem 1′

Then the backward dynamic programming optimality principle [8] yields the following theorem.

Theorem 24. *Let and be an optimal solution of Problem 1′, then , and satisfy the following Bellman equation:
*

*Proof. *By applying the backward dynamic programming optimality principle, we obtain recursive relationships between and as follows:
where is the objective of and of . Let be the objective of and then
Combine the terminal state constraint
thus we can solve the above recursive relations. Furthermore, let generating the dynamic programming algorithm.

*Remark 25. *The dynamic programming algorithm can be applied to solve the stochastic LQ problem with inequality constraint except for a heavy computation. Nonetheless, it makes more sense in theory.

*Example 26. *In Problem 1, let
By Theorem 15, we have
The constraints of Problem 11 are
Via a series of computations, we obtain
In Problem 11, the inequality constraint is
When ,
Hence
This shows that the optimal solution of Problem 11 satisfies the inequality constraint. In this case, the inequality constraint is a nonactive constraint. Because , using Theorem 23, the optimal control of Problem 1 is
and the optimal value is

#### 5. Conclusion

We have studied the indefinite stochastic LQ optimal control problem with unequal terminal state constraint, which can be transformed into a hybrid constrained mathematical programming problem. By applying KKT theorem, we have presented a necessary condition for the constrained indefinite stochastic LQ optimal control problem. By adding some conditions, we obtain a necessary and sufficient condition for indefinite stochastic LQ optimal control problem with inequality constraint. We demonstrate that the solvability of the generalized Riccati equation is sufficient for the well posedness of the indefinite LQ problem. Since this kind of LQ control problems can be transformed into a hybrid constrained mathematical programming, we have given a dynamic programming algorithm.

#### Acknowledgments

This work is supported by the National Natural Science Foundation of China (61174078, 61170183), the Research Fund for the Taishan Scholar Project of Shandong Province of China, the SDUST Research Fund (no. 2011KYTD105), and the State Key Laboratory of Alternate Electrical Power System with Renewable Energy Sources (no. LAPS13018).

#### References

- R. E. Kalman, “Contributions to the theory of optimal control,”
*Boletín de la Sociedad Matemática Mexicana*, vol. 5, no. 2, pp. 102–119, 1960. View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet - W. M. Wonham, “On a matrix Riccati equation of stochastic control,”
*SIAM Journal on Control and Optimization*, vol. 6, pp. 681–697, 1968. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet - C. E. de Souza and M. D. Fragoso, “On the existence of maximal solution for generalized algebraic Riccati equations arising in stochastic control,”
*Systems & Control Letters*, vol. 14, no. 3, pp. 233–239, 1990. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet - W. Zhang, “Study on generalized algebraic Riccati equation and optimal regulators,”
*Journal of Control Theory and Applications*, vol. 20, no. 4, pp. 637–640, 2003. View at Google Scholar - S. Chen, X. Li, and X. Y. Zhou, “Stochastic linear quadratic regulators with indefinite control weight costs,”
*SIAM Journal on Control and Optimization*, vol. 45, no. 6, pp. 1131–1143, 2000. View at Google Scholar - M. A. Rami and X. Y. Zhou, “Linear matrix inequalities, Riccati equations, and indefinite stochastic linear quadratic controls,”
*IEEE Transactions on Automatic Control*, vol. 45, no. 6, pp. 1131–1143, 2000. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet - C. E. de Souza and M. D. Fragoso, “On the existence of maximal solution for generalized algebraic Riccati equations arising in stochastic control,”
*Systems & Control Letters*, vol. 14, no. 3, pp. 233–239, 1990. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet - R. Bellman, “On the theory of dynamic programming,”
*Proceedings of the National Academy of Sciences of the United States of America*, vol. 38, pp. 716–719, 1952. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet - D. D. Yao, S. Zhang, and X. Y. Zhou, “Stochastic linear-quadratic control via semidefinite programming,”
*SIAM Journal on Control and Optimization*, vol. 40, no. 3, p. 801–823 (electronic), 2001. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet - W. Zhang and B.-S. Chen, “On stabilizability and exact observability of stochastic systems with their applications,”
*Automatica*, vol. 40, no. 1, pp. 87–94, 2004. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet - A. Beghi and D. D'Alessandro, “Discrete-time optimal control with control-dependent noise and generalized Riccati difference equations,”
*Automatica*, vol. 34, no. 8, pp. 1031–1034, 1998. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet - D. D. Yao, S. Zhang, and X. Y. Zhou, “Stochastic linear-quadratic control via semidefinite programming,”
*SIAM Journal on Control and Optimization*, vol. 40, no. 3, pp. 801–823, 2001. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet - M. Ait Rami, J. B. Moore, and X. Y. Zhou, “Indefinite stochastic linear quadratic control and generalized differential Riccati equation,”
*SIAM Journal on Control and Optimization*, vol. 40, no. 4, pp. 1296–1311, 2001. View at Publisher · View at Google Scholar · View at MathSciNet - M. Ait Rami, X. Chen, and X. Y. Zhou, “Discrete-time indefinite LQ control with state and control dependent noises,”
*Journal of Global Optimization*, vol. 23, no. 3-4, pp. 245–265, 2002. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet - Y. Huang, W. Zhang, and H. Zhang, “Infinite horizon linear quadratic optimal control for discrete-time stochastic systems,”
*Asian Journal of Control*, vol. 10, no. 5, pp. 608–615, 2008. View at Publisher · View at Google Scholar · View at MathSciNet - G. Wang and Z. Wu, “Stochastic linear quadratic optimal control with partial information and its application to mean-variance hedging problems,” in
*Proceedings of the 27th Chinese Control Conference (CCC '08)*, pp. 601–605, Yunnan, China, July 2008. View at Publisher · View at Google Scholar · View at Scopus - Y. L. Huang and W. H. Zhang, “Study on stochastic linear quadratic optimal control with constraint,”
*Acta Automatica Sinica*, vol. 32, no. 2, pp. 246–254, 2006. View at Google Scholar · View at MathSciNet - H. W. Kuhn and A. W. Tucker, “Nonlinear programming,” in
*Proceedings of the 2nd Berkeley Symposium on Mathematical Statistics and Probability*, pp. 481–492, Berkeley University, California Press, 1951. View at MathSciNet - W. Karush,
*Minima of functions of several variables with inequalities as side constraints [M.S. thesis]*, Department of Mathematics, University of Chicago, Chicago, Ill, USA, 1939. - T. H. Kjeldsen, “A contextualized historical analysis of the Kuhn-Tucker theorem in nonlinear programming: the impact of World War II,”
*Historia Mathematica*, vol. 27, no. 4, pp. 331–361, 2000. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet - D. G. Luenberger,
*Optimization by Vector Space Methods*, John Wiley & Sons, New York, NY, USA, 1968. View at MathSciNet - M. Athans, “The matrix minimum principle,”
*Information and Computation*, vol. 11, pp. 592–606, 1967. View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet