Mathematical Problems in Engineering

Volume 2014 (2014), Article ID 278142, 9 pages

http://dx.doi.org/10.1155/2014/278142

## Discrete-Time Indefinite Stochastic Linear Quadratic Optimal Control with Second Moment Constraints

^{1}College of Electrical Engineering and Automation, Shandong University of Science and Technology, Qingdao 266590, China^{2}College of Mathematics and Systems Science, Shandong University of Science and Technology, Qingdao 266590, China

Received 1 April 2014; Accepted 29 April 2014; Published 18 May 2014

Academic Editor: Xuejun Xie

Copyright © 2014 Weihai Zhang and Guiling Li. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

#### Abstract

This paper studies the discrete-time stochastic linear quadratic (LQ) problem with a second moment constraint on the terminal state, where the weighting matrices in the cost functional are allowed to be indefinite. By means of the matrix Lagrange theorem, a new class of generalized difference Riccati equations (GDREs) is introduced. It is shown that the well-posedness, and the attainability of the LQ problem and the solvability of the GDREs are equivalent to each other.

#### 1. Introduction

LQ control, initiated by Kalman [1] and extended to stochastic systems by Wonham [2], is one of the most important classes of optimal control issues from both theory and application point of view; we refer the reader to [2–8] for representative work in this area. Different from the classical LQ in modern control theory, it was found in [9, 10] that a stochastic LQ problem with indefinite control weighting matrices can still be well-posed, which evoked a series of subsequent researches; see, for example, [11, 12].

It is well known that in practical engineering, the system state and control input are always subject to various constraints, so how to solve the constrained stochastic LQ issue is a more attractive topic; we refer the reader to [13–19]. Reference [14] presented a tractable approach for LQ controller design of the system with additive noise. Reference [16] was about the constrained LQ of deterministic systems with state equality constraints. Reference [13] studied the parametrization of the solutions of finite-horizon constrained LQ control. Reference [15] was devoted to a stochastic LQ optimal control and an application to portfolio selection, where the control variable is confined to a cone, and all the coefficients of the state equation are random processes. Reference [19] studied the indefinite stochastic LQ control problem of continuous-time Itô systems with a linear equality constraint on the terminal state and gave a necessary condition for the existence of an optimal controller. Reference [20] generalized the results of [19] to discrete-time stochastic systems.

In this paper, different from [19, 20] on the constraint conditions, we would like to deal with stochastic LQ control of discrete-time multiplicative noise systems with a second moment constraint and such constraints are often encountered in filtering design; see [21, 22]. By means of Lagrange theorem, we present a necessary condition for the existence of an optimal linear state feedback control with the second moment constraint on the terminal states. It is proved that the solvability of GDRE is necessary and sufficient for the existence of an optimal control under either of the state feedback case or of the open-loop forms. Moreover, we show that the well-posedness and the attainability of the constrained LQ problem, the feasibility of the LMI, and the solvability of the GDRE are equivalent to each other. The novel contribution of this paper is to consider a constrained discrete-time LQ optimal stochastic control, which includes some results of [23] as special cases. A new class of generalized difference Riccati equations (GDREs) is first introduced.

The remainder of the paper is organized as follows. Section 2 gives some definitions and preliminaries. In Section 3, the optimal state feedback control is studied using the matrix Lagrange theorem. We give a necessary and sufficient condition for the well-posedness of the constrained LQ control in Section 4. Section 5 shows the equivalence among the well-posedness and the attainability of the LQ problem, the feasibility of the LMI, and the solvability of the GDRE. The set of all optimal controls is determined. We conclude the paper in Section 6.

Throughout the paper, the following notations are adopted: denotes the transpose of . : is a positive definite (positive semidefinite) symmetric matrix. : the trace of a square matrix . : the space of all matrices. : the space of all symmetric matrices.

#### 2. Problem Setting

Consider the following constrained discrete-time stochastic LQ control problem.

*Problem 1. *Consider
where the state , the control input , and the noise , ,
The process is a sequence of second-order stationary random variables defined on a complete probability space . Without loss of generality, we assume that
where is the Kronecker delta, . is a constant, , and are matrices having appropriate dimensions determined from context, and and are real symmetric indefinite matrices. is a given deterministic vector.

*Definition 2. *Problem 1 is called well-posed, if ,

*Definition 3. *Problem 1 is called attainable, if , there exists a sequence , such that . In this case, is called an optimal control sequence.

Now, let us consider a mathematical programming (MP) problem in a matrix space:

*Definition 4. *Let be a point satisfying
and then is said to be a constraint* regular point* if the gradient vectors , are linearly independent.

Lemma 5 (Lagrange theorem [24]). *Assume that the functions , , are twice continuously differentiable. If a regular point is also a relative minimum point for the original MP, then there exists a vector such that
**
where the Lagrangian function .*

#### 3. A Necessary Condition for State Feedback Control

In this section, by the matrix Lagrange theorem, we present a necessary condition for Problem 1 based on a new type of GDREs.

Let . Through a simple calculation, the following deterministic optimal control Problem 6 is equivalent to the original Problem 1 under the state feedback for .

*Problem 6. *Consider
with

*Remark 7. *If Problem 1 has a linear feedback optimal control solution , , then , , are the optimal solution of Problem 6.

Theorem 8. *If Problem 1 is attainable by , and the regular point is a locally optimal solution of Problem 6, then there exist symmetric matrices , and solving the following GDRE:
**
where is the Moore-Penrose generalized inverse of . Moreover,
**
with , , being any given real matrices:
*

*To prove Theorem 8, we mainly use Lemma 5 to Problem 6 together with the following lemma to obtain GDRE (10) and then apply the technique of completing squares to show (12).*

*Lemma 9 (see [12]). Let be given matrices with appropriate sizes; then the matrix equation
has a solution if and only if
Moreover, any solution to can be represented by
where is any matrix with appropriate size.*

* Proof. *According to Remark 7, is also the optimal solution of Problem 6. Problem 6 is a typical MP problem about and as follows:
where
Let matrices , , be the Lagrangian multipliers of
and let be the Lagrangian multiplier of ; then the Lagrangian function
According to the the matrix Lagrange theorem, we obtain
Based on the partial rule of gradient matrices, (20) can be transformed into
Let
Then we obtain
Applying Lemma 9, we have
Equation (21) yields
Substituting into (26), it follows
Without loss of generality, we can assume that is symmetric. Otherwise, we can take . The objective functional
A completion of square implies
We assert that must satisfy
If it is not so, there is for with a negative eigenvalue . Denote the unitary eigenvector with respect to by . Let be an arbitrary scalar; we construct a control sequence as follows:
The associated cost functional becomes
Let ; then , which contradicts the attainability of Problem 1. So (30) holds.

In view of (29) and (30), (11) and (12) are easily derived. The proof is completed.

*Remark 10. *In Theorem 8, in order to apply matrix Lagrange theorem, we assume the optimal solution is a regular point. Generally speaking, for a given LQ control, it is easy to examine the regular condition.

Below, we present a numerical example to illustrate the effectiveness of Theorem 8.

*Example 11. *In Problem 1, we set
The state and control weighting matrices are as
By the relationship between Problems 1 and 6, we know
Applying Theorem 8, we obtain
*Stage 2. *Consider
*Stage 1. *Consider
The optimal cost value of Problem 1 is
We are able to test the regular condition of as follows. In Problem 6,
which is linear about and quadratic about , while is linear about . By simple calculations, , , and are all nonzero vectors and hence are linearly independent.

*4. Well-Posedness*

*In this section, we first establish the link between the well-posedness of Problem 1 and the feasibility of some LMIs and then prove that the solvability of GDRE (10) is not only necessary but also sufficient to the well-posedness of Problem 1. Moreover, the well-posedness and the attainability of Problem 1, the feasibility of some LMIs, and the solvability of GDRE (10) are equivalent to each other.*

*Theorem 12. Problem 1 is well-posed if there exist symmetric matrices , and solving the following LMIs:
where
*

*Proof. *Note that
By (41), it is easy to deduce that the cost functional is bounded from below by
Hence, Problem 1 is well-posed.

*Remark 13. *Theorem 12 tells us that any symmetric matrices , and satisfying LMIs (41)-(42) provide a lower bound
for the cost function. In what follows, we will show that this lower bound is an exact optimal cost value if and solve GDRE (10).

We have shown that if the LMIs (41)-(42) are satisfied, then the constrained LQ Problem 1 is well-posed. Below, we further show some other equivalent conditions.

*Lemma 14 (extended Schur’s lemma [25]). Let the matrices , , be given with appropriate sizes. Then, the following three conditions are equivalent:
*

*Theorem 15. Problem 1 is well-posed if and only if there exist symmetric matrices , and satisfying GDRE (10). Furthermore, the optimal cost is
*

A key to prove Theorem 15 is the necessity part, where the stochastic optimization principle is used.

*Proof. **Necessity.* For , define
By the stochastic optimization principle, when is finite, then so is for any . Since Problem 1 is assumed to be well-posed at , is finite at any stage . Now let us start with , and let , and we have
Since is finite, using Lemma 4.3 of [23], there exists a symmetric matrix such that
The obtained solution sequence of symmetric matrices , , and to GDRE (10) satisfy
Then by the stochastic optimality principle, the following holds:
Lemma 4.3 of [23] provides necessary and sufficient conditions for the finiteness of :
Moreover,
The above proves the necessity part via mathematical induction.*Sufficiency.* From the proof of Theorem 8, if GDRE (10) admits a solution and , Problem 1 is not only well-posed, but also attainable. The proof of this theorem is complete.

*5. Other Equivalent Conditions*

*5. Other Equivalent Conditions*

*In this section, we present some other equivalent conditions for Problem 1.*

*Theorem 16. For the constrained LQ Problem 1, the following are equivalent:(i)Problem 1 is well-posed.(ii)Problem 1 is attainable.(iii)The LMIs (41)-(42) are feasible.(iv)The GDRE (10) is solvable.*

Furthermore, when any one of the above conditions is satisfied, Problem 1 is attainable by where , are solutions to the GDRE (10).

*Proof. *Applying Theorems 12–15, . is shown by Theorem 8. The rest is to prove and (56). Let , solve the GDRE (10). In view of
a completion of squares yields
which shows

Finally, we present a general expression for the optimal control set based on the solution to GDRE (10).

*Theorem 17. Assume that the GDRE (10) admits a solution. Then the set of all optimal controls is determined by
where and are arbitrary random variables defined on the probability space . Moreover, the optimal cost value is uniquely given by
where , and are the solution to the GDRE (10).*

*Proof. *This theorem can be proved by repeating the same procedure as in Theorem 5.1 of [23].

*6. Conclusion*

*6. Conclusion*

*In this paper, we have investigated a class of indefinite stochastic LQ control problems with second moment constraints on the terminal state. By the matrix Lagrange theorem, we have established a new GDRE (10) associated with the constrained optimization Problem 1. In addition, by introducing LMIs (41)-(42), we show that the well-posedness and the attainability of Problem 1, the feasibility of the LMIs (41)-(42), and the solvability of GDRE (10) are equivalent to each other.*

*Conflict of Interests*

*Conflict of Interests*

*The authors declare that there is no conflict of interests regarding the publication of this paper.*

*Acknowledgments*

*Acknowledgments*

*This work is supported by the National Natural Science Foundation of China (no. 61174078), the Research Fund for the Taishan Scholar Project of Shandong Province of China and SDUST Research Fund (no. 2011KYTD105), and State Key Laboratory of Alternate Electrical Power System with Renewable Energy Sources (Grant no. LAPS13018).*

*References*

*References*

- R. E. Kalman, “Contributions to the theory of optimal control,”
*Boletín de la Sociedad Matemática Mexicana*, vol. 5, no. 2, pp. 102–119, 1960. View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet - W. M. Wonham, “On a matrix Riccati equation of stochastic control,”
*SIAM Journal on Control and Optimization*, vol. 6, no. 4, pp. 681–697, 1968. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet - C. E. de Souza and M. D. Fragoso, “On the existence of maximal solution for generalized algebraic Riccati equations arising in stochastic control,”
*Systems and Control Letters*, vol. 14, no. 3, pp. 233–239, 1990. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet - B. S. Chen and W. Zhang, “Stochastic ${H}_{2}/{H}_{\infty}$ control with state-dependent noise,”
*IEEE Transactions on Automatic Control*, vol. 49, no. 1, pp. 45–57, 2004. View at Publisher · View at Google Scholar · View at MathSciNet - W. Zhang and B. S. Chen, “On stabilizability and exact observability of stochastic systems with their applications,”
*Automatica*, vol. 40, no. 1, pp. 87–94, 2004. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet - B. D. O. Anderson and J. B. Moore,
*Optimal Control Linear Quadratic Mathods*, Prentice-Hall, New York, NY, USA, 1989. - F. L. Lewis,
*Optimal Control*, John Wiley and Sons, New York, NY, USA, 1986. - Y. Huang, W. Zhang, and H. Zhang, “Infinite horizon linear quadratic optimal control for discrete-time stochastic systems,”
*Asian Journal of Control*, vol. 10, no. 5, pp. 608–615, 2008. View at Publisher · View at Google Scholar · View at MathSciNet - S. Chen, X. J. Li, and X. Y. Zhou, “Stochastic linear quadratic regulators with indefinite control weight costs,”
*SIAM Journal on Control and Optimization*, vol. 36, no. 5, pp. 1685–1702, 1998. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet - M. A. Rami and X. Y. Zhou, “Linear matrix inequalities, Riccati equations, and indefinite stochastic linear quadratic controls,”
*IEEE Transactions on Automatic Control*, vol. 45, no. 6, pp. 1131–1143, 2000. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet - D. D. Yao, S. Zhang, and X. Y. Zhou, “Stochastic linear-quadratic control via semidefinite programming,”
*SIAM Journal on Control and Optimization*, vol. 40, no. 3, pp. 801–823, 2001. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet - M. Ait Rami, J. B. Moore, and X. Y. Zhou, “Indefinite stochastic linear quadratic control and generalized differential Riccati equation,”
*SIAM Journal on Control and Optimization*, vol. 40, no. 4, pp. 1296–1311, 2001. View at Publisher · View at Google Scholar · View at MathSciNet - A. Ferrante, G. Marro, and L. Ntogramatzidis, “A parametrization of the solutions of the finite-horizon LQ problem with general cost and boundary conditions,”
*Automatica*, vol. 41, no. 8, pp. 1359–1366, 2005. View at Publisher · View at Google Scholar · View at MathSciNet - D. Bertsimas and D. B. Brown, “Constrained stochastic LQC: a tractable approach,”
*IEEE Transactions on Automatic Control*, vol. 52, no. 10, pp. 1826–1841, 2007. View at Publisher · View at Google Scholar · View at MathSciNet - Y. Hu and X. Y. Zhou, “Constrained stochastic LQ control with random coefficients, and application to portfolio selection,”
*SIAM Journal on Control and Optimization*, vol. 44, no. 2, pp. 444–466, 2005. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet - S. Ko and R. R. Bitmead, “Optimal control for linear systems with state equality constraints,”
*Automatica*, vol. 43, no. 9, pp. 1573–1582, 2007. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet - A. Kojima and M. Morari, “LQ control for constrained continuous-time systems,”
*Automatica*, vol. 40, no. 7, pp. 1143–1155, 2004. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet - Z. Zhou and C. Randy, “An algorithm for state constrained stochastic linear-quadratic control,” in
*Proceedings of the American Control Conference*, pp. 1476–1481, San Francisco, Calif, USA, June 2011. - Y. L. Huang and W. H. Zhang, “Study on stochastic linear quadratic optimal control with constraint,”
*Acta Automatica Sinica*, vol. 32, no. 2, pp. 246–254, 2006 (Chinese). View at Google Scholar · View at MathSciNet - G. Li and W. Zhang, “Discrete-time indefinite stochastic linear quadratic optimal control with equality constraints,” in
*Proceeding of the Chinese Control and Decision Conference*, pp. 4999–5004, Guiyang, China, May 2013. - H. Dong, Z. Wang, D. W. C. Ho, and H. Gao, “Variance-constrained ${H}_{\infty}$ filtering for a class of nonlinear time-varying systems with multiple missing measurements: the finite-horizon case,”
*IEEE Transactions on Signal Processing*, vol. 58, no. 5, pp. 2534–2543, 2010. View at Publisher · View at Google Scholar · View at MathSciNet - L. Ma, Y. Bo, Y. Zhou, and Z. Guo, “Error variance-constrained ${H}_{\infty}$ filtering for a class of nonlinear stochastic systems with degraded measurements: the finite horizon case,”
*International Journal of Systems Science*, vol. 43, no. 12, pp. 2361–2372, 2012. View at Publisher · View at Google Scholar · View at MathSciNet - M. Ait Rami, X. Chen, and X. Y. Zhou, “Discrete-time indefinite LQ control with state and control dependent noises,”
*Journal of Global Optimization*, vol. 23, no. 3-4, pp. 245–265, 2002. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet - D. P. Bertsekas,
*Constrained Optimization and Lagrange Multiplier Methods*, Athena Scientific, Belmont, Mass, USA, 1996. - A. Albert, “Conditions for positive and nonnegative definiteness in terms of pseudoinverses,”
*SIAM Journal on Applied Mathematics*, vol. 17, no. 2, pp. 434–440, 1969. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet

*
*