Mathematical Problems in Engineering

Volume 2012, Article ID 674087, 14 pages

http://dx.doi.org/10.1155/2012/674087

## Indefinite LQ Control for Discrete-Time Stochastic Systems via Semidefinite Programming

College of Information and Electrical Engineering, Shandong University of Science and Technology, Qingdao 266510, China

Received 19 October 2011; Accepted 29 November 2011

Academic Editor: Xue-Jun Xie

Copyright © 2012 Shaowei Zhou and Weihai Zhang. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

#### Abstract

This paper is concerned with a discrete-time indefinite stochastic LQ problem in an infinite-time horizon. A generalized stochastic algebraic Riccati equation (GSARE) that involves the Moore-Penrose inverse of a matrix and a positive semidefinite constraint is introduced. We mainly use a semidefinite-programming- (SDP-) based approach to study corresponding problems. Several relations among SDP complementary duality, the GSARE, and the optimality of LQ problem are established.

#### 1. Introduction

Stochastic linear quadratic (LQ) control problem was pioneered by Wonham [1] and has become one of the most popular research field of modern control theory; see, for example, [2–12] and the references therein. In the most early literature about stochastic LQ issue, it is always assumed that the control weighting matrix is positive definite and the state weighting matrix is positive semidefinite as the deterministic LQ problem does. However, a surprising fact was found that, different from deterministic LQ problem, for a stochastic LQ modeled by a stochastic Itô-type differential system, the original LQ optimization may still be well posed even if the cost weighting matrices and are indefinite [5]. Follow-up research was carried out, and a lot of important results were obtained. In [6–9], continuous-time indefinite stochastic LQ control problem was studied. For the discrete-time case, there have been some works. For example, the system with only control-dependent noises was studied in [10]. The finite time and infinite horizon indefinite stochastic LQ control problem with state- and control-dependent noises were, respectively, studied in [11, 12].

In this paper, we study discrete-time indefinite stochastic LQ control problem over an infinite time horizon. The system involves multiplicative noises in both the state and the control. We mainly use the SDP approach introduced in [9, 13] to discuss the corresponding problem. We first introduce a generalized stochastic algebraic Riccati equation (GSARE) that involves the Moore-Penrose inverse of a matrix. The potential relations among LQ problem, SDP, and GSARE are studied. What we have obtained extends the results of [9] from continuous-time case to discrete-time case.

The remainder of this paper is organized as follows. In Section 2, we formulate the discrete-time indefinite stochastic LQ problem and present some preliminaries including generalized stochastic algebraic Riccati equation, SDP, and some lemmas. Section 3 contains the main results. Some relations among the optimality of the LQ problem, the complementary optimal solutions of the SDP and its dual problem, and the solvability of the GSARE are established. Some comments are given in Section 4.

*Notations*. : -dimensional Euclidean space. : the set of all matrices. : the set of all symmetric matrices. : the transpose of matrix . : is positive semidefinite (positive definite). : the identity matrix. : the set of all real numbers. and . Tr(): the trace of a square matrix . : the adjoint mapping of a mapping .

#### 2. Preliminaries

##### 2.1. Problem Statement

Consider the following discrete-time stochastic system: where , are, respectively, the system state and control input. is the initial state and is the noise. and are constant matrices. is a sequence of real random variables defined on a complete probability space with , which is a wide sense stationary, second-order process with and , where is the Kronecker function. belongs to , the space of all -valued, -adapted measurable processes satisfying . We assume that the initial state is independent of the noise .

We first give the following definitions.

*Definition 2.1. *System (2.1) is called mean square stabilizable if there exists a feedback control such that for any initial state , the closed-loop system
is asymptotically mean square stable, that is, the corresponding state of (2.2) satisfies , where is a constant matrix.

For system (2.1), we define the admissible control set The cost functional associated with system (2.1) is where and are symmetric matrices with appropriate dimensions and may be indefinite. The LQ optimal control problem is to minimize the cost functional over . We define the optimal value function as Since the weighting matrices and may be indefinite, the LQ problem is called an indefinite stochastic LQ control problem.

*Definition 2.2. *The LQ problem is called well posed if
If there exists an admissible control such that , the LQ problem is called attainable and is the optimal cost value. , is called an optimal control, and , corresponding to is called the optimal trajectory.

Stochastic algebraic Riccati equation (SARE) is a primary tool in solving stochastic LQ control problems. In [12], the following discrete SARE: was studied. The constraint that is demanded in (2.7). In fact, the corresponding LQ problem may have optimal control even if the condition is not satisfied. In this paper, we introduce the following generalized stochastic algebraic Riccati equation (GSARE), which weakens the positive definiteness constraint of to positive semidefiniteness constraint and replaces the inverse by Moore-Penrose inverse. Hence, (2.8) is an extension of (2.7).

##### 2.2. Semidefinite Programming

In this subsection, we will introduce SDP and its dual. SDP is a special conic optimization problem and is defined as follows.

*Definition 2.3 (see [14]). *Suppose that is a finite-dimensional vector space with an inner product and is a space of block diagonal symmetric matrices with an inner product . is a linear mapping, and . The following optimization problem:
is called a semidefinite programming (SDP). From convex duality, the dual problem associated with the SDP is defined as
In the context of duality, we refer to the SDP (2.9) as the primal problem associated with (2.10).

Consider the following SDP problem: By the definition of SDP, we can get the dual problem of (2.11).

Proposition 2.4. *The dual problem of (2.11) can be formulated as
*

*Proof. *The objective of the primal problem can be rewritten as maximizing . The dual variable , where . The LMI constraint in the primal problem can be represented as
According to the definition of adjoint mapping, we have , that is, . It follows . By Definition 2.3, the objective of the dual problem is to minimize . On the other hand, we will state that the constraints of the dual problem (2.10) are equivalent to the constraints of (2.12). Obviously, is equivalent to the equality constraint of (2.12). This ends the proof.

The primal problem (2.9) is said to satisfy the Slater condition if there exists a primal feasible solution such that , that is, the primal problem (2.9) is strictly feasible. The dual problem (2.10) is said to satisfy the Slater condition if there is a dual feasible solution satisfying , that is, the dual problem (2.10) is strictly feasible.

Let and denote the optimal values of SDP (2.9) and the dual SDP (2.10), respectively. Let and denote the primal and dual optimal sets. Then, we have the following proposition (see [13, Theorem 3.1]).

Proposition 2.5. * if either of the following conditions holds.*(1)*The primal problem (2.9) satisfies Slater condition.*(2)*The dual problem (2.10) satisfies Slater condition.*

If both conditions hold, the optimal sets and are nonempty. In this case, a feasible point is optimal if and only if there is a feasible point satisfying the complementary slackness condition:

##### 2.3. Some Definitions and Lemmas

The following definitions and lemmas will be used frequently in this paper.

*Definition 2.6. *For any matrix , there exists a unique matrix , called the Moore-Penrose inverse of , satisfying

Lemma 2.7 (extended Schur’s lemma). *Let matrices , , and be given with appropriate dimensions. Then, the following conditions are equivalent:*(1)*, , and ,*(2)*,
*(3)*.*

Lemma 2.8 (see [7]). *For a symmetric matrix , we have*(1)*,
*(2)* if and only if ,*(3)*. *

Lemma 2.9 (see [12]). *In system (2.1), suppose is given, and , is an arbitrary family of matrices, then, for any , we have
**
where
*

Lemma 2.10. *System (2.1) is mean square stabilizable if and only if one of the following conditions holds.*(1)*There are a matrix and a symmetric matrix such that
Moreover, the stabilizing feedback control is given by .*(2)*For any matrix , there is a matrix such that the following matrix equation:
has a unique positive definite solution . Moreover, the stabilizing feedback control is given by .*(3)*The dual problem satisfies the Slater condition.*

*Proof. * and can be derived from Proposition 2.2 in [15]. (3) is a discrete edition of Theorem 6 in [7]. The proof is similar to Theorem 6 in [7] and is omitted.

To this end, we need the following assumptions throughout the paper.

*Assumption 2.11. *System (2.1) is mean square stabilizable.

*Assumption 2.12. *The feasible set of is nonempty.

#### 3. Main Results

In this section, we will establish the relationship among the optimality of the LQ problem, the SDP, and the GSARE.

The following theorem reveals the relation between the SDP complementary optimal solutions and the GSARE.

Theorem 3.1. *If a feasible solution of , , satisfies , and the feedback control
**
is stabilizing, then there exist complementary optimal solutions of and . In particular, is optimal to , and there is a complementary dual optimal solution of , such that .*

*Proof. *By the stability assumption of the control and Lemma 2.10, the equation
has a positive solution . Let
that is,
By (3.2) and (3.3), we have
which shows is a feasible solution of (). because is a feasible solution of . By Lemmas 2.7 and 2.8,
In addition, we have
by and . Therefore, we have
Obviously, and are complementary optimal solutions to and is optimal to , and is optimal to is trivial because .

In above, the assumption that the control in (3.1) is stabilizing is not automatically satisfied. The following theorem reveals that we can obtain a stabilizing feedback control by the dual SDP.

Theorem 3.2. *Suppose that is a feasible solution of with , then the feedback control is stabilizing.*

*Proof. *First, we have because is feasible to . By Lemma 2.7, the inequality holds. By simple calculations, we have
Hence,
Above inequality shows (2.18) has a positive definite solution with . According to Lemma 2.10, is stabilizing.

The following theorem shows the relationship between the optimality of the LQ problem and the solution of GSARE.

Theorem 3.3. *If LQ problem (2.1)–(2.5) is attainable with respect to any , then must have an optimal solution such that .*

*Proof. *Since the LQ problem is attainable, then the optimal value must be of the quadratic form [16]:
Let be an optimal pair for the initial state . Let and in (2.16), where is an any feasible solution of , then we have
Then, a completion square means
where . Since is feasible to , we have and by Lemma 2.7. Then, the inequality
holds for any feasible solution to . This shows that must be optimal to . Moreover, taking in (3.13) and considering , we know that for . Setting and noticing that is arbitrary, it follows that .

Below, we will show is a feasible solution of . We consider the following SDP and its dual under a perturbation :
Obviously, satisfies the Slater condition because we assume that the feasible set of is nonempty and also satisfies the Slater condition by the mean square stabilizability assumption and Lemma 2.10. Hence, the complementary optimal solutions exist by Proposition 2.5. Take any dual feasible solution . By the weak duality in conic optimization problems, we have
Let be a feasible solution of , then is feasible to for all . Similar to Theorem 10 in [7], we conclude that, for any , there exists the unique optimal solution of , denoted by , and .

Together with (3.17), we know that are contained in a compact set with ( is a constant). Then, take a convergent subsequence satisfying with as . Obviously, is feasible to because the feasible region of monotonically shrinks as . Define the perturbed cost functional
where . By (3.13), we have
for any , where and is the form of with and replaced by and . Then, by Theorems 10 and 12 in [7],
Furthermore,
Taking limit, we have . On the other hand, because is feasible to and (3.14). So . The feasibility of is proved. The proof is completed.

The following theorem studies the converse of Theorem 3.3.

Theorem 3.4. *If a feasible solution of , , satisfies and the feedback control is stabilizing, then it must be optimal for LQ problem (2.1)–(2.5).*

*Proof. *For any , we have
by (3.13) and , where . Because is stabilizing, must be optimal.

The following theorem shows we can get the optimal feedback control by SDP dual optimal solution.

Theorem 3.5. *Assume that (P) and (D) have complementary optimal solutions and with . Then, and LQ problem (2.1)–(2.5) has an attainable optimal feedback control given by .*

*Proof. *From the proof of Theorem 3.1, we have
where . By complementary slackness condition and the invertibility of , we have
So . On the other hand, and from and Lemma 2.7. From the equality constraint in (2.12) and the above results, we have
The last inequality holds because from Lemma 2.8. It follows that .

For any , by (3.13), we get
where is any feasible solution of and because of the feasibility of . Then,
On the other hand, is stabilizing by Theorem 3.2. Let and in (3.26), then it follows that
where . Below we prove . Applying complementary slackness condition and above proof, we have
Hence, . Then,
It follows from (3.27) and (3.28) that
The optimality of is proved.

#### 4. Conclusion

In this paper, we use the SDP approach to study discrete-time indefinite stochastic LQ control problem. Some relations are given and are summarized as follows. The condition that LQ problem is attainable can induce that has an optimal solution satisfying GSARE (Theorem 3.3). Theorems 3.4 and 3.5 give two suffcient conditions for LQ problem attainability by GSARE and complementary optimal solutions of and . Moreover, by dual SDP, we can get stabilized feedback control (Theorem 3.2). What we have obtained can be viewed as a discrete-time version of [9]. Of course, there are many open problems to be solved. For instance, the indefinite LQ problems for Markovian jumps or time-variant system merit further study.

#### Acknowledgments

This work was supported by the National Natural Science Foundation of China (61174078), Specialized Research Fund for the Doctoral Program of Higher Education (20103718110006), and Key Project of Natural Science Foundation of Shandong Province (ZR2009GZ001).

#### References

- W. M. Wonham, “On a matrix Riccati equation of stochastic control,”
*SIAM Journal on Control*, vol. 6, no. 4, pp. 681–697, 1968. View at Google Scholar - C. E. de Souza and M. D. Fragoso, “On the existence of maximal solution for generalized algebraic Riccati equations arising in stochastic control,”
*Systems and Control Letters*, vol. 14, no. 3, pp. 233–239, 1990. View at Publisher · View at Google Scholar - W. H. Zhang, H. S. Zhang, and B. S. Chen, “Generalized Lyapunov equation approach to state-dependent stochastic stabilization/detectability criterion,”
*IEEE Transactions on Automatic Control*, vol. 53, no. 7, pp. 1630–1642, 2008. View at Publisher · View at Google Scholar - F. Carravetta and G. Mavelli, “Suboptimal stochastic linear feedback control of linear systems with state- and control-dependent noise: the incomplete information case,”
*Automatica*, vol. 43, no. 5, pp. 751–757, 2007. View at Publisher · View at Google Scholar · View at Zentralblatt MATH - S. P. Chen, X. J. Li, and X. Y. Zhou, “Stochastic linear quadratic regulators with indefinite control weight costs,”
*SIAM Journal on Control and Optimization*, vol. 36, no. 5, pp. 1685–1702, 1998. View at Publisher · View at Google Scholar · View at Zentralblatt MATH - M. A. Rami, X. Chen, J. B. Moore, and X. Y. Zhou, “Solvability and asymptotic behavior of generalized Riccati equations arising in indefinite stochastic LQ controls,”
*IEEE Transactions on Automatic Control*, vol. 46, no. 3, pp. 428–440, 2001. View at Publisher · View at Google Scholar · View at Zentralblatt MATH - M. A. Rami and X. Y. Zhou, “Linear matrix inequalities, Riccati equations, and indefinite stochastic linear quadratic controls,”
*IEEE Transactions on Automatic Control*, vol. 45, no. 6, pp. 1131–1143, 2000. View at Publisher · View at Google Scholar · View at Zentralblatt MATH - M. A. Rami, J. B. Moore, and X. Y. Zhou, “Indefinite stochastic linear quadratic control and generalized differential Riccati equation,”
*SIAM Journal on Control and Optimization*, vol. 40, no. 4, pp. 1296–1311, 2001. View at Publisher · View at Google Scholar - D. D. Yao, S. Z. Zhang, and X. Y. Zhou, “Stochastic linear-quadratic control via semidefinite programming,”
*SIAM Journal on Control and Optimization*, vol. 40, no. 3, pp. 801–823, 2001. View at Publisher · View at Google Scholar · View at Zentralblatt MATH - A. Beghi and D. D'Alessandro, “Discrete-time optimal control with control-dependent noise and generalized Riccati difference equations,”
*Automatica*, vol. 34, no. 8, pp. 1031–1034, 1998. View at Publisher · View at Google Scholar · View at Zentralblatt MATH - M. A. Rami, X. Chen, and X. Y. Zhou, “Discrete-time indefinite LQ control with state and control dependent noises,”
*Journal of Global Optimization*, vol. 23, no. 3-4, pp. 245–265, 2002. View at Publisher · View at Google Scholar · View at Zentralblatt MATH - Y. L. Huang, W. H. Zhang, and H. S. Zhang, “Infinite horizon linear quadratic optimal control for discrete-time stochastic systems,”
*Asian Journal of Control*, vol. 10, no. 5, pp. 608–615, 2008. View at Publisher · View at Google Scholar - L. Vandenberghe and S. Boyd, “Semidefinite programming,”
*SIAM Review*, vol. 38, no. 1, pp. 49–95, 1996. View at Publisher · View at Google Scholar · View at Zentralblatt MATH - V. Balakrishnan and L. Vandenberghe, “Semidefinite programming duality and linear time-invariant systems,”
*IEEE Transactions on Automatic Control*, vol. 48, no. 1, pp. 30–41, 2003. View at Publisher · View at Google Scholar - A. El Bouhtouri, D. Hinrichsen, and A. J. Pritchard, “${H}_{\infty}$-type control for discrete-time stochastic systems,”
*International Journal of Robust and Nonlinear Control*, vol. 9, no. 13, pp. 923–948, 1999. View at Publisher · View at Google Scholar · View at Zentralblatt MATH - B. D. O. Anderson and J. B. Moore,
*Optimal Control-Linear Quadratic Methods*, Prentice-Hall, Englewood Cliffs, NJ, USA, 1989.