Maximum Principle for Stochastic Recursive Optimal Control Problems Involving Impulse Controls
Zhen Wu1and Feng Zhang2
Academic Editor: Zhenya Yan
Received13 Jan 2012
Accepted11 Apr 2012
Published20 Jun 2012
Abstract
We consider a stochastic recursive optimal control problem in which the control variable has two components: the regular control and the impulse control. The control variable does not enter the diffusion coefficient, and the domain of the regular controls is not necessarily convex. We establish necessary optimality conditions, of the Pontryagin maximum principle type, for this stochastic optimal control problem. Sufficient optimality conditions are also given. The optimal control is obtained for an example of linear quadratic optimization problem to illustrate the applications of the theoretical results.
1. Introduction
The nonlinear backward stochastic differential equations (BSDEs for short) were first introduced by Pardoux and Peng [1]. Independently, Duffie and Epstein [2] introduced BSDEs under economic background. In [2], they presented a stochastic recursive utility which is an extension of the standard additive utility with the instantaneous utility depending not only on the instantaneous consumption rate but also on the future utility. Actually, it corresponds to the solution of a particular BSDE whose generator does not depend on the variable . And then, El Karoui et al. [3] gave the formulation of recursive utilities from the BSDE point of view. The problem that the cost function of the control system is described by the solution of BSDE is called the stochastic recursive optimal control problem. In this case, the control systems become forward-backward stochastic differential equations (FBSDEs).
One fundamental research direction for optimal control problem is to establish the necessary optimality conditionsβPontryagin maximum principle. Stochastic maximum principle for forward, backward, and forward-backward systems has been studied by many authors, including Peng [4, 5], Tang and Li [6], Wang and Yu [7], Wu [8], and Xu [9] for full information and Huang et al. [10], Wang and Wu [11], Wang and Yu [12], and Wu [13] for partial information case. However, in these papers, there are only regular controls in the control systems and impulse controls are not included.
Stochastic impulse control problems have received considerable research attention in recent years due to wide applicability in a number of different areas, especially in mathematical finance; see, for example, [14β17]. In most cases, the optimal impulse control problem was studied through dynamic programming principle. It was shown in particular that the value function is a solution of some quasi-variational inequalities.
The first result in stochastic maximum principle for singular control problem was obtained by Cadenillas and Haussmann [18], in which linear dynamics, convex cost criterion, and convex state constraint are assumed. Bahlali and Chala [19] generalized [18] to the nonlinear dynamics case with a convex state constraint. Bahlali and Mezerdi [20] considered a stochastic singular control problem in which the control system is governed by a stochastic differential equation where the regular control enters the diffusion coefficient and the control domain is not necessarily convex. The stochastic maximum principle was obtained with the approach developed by Peng [4]. Dufour and Miller [21] studied a stochastic singular control problem in which the admissible control is of bounded variation. It is worth pointing out that the control systems in these works are stochastic differential equations with singular control, and few examples are given to illustrate the theoretical results. Wu and Zhang [22] were the first to study stochastic optimal control problems of forward-backward systems involving impulse controls, and they obtained both the maximum principle and sufficient optimality conditions for the optimal control problem.
In this paper, we continue to study stochastic optimal control problem involving impulse controls, in which the control system is described by a forward-backward stochastic differential equation and the control variable consists of regular control and impulse control. Different from [22], it is assumed in this paper that the domain of the regular controls is not necessarily convex and the control variable does not enter the diffusion coefficient. Thus the result of this paper and that of [22] do not contain each other. We obtain the stochastic maximum principle by using a spike variation on the regular part of the control and a convex perturbation on the impulsive one. Sufficient optimality conditions are also obtained which can help to find the optimal control in applications.
The rest of this paper is organized as follows. In Section 2 we give some preliminary results and the formulation of our stochastic optimal control problem. In Section 3 we obtain the maximum principle for our stochastic optimal control problem. Sufficient optimality conditions for the optimal control problem is established in Section 4, and an example of linear quadratic optimization problem is also given to illustrate the applications of our theoretical results.
2. Formulation of the Stochastic Optimal Control Problem
Firstly we introduce some notations. Let () be a probability space and the expectation with respect to . Let be a finite time horizon and the natural filtration of a -dimensional standard Brownian motion augmented by the -null sets of . For and , denote by the set of -dimensional adapted processes such that , and denote by the set of -dimensional adapted processes such that .
Let be a nonempty subset of and a nonempty convex subset of . Let be a given sequence of increasing -stopping times such that as . We denote by the class of right continuous processes such that each is an -measurable random variable. It is worth noting that the assumption implies that at most finitely many impulses may occur on . Denote by the class of adapted processes such that , and denote by the class of -valued impulse processes such that . We call the admissible control set. In what follows, for a continuous function , the integration is understood as follows:
Given and , we consider the following SDE with impulses:
where , , and are measurable mappings. Similar to [22, Proposition 2.1], we have the following.
Proposition 2.1. Let be continuous and , uniformly Lipschitz in . Assume that , , and for some . Then SDE (2.2) admits a unique solution .
For , let us consider the following BSDE with impulses:
where , and are measurable mappings. Similar to [22, Proposition 2.2], we have the following.
Proposition 2.2. Let be continuous and Lipschitz in . Assume that , , and for some . Then BSDE (2.3) admits a unique solution .
The control system of our stochastic optimal control problem is subject to the following FBSDE:
where , , , are measurable mappings, and , are continuous functions. The objective is to minimize the following cost functional over the class :
where , , , and are measurable mappings.
In what follows we assume the following., , , are continuous, and they are continuously differentiable in , with derivatives continuous and uniformly bounded. Moreover, assume that and have linear growth in ., , , are continuous, and they are continuously differentiable in , with derivatives continuous and bounded by , , , and , respectively. Moreover, we assume for any .
From Propositions 2.1 and 2.2, it follows that FBSDE (2.4) admits a unique solution for any , and the functional is well defined.
3. Stochastic Maximum Principle for the Optimal Control Problem
Let be an optimal control and the corresponding trajectory. We introduce the spike variation with respect to as follows:
where is an arbitrarily fixed time, is a sufficiently small constant, and is an arbitrary -valued -measurable random variable such that . Let be such that . Then it is easy to check that , is also an element of . Let us denote by the trajectory associated with . For convenience, denote , for . In what follows, we use to denote a positive constant which can be different from line to line.
Let us introduce the following FBSDE (called the variational equation):
By Propositions 2.1 and 2.2, FBSDE (3.2) admits a unique solution .
Similar to [9, Lemma 1], we can easily obtain the following.
Lemma 3.1.
We have
We proceed to give the following lemma.
Lemma 3.2. The following estimations hold:
where as .
Proof. It is easy to check that
where
Since , are uniformly bounded, we have . Hence, if we can obtain
then the estimation (3.4) can be obtained from Gronwall's lemma and (3.7). Let us take the term for example. By the definition of and HΓΆlder's inequality, we have
From HΓΆlder's inequality, Lemma 3.1, and the dominated convergence theorem, it follows that
Since is uniformly bounded, by Lemma 3.1 we get
Thus we obtain . In the same way we can get
Hence, the estimation (3.4) is proved. Now we prove (3.5) and (3.6). Set
It is easy to obtain
where
We have
Since is uniformly bounded, it follows from (3.4) that . Since is continuous and uniformly bounded, from Lemma 3.1 and the dominated convergence theorem it follows that
Consequently,
From Lemma 3.1 and the dominated convergence theorem, it follows that
Since , , and are uniformly bounded, we have
Similar to the proof of Lemma 1 in [9] for the BSDE part, we can obtain (3.5) and (3.6) with the iterative method.
We are now ready to state the variational inequality.
Lemma 3.3. The following variational inequality holds:
Proof. From the optimality of , we have
From Lemmas 3.1 and 3.2, it follows that
Hence,
Similarly we get
while
Since have linear growth, it follows from Lemma 3.2 and HΓΆlder's inequality that
By Lemma 3.1 and the dominated convergence theorem, we have
where , . It follows from HΓΆlder's inequality that
Using Lemma 3.1 again, we get
Consequently,
The variational inequality follows from (3.25)β(3.32).
Now we introduce the following FBSDE (called the adjoint equation):
It is easy to check that the adjoint equation admits a unique solution .
We are now in a position to state the stochastic maximum principle.
Theorem 3.4. Let be an optimal control, the corresponding trajectory, and the solution of the adjoint equation. Then for any and it holds that
where is defined by
Proof. Applying ItΓ΄'s formula to , by Lemma 3.3 we derive
where satisfies . Dividing (3.37) by and letting go to 0, we obtain
By choosing in (3.38) we obtain the conclusion (3.35). If we choose , then for satisfying we have
Now let us set for any and . Then it is obvious that and . So from (3.39) it follows that, for any ,
Hence,
Since the quantity inside the conditional expectation is -measurable, the conclusion (3.34) can be obtained easily.
Similar to [22, Corollary 3.1], by Theorem 3.4 we can easily obtain the following
Corollary 3.5. Assume . Then for the optimal control it holds that
Remark 3.6. We can still obtain the stochastic maximum principle if the assumptions are relaxed in the following way.(i)The regular control process and the impulse control process are assumed to satisfy and for some .(ii)The assumption in Hypothesis (H2) can be weakened as .(iii)In the spike variation setting, the random variable is assumed to satisfy . In fact, under these new assumptions both the solutions of the control system (2.4) and the variational equation (3.2) belong to . The conclusion of Lemma 3.1 becomes
And Lemmas 3.2 and 3.3 still hold true.
4. Sufficient Optimality Conditions for Optimal Controls
We still denote by the trajectory corresponding to . Let us first introduce an additional assumption.
(H3) The control domain is a convex body in . The maps , , and are locally Lipschitz in the regular control variable .
Theorem 4.1. Let (H1)β(H3) hold. Assume that the functions , , and are convex. Moreover, has the following particular form: for and . Let be the solution of the adjoint equation associated with . Then is an optimal control of the stochastic optimal control problem if it satisfies (3.34) and (3.35).
Proof. Set . Since , , are convex, we have
Thus,
Set . Then by ItΓ΄'s formula applied to , we get , where
From (3.35) we have . By (3.34) and [23, Lemma 2.3-(iii); Chapter 3], we have . By [23, Lemma 2.4; Chapter 3], we further conclude that
Then, by [23, Lemma 2.3-(v); Chapter 3] and the convexity of , we obtain
from which it follows immediately that . Thus we obtain and the proof is complete.
We now give an example of linear quadratic optimal control problem involving impulse controls to illustrate the application of our theoretical results.
Example 4.2. For simplicity, assume that the variables and coefficients are scalar-valued. Let us take and . There are only two values β1 and 1 in which is a usual case in practice and represents only two control states: βonβ and βoffβ. For , the controlled system is subject to the following linear FBSDE:
and the cost functional is given by
The coefficients are deterministic constants such that and . By Propositions 2.1 and 2.2 we know that the control system admits a unique solution for any . And the functional is well defined from into . Let be an optimal control and the corresponding trajectory. Then the following adjoint equation
admits a unique solution . The Hamiltonian is given by
Then by Corollary 3.5 we obtain
From (4.10) we get
From (4.11) we obtain that
Hence, if is an optimal control of this linear quadratic control problem, then it satisfies (4.12) and (4.13). We can prove that obtained in (4.12) and (4.13) is indeed an optimal control of this linear quadratic optimization problem. Note that Theorem 4.1 does not hold now since is not convex in this example. In what follows, we use the same notations as those in the proof of Theorem 4.1. In fact, as in the proof of Theorem 4.1, we can still derive . On the one hand, it follows from (4.13) that . On the other hand, we have
where
From (4.12) and the definition of , it is easy to get
Since , , is convex in , and thus , so we obtain . Consequently, it follows that and the optimality of is proved.
Remark 4.3. For the classical linear quadratic optimal control problem, one can usually obtain an optimal control in a linear state feedback form by virtue of the so-called Riccati equation, and along this line the solvability of the Riccati equation leads to that of the linear quadratic problem. However, it is difficult to obtain a state feedback optimal control in terms of the Riccati equation in Example 4.2 mainly due to the particular form of the regular control domain and the appearance of the impulse control in the control system.
Acknowledgments
The authors would like to thank the referees for valuable suggestions which helped to improve the first version of this paper. Z. Wu acknowledges the financial support from the National Natural Science Foundation of China (10921101 and 61174092) and the Natural Science Fund for Distinguished Young Scholars of China (11125102). F. Zhang acknowledges the financial support from the Natural Science Foundation of Shandong Province, China (ZR2011AQ018), and the Foundation of Doctoral Research Program, Shandong University of Finance and Economics.
References
E. Pardoux and S. G. Peng, βAdapted solution of a backward stochastic differential equation,β Systems & Control Letters, vol. 14, no. 1, pp. 55β61, 1990.
N. El Karoui, S. G. Peng, and M. C. Quenez, βBackward stochastic differential equations in finance,β Mathematical Finance, vol. 7, no. 1, pp. 1β71, 1997.
S. G. Peng, βA general stochastic maximum principle for optimal control problems,β SIAM Journal on Control and Optimization, vol. 28, no. 4, pp. 966β979, 1990.
S. G. Peng, βBackward stochastic differential equations and applications to optimal control,β Applied Mathematics and Optimization, vol. 27, no. 2, pp. 125β144, 1993.
S. J. Tang and X. J. Li, βNecessary conditions for optimal control of stochastic systems with random jumps,β SIAM Journal on Control and Optimization, vol. 32, no. 5, pp. 1447β1475, 1994.
G. C. Wang and Z. Y. Yu, βA Pontryagin's maximum principle for non-zero sum differential games of BSDEs with applications,β IEEE Transactions on Automatic Control, vol. 55, no. 7, pp. 1742β1747, 2010.
Z. Wu, βMaximum principle for optimal control problem of fully coupled forward-backward stochastic systems,β Systems Science and Mathematical Sciences, vol. 11, no. 3, pp. 249β259, 1998.
W. S. Xu, βStochastic maximum principle for optimal control problem of forward and backward system,β Journal of the Australian Mathematical Society B, vol. 37, no. 2, pp. 172β185, 1995.
J. H. Huang, G. C. Wang, and J. Xiong, βA maximum principle for partial information backward stochastic control problems with applications,β SIAM Journal on Control and Optimization, vol. 48, no. 4, pp. 2106β2117, 2009.
G. C. Wang and Z. Wu, βThe maximum principles for stochastic recursive optimal control problems under partial information,β IEEE Transactions on Automatic Control, vol. 54, no. 6, pp. 1230β1242, 2009.
G. C. Wang and Z. Y. Yu, βA partial information non-zero sum differential game of backward stochastic differential equations with applications,β Automatica, vol. 48, no. 2, pp. 342β352, 2012.
Z. Wu, βA maximum principle for partially observed optimal control of forward-backward stochastic control systems,β Science China information sciences, vol. 53, no. 11, pp. 2205β2214, 2010.
A. Bensoussan and J.-L. Lions, Impulse Control and Quasivariational Inequalities, Gauthier-Villars, Paris, France, 1984.
M. H. A. Davis and A. R. Norman, βPortfolio selection with transaction costs,β Mathematics of Operations Research, vol. 15, no. 4, pp. 676β713, 1990.
R. Korn, βSome applications of impulse control in mathematical finance,β Mathematical Methods of Operations Research, vol. 50, no. 3, pp. 493β518, 1999.
A. Cadenillas and U. G. Haussmann, βThe stochastic maximum principle for a singular control problem,β Stochastics and Stochastics Reports, vol. 49, no. 3-4, pp. 211β237, 1994.
S. Bahlali and A. Chala, βThe stochastic maximum principle in optimal control of singular diffusions with non linear coefficients,β Random Operators and Stochastic Equations, vol. 13, no. 1, pp. 1β10, 2005.
S. Bahlali and B. Mezerdi, βA general stochastic maximum principle for singular control problems,β Electronic Journal of Probability, vol. 10, no. 30, pp. 988β1004, 2005.
F. Dufour and B. Miller, βMaximum principle for singular stochastic control problems,β SIAM Journal on Control and Optimization, vol. 45, no. 2, pp. 668β698, 2006.
Z. Wu and F. Zhang, βStochastic maximum principle for optimal control problems of forward-backward systems involving impulse controls,β IEEE Transactions on Automatic Control, vol. 56, no. 6, pp. 1401β1406, 2011.