Abstract
We get a new type of controlled backward stochastic differential equations (BSDEs), namely, the BSDEs, coupled with value function. We prove the existence and the uniqueness theorem as well as a comparison theorem for such BSDEs coupled with value function by using the approximation method. We get the related dynamic programming principle (DPP) with the help of the stochastic backward semigroup which was introduced by Peng in 1997. By making use of a new, more direct approach, we prove that our nonlocal Hamilton-Jacobi-Bellman (HJB) equation has a unique viscosity solution in the space of continuous functions of at most polynomial growth. These results generalize the corresponding conclusions given by Buckdahn et al. (2009) in the case without control.
1. Introduction
In the recent years, many authors (see [1–6]) have studied models of large stochastic particle systems with mean-field interaction. Lasry and Lions studied mean-field limits of problems of stochastic differential games ([4] and the references inside). Inspired by them, Buckdahn et al. [1] got in a purely stochastic approach, a new type of backward stochastic differential equations (BSDEs), namely, mean-field BSDEs. In [7], Buckdahn et al. deepened the investigation of such mean-field BSDEs. They obtained some central results for the mean-field BSDEs such as the existence and the uniqueness theorem, as well as a comparison theorem.
On the other hand, the modern optimal control theory has been developing very quickly since the works on the maximum principle by Pontryagin et al. [8] and the dynamic programming approach proposed by Bellman [9]. Since then, there have been a lot of works published on the stochastic maximum principle; refer to, for example, Kushner [10, 11], Bensoussan [12], Haussmann [13], Peng [14], Tang and Li [15], and Zhou [16]. There are also many works on the stochastic maximum principle for optimal control problems in the mean-field case; see, for example, Bensoussan et al. [17], Buckdahn et al. [18], Li [19], Meyer-Brandis et al. [20], and Yong [21]. There have also been a lot of works published on the dynamic programming approach, which gives with the help of dynamic programming principle (DPP) a stochastic interpretation to the associated partial differential equations (PDEs); we refer, for instance, to Buckdahn and Li [22], Peng [23, 24], and Yong and Zhou [25]. But to the best of our knowledge, there are no works relating optimal control problems in the mean-field case to nonlocal PDEs of Hamilton-Jacobi-Bellman (HJB) type.
In [7], the authors also considered the following decoupled forward-backward stochastic differential equation (FBSDE) with the initial given data and frozen : and they defined the value function which turns out to deterministic when all coefficients are deterministic. In (1), we have denoted by an independent copy of , and by the expectation taken only with respect to ; for more details, refer to Section 2. The authors of [7] proved that is the unique viscosity solution of the following nonlocal PDE in the space (the space of continuous functions over with polynomial growth): Here we have defined
In this paper we will investigate an optimal stochastic control problem involving mean-field BSDEs. Two major obstacles can be observed; let us explain them. We consider the following controlled decoupled FBSDE with the initial given data and frozen and the control : where is an admissible control over the time interval . In order to determine , we consider in a first step (4) for . The thus obtained equation is a mean-field one and was studied in [7]; for more details, the reader is referred to (22) and (26).
A first idea for the above introduced control problem could be to consider as a value function. Consider But, in fact, such a value function does not satisfy the DPP because of the expectation terms in (4). For this reason, similar to [7] we have to freeze, here, not only but also , and we consider the following value function: However, the fact that the control is frozen in (4) has as a consequence that the value function is a viscosity solution of the following classical PDE (there are many references, such as [22–24]): with
But is not a viscosity solution of the following nonlocal HJB equation:
However, it is this latter nonlocal HJB equation which we want to give a stochastic interpretation. PDE (9) is nonlocal in its solution . Indeed; that is, this coefficient depends nonlinearly not only of the value of at but also on the whole function .
On the other hand, we observe that, in the case without control, that is, in [7], we have , and then we can rewrite the backward SDE of (1) as follows: which can be regarded as an equation with the solution in some sense.
Inspired by this idea, we change to study the following BSDE coupled with value function: Here and satisfy (H3.2). Notice that if one of the coefficients is not deterministic, then usually the value function is also not deterministic. On the other hand, the assumption that is bounded in plays an important role in our work. However, in the case without control, that is, when the control state space is a singleton, the boundedness assumption on can be deleted (see Remark 5(i) or [7]). The solution of BSDE coupled with value function (12) is a triplet (see Theorem 9). We use a new iteration method to prove that (12) has a unique solution .
One of the main objectives of our paper is to study the stochastic interpretation of our nonlocal HJB equation (9). Firstly, unlike [7, 22, 24] in which the authors use the BSDE method to prove the existence and the uniqueness of the viscosity solution for the related PDEs, our approach here is quite different and more direct. Secondly, in [7], the authors have to consider the uniqueness of the viscosity solution in a smaller space in which the continuous functions are of at most polynomial growth. But, in our work, since is bounded in , we have the uniqueness of viscosity solution in such that , uniformly in . On the other hand, for the existence and the uniqueness of the viscosity solution of our nonlocal HJB equation (9), we do not need the monotonicity assumption on in , because once knowing , the driving coefficient satisfies the usual assumptions for classical BSDEs. From this point of view, we generalize Theorem 6.1 and Theorem 7.1 in [7] for the case without control (see Remark 23).
Our paper is organized as follows. Section 2 introduces the theory of mean-field SDEs and mean-field BSDEs which are used in what follows. In Section 3, a new type of BSDEs, namely, BSDEs, coupled with value function is studied. The existence and the uniqueness theorem, as well as a comparison theorem, for this type of BSDEs are proved (Theorems 9 and 11). We also show that is Lipschitz and has linear growth in (Theorem 9). Section 4 is devoted to prove the DPP and to show that is -Hölder continuous in . The existence and the uniqueness of the viscosity solution of our nonlocal HJB equation in the space is studied in Section 5. Finally, we give two examples.
2. Mean-Field SDEs and Mean-Field BSDEs
Our probability space is the classical Wiener space; that is, is the set of all continuous functions from to beginning from 0; is the Wiener measure such that the coordinate process , , , becomes a -dimensional Brownian motion; is the Borel -field over , completed by the set of all -null sets, and is the natural filtration generated by and completed by ; that is, .
We introduce the following spaces which will be used frequently: for ,
real-valued -adapted càdlàg process: ;
-valued -progressively measurable process: .
For the reader's convenience, let us first introduce the framework of mean-field SDEs—also called McKean-Vlasov SDEs (MV SDEs for short) and mean-field BSDEs which we will use in our work. For more details about them we refer to [1, 7].
Let be the (noncompleted) product of with itself. In this space, we use the filtration . A random variable (the space of all real-valued random variables over ) defined on can be extended to by putting , . For any (the space of integrable random variables of ), is in , -a.s., and we define . Then we can calculate the expectation of with the help of the Fubini Theorem:
We suppose that the following are given measurable functions: and , which satisfy the following:
(H2.1) (i) are -progressively measurable processes;
(ii) and have linear growth and are globally Lipschitz in ; that is, there exist some constant , such that, for all -a.s.,(1);(2).
For any , the mean-field SDE has a unique strong solution . For the proof, we refer to Theorem 4.1 in [7]. Notice that
Let us now introduce the mean-field BSDEs (see [7] for more details). We suppose that is -progressively measurable, for all and satisfies the following assumptions:
(H2.2) (i) there exists a constant such that, -a.s., for all ,,
(ii) .
Lemma 1. Under the assumption (H2.2), for any random variable the mean-field BSDE has a unique adapted solution .
Lemma 2 (comparison theorem). Let , , be two generators satisfying the assumption (H2.2). Furthermore, we assume the following:(i)one of the two coefficients is independent of ;(ii)one of the two coefficients is nondecreasing with respect to .
Let and and be the solutions of the mean-field BSDE (17) with data and , respectively. Then, if , -a.s., and , -a.s., we have , , -a.s.
Now, we want to introduce the decoupled forward-backward SDEs in the mean-field case. We suppose is -progressively measurable, for all and satisfies the following:
(H2.3) (i) is Lipschitz with respect to ; that is, there exists a constant such that, -a.s., for all ,
(ii) .
We suppose that and satisfy (H2.1) and (H2.3), respectively. Let be arbitrarily given. For any data , we consider the following decoupled forward-backward SDE in the mean-field case:
Notice that with the choice of the initial data , (19) becomes a decoupled mean-field FBSDE, the forward equation is a mean-field SDE which has a unique strong solution . Then, from Lemma 1, it follows that the backward equation is the mean-field BSDE which has a unique solution . Once having got , (19) becomes a classical decoupled forward-backward SDE with the coefficients , , and , respectively, which means that (19) has a unique solution under the assumptions (H2.1) and (H2.3).
Lemma 3. For and , we let and , be the solutions of FBSDE (19) with the initial data and , respectively. Then there exists a constant such that Here the constant depends only on the Lipschitz and the linear growth constants of , and .
For Lemmas 1–3, the reader is referred to [7].
3. BSDEs Coupled with Value Function
In this section, we will investigate a new type of BSDEs, namely, the BSDEs, coupled with value function. We will first prove the existence and the uniqueness theorem of the solution for this type of BSDEs. For this we first consider the associate forward equation and we study later the BSDEs coupled with value function by an iteration approach.
Let be a compact metric space. An admissible control process on is an -progressively measurable process taking its values in . The set of all admissible controls on is denoted by .
We assume that the coefficients and satisfy the following conditions:
(H3.1) (i) for every fixed , , and are continuous in ;
(ii) there exists a such that, for all , , ,
From the above assumption (H3.1), we get immediately that, for all , , , .
For what follows is chosen arbitrarily but fixed. Under the assumption (H3.1), for any , , and , the SDE has a unique strong solution. We emphasize that, for , SDE (22) is a mean-field equation with as the unique solution. Once having , SDE (22) becomes a classical SDE with the coefficients and , which satisfy the linear growth and the Lipschitz assumptions.
Remark 4. For any , there exist the constants and such that, for all , and . Consider , for all , where only depend on the Lipschitz constant and the linear growth constant of and (for it also depends on ). The reader is referred to Remark 4.1 in Buckdahn et al. [7].
We assume that the both mappings and satisfy the following conditions:
(H3.2) (i) for each fixed , is continuous in , and there exists a constant , such that, for all , , , , and ,
(ii) is -measurable, for all ; and there exists a constant , such that, for all ,
(iii) there exists a constant , such that , for all .
From (H3.2)-(i), we get directly that there exists some constant such that, for all , , and , , .
For and satisfying the assumption (H3.2), we consider the following new type of BSDE, namely, the BSDE, coupled with value function. Consider
Remark 5. (i) When the coefficients , and do not depend on controls, we consider (19) as in [7] with the initial data . We define . From [7], we have ,-a.s.; that is, the backward equation of (19) becomes now
coupled with the associated value function . It means that (27) (i.e., (26)) has a unique solution . However, this approach is not possible anymore in the case we study here. Moreover, we emphasize that, for the case without control, we do not need to be bounded in to make sure that (26) has a unique solution . On the other hand, when all the coefficients are deterministic, is the unique viscosity solution of the associated nonlocal PDE under the standard assumptions; refer to [7].
(ii) When the coefficients , and do not depend on , and is deterministic and does not depend on , then the SDE (22) becomes the classical SDE:
The BSDE (26) becomes the following classical BSDE:
decoupled with the associated value function , where is the solution of SDE (28). is the unique viscosity solution of the associated HJB equation under the standard assumptions; we refer to [24] or [22].
Remark 6. Recall that due to (H3.2) (ii) the terminal condition is a random variable. This has as a consequence that, here, in general, cannot be expected to be a deterministic function, but is an -measurable random variable.
In order to make things clearer and to show the existence of a solution, we choose an iterative approach. Putting ; we consider, for , ,
Lemma 7. For all , (30) admits a unique solution . Moreover, is a measurable random field such that(i) is -measurable, ;(ii)there exists a constant independent of , such that, a.s., for all ,
Proof. For and ; that is, the driving coefficient of (30) takes the form
it satisfies the Lipschitz and the growth conditions for classical BSDEs (see [26, 27] or refer to Lemma 1); therefore, (30) admits a unique solution . Moreover, from (20), for some constant , we have -a.s., for all ,
Consequently, -a.s., for all ,
and, obviously, is -measurable.
We suppose now that, for , is a measurable random field with, for some constant , -a.s., for all ,
Then, putting
it follows from (H3.2) that, for some constant independent of ,
-a.s., for all . Consequently, due to (20), for some constant independent of , -a.s., for all , we have
Therefore, with respect to the same constant , we have, -a.s., for all ,
and, obviously, is -measurable. The proof of the lemma is complete.
Theorem 8. There are processes and a measurable random field which is -measurable, for all , such that converges to in , for all , and converges to in ; solves BSDE (26). Moreover, there exists some constant , such that, -a.s., for all , we have
Proof. Putting
by applying Itô's formula to and taking the conditional expectation, we get for all , -a.s.,
where is arbitrarily small and is a constant depending on the Lipschitz constant and on . As is independent of , we can choose such that . Then, for some constant depending only on and ,
.
Thus, in particular, for ,
And, from the definition of , and , it follows that
Therefore,
which means
. Iterating this inequality and denoting the bound of by , we obtain that
Therefore,
It follows that there exists some process such that
On the other hand, from (49) and (45), we get
Then, obviously, there exists a random field measurable, such that is -measurable, and
Moreover, from Lemma 7, we deduce that, -a.s., for all ,
We also remark that, from (52), it follows that
; that is, , dtd -a.s.
From (42) and (43), we deduce now that there are processes such that in as . This allows taking the limit in (30), as , and we obtain
dsd . Consequently, there is a modification of belonging to , which is also denoted by , such that (55) still holds, for all . Moreover, using the convergence of to , we can deduce that converges to even in .
In order to complete the proof of the lemma, we still have to show that . In analogy to (43), from (30) and (55), we get
.
Therefore,
(see (52)). Notice that ; therefore, taking into account (52), we get .
We can state now one of our main results.
Theorem 9. Under the assumptions (H3.1) and (H3.2), the BSDE coupled with value function (26) has a unique solution , with and is a random field such that the associated random field satisfies for some constant .
Proof. The existence of a solution has been proved above; let us prove the uniqueness. Let , , be two solutions of BSDE coupled with value function (26) and let
satisfy (60). Then, by following the computations for (42) and (43), but now for and for , we obtain
for all .
Consequently,
.
Notice that, and both satisfy (60); then, from (64), we have
Furthermore, from (64), we also get
and, from Gronwall's inequality; we get
that is, .
Hence,
This has the consequence that the coefficients
of BSDE (26) coincide. Since and both satisfy (60), the coefficient satisfies the standard conditions of a coefficient of a classical BSDE, and the consequence is that , coincide, for all .
Remark 10. From the standard estimates for BSDEs (see, e.g., Proposition 4.1 in Peng [24]), there exists a constant such that, for any , and , we get and, in particular, where and are the solutions of BSDE (26) with the initial data and , respectively. Here, the constant depends only on the Lipschitz and the linear growth constants of , and .
After having studied the existence and the uniqueness of our BSDE coupled with value function (26), let us establish now a comparison theorem. For this, we need the following additional assumption:
(H3.3) the function is nondecreasing, for all .
Theorem 11 (comparison theorem). Let be two generators satisfying the assumptions (H3.2) and (H3.3) and let , . By , we denote the solution of the following BSDEs coupled with value function, respectively, with satisfying (60), i = 1, 2. And is the solution of SDE (22) with the coefficients and satisfying (H3.1). Then, if , ., and , we have that Moreover, , -a.s., .
Proof. Let be the solution of the following iterating BSDE:
, where for , , . From Lemma 7, we know that
Due to , we have . For , we define
From the comparison theorem for classical BSDEs (or refer to Lemma 2), we have
This implies
Now, for , we define
Thanks to the monotonicity assumption (H3.3), we have
Again from the comparison theorem for classical BSDEs, we get
and thus also
By iterating this argument we obtain that
It follows from Theorem 8 that, by letting , we obtain
Furthermore, we have , P-a.s., .
Remark 12. It is not hard to prove that the corresponding results still hold, when also the coefficients , and are random.
We now study the special case, when the terminal function is deterministic.
Proposition 13. Let us suppose now in addition to (H3.1) and (H3.2) that is a deterministic function. Then, the random field in Theorem 9 has a deterministic modification; that is, , with which we identify .
Proof. We use the iteration made in the frame of the proof of Lemma 7, but take now into account that and also the other coefficients are deterministic. Obviously, is deterministic. Let us suppose now that is deterministic, then is the unique solution of the controlled BSDE (30) with driving coefficient Since all the coefficients involved , and are deterministic, it follows now from the argument in [22] (see Proposition 3.3 in [22]) that the value function has a modification which is deterministic. Identifying with this deterministic modification and, recalling that converges to in , we have that also has a deterministic modification.
Remark 14. In all what follows, we consider to be deterministic and we identify with its deterministic modification.
4. The Dynamic Programming Principle
The objective of this section is to state the dynamic programming principle for our control problem, in which value function is defined through a controlled BSDE coupled with value function.
For this, we shall introduce the stochastic backward semigroup, whose definition is inspired by Peng [24]. Let us suppose that the assumptions (H3.1) and (H3.2) hold and is deterministic. Given the initial data and a random terminal condition , we introduce the stochastic backward semigroup by setting where is the solution of the following BSDE with the time horizon : where is the unique solution of SDE (22) and is the value function given by the BSDE coupled with value function (26) in Theorem 9. From Proposition 13 and Theorem 9, we know that is deterministic, and .
We define the function . From (H3.2), we know that there exists a constant independent of controls, such that for all .
Obviously, knowing given by the BSDE coupled with value function (26), BSDE (26) coincides with the following BSDE: with .
Based on BSDE (26) driven by the forward SDE (22), we see that (88) writes
But this places us in a classical setting, for which the dynamic programming principle is well studied (see, e.g., [24]). Therefore, we have the following theorem.
Theorem 15 (DPP). Under the assumptions (H3.1) and (H3.2), we have, for all , P-a.s.,
With the help of the DPP, we show the following proposition.
Proposition 16. The value function given by the BSDE coupled with value function (26) is -Hlder continuous in . More precisely, for some constant ,
Proof. Under the above properties (i) and (ii) of combined with the fact that is a continuous, deterministic function, it is standard that the value function has the property (94), since, in the proof of this classical result, the special structure of does not play a role, only (i) and (ii) and the continuity of are used; see [24] or [7, 22].
5. Associated Nonlocal Hamilton-Jacobi-Bellman Equations
In this section, we consider the following nonlocal Hamilton-Jacobi-Bellman (HJB) equation: where and are the gradient and the Hessian matrix of with respect to , respectively. Here we have used the following notations: The deterministic functions , and are supposed to satisfy (H3.1) and (H3.2), respectively, and is the solution of mean-field SDE (22) with the initial data and the given control .
As already mentioned, (96) is a nonlocal HJB equation and has the coefficient
A special case of such nonlocal PDEs, that is, without controls (or not of Hamilton-Jacobi-Bellman type), was studied by Buckdahn et al. [7]. In their approach the authors extended Peng's BSDE method [24] to the study of such nonlocal PDEs. However, our approach here is quite different and more direct.
Let us begin with recalling the definition of the viscosity solution, which is formulated directly for PDE (96). (For more details see [28].)
Definition 17. A real-valued continuous function is called(i)a viscosity subsolution of (96), if , for all , and if for all function and such that attains a local minimum at , (ii)a viscosity supersolution of (96), if , for all , and if for all function and such that attains a local maximum at , (iii)a viscosity solution of (96), if it is both a viscosity subsolution and supersolution of (96).
Remark 18. Inspired by [29], the authors of [7] (that is, the case without control) showed that for a function , the coefficient , may not be well defined, where such that , uniformly in . For this, they gave an example. Hence, they are restricted to the smaller space (the space of continuous functions with at most polynomial growth) for the proof of the uniqueness of the viscosity solution. Obviously, the value function is in , since it is of at most linear growth. They proved the existence and the uniqueness of the viscosity solution in the space . This is related to the fact that their driving coefficient is supposed to have linear growth in . Indeed, if grows linearly in , a correct definition of the coefficient of the nonlocal PDE makes it necessary that, for a viscosity solution , we have . This is the case if is of polynomial growth, but may not be satisfied, if . However, since we suppose our is bounded in , we do not meet this difficulty here.
Remark 19. It is by now standard that the space of test functions used in the above definition can be substituted by any other space of sufficiently regular functions containing , as long as the test functions constructed in the proof of the uniqueness are of class . Often, related with the needs of Peng's BSDE method, the class is used. We denote by the set of the real-valued functions that are continuously differentiable up to the third order and whose derivatives of order from 1 to 3 are bounded.
Theorem 20 (existence). Under the assumptions (H3.1) and (H3.2), the value function given by the BSDE coupled with value function (26) in Theorem 9 is a viscosity solution of PDE (96).
Proof. From Theorem 9 and Propositions 13 and 16, we know that is deterministic and there exists some constant , such that, , Consequently, is of linear growth. With the help of we define the coefficient ; see (89). Then is the value function of the classical control problem composed of SDE (22) (with the coefficients ) and BSDE (26) (with the coefficients and ). The coefficients satisfy the usual Lipschitz and linear growth conditions, and they are continuous. Under these conditions, it is by now well known that is the viscosity solution of the following HJB equation: But this means that is a viscosity solution of (96).
Theorem 21 (uniqueness). The function is the unique viscosity solution of PDE (96) in the class .
Remark 22. As we told above, in [7], the authors had to restrict the class of functions in which one has the uniqueness of the viscosity solution to the continuous functions of polynomial growth. But, in our work, is bounded in . Therefore, we can have the uniqueness in .
Proof. Let be two viscosity solutions of PDE (96). We put
. Then is a viscosity solution of the PDE. Consider
However, since the coefficients , and satisfy the usual conditions (Lipschitz continuity in , uniformly with respect to , and continuity), we know that is the unique viscosity solution in ; it is of at most linear growth and has the stochastic interpretation as value function:
of the stochastic control problem described by the following BSDE decoupled with the SDE (22):
Moreover, satisfies (60), (see [30] or [22]). Taking into account the definition of and , this latter BSDE (106) is nothing else but
with satisfing (60), .
But due to Theorem 9, the solution of (107) is unique; i.e., in particular, .
Remark 23. For the existence and the uniqueness of the viscosity solution of PDE (96), we do not need the monotonicity assumption (H3.3), because is as a part of the solution of (26), and once is known, we have a driver satisfying the usual assumptions for the classical BSDEs. That is, for the case without control we generalize Theorem 6.1 and Theorem 7.1 in [7] by deleting the monotonicity assumption.
Now we give two examples. In the first one, we consider the case without control. For notational convenience; let .
Example 24. For any arbitrarily chosen but fixed , it is easy to check that is the solution of the following SDE:
Let us consider BSDE:
which is equivalent to
We can check that is the solution of (110). From Theorems 20 and 21, we know that is the unique viscosity solution of the following PDE:
where and .
Remark 25. As already discussed in Remark 5, in the case without control, does not need to be bounded in .
In the second example, we suppose that , and depend on a control parameter.
Example 26. Let be given. For any ; let be the solution of the following controlled SDE: We associate it with the following controlled BSDE coupled with value function: Obviously, , and satisfy (H3.1) and (H3.2). From Theorem 9, BSDE (113) has a unique solution . According to Theorems 20 and 21, the value function defined by the BSDE coupled with value function (113) is the unique viscosity solution of the following HJB equation:
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.
Acknowledgments
This work has been supported by the NSF of P.R.China (nos. 11071144, 11171187, and 11222110), Shandong Province (nos. BS2011SF010 and JQ201202), SRF for ROCS (SEM), Program for New Century Excellent Talents in University (no. NCET-12-0331), and 111 Project (no. B12023).