Linearization of a Matrix Riccati Equation Associated to an Optimal Control Problem

Abstract

The matrix Riccati equation that must be solved to obtain the solution to stochastic optimal control problems known as LQG homing is linearized for a class of processes. The results generalize a theorem proved by Whittle and the one-dimensional case already considered by the authors. A particular two-dimensional problem is solved explicitly.

1. Introduction

Let be a one-dimensional standard Brownian motion and consider the controlled diffusion process that satisfies the stochastic differential equation where is the control variable and , , and are Borel measurable functions.

The problem of finding the control that minimizes the expected value of the cost function in which is a Borel measurable function, is a real parameter, and is a first passage time defined by with , is a special case of what Whittle [1] (p. 289) termed LQG homing. This type of problem has applications in financial mathematics (see Makasu [2]). In Lefebvre and Zitouni [3], the authors considered an optimal landing problem. They also mentioned another possible application in which one wants to optimally control a dam.

In order to obtain , we define the value function Using dynamic programming, one only has to determine the value of . We find that is such that that is, It follows that so that The boundary conditions are

Whittle has shown that if the relation holds for a positive constant , then the function satisfies the linear second-order ordinary differential equation subject to Equation (12) is actually the Kolmogorov backward equation satisfied by the moment-generating function (or the Laplace transform of the density function) of the random variable that corresponds to but for the uncontrolled process obtained by setting in (1). Moreover, the above boundary conditions are the appropriate ones. Thus, Whittle was able to sometimes transform the optimal control problem into a purely probabilistic problem.

Remark 1. When , , and are (positive) constant functions, the relation in (10) is obviously satisfied. Therefore, it is then always possible to linearize (8) in such a case. However, if two (or all) of these functions are not constant, we can say that it is a special case when (10) does hold. When only one of these three functions is not a constant, the relation cannot be satisfied.
Next, notice that the optimal control is expressed in terms of the derivative of the value function, which satisfies the Riccati equation However, in general, we do not have a condition that would enable us to determine the value of the arbitrary constant that appears in the solution of (14). Therefore, we must solve either the nonlinear second-order differential equation (8) or the Kolmogorov equation (12).
In Lefebvre and Zitouni [3], the authors generalized Whittle’s result. They showed that if is different from zero, then the function defined through is a solution of the linear second-order ordinary differential equation They then gave a method that can be used to obtain an explicit expression for , hence the optimal control .
Now, Whittle actually considered LQG homing problems in dimensions. is then an -dimensional controlled diffusion process defined by where the noise matrix is symmetric and positive definite. The cost function is replaced by The matrix is positive definite and where denotes the continuation region.

Remark 2. In the general formulation, is an matrix, is a (column) vector of dimension , and is an matrix. Here, we assume that .
The optimal control is given by and the value function satisfies where is the derivative of with respect to the vector . The equation is subject to in which denotes the boundary of the stopping region (the complement of continuation region ).
The relation that generalizes (10) and that must hold between the control matrices and and the noise matrix in order to be able to linearize the nonlinear partial differential equation (21) is the following: In practice, it is difficult to satisfy (exactly) the above relation for , especially if the matrices involved are not constant matrices. In fact, even in the case when the various matrices in (23) are indeed constant, it is rather rare that this relation is satisfied. Problems for which (23) holds must be symmetrical. For instance, an important particular case is the one when N, B, and Q are all proportional to the identity matrix of order (and f is identical to zero), so that we want to optimally control an -dimensional Brownian motion.
Because of the importance of the matrix Riccati equation in many applications, the problem of linearizing this equation has been considered by a number of authors. Grasselli and Tebaldi [4], in particular, proposed a method that enabled them to transform the matrix Riccati equation that appeared in their work into linear equations; see also Gourieroux and Sufana [5].
The aim of this paper is to generalize the theorem proved by Whittle [1] and, at the same time, the results in Lefebvre and Zitouni [3]. In the next section, first the two-dimensional case will be presented. Then, the results will be extended to the -dimensional case. In Section 3, a particular two-dimensional problem will be solved explicitly. Finally, we will end with a few concluding remarks in Section 4.

2. Linearization of the Matrix Riccati Equation in Two Dimensions

Let If the relation in (23) holds, then the matrix is symmetric and invertible. To generalize Whittle’s theorem, we will assume that is indeed symmetric and invertible but not necessarily proportional to .

For simplicity, we will present the linearization technique that we propose in the case of two-dimensional controlled diffusion processes. Equation (17) can then be rewritten as follows [omitting the dependence of all functions on ]: and the cost function becomes The two standard Brownian motions are assumed to be independent.

Next, let where we have assumed that .

The optimal control is given by Moreover, the matrix Riccati equation satisfied by the derivative of the value function with respect to and that we want to linearize is given by [see (21)]

Proposition 3. Assume that the matrix defined in (24) is symmetric and invertible. Then, the function defined by the transformation where   for and satisfies the linear partial differential equation where Furthermore, for the transformation to be valid, there must exist functions and such that

Proof. We compute where the function is defined in (33). Substituting these expressions into (29), we find that the differential equation satisfied by will indeed be the linear equation (32) if
Now, the three equations in this system hold simultaneously if But this relation is verified for all matrices , , and that satisfy the conditions mentioned above. Notice, however, that from (30) we deduce two expressions for the value function . For the transformation to be valid, these two expressions must of course be compatible, which yields (34).

Remarks 4. (i) When (23) holds, so that we can apply Whittle’s theorem, the matrix is given by where denotes the identity matrix of order 2. The condition in (34) becomes that is, We can obviously choose , a constant. Therefore, this condition is always satisfied when Whittle’s theorem can be used. Moreover, it is clearly more likely to satisfy (23) [and (34)] when all the matrices are constant.
(ii) The function must be strictly positive. In one dimension, is indeed strictly positive since, making use of (15), it can be expressed as an exponential function. When (23) (in two dimensions) is satisfied, it is also easy to prove that is strictly positive.
(iii) The simplest problems that can be considered are such that the two controlled processes defined by (25) are independent, so that , and all the matrices are constant, which implies that is also a constant, for . The linear function then reduces to
(iv) Proposition 3 does not give us the function from which one deduces the optimal control. Similarly to Whittle’s theorem, it rather simplifies the optimal control problem. Indeed, it is generally easier to solve a linear than a nonlinear differential equation.
Since the proof is a simple extension of that of Proposition 3, we can state the following corollary.

Corollary 5. In the -dimensional case, the matrix Riccati equation [see (21)] is transformed into a linear partial differential equation for the function defined through where provided that the expressions that we deduce from (43) for the value function are compatible.

Remarks 6. (i) For the sake of brevity, we did not give the linear equation satisfied by , but it is a simple matter to derive it.
(ii) The larger is, the more difficult it should be to obtain compatible expressions for the value function. Nevertheless, the result is clearly an improvement over Whittle’s theorem.
(iii) If we define instead through the equation with then we find that satisfies a linear partial differential equation if which does not always hold true. Therefore, the transformation that we used is more appropriate.

3. Explicit Solution to a Particular Problem

In this section, we will make use of Proposition 3 to help us solve a particular LQG homing problem in two dimensions.

Assume that and let where with . We assume that is positive.

We calculate

Remark 7. It is important to notice that, in this particular problem, the relation in (23) does not hold, since the matrix is not proportional to the identity matrix of order 2. Hence, we could not appeal to Whittle’s theorem to linearize the differential equation satisfied by the value function.
From Proposition 3, we deduce that the function here satisfies the linear partial differential equation To solve this differential equation, we will use the method of similarity solutions. That is, we assume that the function can actually be written as with . Equation (52) is then transformed into the second-order ordinary differential equation whose general solution can be written as so that
Let us take , for simplicity. We then deduce from the preceding equation that We can now compare the two expressions that we obtain for the value function . First, we have which implies that
Similarly, we obtain that so that Since depends on both and in our problem, we conclude that we must set .
Next, making use of the boundary conditions if or , we can write that If , we find that Hence,
When , we deduce from (62) that Thus, we obtain the following expression for the value function: which is valid for .
Now, remember that we do not need to determine explicitly to obtain the optimal controls and . Only is needed. Here, we compute for any constant (and any ). It follows that the optimal controls are given by [see (20)]

Remarks 8. (i) If tends to infinity, we find that tends to , for . Notice that the optimal controls and are then negative, which is logical because and the process cannot reach the line when tends to infinity. Therefore, the optimizer tries to hit the line as soon as possible, but taking the quadratic control costs into account.
(ii) The reason why is the fact that , while . Thus, controlling is more expensive than controlling . Moreover, .

4. Conclusion

In this paper, we have generalized a theorem due to Whittle that sometimes enables us to linearize the matrix Riccati equation satisfied by the derivative of the value function in the so-called LQG homing problems. In Section 3, we were able to use our result to obtain the explicit solution to such a problem in two dimensions, for which Whittle’s theorem did not apply. Solving this type of problem explicitly is usually very difficult. Furthermore, notice that the optimal control that we obtained depends on and , for , rather than being constant. Therefore, the solution was not evident at all.

One cannot expect to be able to linearize the matrix Riccati equation (21) in all cases. Whittle’s theorem depends on a quite restrictive condition. Here, we showed that there are other cases than the ones to which Whittle’s result applies. We saw in Proposition 3 that the transformation that we propose will linearize (21) (with ), provided that (34) is satisfied. Since, as we mentioned above, our transformation will work every time Whittle’s theorem applies, we can at least claim that we have improved Whittle’s result.

Apart from the case when we can transform the stochastic optimal control problem into a purely probabilistic problem, few particular cases have been solved so far in two or more dimensions. The authors (see Lefebvre and Zitouni [6]) treated the general case in one dimension. They used symmetry, when possible, to obtain the exact optimal control, and they proposed a technique that yields a very good approximation to this optimal control when symmetry arguments cannot be used. It would be interesting to generalize these results in the -dimensional case.

Finally, we could try to apply the technique developed by Grasselli and Tebaldi [4], as well as other techniques proposed by various authors, to linearize the matrix Riccati equation in the case of LQG homing problems.

Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.

Acknowledgment

The authors want to thank the anonymous reviewer for the constructive comments.