Abstract
Let be a controlled one-dimensional diffusion process having constant infinitesimal variance. We consider the problem of optimally controlling until time , where is the first-passage time of the process to a given boundary and is a fixed constant. The optimal control is obtained explicitly in the particular case when is a controlled Wiener process.
1. Introduction
Let be the one-dimensional controlled diffusion process defined by the stochastic differential equation where is a real function, is the control variable, , and are constants and is a standard Brownian motion. We define the first-passage time where , and the random variable where is a constant.
Next, we consider the cost criterion where and are constants. We want to find the control that minimizes the expected value of . This type of problem is a special case of the ones that Whittle [1, page 289] termed LQG homing. Notice that if the constant is negative, then the optimizer tries to maximize the survival time of the process in the interval , taking the quadratic control costs into account. LQG homing problems have been treated by various authors; see Kuhn [2], Lefebvre [3], and Makasu [4]. Kuhn and Makasu used a risk-sensitive cost criterion (see also Whittle [5, page 222]).
In the general formulation given by Whittle, is an -dimensional process and the random variable is the moment of first entry of the joint variable into a stopping set . However, in practice, it is very difficult to obtain explicit solutions to problems in two or more dimensions (except in special instances). Moreover, in the papers published so far on homing problems, the hitting time was defined only in terms of . Here, we consider the case when the optimizer stops controlling the diffusion process at most at time .
Using a theorem in Whittle [1], we can state that the optimal control can be expressed as follows: where In the above formula, is a random variable defined by with and is the uncontrolled process that satisfies the stochastic differential equation That is, is the random variable that corresponds to for the diffusion process obtained by setting in (1).
Hence, the optimal control problem is reduced to the computation of the mathematical expectation . Actually, for this result to hold, we must have . However, in our case this condition is trivially satisfied because .
In Section 2, we will obtain an explicit solution in the case when , so that Notice that the uncontrolled process is then a Wiener process with drift and diffusion parameter . Furthermore, we will choose the constant . With this choice, the mathematical expectation simplifies to
2. Optimal Control of a Wiener Process
Let denote the expected value of the first-passage time . In the case of the Wiener process defined by the function satisfies the ordinary differential equation and is such that if . We find (see Lefebvre [6, page 220]) that Therefore, in the case when tends to infinity, the function is given by It follows from (5) that
Now, to obtain the expected value of the random variable , we can condition on : We may write that Moreover, because the conditional probability density function of , given that , is given by we have:
The function is known to be (see Lefebvre [6, page 219]) Making use of this formula, we can obtain an explicit expression for the mathematical expectation and hence, for the optimal control .
To illustrate the results, we computed (numerically) the optimal control when and . Looking at Figure 1, we see that the optimal control tends to zero much faster than as tends to . However, for close to the boundary at , the two functions are similar.
Next, to see the effect of the constant on the optimal control, we computed the value of when and varies from 0 to 20. This value is compared to in Figure 2. When decreases to 0, so does , as it should be. For (approximately), we have .
Finally, in Figure 3, we show the optimal control when and . Because (see above), when (approximately), it is very unlikely that the uncontrolled process will hit the boundary at before time . Therefore, the optimal control is close to 0. Notice that (for a finite value of ). Indeed, we can write that . Hence, we deduce from (17) that , which implies that , and thus .
3. Conclusion
We have considered LQG homing problems for which the optimizer controls the diffusion process in the time interval , where the random variable is smaller than or equal to a fixed constant . Moreover, the termination cost function was chosen so that the optimal control was expressed in terms of the expected value of a first-passage time for the corresponding uncontrolled process.
An application of this type of problem is the following: suppose that denotes the wear of a machine. The optimizer wants to maximize the lifetime of the machine. However, it is natural to assume that the machine will be replaced after a certain time, even if it is still in working order, because it might become obsolete.
To obtain a more realistic model for the wear of a device, we could use a degenerate two-dimensional diffusion process, as in Lefebvre [7]. The most difficult problem would then be to compute the probability density function of the first-passage time .