General LQG Homing Problems in One Dimension

Abstract

Optimal control problems for one-dimensional diffusion processes in the interval (𝑑₁,𝑑₂) are considered. The aim is either to maximize or to minimize the time spent by the controlled processes in (𝑑₁,𝑑₂). Exact solutions are obtained when the processes are symmetrical with respect to 𝑑^∗∈(𝑑₁,𝑑₂). Approximate solutions are derived in the asymmetrical case. The one-barrier cases are also treated. Examples are presented.

1. Introduction

Let {𝑋(𝑡),𝑡≥0} be a one-dimensional controlled diffusion process defined by the stochastic differential equation 𝑑𝑋(𝑡)=𝑚[𝑋(𝑡)]𝑑𝑡+𝑏₀𝑋^𝑘(𝑡)𝑢[𝑋(𝑡)]𝑑𝑡+{𝑣[𝑋(𝑡)]}^1/2𝑑𝐵(𝑡), (1.1) where 𝑢(⋅) is the control variable, 𝑚(⋅) and 𝑣(⋅)>0 are Borel measurable functions, 𝑏₀≠0 and 𝑘∈{0,1,…} are constants, and {𝐵(𝑡),𝑡≥0} is a standard Brownian motion. The set of admissible controls consists of Borel measurable functions.

Remark 1.1. We assume that the solution of (1.1) exists for all 𝑡∈[0,∞) and is weakly unique.

We define the first-passage time 𝑇(𝑥)=inf𝑡>0∶𝑋(𝑡)=𝑑₁or𝑑₂∣𝑋(0)=𝑥, (1.2) where 𝑥∈(𝑑₁,𝑑₂). We want to find the control 𝑢^∗ that minimizes the expected value of the cost function 𝐽(𝑥)=₀^𝑇(𝑥)12𝑞₀𝑢²[𝑋(𝑡)]+𝜆𝑑𝑡, (1.3) where 𝑞₀>0 and 𝜆≠0 are constants. Notice that if 𝜆 is negative, then the optimizer wants to maximize the survival time of the controlled process in the interval (𝑑₁,𝑑₂), taking the quadratic control costs into account. In general, there is a maximal value that the parameter 𝜆 can take. Otherwise, the expected reward becomes infinite.

When the relation [𝑋(𝑡=𝑏𝛼𝑣)]²⁰𝑞₀𝑋^2𝑘(𝑡) (1.4) holds for some positive constant 𝛼, using a theorem in Whittle [1, p. 289], we can express the value function 𝐹(𝑥)∶=inf_{𝑢[𝑋(𝑡(𝑥))],0≤𝑡≤𝑇}𝐸[𝐽(𝑥)] (1.5) in terms of a mathematical expectation for the uncontrolled process obtained by setting 𝑢[𝑋(𝑡)]≡0 in (1.1). Actually, for the result to hold, ultimate entry of the uncontrolled process into the stopping set must be certain, which is not a restrictive condition in the case of one-dimensional diffusion processes considered in finite intervals.

In practice, the theorem in Whittle [1] gives a transformation that enables us to linearize the differential equation satisfied by the function 𝐹(𝑥).

In Lefebvre [2], using symmetry, the author was able to obtain an explicit and exact expression for the optimal control 𝑢^∗ when {𝑋(𝑡),𝑡≥0} is a one-dimensional controlled standard Brownian motion process (so that 𝑚[𝑋(𝑡)]≡0 and 𝑣[𝑋(𝑡)]≡1), 𝑑₂=−𝑑₁=𝑑 and 𝑘=1. Notice that the relation in (1.4) does not hold in that case. The author assumed that the parameter 𝜆 in the cost function is negative, and he found the maximal value that this parameter can take.

Previously, Lefebvre [3] had computed the value of 𝑢^∗ when 𝑘=0, but with the cost function 𝐽₁(𝑥)=₀^𝑇(𝑥)12𝑞₀𝑋²(𝑡)𝑢²[𝑋(𝑡)]+𝜆𝑑𝑡 (1.6) rather than the function 𝐽(𝑥) defined above. We cannot appeal to the theorem in Whittle [1] in that case either. However, the author expressed the function 𝐹(𝑥) in terms of a mathematical expectation for an uncontrolled geometric Brownian motion.

In Section 2, we will generalize the results in Lefebvre [2] to one-dimensional diffusion processes for which the functions 𝑚[𝑋(𝑡)] and 𝑣[𝑋(𝑡)] are symmetrical with respect to zero and 𝑑₁=−𝑑₂. Important particular cases will be considered.

Next, in Section 3, we will treat the general symmetrical case when 𝑑₁ is not necessarily equal to −𝑑₂. In Section 4, we will consider processes for which the functions 𝑚[𝑋(𝑡)] and 𝑣[𝑋(𝑡)] are not symmetrical with respect to a certain 𝑑^∗∈(𝑑₁,𝑑₂). An approximate solution will then be proposed. In Section 5, we will present possible extensions, including the case of a single barrier. Finally, we will make some concluding remarks in Section 6.

2. Optimal Control in the Symmetrical Case with 𝑑₁=−𝑑₂

In this section, we take 𝑑₂=−𝑑₁=𝑑. Assuming that it exists and that it is twice differentiable, we find that the value function 𝐹(𝑥) defined in (1.5) satisfies the dynamic programming equation inf_𝑢(𝑥)12𝑞₀𝑢²(𝑥)+𝜆+𝑚(𝑥)+𝑏₀𝑥^𝑘𝑢(𝑥)𝐹^′(𝑥)+12𝑣(𝑥)𝐹^′′(𝑥)=0, (2.1) where 𝑥=𝑋(0). Differentiating with respect to 𝑢(𝑥) and equating to zero, we deduce that the optimal control 𝑢^∗(𝑥) can be expressed as follows: 𝑢^∗(𝑥)𝑏=−⁰𝑥^𝑘𝑞₀𝐹^′(𝑥). (2.2) Substituting the optimal control into the dynamic programming equation (2.1), we obtain that the function 𝐹(𝑥) satisfies the nonlinear second-order ordinary differential equation 𝜆+𝑚(𝑥)𝐹^′(𝑥)−𝑏²₀𝑥^2𝑘2𝑞₀𝐹^′(𝑥)²+12𝑣(𝑥)𝐹^′′(𝑥)=0, (2.3) subject to the boundary conditions 𝐹(𝑑)=𝐹(−𝑑)=0. (2.4)

Now, in general solving nonlinear second-order differential equations is not an easy task. As mentioned previously, when the relation in (1.4) holds, there exists a transformation that enables us to linearize (2.3). Notice, however, that in order to obtain an explicit expression for the optimal control 𝑢^∗(𝑥), one only needs the derivative of the value function 𝐹(𝑥). Hence, if we can find a boundary condition in terms of 𝐹′(𝑥), rather than the boundary conditions in (2.4), then we could significantly simplify our problem, since we would only have to solve the first-order nonlinear (Riccati) differential equation: (𝑥)𝐺(𝑥)−𝑏𝜆+𝑚²⁰𝑥^2𝑘2𝑞₀𝐺²(𝑥)+12𝑣(𝑥)𝐺^′(𝑥)=0, (2.5) where 𝐺(𝑥)∶=𝐹^′(𝑥). (2.6)

Proposition 2.1. Assume that the function 𝑚(𝑥) is odd and that the function 𝑣(𝑥) is even. Then the optimal control 𝑢^∗(𝑥) is given by 𝑢^∗(𝑥)𝑏=−⁰𝑥^𝑘𝑞₀𝐺(𝑥), (2.7) where 𝐺(𝑥) satisfies (2.5), subject to the condition 𝐺(0)=0. (2.8)

Proof. The condition (2.8) follows from the fact that, by symmetry, when the parameter 𝜆 is positive, then 0 is the value of 𝑥 for which the function 𝐹(𝑥) has a maximum, whereas 𝐹(𝑥) has a minimum at 𝑥=0 when 𝜆 is negative. Indeed, the origin is the worst (resp., best) position possible when the optimizer is trying to minimize (resp., maximize), taking the quadratic control costs into account, the time spent by 𝑋(𝑡) in the interval (−𝑑,𝑑).

Remarks 2.2. (i) The solution to (2.5), subject to (2.8), might not be unique.
(ii) Notice that the symmetrical case includes the one when 𝑚(𝑥) is identical to 0, and 𝑣(𝑥) is a constant, so that the uncontrolled process is a Wiener process with zero drift.

The previous proposition can be generalized as follows.

Corollary 2.3. If 𝑋^𝑘(𝑡) is replaced by ℎ[𝑋(𝑡)] in (1.1), where ℎ²(𝑥) is even, and if the hypotheses in Proposition 2.1 are satisfied, then the optimal control 𝑢^∗(𝑥) can be expressed as 𝑢^∗(𝑥)𝑏=−⁰ℎ(𝑥)𝑞₀𝐺(𝑥), (2.9) where 𝐺(𝑥) is a solution of (𝑥)𝐺(𝑥)−𝑏𝜆+𝑚²⁰ℎ²(𝑥)2𝑞₀𝐺²(𝑥)+12𝑣(𝑥)𝐺^′(𝑥)=0 (2.10) that satisfies the condition 𝐺(0)=0.

We will now present an example for which we can determine the optimal control 𝑢^∗ explicitly.

We consider the case when {𝑋(𝑡),𝑡≥0} is a controlled Bessel process, so that 𝑑𝑋(𝑡)=𝜃−12𝑋(𝑡)𝑑𝑡+𝑏₀𝑋^𝑘(𝑡)𝑢[𝑋(𝑡)]𝑑𝑡+𝑑𝐵(𝑡). (2.11) Moreover, we assume that the parameter 𝜃 belongs to the interval (0,2). The origin is then a regular boundary (see [4, p. 239]) for the uncontrolled process {𝑋₀(𝑡),𝑡≥0} obtained by setting 𝑢[𝑋(𝑡)]≡0 in the stochastic differential equation above. Notice that if the parameter 𝜃 is equal to 1, then {𝑋₀(𝑡),𝑡≥0} becomes a standard Brownian motion, which is the process considered in Lefebvre [2]. Therefore, this example generalizes the results in Lefebvre’s paper.

Here, the relation in (1.4) holds if there exists a positive constant 𝛼 such that 𝑏𝛼=²⁰𝑞₀𝑋^2𝑘(𝑡). (2.12) Hence, we can appeal to the theorem in Whittle [1] when 𝑘 is equal to zero. We will treat the case when 𝑘>0 instead.

The differential equation that we must solve is 𝜆+𝜃−1𝐺(𝑥)−𝑏2𝑥²⁰𝑥^2𝑘2𝑞₀𝐺²(𝑥)+12𝐺^′(𝑥)=0. (2.13) We find that 𝐺(𝑥)=−2𝜆𝑞₀𝑏₀𝑥^𝑘𝐽_𝜈𝑐𝑥^𝑘+1+𝑐₀𝑌_𝜈𝑐𝑥^𝑘+1𝐽_𝜈−1𝑐𝑥^𝑘+1+𝑐₀𝑌_𝜈−1𝑐𝑥^𝑘+1, (2.14) where 𝐽_𝜈 and 𝑌_𝜈 are Bessel functions and 𝑐₀ is an arbitrary constant, 𝜃𝜈∶=2(),𝑏𝑘+1𝑐∶=⁰−2𝜆/𝑞₀.𝑘+1 (2.15)

The expression above for the function 𝐺(𝑥) is appropriate when the parameter 𝜆 is negative. However, when 𝜆>0, it is better to rewrite it as follows: 𝐺(𝑥)=2𝜆𝑞₀𝑏₀𝑥^𝑘𝑐₀𝐾_𝜈𝑐^∗𝑥^𝑘+1−𝐼_𝜈𝑐^∗𝑥^𝑘+1𝑐₀𝐾_𝜈−1𝑐^∗𝑥^𝑘+1+𝐼_𝜈−1𝑐^∗𝑥^𝑘+1, (2.16) where 𝐼_𝜈 and 𝐾_𝜈 are modified Bessel functions and 𝑐^∗𝑏∶=⁰2𝜆/𝑞₀𝑘+1=𝑖𝑐. (2.17)

In order to determine the value of the constant 𝑐₀, we will use the condition 𝐺(0)=0. First, we consider the special case when 𝑘=1, 𝜃=1 and 𝜆 is negative. Then, we have that 𝜈=1/4 and 𝐺(𝑥)=−2𝜆𝑞₀𝑏₀𝑥𝐽_1/4𝑐𝑥²+𝑐₀𝑌_1/4𝑐𝑥²𝐽_−3/4𝑐𝑥²+𝑐₀𝑌_−3/4𝑐𝑥², (2.18) where 𝑐=𝑏₀−𝜆/(2𝑞₀).

Next, when 𝜈≠−1,−2,…, we have the formula (see [5, p. 358]) 𝑌_𝜈(𝑧)=𝐽_𝜈(𝑧)cos(𝜈𝜋)−𝐽_−𝜈(𝑧)().sin𝜈𝜋 (2.19) It follows, with 𝜈=1/4 and 𝜈=−3/4, that 𝐺(𝑥)=−2𝜆𝑞₀𝑏₀𝑥1+𝑐₀𝐽_1/4𝑐𝑥²−√2𝑐₀𝐽_−1/4𝑐𝑥²1+𝑐₀𝐽_−3/4𝑐𝑥²+√2𝑐₀𝐽_3/4𝑐𝑥². (2.20)

Finally, making use of the limiting form of the function 𝐽_𝜈(𝑧) when 𝑧→0 (see [5, p. 360]): 𝐽_𝜈(𝑧)∼𝑧2^𝜈1Γ()(),𝜈+1if𝜈≠−1,−2,… (2.21) we obtain that lim_𝑥→0𝐺(𝑥)=−−2𝜆𝑞₀𝑏₀Γ(1/4)Γ()𝑐3/4^1/2𝑐₀1+𝑐₀. (2.22) It follows that we must set the constant 𝑐₀ equal to 0, which implies that 𝐺(𝑥)=−2𝜆𝑞₀𝑏₀𝑥𝐽_1/4𝑐𝑥²𝐽_−3/4𝑐𝑥². (2.23) We then deduce from (2.7) (with 𝑘=1) that the optimal control is given by 𝑢^∗(𝑥)√=−−2𝜆√𝑞₀𝐽_1/4√−𝜆/2𝑞₀𝑏₀𝑥²𝐽_−3/4√−𝜆/2𝑞₀𝑏₀𝑥²for−𝑑<𝑥<𝑑. (2.24) This formula for the optimal control is the same as the one obtained by Lefebvre [2].

Now, in the general case, proceeding as previously we find that when 𝜆<0, the function 𝐺(𝑥) defined in (2.14) is such that when 𝑥 decreases to zero, 𝐺(𝑥)∼−2𝜆𝑞₀𝑏₀𝑥^𝑘×𝑐𝑥^𝑘+1/2^𝜈Γ(𝜈+1)1+𝑐₀cos(𝜈𝜋)()−𝑐sin𝜈𝜋sin(⁰𝜈𝜋)𝑐𝑥^𝑘+1/2^−𝜈Γ(−𝜈+1)𝑐𝑥^𝑘+1/2^𝜈−1Γ(𝜈)1+𝑐₀cos[(𝜈−1)𝜋])𝜋]−𝑐sin[(𝜈−1sin[(𝜈⁰−1)𝜋]𝑐𝑥^𝑘+1/2^{−(𝜈−1)}Γ().−𝜈+2 (2.25) This expression may be rewritten as follows: 𝐺(𝑥)∼−2𝜆𝑞₀𝑏₀𝑐₁𝑥^𝜃/2+𝑐₂𝑥^−𝜃/2𝑐₃𝑥^(𝜃/2)−1+𝑐₄𝑥^{−(𝜃/2)+2𝑘+1}, (2.26) where 𝑐_𝑖 is a constant, for 𝑖=1,2,3,4. Multiplying the numerator and the denominator by 𝑥^𝜃/2, we obtain that 𝐺(𝑥)∼−2𝜆𝑞₀𝑏₀𝑐₁𝑥^𝜃+𝑐₂𝑐₃𝑥^𝜃−1+𝑐₄𝑥^2𝑘+1. (2.27) Hence, we deduce that if 𝜃∈(1,2), we must set the constant 𝑐₂ equal to 0. This implies that 𝑐₀=0, so that the constant 𝑐₄ is equal to 0 as well. It follows that the function 𝐺(𝑥) is given by 𝐺(𝑥)=−2𝜆𝑞₀𝑏₀𝑥^𝑘𝐽_𝜈𝑐𝑥^𝑘+1𝐽_𝜈−1𝑐𝑥^𝑘+1. (2.28) This expression is valid as long as the denominator is positive. This is tantamount to saying that the parameter 𝜆, which represents the instantaneous reward given for survival in the interval (−𝑑,𝑑), must not be too large.

Finally, the optimal control is 𝑢^∗(𝑥)√=−−2𝜆√𝑞₀𝐽_𝜈𝑐𝑥^𝑘+1𝐽_𝜈−1𝑐𝑥^𝑘+1. (2.29)

Now, if 𝜃∈(0,1), it turns out that lim_𝑥→0𝐺(𝑥)=0 for any constant 𝑐₀, so that the solution is not unique. However, this does not entail that we can choose any 𝑐₀. For instance, in the particular case when 𝜃=1/2, 𝑘=1, 𝜆=−1, 𝑏₀=1 and 𝑞₀=1/2, we find that 𝐺(𝑥)=1𝑥𝐽_1/8𝑥²+𝑐₀𝑌_1/8𝑥²𝐽_−7/8𝑥²+𝑐₀𝑌_−7/8𝑥². (2.30) If we choose 𝑐₀=0, as in the case when 𝜃∈(1,2), then the expression for the optimal control is 𝑢^∗(𝑥)𝐽=−2^1/8𝑥²𝐽_−7/8𝑥². (2.31) One can check that if 𝑑=1/2, then 𝑢^∗(𝑥)<0 for 0<|𝑥|<1/2, which is logical because the optimizer wants to maximize the survival time in (−1/2,1/2). But if we let 𝑐₀ tend to infinity, the optimal control becomes 𝑢^∗(𝑥)𝑌=−2^1/8𝑥²𝑌_−7/8𝑥², (2.32) which is strictly positive for 0<|𝑥|<1/2. Thus, when the solution to (2.5), (2.8) is not unique, one must use other arguments to find the optimal control. One can obviously check whether the expression obtained for the optimal control does indeed correspond to a minimum (or a maximum in absolute value). In the particular case considered previously, if we let 𝐹(𝑥)=^𝑥_−𝑑1𝑦𝐽_1/8𝑦²𝐽_−7/8𝑦²𝑑𝑦, (2.33) we find that this function satisfies all the conditions of the optimal control problem set up in Section 1 and leads to a valid expression for the optimal control.

Next, if the parameter 𝜆 is positive, we deduce from the function 𝐺(𝑥) in (2.16) that the optimal control is given by 𝑢^∗(𝑥)√=−2𝜆√𝑞₀𝐼_𝜈𝑐^∗𝑥^𝑘+1𝐼_𝜈−1𝑐^∗𝑥^𝑘+1 (2.34) when 𝜃∈(1,2). However, when 𝜃∈(0,1), again we do not obtain a unique solution to (2.5) and (2.8).

Moreover, contrary to the case when 𝜆 is negative, there is no constraint on this parameter when it is positive. That is, we can give as large a penalty as we want for survival in the continuation region.

3. Optimal Control in the Symmetrical General Case

In this section, we assume that 𝑑₁ and 𝑑₂ are not necessary such that 𝑑₁=−𝑑₂. Moreover, we assume that there is a transformation 𝑌(𝑡)=𝑔[𝑋(𝑡)] of the stochastic process {𝑋(𝑡),𝑡≥0} such that the functions 𝑚[𝑌(𝑡)] and 𝑣[𝑌(𝑡)] are symmetrical with respect to zero. Then the optimal control problem is reduced to the one presented in the previous section.

A simple example of such a situation is the case when {𝑋(𝑡),𝑡≥0} is a one-dimensional controlled standard Brownian motion and 𝑑₁≠−𝑑₂. Then one can simply define 𝑌(𝑡)=𝑎𝑋(𝑡)+𝑏, (3.1) with 𝑎=2𝑑𝑑₂−𝑑₁𝑑𝑑,𝑏=−²+𝑑₁𝑑₂−𝑑₁ (3.2) to obtain a controlled Brownian motion with zero drift and variance parameter 𝜎²=𝑎² in the interval (−𝑑,𝑑). We can then apply Proposition 2.1 to find the optimal control.

A more interesting example is the following one: assume that the controlled process {𝑋(𝑡),𝑡≥0} is defined by (𝑡)=1𝑑𝑋2𝑋(𝑡)𝑑𝑡+𝑏₀[𝑋(𝑡)]^𝑘𝑢[𝑋(𝑡)]𝑑𝑡+𝑋(𝑡)𝑑𝐵(𝑡). (3.3) That is, {𝑋(𝑡),𝑡≥0} is a controlled geometric Brownian motion. Since this process is strictly positive, we cannot have 𝑑₁=−𝑑₂. Let us define 𝑇(𝑥)(𝑡)=1=inf𝑡>0∶𝑋𝑑(0)1or𝑑∣𝑋=𝑥for𝑥∈𝑑,𝑑, (3.4) where 𝑑>1.

Notice that the relation in (1.4) only holds in the case when 𝑘=1. To obtain the control that minimizes the expected value of the cost function defined in (1.3), we will transform the geometric Brownian motion process into a Wiener process by setting 𝑌(𝑡)=ln[𝑋(𝑡)]. (3.5)

The infinitesimal parameters of the process {𝑌(𝑡),𝑡≥0} are given by (see, e.g., [6, p. 64]) 𝑚_𝑌𝑦=12𝑥+𝑏₀𝑥^𝑘𝑢1𝑥+12𝑥²−1𝑥²=𝑏₀𝑥^𝑘−1𝑢=𝑏₀𝑒^{(𝑘−1)𝑦}𝑣𝑢,_𝑌𝑦=𝑥²1𝑥²≡1. (3.6)

Hence, we can write that 𝑌(𝑡) satisfies the stochastic differential equation 𝑑𝑌(𝑡)=𝑏₀𝑒^{(𝑘−1)𝑌(𝑡)}𝑢[𝑌(𝑡)]𝑑𝑡+𝑑𝐵(𝑡). (3.7) That is, {𝑌(𝑡),𝑡≥0} is a controlled standard Brownian motion. Moreover, the random variable 𝑇(𝑥) becomes 𝑇_𝑌𝑦=inf𝑡>0∶|𝑌(𝑡)|=ln(𝑑)∣𝑌(0)=𝑦, (3.8) where 𝑦∈(−ln(𝑑),ln(𝑑)).

We can find the function 𝐺(𝑦), from which the optimal control 𝑢^∗ is obtained at once, for any choice of 𝑘∈{0,1,…}. We will present the solution in the case when 𝑘=0. Furthermore, we let 𝜆=−1, 𝑏₀=1 and 𝑞₀=1/2. We then must solve the nonlinear ordinary differential equation 12𝐺^′𝑦−𝑒^−2𝑦𝐺²𝑦−1=0. (3.9) The solution that satisfies the condition 𝐺(0)=0 is 𝐺𝑦=𝑒^𝑦𝑌₀(2)𝐽₀2𝑒^−𝑦−𝐽₀(2)𝑌₀2𝑒^−𝑦𝑌₀(2)𝐽₁(2𝑒^−𝑦)−𝐽₀(2)𝑌₁(2𝑒^−𝑦). (3.10) Hence, from Corollary 2.3, we can state that the optimal control is given by 𝑢^∗=−2𝑒^−𝑦𝐺(𝑦). In terms of the original process, we have that 𝑢^∗(𝑥)𝑌=−2⁰(2)𝐽₀(2/𝑥)−𝐽₀(2)𝑌₀(2/𝑥)𝑌₀(2)𝐽₁(2/𝑥)−𝐽₀(2)𝑌₁().2/𝑥 (3.11) Since 𝜆<0, this solution is only valid as long as it remains finite. That is, because we chose the value of 𝜆, the constant 𝑑 must not be too large. The optimal control is plotted in Figure 1 when 𝑑=3/2. Notice that 𝑢^∗(𝑥) is positive when 𝑥<1 and negative when 𝑥>1, which is logical because the optimizer wants to maximize the survival time in the interval (2/3,3/2). However, the optimal control is not symmetrical with respect to 1.

Figure 1 

Optimal control when 𝑘=0, 𝜆=−1, 𝑏0=1, 𝑞0=1/2 and 𝑑=3/2.

4. Approximate Optimal Control in the Asymmetrical Case

We will now consider the case when the infinitesimal parameters of the controlled process {𝑋(𝑡),𝑡≥0} do not satisfy the hypotheses in Proposition 2.1. In order to obtain the optimal control without having to find the function 𝐹(𝑥) explicitly, we need a condition on 𝐺(𝑥). If we could determine the value 𝑥₀ of 𝑥 in the interval (𝑑₁,𝑑₂) for which the function 𝐺(𝑥) has a maximum or a minimum, then we would set 𝐺(𝑥₀)=0.

An approximate solution can be obtained by finding the value of 𝑥 that maximizes the expected value of the time it takes the uncontrolled process that corresponds to {𝑋(𝑡),𝑡≥0} to leave the interval (𝑑₁,𝑑₂). Let 𝑒(𝑥) denote this expected value. This function satisfies the ordinary differential equation (see [6, p. 220]) 12𝑣(𝑥)𝑒′′(𝑥)+𝑚(𝑥)𝑒′(𝑥)=−1. (4.1) The boundary conditions are obviously 𝑒𝑑₁=𝑒𝑑₂=0. (4.2)

We can state the following proposition.

Proposition 4.1. Let 𝑥₀ be the value of 𝑥 that maximizes the function 𝑒(𝑥) defined previously. The optimal control 𝑢^∗(𝑥) is approximately given by (2.7), where the function 𝐺(𝑥) satisfies (2.5), subject to the condition 𝐺(𝑥₀)=0.

To illustrate this result, we will present an example for which we can find the exact optimal control. We will then be able to assess the quality of the approximation proposed previously.

Let {𝑋(𝑡),𝑡≥0} be the controlled Wiener process with drift 𝜇≠0 and variance parameter 𝜎² defined by 𝑑𝑋(𝑡)=𝜇𝑑𝑡+𝑏₀𝑢[𝑋(𝑡)]𝑑𝑡+𝜎𝑑𝐵(𝑡). (4.3) Because the relation in (1.4) holds with 𝑏𝛼=²⁰𝑞₀𝜎²>0, (4.4) and ultimate entry of the uncontrolled process into the set {𝑑₁,𝑑₂} is certain, we can indeed appeal to Whittle’s theorem to obtain the control that minimizes the expected value of the cost function 𝐽(𝑥) defined in (1.3).

Assume that the parameter 𝜆 is positive, so that the optimizer wants 𝑋(𝑡) to leave the interval (𝑑₁,𝑑₂) as soon as possible. We deduce from Whittle’s theorem that the value function 𝐹(𝑥) can be expressed as follows: 𝐹(𝑥)1=−𝛼ln[𝑀(𝑥)], (4.5) where 𝑀(𝑥)∶=𝐸𝑒^{−𝛼𝜆𝜏(𝑥)}, (4.6) in which 𝜏(𝑥) is the same as the random variable 𝑇(𝑥) in (1.2), but for the uncontrolled process {𝜉(𝑡),𝑡≥0} defined by 𝑑𝜉(𝑡)=𝜇𝑑𝑡+𝜎𝑑𝐵(𝑡). (4.7)

It is a simple matter to find that 𝑀(𝑥)∝𝑓₋𝑑₁−𝑓₋𝑑₂+𝑓₊𝑑₁−𝑓₊𝑑₂, (4.8) where 𝑓_±𝑑_𝑖−1∶=exp𝜎²𝑥+𝑑_𝑖𝜇±𝑥−𝑑_𝑖Δ (4.9) for 𝑖=1,2, and Δ∶=𝜇²+2𝛼𝜆𝜎^21/2. (4.10) We then obtain that the exact optimal control is given by 𝑢^∗(𝑥)𝜇=−𝑏₀−Δ𝑏₀𝑓₋𝑑₁−𝑓₋𝑑₂+𝑓₊𝑑₁−𝑓₊𝑑₂𝑓₋𝑑₂−𝑓₋𝑑₁+𝑓₊𝑑₁−𝑓₊𝑑₂. (4.11)

Now, the function 𝑒(𝑥)∶=𝐸[𝜏(𝑥)] satisfies the ordinary differential equation 𝜎²2𝑒^′′(𝑥)+𝜇𝑒^′(𝑥)=−1. (4.12) The unique solution for which 𝑒(𝑑₁)=𝑒(𝑑₂)=0 is 𝑒(𝑥)∝𝑑₂−𝑑₁𝑒^{−2𝜇𝑥/𝜎2}−𝑥−𝑑₁𝑒^{−2𝜇𝑑2/𝜎2}−𝑑₂−𝑥𝑒^{−2𝜇𝑑1/𝜎2}. (4.13) The value of 𝑥 that maximizes 𝑒(𝑥) is obtained by differentiation: 𝑒^′(𝑥)𝜎=0⟺𝑥=−²𝜎2𝜇ln²𝑒2𝜇^{−2𝜇𝑑2/𝜎2}−𝑒^{−2𝜇𝑑1/𝜎2}𝑑₂−𝑑₁∶=𝑥₀. (4.14)

Next, from (2.2), the optimal control is given by 𝑢^∗(𝑥)𝑏=−⁰𝑞₀𝐹^′(𝑥), (4.15) where 𝐺(𝑥)∶=𝐹^′(𝑥) satisfies the nonlinear differential equation 1𝜆−2𝑏²₀𝑞₀𝐺²(𝑥)(𝑥)+𝜎+𝜇𝐺²2𝐺^′(𝑥)=0. (4.16) We find that the solution of this equation is the following: 𝐺(𝑥)=𝜇𝑞₀𝑏²₀−2𝜆𝑞₀𝑏²₀+𝜇²𝑞²₀𝑏²₀tanh2𝜆𝑞₀𝑏²₀+𝜇²𝑞²₀𝜎²𝑞₀𝑥+𝑐₁. (4.17) The constant 𝑐₁ is uniquely determined from the condition 𝐺(𝑥₀)=0. We have that 𝐺(𝑥)≃𝜇𝑞₀𝑏²₀−2𝜆𝑞₀𝑏²₀+𝜇²𝑞²₀𝑏²₀×tanh2𝜆𝑞₀𝑏²₀+𝜇²𝑞²₀𝜎²𝑞₀𝑥−𝑥₀+arctanh𝜇𝑞₀2𝜆𝑞₀𝑏²₀+𝜇²𝑞²₀. (4.18)

The expression that we obtain for the approximate optimal control by multiplying the function 𝐺(𝑥) by −𝑏₀/𝑞₀ is quite different from the exact optimal control. To compare the two solutions, we consider the special case when 𝜇=𝜎=𝑏₀=𝑞₀=𝜆=1, 𝑑₁=0 and 𝑑₂=2. We then find that the constant Δ is equal to √3, and the value that maximizes 𝐺(𝑥) is approximately 𝑥₀≃ 0,7024. We plotted the two controls in Figure 2. Notice how close the two curves are.

Figure 2 

Exact (solid line) and approximate (dotted line) optimal controls when 𝜇=𝜎=𝑏0=𝑞0=𝜆=1, 𝑑1=0, and 𝑑2=2.

5. Extensions

To complete this work, we will consider two possible extensions of the results presented. First, suppose that the random variable 𝑇(𝑥) defined in (1.2) is replaced by 𝑇_𝑑(𝑥)=inf{𝑡>0∶𝑋(𝑡)=𝑑∣𝑋(0)=𝑥}. (5.1) That is, we want to solve a one-barrier, rather than a two-barrier problem. To do so, we can introduce a second barrier, at 𝑥=𝑑^∗. In general, it will be necessary to find a transformation 𝑌(𝑡)=𝑔[𝑋(𝑡)] for which the infinitesimal parameters of the uncontrolled process {𝑌₀(𝑡),𝑡≥0} that corresponds to {𝑌(𝑡),𝑡≥0} satisfy the hypotheses in Proposition 2.1. If we can find such a transformation, then we can try to obtain the optimal control 𝑢^∗(𝑦) for the transformed process. Finally, we must express the optimal control in terms of the original variable 𝑥 and take the limit as 𝑑^∗ tends to ∞ (resp., −∞) if 𝑑^∗>𝑑 (resp., 𝑑^∗<𝑑).

Remark 5.1. If there is a natural boundary at the origin, for example, and if 𝑑^∗<𝑑, then we would take the limit as 𝑑^∗ decreases to zero.

We will now present an example where the technique described previously is used. We consider the controlled geometric Brownian motion defined in Section 3 by (𝑡)=1𝑑𝑋2𝑋(𝑡)𝑑𝑡+𝑏₀[𝑋(𝑡)]^𝑘𝑢[𝑋(𝑡)]𝑑𝑡+𝑋(𝑡)𝑑𝐵(𝑡), (5.2) and we assume that 𝑋(0)=𝑥>𝑑∈(0,1) and that 𝑏₀>0. Let 𝑋_𝑎(𝑡)=𝑎𝑋(𝑡), (5.3) where 1𝑎∶=√𝑑𝑑^∗. (5.4) Notice that the boundaries 𝑥=𝑑 and 𝑥=𝑑^∗>𝑑 become, respectively: 𝑑𝑑^∗1∶=𝛿,𝑑^∗𝑑=𝛿. (5.5)

Next, we set 𝑌(𝑡)=ln[𝑋_𝑎(𝑡)]. We then find that {𝑌(𝑡),𝑡≥0} is a controlled standard Brownian motion, and the first-passage time 𝑇_𝑑,𝑑^∗(𝑥)∶=inf𝑡>0∶𝑋(𝑡)=𝑑or𝑑^∗∣𝑋(0)=𝑥 (5.6) becomes 𝑇_𝛿𝑦∶=inf𝑡>0∶|𝑌(𝑡)|=ln(𝛿)∣𝑌(0)=𝑦. (5.7) Hence, we can appeal to Proposition 2.1 to determine the optimal value of the control 𝑢(𝑦).

Assume that 𝑘=1 in (5.2). Then Whittle’s theorem applies with 𝑏𝛼=²⁰𝑞₀>0. (5.8) The optimal control is given by 𝑢^∗(𝑥)𝑏=−⁰𝑞₀𝑥𝐹^′(𝑥), (5.9) and, as in Section 4, the value function 𝐹(𝑥) can be expressed as 𝐹(𝑥)1=−𝛼ln[𝑀(𝑥)], (5.10) where 𝑀(𝑥)∶=𝐸𝑒^{−𝛼𝜆𝜏𝑑(𝑥)}, (5.11) in which 𝜏_𝑑(𝑥) is the same as 𝑇_𝑑(𝑥) for the uncontrolled process that corresponds to {𝑋(𝑡),𝑡≥0}.

The function 𝑀(𝑥) satisfies the second-order ordinary differential equation 12𝑥²𝑀^′′(𝑥)+12𝑥𝑀^′(𝑥)=𝛼𝜆𝑀(𝑥). (5.12) The general solution of this equation can be written as 𝑀(𝑥)=𝑐₁𝑥^√2𝛼𝜆+𝑐₂𝑥^{−√2𝛼𝜆}. (5.13) We assume that the parameter 𝜆 is positive. Then, we can write that lim_𝑥→∞𝑀(𝑥)=0. (5.14) It follows that we must choose the constant 𝑐₁=0 in the general solution. Finally, making use of the boundary condition 𝑀(𝑑)=1, we obtain that 𝑀(𝑥)=𝑑𝑥^√2𝛼𝜆, (5.15) from which we deduce that the optimal control is constant: 𝑢^∗(𝑥)√≡−2𝜆√𝑞₀. (5.16)

If we do not appeal to Whittle’s theorem, we must solve the nonlinear first-order differential equation (see Section 3) 𝑏𝜆−²⁰2𝑞₀𝐺²𝑦+12𝐺^′𝑦=0, (5.17) subject to the boundary condition 𝐺(0)=0. We find that 𝐺𝑦=−2𝜆𝑞₀𝑏₀√tanh2𝜆𝑏₀𝑦√𝑞₀. (5.18) It follows that 𝑢^∗𝑦𝑏=−⁰𝑞₀𝐺𝑦=√2𝜆√𝑞₀√tanh2𝜆𝑏₀𝑦√𝑞₀. (5.19)

In terms of the original variable 𝑥=𝑒^𝑦/𝑎=√𝑑𝑑^∗𝑒^𝑦, we can write that 𝑢^∗(𝑥)=√2𝜆√𝑞₀√tanh2𝜆𝑏₀ln𝑥/√𝑑𝑑^∗√𝑞₀. (5.20) Since lim_𝑐→∞tanh𝑐₀𝑥ln√𝑐=−1 (5.21) for any positive constant 𝑐₀, we obtain that lim_{𝑑∗→∞}𝑢^∗(𝑥)√=−2𝜆√𝑞₀. (5.22) Thus, we retrieve the formula for the optimal control.

Next, we will treat the case when 𝑘=0 in (5.2), so that Whittle’s theorem does not apply. The optimal control becomes 𝑢^∗(𝑥)𝑏=−⁰𝑞₀𝐺(𝑥). (5.23)

In terms of the transformed variable 𝑦, we have that 𝑢^∗𝑦𝑏=−⁰𝑞₀𝑒^−𝑦𝐺𝑦, (5.24) where 𝐺(𝑦) is a solution of (see Section 3) 12𝐺^′𝑦−𝑏²₀2𝑞₀𝑒^−2𝑦𝐺²𝑦+𝜆=0. (5.25) When 𝜆 is positive, the solution that satisfies the condition 𝐺(0)=0 is 𝐺𝑦𝑞=−⁰𝑏²₀𝜅𝑒^𝑦𝐼₀(𝜅)𝐾₀𝜅𝑒^−𝑦−𝐾₀(𝜅)𝐼₀𝜅𝑒^−𝑦𝐼₀(𝜅)𝐾₁(𝜅𝑒^−𝑦)+𝐾₀(𝜅)𝐼₁(𝜅𝑒^−𝑦), (5.26) where √𝜅∶=2𝜆𝑏₀√𝑞₀, (5.27) from which we deduce that 𝑢^∗𝑦=𝜅𝑏₀𝐼₀(𝜅)𝐾₀𝜅𝑒^−𝑦−𝐾₀(𝜅)𝐼₀𝜅𝑒^−𝑦𝐼₀(𝜅)𝐾₁(𝜅𝑒^−𝑦)+𝐾₀(𝜅)𝐼₁(𝜅𝑒^−𝑦). (5.28) It follows that 𝑢^∗(𝑥)=𝜅𝑏₀𝐼₀(𝜅)𝐾₀𝜅√𝑑𝑑^∗/𝑥−𝐾₀(𝜅)𝐼₀𝜅√𝑑𝑑^∗/𝑥𝐼₀(𝜅)𝐾₁𝜅√𝑑𝑑^∗/𝑥+𝐾₀(𝜅)𝐼₁𝜅√𝑑𝑑^∗/𝑥. (5.29) Finally, using the asymptotic expansions for large arguments of the functions 𝐼_𝜈(𝑧) and 𝐾_𝜈(𝑧) (see [5, p. 377]), we find that lim_{𝑑∗→∞}𝑢^∗(𝑥)𝜅≡−𝑏₀√=−2𝜆√𝑞₀, (5.30) which is the same optimal control as in the case when 𝑘=1.

Remarks 5.2. (i) If we take the limit as 𝑑^∗ decreases to zero in 𝑢^∗(𝑥) instead, then making use of the formulas (see [5, p. 375]) 𝐼₀(𝑧)∼1,𝐼₁(𝑧)∼𝑧2,𝐾₀(𝑧)∼−ln(𝑧),𝐾₁(𝑧)∼1𝑧as𝑧↓0, (5.31) we obtain that lim_𝑑∗↓0𝑢^∗(𝑥)≡0. (5.32) (ii) We can also try to solve the differential equation 𝑥²2𝐺^′(𝑥)+𝑥2𝐺(𝑥)−𝑏²₀2𝑞₀𝐺²(𝑥)+𝜆=0 (5.33) satisfied by the function 𝐺(𝑥) directly. However, the solution that we are looking for must be such that 𝐺(√𝑑𝑑^∗)=0, because 𝑥=√𝑑𝑑^∗ is the value of the original variable 𝑥 that corresponds to 𝑦=0.

Now, in Corollary 2.3 we mentioned that Proposition 2.1 could be generalized by replacing 𝑋^𝑘(𝑡) by ℎ[𝑋(𝑡)] in (1.1). Another extension of Proposition 2.1 is to generalize the cost function 𝐽(𝑥) defined in (1.3) to 𝐽(𝑥)=₀^𝑇(𝑥)12𝑞[𝑋(𝑡)]𝑢²[𝑋(𝑡)]+𝜆𝑑𝑡, (5.34) where the function 𝑞(⋅)≥0 is even.

To illustrate this result, we consider a particular controlled Ornstein-Uhlenbeck process defined by 𝑑𝑋(𝑡)=−𝑋(𝑡)𝑑𝑡+𝑢[𝑋(𝑡)]𝑑𝑡+𝑑𝐵(𝑡), (5.35) and we take 𝐽(𝑥)=₀^𝑇(𝑥)𝑢²[𝑋(𝑡)]|𝑋(𝑡)|+1+1𝑑𝑡, (5.36) in which 𝑇(𝑥)=inf{𝑡>0∶|𝑋(𝑡)|=2∣𝑋(0)=𝑥∈(−2,2)}. (5.37) Then, we find that the optimal control is given by 𝑢^∗(𝑥)1=−2(|𝑥|+1)𝐺(𝑥), (5.38) and that the function 𝐺(𝑥) satisfies the nonlinear differential equation 1−𝑥𝐺(𝑥)−(|𝑥|+1)𝐺²(𝑥)+12𝐺^′(𝑥)=0, (5.39) subject to the condition 𝐺(0)=0.

Next, by symmetry, we can write that 𝐺(−𝑥)=−𝐺(𝑥) (and that 𝑢^∗(−𝑥)=−𝑢^∗(𝑥)). Hence, we can restrict ourselves to the interval [0,𝑑]. The solution of the differential equation that is such that 𝐺(0)=0 is 𝐺(𝑥)=√𝜋[erf(𝑥+2)−erf(2)]√𝜋[erf(2)−erf(𝑥+2)]−𝑒^−(𝑥+2)2, (5.40) where “erf” is the error function. It follows that the optimal control is given by 𝑢^∗(𝑥)=12()√𝑥+1𝜋[erf(2)−erf(𝑥+2)]√𝜋[erf(2)−erf(𝑥+2)]−𝑒^−(𝑥+2)2 (5.41) for 0≤𝑥≤2. This function is plotted in Figure 3.

Figure 3 

Optimal control in the interval [0,2].

6. Conclusion

We have shown that when the LQG homing problem that we want to solve possesses a certain symmetry, then it is not necessary to obtain the value function 𝐹(𝑥) explicitly; only the derivative of 𝐹(𝑥) is needed to determine the optimal control. Using this result, we were able to solve various problems for which Whittle’s theorem does not apply. In Section 4, we proposed an approximate solution in the case when the infinitesimal parameters of the controlled processes are not symmetrical with respect to the origin.

Many papers have been written on LQG homing problems, in particular by the first author (see, e.g., [7]) and recently by Makasu [8]. In most cases, the problems considered were only for one-dimensional processes, because to apply Whittle’s theorem a certain relation must hold between the noise and control terms. This relation is generally not verified in two or more dimensions. Furthermore, even if the relation in question holds, we still must solve a nontrivial probability problem. More precisely, we need to evaluate the moment-generating function of a first-passage time. To do so, we must find the solution of a Kolmogorov backward equation that satisfies the appropriate boundary conditions.

Proceeding as we did in this paper, we could simplify, at least in the symmetrical case, the differential equation problem, even in more than one dimension. Therefore, we should be able to solve more realistic problems. Such problems will also have interesting applications.

Finally, in order to be able to treat real-life applications, we should try to find a way to solve problems that are not symmetrical and for which Whittle’s theorem does not apply. This could be achieved by finding a transformation that linearizes the differential equations that we need to solve.

Acknowledgments

The authors are very grateful to the anonymous reviewer whose constructive comments helped to improve their paper.