A General Setting and Solution of Bellman Equation in Monetary Theory

Abstract

As an important tool in theoretical economics, Bellman equation is very powerful in solving optimization problems of discrete time and is frequently used in monetary theory. Because there is not a general method to solve this problem in monetary theory, it is hard to grasp the setting and solution of Bellman equation and easy to reach wrong conclusions. In this paper, we discuss the rules and problems that should be paid attention to when incorporating money into general equilibrium models. A general setting and solution of Bellman equation in monetary theory are provided. The proposed method is clear, is easy to grasp, is generalized, and always leads to the correct results.

1. Introduction

In recent years, many economists applied business cycle approaches to macroeconomic modeling so that monetary factors could be modeled into dynamic general equilibrium models. As an important method of monetary economics modeling, infinite horizon representative-agent models provide a close link between theory and practice; its research framework can guide practice behavior and be tested by actual data. This method can link monetary economics and other popular models for studying business cycle phenomena closely. There are three basic monetary economics approaches introducing money to economic general equilibrium models in the infinite horizon representative-agent framework. First, it is assumed that utility could be yielded by money directly so that the money variable has been incorporated into utility function of the representative-agent models [1]. Second, assuming asset exchanges are costly, transaction costs of some form give rise to a money demand [2, 3]. Clower [4] considers that money is used for some types of transactions. Brock [5], McCallum and Goodfriend [6], and Croushore [7] assume that time and money can be combined to yield transaction services. Third, money is treated as an asset that can be used to transfer resources between generations [8].

Dynamic optimization is the main involved issue during the modeling process. It is represented and solved by Bellman equation method, namely, the value function method. The method will obtain a forward-looking household’s path to maximize lifetime utility through the optimal behavior and further relevant conclusions. The setting of Bellman equation is the first and crucial step to solve dynamic programming problems. It is hard to grasp the setting and solution of Bellman equation and easy to reach wrong conclusions since there is not a general method to set Bellman equation or the settings of Bellman equation are excessively flexible. Walsh [9] used Bellman equation to set and solve dynamic general equilibrium models of money. However he does not give a general law of Bellman equation setting. The setting and solution of the equation in his book are ambiguous and not clear. To the best of the authors’ knowledge, there have been no studies of uniform setting of Bellman equation.

In this paper, we provide a set of general setting and solution methods for Bellman equation with multipliers. It is very clear and easy to grasp. The most important thing is that the proposed method can always lead to correct results. We apply our method to monetary general equilibrium models that are in the framework of the first two basic monetary economic approaches which incorporate money into economic general equilibrium model and make some extensions. At the same time, we compare our method with other current methods of setting and solution for Bellman equation to display the clarity and correctness of the proposed method. It is assumed that all the relevant assumptions of applying Bellman equation are satisfied. This is to ensure the feasibility of analysis and solution. For the convenience of discussion and suitable length of the paper, we mainly discuss the certainty linear programming problems. The Bellman equation’s setting and solution of uncertainty problems are similar to those with certainty problems essentially.

2. A New Method of Setting and Solution of Bellman Equation

First of all, we provide the theoretical details of the new general method and steps when applying Bellman equation to solve problems. It has many advantages. The relationship of the results is very clear through the connection of multipliers. There are no tedious expansions during the derivations. It differs from the expansion method because it does not need to consider which control variable should be replaced. The technical details about the equivalence of Bellman equations and dynamic programming problems and the solvability of set problems can be found in [10]. Actually, it is easy to reach wrong conclusions using other settings and solutions of Bellman equation with multipliers, or it has to find some particular skills to get the right results. But our method does not meet the problem. The proposed method can be described as the following five steps.

Step 1. List the expression of target problem.
Consider the dynamic programming problem as where the objective function (1) represents the maximum sum of present value of a forward-looking behavioral agent’s every period objective function . In monetary theory, usually represents the utility function of a family and is a subjective rate of discount. Equation (2) is constraints; are dynamic constraint functions and usually are intertemporal budget constraints. Let control variables ; the remaining variables are state variables.

Step 2. Set up Bellman equation with multipliers to express dynamic optimization problem in Step 1: where is the value function and is the multiplier of the th constraint , .

Step 3. Compute the partial derivatives of all control variables on the right side of the equation at Step 2 to derive first-order conditions:
During the derivation, it should be taken into account that next period state variables can be represented by other control variables according to the constraints, that is, to expand . Because the constraints are dynamic and are usually intertemporal budget constraints, those control variables at time can often be represented as expressions of the previous period control variables. Attention should be paid to compute the derivatives of control variables hiding in . But there is no need to expand after the multipliers during the derivation. Other than that, no particular attention should be paid to other current skills; for example, some control variables should be replaced after represented by other control variables in and so on.

Step 4. By the envelope theorem, take the partial derivatives of control variables at time on both sides of Bellman equation to derive the remaining first-order conditions:
Still, it does not need to expand after the multipliers during the derivations. Requirements are the same as Step 3.

Step 5. Obtain the new relevant results about target problem through recursion and substitution according to the above results.
We could combine several state variables to one as you need according to specific economic significance and constraints when there is no need to describe the relevant economic significance of state variables one by one. For example, , could be combined as . is a state variable in value function , that is, . And then we follow the above five steps to solve specific problem. The rest of state variables could be operated similarly. But combinations should not be overlapped over state or combined state variables. For instance, suppose that , have been combined as . It is inflexible to take as an independent state variable in value function or put the combination of , , in value function as a state variable.

The steps above give a general setting and solution of Bellman equation. These can be summarized as follows: first, set Bellman equation with multipliers of target dynamic optimization problem under the requirement of no overlaps of state variables; second, extend the late period state variables in on the right side of Bellman equation and there is no need to expand these variables after the multipliers; third, let the derivatives of state variables of time equal zero and take the partial derivatives of these variables on both sides of Bellman equation to derive first-order conditions; finally, get more needed results for analysis from these conditions.

Different from some current settings which allow overlap of state variables in value function, our method does not permit overlaps. In fact, overlap of state variable is easy to reach wrong conclusions or it has to find some particular skills to get the right results [9]. These methods are hard to generalize. More details will be seen in the following sections.

Now we will take several discrete time dynamic optimization problems that are under framework of the basic two methods of incorporating money into general equilibrium models as examples to show the applications of our setting and solution method and compare with current popular methods.

3. Setting and Solution of Bellman Equation in Basic Money-In-Utility Model

3.1. Model and Solution Using Previous Approach

The basic Money-In-Utility model has few features. The labor-leisure options of families are ignored temporarily in utility function. Only family consumption and real money balances are involved in utility function; that is, real money balances yield utility directly. We ignore the uncertainty of currency impact and technological changes temporarily for convenience.

The total present utility value of family life cycle is . Usually the per capita version of intertemporal budget constraint is where , , , , , , and are the per capita consumption, bonds, stock of capital, money balances, net lump-sum transfer received from the government, nominal interest rate, and inflation rate, respectively, at time . is the rate of depreciation of physical capital, is the population growth rate (assumed to be constant), and is the production function assumed to be continuously differentiable and to satisfy the usual Inada conditions (see [11]). Note that control variables can be changed and affect the total discounted value of utility through consumption in period ; state variables cannot be changed. Determination of state and control variables is crucial to derive the first-order conditions and other results. The control variables for this problem are , , and, and the state variables are , , , , and , the household’s initial level of resources.

This problem can be solved by the expansion method below according to Walsh [9]. It is started with setting up value function

Using (6), value function can be expanded as

Replace as ; according to (6), (8) can be written as an expansion equation as follows:

Compute the partial derivatives of control variables to derive first-order conditions: and the envelope theorem yields

Another expansion method will replace as ; according to (6), (8) can be written as

Compute the partial derivatives of control variables to derive first-order conditions as

Using the envelope theorem and computing the derivative with respect to state variable , we get

3.2. Our Solving Approach

Now, we use our proposed steps of setting and solution of Bellman equation to solve the above basic Money-In-Utility problem. First, let the Bellman equation with multiplier be

Second, computing the partial derivatives for the control variables, we obtain the first-order conditions as

Third, using the envelope theorem and computing the derivatives of both sides of Bellman equation with respect to state variable , we get

Finally, based on the above results, it is easy to get

This expression indicates that the marginal benefit of increased money holdings should be equal to the marginal utility of consumption on period . Similarly, we can easily obtain the allocation of initial level of resources among consumption, capital, bonds, and money balances, and the same marginal benefit must be yielded by each use at an optimum allocation.

These results with multiplier are open-and-shut, and it is easy to find the economic significance of marginal utility of consumption. Comparing with current approach mentioned above, during the process of solving problem, there is no need to consider which variable should be replaced. There are no tedious expanded expressions. Using expansion method, if we replace , the derived first-order conditions are seen to be messy and it is not easy to find the relationships of results. If we replace , the derived results are a bit more clear, but we might not think of replacing at the beginning. So expansion methods are not easy to operate, and it is not clear to replace variable.

4. Setting and Solution of Bellman Equation in Shopping-Time Model

4.1. Model and Solution Using Previous Approach

In Shopping-Time Model, shopping time is a function of consumption and money balances. Because consumption needs shopping time, leisure is reduced. Household utility is assumed to depend on consumption and leisure. Consumption can not only yield utility directly but also decrease utility indirectly. In this section, is time spent in market employment and is time spent shopping. is the function of consumption and money balances; that is, , , . Total time available is normalized to equal 1. Growth rate of population is assumed to be 0 for convenience. Let be the leisure time. The utility function is . It donates utility as a function of consumption, labor supply, and money holdings.

The household’s intertemporal objective is maximum discounted utility subject to resource constraint as

Because controllable factors, labor supply, and consumption affect labor supply, output is affected not only by state variable but also by these controllable factors. The left side of the budget constraint is unsuitable to be combined as a state variable . Combining the money holdings, bonds, and transfers to , the household’s financial assets, value function can be written as

According to Walsh [9], after replacing , , value function can be expanded as

Computing the partial derivatives with respect to control variables, we get

Computing the partial derivatives of both sides of the value equation with respect to the state variables , , we get

4.2. Our Solving Approach

Now, we use our proposed method to solve the above Shopping-Time Model problem. First, let the Bellman equation with multiplier be where state variable . Then, compute the partial derivatives with respect to control variables to derive first-order conditions,

Using the envelope theorem and computing the derivatives with respect to the state variables , , we get

Now, we use the above results to compute the opportunity cost of holding money. Since the utility function is and based on (6), (25), and (26), we obtain

Obviously, the expressions of the derived first-order conditions by previous method seem to be tedious and messy, and it is not so easy to compute the relevant results such as the opportunity cost of holding money. Our proposed method is comparatively neat and can easily obtain relevant results correctly.

5. Setting and Solution of Bellman Equation in Cash-In-Advance Model

5.1. Model and Solution Using Previous Approach

In basic Cash-In-Advance model, money is used to purchase goods. Money cannot yield utility itself, but the consumption of future can yield utility. Svensson [12] assumed that agents are available for spending only the cash carried over from the previous period. This is essentially different from Money-In-Utility model. Consider a simple form of discounted household utility value . More complicated utility function will be discussed in the following sections. Take Cash-In-Advance constraint form as due to Svensson [12]. This means that the agent enters the period with money holdings and receives a lump-sum transfer (in real currency terms) for consumption goods. Bonds and capital may not be purchased by currency. If capital is assumed to be purchased by money, the Cash-In-Advance constraint will become .

The budget constraint is rewritten in real terms as

In monetary theory, constraints are expressed as inequality frequently. This constraint describes that the representative agent’s time real resources should be more than or equal to the use of it, that is, purchasing consumption, capital, bonds, and money holdings that are then carried into period . Because of the assumption of rational agents, the certainty problems we are discussing, and the positive opportunity cost of money holdings, the constraints become equations in equilibriums.

Output is only affected by state variable without considering the effect of labor supply. Following the solution methods of Walsh [9], let the left side of the budget constraint be a state variable . It will be very tedious adopting expansion method with two constraints. Setting Bellman equation with multipliers will be better. Let state variable in value function get the economic significance of money in first-order conditions. However, there is an overlapped setting with ,

5.2. Our Solving Approach

Let . There is no overlap with in the value function. The Bellman equation is set as

The solution of this problem is similar to the application of the proposed method in Sections 3 and 4. We will not give unnecessary details here.

Note that Walsh [9] puts overlapping state variables and in value function; it is easy to get wrong results in those problems with Cash-In-Advance constraints although it is no problem setting with no more results required now. This will be seen in the following example of Section 6.

6. Setting and Solution of Bellman Equation in Model considering Both Labor Time and Cash-In-Advance Constraints

6.1. Model and Solution Using Previous Approach

Assuming that money is used to purchase consumption goods and investments, the Cash-In-Advance constraint becomes , where is investment and . In considering of the effects of labor supply on output and leisure on utility, we have . The agent’s objective becomes

Output is affected not only by state variable but also by controllable factor ; the left side of the budget constraint is unsuitable to be combined as a state variable. Thinking of the setting of the Shopping-Time Model and Cash-In-Advance model above, it is plausible to put overlapped state variables and in value function following the solution methods of Walsh [9]. The Bellman equation will be

The first-order conditions are

Using the envelope theorem and computing the derivatives with respect to state variables, we get

When the question is to derive the effect of inflation rate on the steady-state capital-labor ratio, that is, the steady-state relationship of and , assume that the aggregate production takes the form . Using (34) and (35),

Then

From (35),

Rewriting the aggregate production as , we have

Using (37), we obtain

Rewriting this equation, we get

In steady-state, , , , , , from (41), we have

This steady-state capital-labor ratio is derived by using current prevailing methods. However, it is a wrong result. We will provide the correct result by using the proposed method of this paper.

6.2. Our Solving Approach

Output is affected by state variable and control variable ; it is unsuitable to let the left side of the budget constraint be combined as a state variable. We should separate from it. Let state variable in value function alone get the economic significance of money in first-order conditions. Because overlaps of state variables are not allowed according to our proposed method, we put in value function as a state variable.

First, set up value function with multipliers:

Second, compute the derivatives of state variables and derive first-order conditions:

Third, compute the partial derivatives with respect to , , and and the envelope theorem yields

Finally, derive the steady-state capital-labor ratio by the results above:

The process of deriving this ratio is similar to the deriving process of (42). We will not give unnecessary details here. Compare (46) with (42); the result of (42) is times that of (46). Equation (46) is the correct result. It is the different partial derivatives with respect to in (34) and (44) that cause this deviation. In (34), . In (44), . Fundamentally speaking, the reason for the deviation is the overlapped setting of state variables and versus the nonoverlapped setting of state variables , , and . One has to use the particular skills in [9] that is hard to think about and grasp to obtain the right answer or get a wrong result adopting the overlapped setting . For this reason, the setting of value function by Walsh could not be generalized. In this model considering both Labor Time and Cash-In-Advance constraints, the basic methods of incorporating money into general equilibrium models are generalized, and our proposed method of setting and solution of Bellman equation demonstrates its clarity and validity and corrects the defect that some other current methods of setting and solution are easy to get wrong results.

7. Conclusions

As an important tool in theory economics, Bellman equation is very powerful in solving optimization problems of discrete time and is frequently used in monetary theory. It is hard to grasp the setting and solution of Bellman equation and easy to reach wrong conclusions since there is no general method to set Bellman equation or the settings of Bellman equation are excessively flexible. In this paper, we provide a set of general setting and solution methods for Bellman equation with multipliers. In the processes of solving monetary problems, comparing with other current methods in classic reference, our proposed method demonstrates its features of clarity, validity, correct results, easy operation, and generalization. Bellman equation is used not only in monetary problems but also in almost every dynamic programming problem associated with discrete time optimization. Our future work is to study the applicability of the proposed method in this paper in other areas.

Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.

Acknowledgments

Wanbo Lu’s research is sponsored by the National Science Foundation of China (71101118) and the Program for New Century Excellent Talents in University (NCET-13-0961) in China.