Mathematical Problems in Engineering

Volume 2019, Article ID 9490512, 17 pages

https://doi.org/10.1155/2019/9490512

## Linear Quadratic Optimal Control Design: A Novel Approach Based on Krotov Conditions

Correspondence should be addressed to Avinash Kumar; moc.liamg@upmihsiva

Received 9 March 2019; Accepted 16 August 2019; Published 13 October 2019

Academic Editor: Alberto Olivares

Copyright © 2019 Avinash Kumar and Tushar Jain. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

#### Abstract

This paper revisits the problem of synthesizing the optimal control law for linear systems with a quadratic cost. For this problem, traditionally, the state feedback gain matrix of the optimal controller is computed by solving the Riccati equation, which is primarily obtained using calculus of variations- (CoV-) based and Hamilton–Jacobi–Bellman (HJB) equation-based approaches. To obtain the Riccati equation, these approaches require some assumptions in the solution procedure; that is, the former approach requires the notion of costates and then their relationship with states is exploited to obtain the closed-form expression of the optimal control law, while the latter requires *a priori* knowledge regarding the optimal cost function. In this paper, we propose a novel method for computing linear quadratic optimal control laws by using the global optimal control framework introduced by V. F. Krotov. As shall be illustrated in this article, this framework does not require the notion of costates and any *a priori* information regarding the optimal cost function. Nevertheless, using this framework, the optimal control problem gets translated to a nonconvex optimization problem. The novelty of the proposed method lies in transforming this nonconvex optimization problem into a convex problem. The convexity imposition results in a linear matrix inequality (LMI), whose analysis is reported in this work. Furthermore, this LMI reduces to the Riccati equation upon imposing optimality requirements. The insights along with the future directions of the work are presented and gathered at appropriate locations in this article. Finally, numerical results are provided to demonstrate the proposed methodology.

#### 1. Introduction

Optimal control is a heavily explored and still developing area of control engineering where the objective is to design a control law so as to optimize (maximize or minimize) a given performance index (cost functional) while driving the states of a dynamical system to zero (regulation problem) or to make output track a reference trajectory (tracking problem) [1]. The generic optimal control problem (GOCP) is given as follows: notation: throughout this article, lowercase alphabets represent scalar quantities, lowercase bold alphabets represent vector quantities, and uppercase alphabets represent matrices, and also, the quantities with “star” in the superscript correspond to the optimal trajectory.

The GOCP computes an optimal control law which minimizes the performance index/cost functionalsubject to the system dynamics to give the desired optimal trajectory . Here, is the running cost, is the terminal cost, is the state vector, and is the control input vector to be designed. Also, and are continuous.

Since, the aforementioned problem corresponds to optimization of the cost functional subject to dynamics of the system considered and possibly constraints on input(s) and/or state(s), the calculus of variations (CoV) is generally employed to address optimal control design problems [1, 2]. The assumption of existence of optimal control is usually the first step while using CoV techniques. Subsequently, the conditions which must be satisfied by such an optimal control law are derived. Hence, only necessary conditions are found, and sufficiency of these conditions is not guaranteed. Furthermore, the obtained control law is usually only locally optimum. Nevertheless, there are results available in the literature which provide restrictions under which the necessary conditions indeed become sufficient and the global optimal control law is obtained [3–5]. Note that, in solving optimal control design problems, the CoV-based methods use the notion of so-called costates (which are not actually present in the system). Moreover, in the solution procedure, the existence of a linear relationship between the states and the costates is exploited to compute the closed-form expression of the optimal control law (this is particularly true for linear quadratic problems) (see [6] for more details).

Alongside CoV, another tool, namely, dynamic programming (DP) (introduced by Bellman), has also been explored to solve optimal control problems. The application of DP to optimal control design problems for continuous linear systems leads to the celebrated Hamilton–Jacobi–Bellman (HJB) equation which also gives a necessary condition for optimality [1]. Nevertheless, this equation also provides sufficiency under the following mild conditions [2, 7]: there exists a continuously differentiable optimal cost function and the gradient of optimal cost function with respect to the state vector equals the costate which corresponds to the optimal trajectory. For example, consider the optimal control design problem for the system [8] with the performance measure as . For this problem, the optimal cost function is , and hence, the HJB equation is not defined at because of nondifferentiability of . From the aforementioned observations, a solution method for optimal control design which does not require these conditions is desirable. As shall be demonstrated in this article, the Krotov solution methodology is indeed such a methodology. In fact, this methodology provides sufficient conditions for existence of the global optimal control law without using the notion of costates and any a priori information regarding the optimal cost function [2].

Starting from the sixties, the results on sufficient conditions for the global optimum of the optimal control problem were published by Krotov [9, 10]. These conditions have been derived from the so-called extension principle [11]. The first step, while employing these conditions, is a total decomposition of the OCP with respect to time via an appropriate selection of the so-called Krotov function [2, 12, 13]. Once such a decomposition is obtained, the problem is reduced to a family of independent elementary optimization problems parameterized in time *t*. It has been shown in [2] that the two problems—the original OCP and the optimization problem resulting from decomposition—are completely equivalent. The method, however, is abstract in the sense that the selection of the Krotov function is not straightforward and is very problem specific [11]. A number of works have used the Krotov methodology for solving OCPs encountered in control of structural vibration problems in buildings [12], such as MEMS-based energy harvesting problem [13], magnetic resonance systems [14], quantum control [15], and computation of extremal space trajectories [16, 17]. However, the equivalent optimization problems in all these articles are nonconvex, and hence, the iterative methods, one of them being the Krotov method, are employed to obtain their solutions. To address this issue, we propose a novel method to directly (noniteratively) synthesize optimal controllers for linear systems using Krotov sufficient conditions. The innovation in our approach lies in transforming the nonconvex optimization problem into a convex optimization problem by a proper selection of Krotov functions. Upon convexity imposition, for the linear quadratic regulation (LQR) problem, a matrix inequality is obtained which can be easily converted to a linear matrix inequality (LMI) using Schur complement. Following the argument “However, it is much less appreciated how the other relations (LMIs) enter in to the theory” in [18] (regarding the LMIs whose derivation and analysis are less appreciated as compared to those of the famous Riccati equation), this work demonstrates how this LMI enters into the theory. Specifically, it results in the convexity imposition on the equivalent optimization problem (which is obtained by decomposition of the original optimal control problem using the Krotov function). Moreover, analysis of the obtained LMI is also reported in this article. In [19], the methodology was demonstrated for finite horizon linear quadratic optimal control problems, and the Krotov function was taken to be a positive definite quadratic function. This article presents a rather detailed discussion of the results along with extension of the methodology to infinite horizon problems. It differs from the study [19] in that the Krotov function here is taken as a quadratic function; neither symmetry nor positive definiteness is imposed upon this function. Moreover, the analysis of the LMI which results from convexity imposition is also presented. Considering the aforementioned points and given the fact that the Krotov framework remains highly unexplored in the literature (to the best of authors’ knowledge), this article may serve as a background for further exploration of this framework to more involved control problems, viz., nonlinear optimal control problems, distributed optimal control problems, etc. In summary, the contribution of this work is as follows: it exhaustively describes the methodology for solving the standard linear quadratic optimal control problems (both finite and infinite horizon) using Krotov sufficient conditions, it solves the equivalent optimization problems via convexity imposition: a technique which is not used in the previous work which uses Krotov conditions and then the analysis of resulting LMI is presented, and it provides the insights which result upon synthesizing the optimal control laws and may also lay the foundation upon which the Krotov sufficient conditions may be employed for solving more complex optimal control problems, viz., nonlinear optimal control problems.

The rest of this article is organized as follows: In Section 2, the preliminaries of linear quadratic optimal control problems are discussed and the solution methodologies based on the calculus of variations- (CoV-) based method and Hamilton–Jacobi–Bellman (HJB) equation-based method are outlined. The assumptions as encountered in these approaches are also discussed. In Section 3, the background literature of Krotov sufficient conditions and their application to the problems considered are detailed. This section also discusses a number of insights which result while solving the considered optimal control problems. These insights are gathered as remarks at appropriate locations in this section. In Section 4, the analysis of LMI obtained upon convexity imposition is presented. The Krotov iterative method is also discussed in brief in this section. In Section 5, the proposed method is demonstrated through numerical examples. Finally, the concluding remarks and future scope of the work are presented in Section 6.

#### 2. Preliminaries and Problem Formulation

In this section, solution procedures of linear quadratic regulation (LQR) and linear quadratic tracking (LQT) problems using CoV- and HJB equation-based approaches are briefly discussed in order to highlight the assumptions used in these approaches.

##### 2.1. OCP 1 (Finite Horizon LQR Problem)

Compute an optimal control law which minimizes the quadratic performance index/cost functionalsubject to the system dynamics and drives the states of the system to zero (regulation). Here, is given, is free, and is fixed. Also,

The solution using the CoV technique comprises the following four major steps:(i)Formulation of Hamiltonian function: the Hamiltonian for the considered problem is given as where is the costate vector.(ii)Obtaining the optimal control law using the first-order necessary condition: the optimal control law is obtained as(iii)Use of state and costate dynamics and a transformation to connect states and costates for all : the boundary conditions (i.e., being fixed and being free) lead to the following boundary condition on : . Then, the transformation to connect costate and state vectors is used to compute the optimal control law as(iv)Obtaining the matrix differential Riccati equation: finally, taking the derivative of equation (5) and substituting the state and costate relationship, the following matrix differential Riccati equation (MDRE) is obtained which must satisfy for all :

Furthermore, the solution using the HJB equation requireswhere is the optimal cost function and is the optimal control law. To solve (8), the boundary condition is used with assumed to bewhere is a real, symmetric, positive-definite matrix to be determined. Substituting (9) into (8), we get

This equation is valid for any if

Finally, , and thus, the solution is the same as that obtained using CoV. Summarizing the above, the global optimal control law is given bywhere is the solution ofwith the boundary condition .

##### 2.2. OCP 2 (Finite Horizon LQT Problem)

Compute an optimal control law which minimizes the following quadratic performance index/cost functional:where subject to the system dynamicssuch that the output tracks the desired reference trajectory . Here, is the error vector, is given, is free, and is fixed. Also,

Similarly to the solution of LQR, the CoV- and HJB equation-based approaches yield the optimal control law aswhere and satisfywith boundary conditions and , respectively. Note that, in the HJB approach, the optimal cost function has to be guessed.

Although the CoV- and HJB equation-based approaches as described above are widely employed for solving OCPs, there are some assumptions associated with these approaches in their solution procedure. Specifically, the CoV-based approach uses the notion of costates and their relationship with states for all time (5) to compute the optimal control law. Similarly, the HJB equation-based approach requires the existence of the continuously differentiable optimal cost function, and its gradient with respect to the state is the costate corresponding to the optimal trajectory [7]. Thus, the information about the optimal cost function must be known *a priori*. The angle of our attack is to synthesize an optimal control law using Krotov sufficient conditions, where the above issues are not encountered in the solution procedure. However, it is well known that the control law using these conditions is synthesized through an iterative procedure. The main nontrivial issue to be tackled is how to obtain noniterative solutions of optimal control problems using Krotov conditions. The next section answers this question for linear quadratic optimal control problems.

#### 3. Computation of Optimal Control Laws

In this section, solutions of the LQR and LQT problems using Krotov sufficient conditions are detailed.

##### 3.1. Krotov Sufficient Conditions in Optimal Control

The underlying idea behind Krotov sufficient conditions for global optimality of control processes is the total decomposition of the original OCP with respect to time using the so-called extension principle [2].

###### 3.1.1. Extension Principle

The essence of the extension principle is to replace the original optimization problem with complex relations and/or constraints with a simpler one such that they are excluded in the new problem definition but the solution of the new problem still satisfies the discarded relations [11].

Consider a scalar-valued functional defined over a set (i.e., ) and the optimization problem as follows.

*Problem 1. *Find such that where . Instead of solving Problem 1, another equivalent optimization problem is solved. Let *L* denote the equivalent representation of the original cost functional. Then, a new problem is formulated over , a superset of , as follows. Equivalent Problem 1. Find such that where .The equivalent problem is also called the extension of the original problem. The method of choosing of the *equivalent functional L* is not unique, and the selection is generally made according to specifications of the problem under consideration. This *freedom* in the selection of the equivalent functional can be exploited to tackle the generic nonconvex optimization problems. Also, it is necessary to ensure that , so that the optimizer is actually the optimizer of original Problem 1 [2]. The idea behind the extension principle is illustrated in Figure 1. Clearly, the application of the extension principle requires *appropriate* selection of the equivalent functional *L* and the set .