#### Abstract

This work introduces the empirical cross-gramian for multiple-input-multiple-output systems. The cross-gramian is a tool for reducing the state space of control systems, by conjoining controllability and observability information into a single matrix and does not require balancing. Its empirical gramian variant extends the applicability of the cross-gramian to nonlinear systems. Furthermore, for parametrized systems, the empirical gramians can also be utilized for sensitivity analysis or parameter identification and thus for parameter reduction. This work also introduces the empirical joint gramian, which is derived from the empirical cross-gramian. The joint gramian allows not only a reduction of the parameter space but also the combined state and parameter space reduction, which is tested on a linear and a nonlinear control system. Controllability- and observability-based combined reduction methods are also presented, which are benchmarked against the joint gramian.

#### 1. Introduction

The evaluation of large-scale dynamical systems, which arise, for example, from complex networks or discretized partial differential equations, may require model reduction due to limitations in computing power or memory. A reduction of the state space generates a surrogate model resembling the same dynamics up to a small error. For parametrized systems, the model order reduction has to take into account the associated parameter space to ensure the validity of the reduced order model. If the parameter space is of high dimension, a repeated evaluation at various locations of the parameter space, for example, during optimization of inverse problems, may also necessitate a model reduction, yet for the parameter space. This contribution is concerned with combined state and parameter reduction, targeting models with high-dimensional state and parameter spaces.

The efficient reduction of large-scale nonlinear control systems is a challenging task, especially in the case of parametrized systems, with high-dimensional state and parameter spaces, where a combined reduction of parameters and states may be required to allow repeated evaluation. For instance, an inverse problem on a neural network with many nodes and unknown connectivity, modeled as a parametrized nonlinear control system, requires long runtimes during parameter estimation due to system size and parameter count. Large-scale neural networks have widespread use, such as forward control problems on artificial neural networks or inverse problems on biological neural networks. A real-life example is the reconstruction of connectivity between brain regions from activity measurements like EEG or fMRI (see, e.g., [1]).

To lower computational complexity, the parameter and state spaces are to be confined to low-dimensional subspaces without affecting the systems dynamics significantly. Projection-based model order reduction techniques are concerned with determining projections to such subspaces, mapping the high-dimensional model to a reduced order low-dimensional surrogate model.

The methods presented in this work are rooted in balanced truncation [2] and proper orthogonal decomposition (POD) [3]. As an alternative to the here presented method using empirical gramians, another class of balancing-related approaches focuses on solving Lyapunov and Sylvester equations (see, e.g., [4]). For parametrized systems, the reduced-basis also method [5] should be noted here.

Since the number of a systems inputs and outputs usually remains fixed, the maps to and from the intermediary states characterize the reducibility of a system [6]. The balanced truncation approach, introduced in [2], balances a system in terms of controllability and observability, where controllability quantifies how well a state is driven by the input and observability quantifies how well changes in a state are reflected in the output. Excluding the least controllable and observable states by truncating the balanced system, a reduced order mapping from inputs to outputs is approximated.

A linear time-invariant control system is composed of a linear dynamic system and a linear output transformation: with states , input or control , and outputs . The system matrix transforms the states, the input matrix introduces external input or control, and the output matrix transforms the states to the outputs.

Controllability and observability can be assessed through the associated controllability gramian and observability gramian . Classically, and are computed as the smallest positive semi-definite solutions of the Lyapunov equations and , respectively. To make a compound statement about controllability and observability, and have to be balanced [7]. The singular values of the resulting balanced gramian correspond to the Hankel singular values of the system, with their magnitude describing how controllable and observable the associated state is.

This work focuses on cross-gramian-based methods for model reduction, which combine controllability and observability information into one gramian and is elaborately described in [8]. The cross-gramian was introduced in [9] and corresponds to a solution of the Sylvester equation .

An alternative to solving the Lyapunov or Sylvester matrix equations, apart from the analytic approaches, for example, in [10], is the method of empirical gramians, which was introduced in the works [11, 12] and enables the computation of gramian matrices also for nonlinear systems by mere basic vector and matrix operations. This concept was extended among others in [13] providing more general input signals. Particularly noted should be [14, 15] for developing the empirical cross-gramian for single-input-single-output (SISO) systems in the context of sensitivity analysis.

In this paper the empirical cross-gramian is generalized to be applicable to multiple-input-multiple-output (MIMO) systems. For the gramian-based parameter reduction, the groundwork has been laid by [16] from the observability and by [17] from the controllability point of view. From the cross-gramian perspective of parameter reduction, a new gramian, namely, the joint gramian, is introduced in this work. Furthermore, the concept of gramian-based combined state and parameter reduction is established. Using empirical gramians, it is shown that combined reduction allows efficient model order reduction of linear and nonlinear control systems.

To begin, the cross-gramian and its properties are reviewed in Section 2. Next, the empirical cross-gramian for MIMO systems is developed in Section 3. Section 4 introduces combined state and parameter reduction in two variants: first, an observability- and, second, a controllability-based approach; the former is enhanced to a cross-gramian-based combined reduction, which is presented in Section 5. Finally, numerical experiments are conducted in Section 6 comparing the newly presented methods for a linear and nonlinear neural network as well as a nonlinear benchmark problem.

#### 2. Review of the Cross-Gramian

A brief review of the cross-gramian along with its application to model reduction of linear time-invariant control systems is given next. The cross-gramian (also known by the symbol ) was introduced in a sequence of works [9, 18–23] and encodes controllability and observability into a single gramian matrix, defined as the product of controllability and observability operator, it can only be computed for square (a system with the same number of inputs and outputs) and asymptotically stable systems: Equivalently, the cross-gramian is given as a solution to the Sylvester equation . Approximate solutions for the Sylvester equation were discussed in [24–26]. If the system is also symmetric, the following relation between the cross, the controllability, and observability gramian holds [9]: While a SISO system is always symmetric [9], a linear MIMO system not only requires the same number of inputs and outputs, but also the system gain has to be symmetric [24]; then a symmetric transformation , with and exist. Trivially, for the system would be restricted by and ; such a system is called state space symmetric.

As presented in [9], the trace of the cross-gramian equals half the gain of a SISO system where now and : Because the trace equals the sum of eigenvalues, the cross-gramians eigenvalues are associated with the system gain (4). This result was used in [14, 15] for parameter identification purposes, using the system gain as a sensitivity measure. An extension of (4) from [9, Theorem 3] for MIMO systems is developed (see also [27]) next.

Corollary 1. *Given a linear, square, asymptotically stable MIMO system, then the trace of the cross-gramian relates to the system gain as follows:
*

*Proof. *For an asymptotically stable system, the trace of the cross-gramian, in the form of (2), is given by

Employing the cross-gramian instead of controllability and observability gramian means only a single gramian has to be computed. And since no balancing is required, the truncation procedure can be simplified to a direct truncation ([28], [8, Ch. 12.3]). A balancing transformation can be approximated by the singular value decomposition (SVD) of the cross-gramian. The approximated Hankel singular values of the diagonal matrix are sorted by the controllability and observability of the states. A projection to a subspace of the state space is then given by truncation of and : The matrices are partitioned based on a threshold into , and , . This leads to the following reduced order model (here the (one-sided) Galerkin projection is used, since the (two-sided) Petrov-Galerkin projection may produce unstable reduced order models):

Apart from truncation-based model reduction, the cross-gramian has applications, for example, in system identification [10] and decentralized control ([29–31]) by computing a participation matrix (see [32]) based on the cross-gramian. Lastly, the cross-gramian also has the benefit of conveying more information than controllability and observability gramian, since the system’s Cauchy index is given by the cross-gramian’s signature [19].

#### 3. Empirical Cross-Gramian

In this section the empirical cross-gramian for MIMO systems is introduced. For general, possibly nonlinear, control systems of the form with states , input or control , outputs , a vector field , and an output function , the procedure from Section 2 is not viable. In [11, 12, 33] the concept of empirical (controllability and observability) gramians was introduced. This is a POD method based solely on state space simulations of the system [34]. These empirical gramians correspond to the classic gramians for linear systems as shown in [11]. Subsequently this approach and its field of application were advanced by [32, 35–37]. Because the empirical gramians can be aligned to the operating region of the underlying system in terms of initial states and input or control, the empirical gramians carry more detailed information on the system [38] than the gramians computed as solutions of matrix equations.

Empirical gramians are based on averaging the response of a system that is perturbed in inputs and initial states. Initially, the perturbed input was restricted to a delta impulse , which was broadened to more general input configurations in [13] under the name of empirical controllability covariance matrix and empirical observability covariance matrix.

The necessary perturbation sets are systematically defined next; these should reflect the operating range of the underlying system. and are sets of standard directions for the inputs and initial states. Sets and are orthogonal transformations (rotations) to these standard directions of inputs and initial states, respectively, while and hold scales to these directions:

Along the lines of the empirical controllability gramian and empirical observability gramian [11], the empirical cross-gramian for SISO systems was introduced in [14]. In this work, as a new contribution, the empirical cross-gramian is generalized to square MIMO systems. Hence, the scope of the cross-gramian is extended to nonlinear control systems and provides an alternative nonlinear cross-gramian to [10]. For a general (possibly nonlinear) MIMO system with the empirical cross-gramian is defined as follows.

*Definition 2 (empirical cross-gramian). *For sets , , , , , and , input during steady state with output , the empirical cross-gramian relating the states of input to output of , is given by

Essentially, the empirical cross-gramian is an averaged cross-gramian over snapshots with the specified perturbations in input and initial states around steady state input and steady state output .

Next, similar to [11, 14], the equality of the cross-gramian and the empirical cross-gramian for linear MIMO control systems are shown next.

Lemma 3 (empirical cross-gramian). *For any nonempty sets , , , and the empirical cross-gramian of an asymptotically stable linear control system is equal to the cross-gramian.*

*Proof. *For an asymptotically stable linear control system, the input-to-state and state-to-output maps are given by
thus,

As for the other empirical gramians, this proof is only valid for impulse input, yet a similar approach to [13] can be used to extend the empirical cross-gramian, yielding an empirical cross-covariance matrix by allowing general discrete input signals. The snapshots and can not only be computed as simulations on demand but also be included from observed experimental data. In case the data is collected at discrete times in regular intervals , following [39], a discrete representation of the empirical cross-gramian is given here too.

*Definition 4 (discrete empirical cross-gramian). *For sets , , , , , and , input during steady state with output , the discrete empirical cross-gramian relating the states of input to output of , is given by
Computational complexity depends largely on the number of scales and rotations of perturbations as well as the order of integration used to generate the snapshots and .

The empirical cross-gramian enables state reduction for square nonlinear control systems without an additional balancing procedure using direct truncation, where the approximately balancing projection is computed by an SVD and truncated to , analogous to (7):

Like POD, to quantify how close the subspace obtained by reduction is approximating the state space, a measure of total preserved energy [11, 40] can also be employed here: for retained states of an -dimensional model with truncated states related to the lowest singular values of the empirical cross-gramian .

#### 4. Combined State and Parameter Reduction

Two methods for combined reduction, allowing simultaneous reduction of state and parameter spaces, are proposed: an observability-based and a controllability-based ansatz. For parametrized general control systems with parameters : in [16], the identifiability gramian was introduced (in [41] a similar concept is presented), which extends the concept of observability from states to parameters. Augmenting the states of a given system by its parameters as constant components, the parameters are treated like states: The initial states are also augmented by the given parameter value , yielding: . The identifiability of the parameters is obtained through the observability of these parameter-states by the augmented observability gramian: with the state observability gramian , the parameter observability gramian , and a mixture matrix . From the observability gramian of this augmented system, the identifiability gramian can be extracted via the Schur complement: For an approximation of the identifiability gramian , the parameter observability gramian itself is often sufficient. The identifiability is then given as the observability of the parameters, through the singular values of or approximately by , respectively. Instead of using the identifiability information for parameter identification as in [16, 42], a projection to the dominant parameter subspace is computed from . Similar to the cross-gramian approach, a singular value decomposition of the approximate identifiability gramian yields the reduced parameters : The partitioning depends on the singular values in . A truncation of the projection results in the reduced parameters and the associated parameter reduced order model:

Next, a combined reduction of state and parameter space is introduced. With the identifiability gramian-based parameter reduction, first, the parameter space of the system is reduced. The observability of the states is encoded in the augmented observability gramian too; it can be extracted as the upper-left matrix from . Then, after computation of a controllability gramian , the state space is reduced by balanced truncation (balanced truncation provides two-sided truncated projection matrices and ; see [43]). This results in an observability-based combined state and parameter reduction:

Similarly, a controllability-based combined reduction can be achieved by a parameter reduction using the sensitivity gramian from [44] for additive partitionable systems: which is based on [17, 45] and treats the parameters as additional inputs. By the sensitivity gramian, controllability information on the parameters is provided, which also allows a parameter reduction, again by a singular value decomposition of . Since an approximate controllability gramian is also computed in process (24), after computation of an observability gramian , the parameter reduced system is reduced in states by balanced truncation. This results in a controllability-based combined state and parameter reduction. The controllability-based combined reduced order model has the same form as the observability-based reduced model (23).

#### 5. Joint Gramian and Combined Reduction

In addition to controllability- and observability-based combined reduction, a cross-gramian-based combined reduction is proposed next, which, for symmetric control systems, is enabled by the empirical cross-gramian for MIMO systems from Section 3. Aggregating the computation of controllability and observability, not only for states like the cross-gramian but also for identifiability of parameters, leads to a reduction and identification method requiring a new single gramian. Here, the same augmented system (18) is used. The systems symmetry is not affected by the augmentation with the constant (parameter-) states, since in terms of a linear system the system components are expanded with zeros. This leads to the following new gramian matrix, which utilizes the cross-gramian and thus unifies controllability and observability of states and parameters.

*Definition 5 (joint gramian). *The joint gramian is the cross-gramian of a square augmented system; see (18).

The joint gramian also has a block structure: with the state cross-gramian , the parameter cross-gramian , and the mixture matrices . The parameter-states are uncontrollable, yielding and , since no inputs affect the augmented states. Thus a Schur complement of the joint gramian to extract the parameter associated lower right block matrix will always be zero: Yet, the identifiability information on the parameters is encoded in the nonzero mixture matrix . By taking the symmetric part of the joint gramian , one obtains the cross-identifiability gramian :

Taking the inverse of the symmetric part of the cross-gramian is too costly in a large-scale setting. But the Schur complement can be approximated by using as a coarse approximation (this approximation of the inverse is of complexity ) to the inverse from [46]. Thus, a more efficient cross-identifiability gramian is given by

A reduced set of parameters is again computed by a truncated projection obtained from the singular value decomposition of the cross-identifiability gramian:

After a parameter reduction, the states can be reduced with a state reduction by direct truncation, employing the usual cross-gramian , a byproduct of the joint gramian , which is the upper-left block matrix of :

For this combined reduction of states and parameters no further gramians have to be computed and no balancing transformation is required.

#### 6. Implementation and Numerical Results

For an efficient implementation, the structure of the gramian computation is exploited. First, the empirical gramians allow extensive parallelization. Each combination of direction, transformation, and scale can be processed separately yielding a sub-gramian. Second, the assembly of each sub-gramian can be comprehensively vectorized, since it consists of vector additions, inner products, and outer products. In the special case of the empirical cross-gramian and, thus, the empirical joint gramian which is an encapsulation of the empirical cross-gramian, organizing the observability snapshots into a third-order-tensor and exploiting generalized transpositions result in a very efficient gramian assembly (see emgr.m). The final resulting gramian is the normalized accumulation over all sub-gramians. For further details about the implementation see [44].

All gramians for the numerical results are computed by the empirical gramian framework introduced in [44]. The empirical gramian framework* emgr* [47] can be found at http://gramian.de and is compatible with Octave [48] and Matlab [49]. The source code, used for the following experiments, is provided as supplementary material.

The error measure employed in the following experiments is the relative -error for a vector valued time series [50] and is defined as follows.

*Definition 6. *The relative -error for two vector valued time series and is given by
with the -norm of a (discrete) time series

Numerical results for three models are presented next. First, state reduction is applied to a nonlinear benchmark problem to validate the applicability of the empirical cross-gramian. Then, a linear and nonlinear parametrized control system is considered for the state, parameter, and combined reduction.

##### 6.1. Nonlinear Benchmark

Introduced in [51], this nonlinear benchmark (this benchmark is also listed in the MORwiki [52]) has been used in [37, 40, 53] as a test problem for the assessment of the empirical controllability gramian and empirical observability gramian in a balanced truncation model order reduction setting. This benchmark system models a circuit consisting of capacitors and nonlinear resistors (a resistor parallel-connected to a diode). Its mathematical model is given by the following nonlinear SISO control system: with the nonlinear function :

In this setting with , a zero initial state and a decaying exponential input are employed. Figure 1 shows the relative -error in the reduced order models outputs reduced by balanced truncation of empirical controllability and observability gramian and direct truncation of the empirical cross-gramian.

After a steep initial drop in the error, the relative output error remains near the machine precision. Both methods, balanced truncation and direct truncation, perform very similarly and require less than of the full order model’s state dimension to reach the error plateau. The empirical cross-gramian-based direct truncation exhibits a slightly lower output error.

##### 6.2. State Reduction

Next, a parametrized linear control system and a parametrized nonlinear control system are reduced in states and parameters and combined in states and parameters. The parametrized linear control system model [17] is given by with an additional parametrized source term .

The system is ensured to be asymptotically stable; thus , which is a central requirement for all empirical gramians to be computable. Furthermore, the system is chosen to be state space symmetric: and . A state space symmetric system provides that all system gramians, the controllability gramian, the observability gramian, and the cross-gramian are equal [54]. This allows the verification of the empirical cross-gramian.

The parametrized nonlinear control system model is given by with a hyperbolic tangent nonlinearity using the same system matrices as in (37). This model is related to the hyperbolic network model from [55].

The linear and nonlinear model are subject to zero initial states and impulse input . For the following experiments a state space dimension ; thus is assumed as well as an input and output dimension of . The parameters are drawn from a uniform distribution ; the system matrix is generated as a sparse uniformly random matrix with ensured stability, and the input matrix is a dense uniformly random matrix, yielding the output matrix .

The linear model in (37) and the nonlinear model in (38) are first reduced in states using the following methods: balanced truncation utilizing the empirical controllability gramian and empirical observability gramian; direct truncation of the empirical cross-gramian presented in Section 3. Furthermore, a method closely related to balanced POD [56, 57] is tested too. For the linear model an approximate cross-gramian is computed using the approach from [22, 58], which obtains the cross-gramian by computing the controllability gramian of the system augmented with its adjoint system (this approach is also implemented in the empirical gramian framework as empirical linear cross gramian). For the nonlinear model an approximate cross-gramian is computed following [56] by the product . In Figure 2 the relative -error in the outputs is plotted for reduced orders .

**(a)**

**(b)**

The state reduced models in the linear setting, generated by balanced POD, balanced truncation, and direct truncation, are of similar quality; yet balanced POD and direct truncation perform slightly better than balanced truncation. In the nonlinear setting balanced truncation outperforms balanced POD and direct truncation for which the output error flattens above a reduced order of about of the original model order.

The better performance of balanced truncation for the nonlinear model is due to use of two-sided projections as opposed to the one-sided projections used for balanced POD and direct truncation here.

##### 6.3. Parameter Reduction

The linear model in (37) and the nonlinear model in (38) are reduced in parameters using the empirical sensitivity gramian, the empirical identifiability gramian, and empirical cross identifiability gramian from the empirical joint gramian. Figure 3 depicts the relative -error in the outputs for the reduced orders .

**(a)**

**(b)**

For the linear and the nonlinear model, the controllability-based sensitivity gramian performs worst with the slowest decline in output error. The observability-based identifiability gramian and the cross-gramian-based cross-identifiability gramian show a sharper descent of the output error of similar quality in the linear and nonlinear setting, yet the observability-based parameter reduction exhibits a steeper drop of the output error for reduced orders up to of the original parameter space.

Since the sensitivity gramian is a diagonal matrix, the associated projections reorder the parameters and thus exclude all effects of the truncated parameters, while the identifiability and cross-identifiability gramian use linear combinations of parameters to be truncated.

##### 6.4. Combined Reduction

The linear model in (37) and the nonlinear model in (38) are next reduced in states and parameters employing the methods presented in Section 4 and Section 5. Controllability-based combined reduction uses the empirical sensitivity gramian for parameter reduction and balanced truncation for the state reduction. Observability-based combined reduction uses the empirical identifiability gramian for parameter reduction and also balanced truncation for the state reduction. The cross-gramian-based combined reduction utilizes the empirical joint gramian; the cross-identifiability gramian is used for the parameter reduction and the direct truncation of the cross-gramian is used for the state reduction. In Figures 4 and 5 the relative -error in the outputs is plotted for reduced orders and .

**(a)**

**(b)**

**(c)**

**(a)**

**(b)**

**(c)**

For a better comparison, a cross-section of the surfaces in Figures 4 and 5 along the diagonals are plotted in Figure 6 showing the reduced order model’s output error for the same reduced order in states and parameters: .

**(a)**

**(b)**

For all reduced order models obtained by combined state and parameter reduction, the parameter reduction error dominates the output error. As for the parameter reduction, the controllability-based combined reduction by sensitivity gramian and balanced truncation performs worst with a slow descent in the output error. In the linear setting the cross-gramian-based joint gramian performs significantly better than the combination of identifiability gramian and balanced truncation. In the nonlinear setting the reduced order models of these methods exhibit similar behavior, which is due to the higher state reduction error (Figure 2) in the cross-gramian-based approach.

To assess the efficiency of the presented methods, the offline times (the time required to assemble the necessary empirical gramian matrices) are compared in Figure 7.

The state reduction of the linear model is accomplished fastest by the balanced-POD-related method, since the computational effort required for computation is equivalent to a single empirical controllability gramian. For the nonlinear model this advantage does not exist, since no adjoint system for the nonlinear model is provided. Notably, the cross-gramian, for the direct truncation, is computed slightly faster than controllability, observability gramian, and balancing transformation for the balanced truncation.

Among the empirical gramians for parameter reduction the sensitivity gramian is computed fastest, yet due to the high error this is the least applicable. Between the identifiability and cross-identifiability gramian, which exhibit a comparable output error, the cross-gramian-based approach is considerably faster.

For the combined reduction the offline times are similar to the offline times of the parameter reduction, with the exception of the controllability-based combined reduction, which now takes longer than the cross-gramian-based joint gramian because of the additional observability gramian required for the balanced truncation state reduction. Thus the empirical joint gramian is the fastest of the tested methods for combined reduction and provides a competitive output error.

#### 7. Conclusion

In this paper the empirical cross-gramian for MIMO systems and the empirical joint gramian for parameter and combined state and parameter reduction have been introduced and benchmarked. The empirical cross-gramian allows a state reduction of linear and nonlinear systems (for a transfer of the symmetry classification to nonlinear systems see [10]). The empirical joint gramian enables not only cross-gramian-based parameter reduction but also an efficient combined state and parameter reduction. Both, the empirical cross-gramian and the empirical joint gramian have been shown to be a viable alternative to balanced truncation and balancing-based combined reduction approaches.

Further research has to be conducted on two-sided projections and error bounds for the reduced order models and on applying the empirical cross-gramian to nonlinear or nonsymmetric systems. Existing extensions for nonsymmetric systems are generalizing the symmetry constraint to orthogonal symmetry [59] or embedding into a symmetric system [24].

The empirical cross-gramian for MIMO systems completes the set of empirical gramians for state reduction, while the joint gramian completes the body of parameter identification gramians and enables combined reduction without balancing.

#### Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.

#### Acknowledgments

This work was supported by the Deutsche Forschungsgemeinschaft, the Open Access Publication Fund of the University of Münster, DFG EXC 1003 Cells in Motion Cluster of Excellence, Münster, Germany, as well as the Center for Developing Mathematics in Interaction, DEMAIN, Münster, Germany.

#### Supplementary Materials

Gzipped Tape Archive files contain one or more files that have been packaged into an archive file and compressed.