Abstract
This study deals with the external validation of simulation models using methods from differential algebra. Without any system identification or iterative numerical methods, this approach provides evidence that the equations of a model can represent measured and simulated sets of data. This is very useful to check if a model is, in general, suitable. In addition, the application of this approach to verification of the similarity between the identifiable parameters of two models with different sets of input and output measurements is demonstrated. We present a discussion on how the method can be used to find parameter deviations between any two models. The advantage of this method is its applicability to nonlinear systems as well as its algorithmic nature, which makes it easy to automate.
1. Introduction
The external validation of simulation models is an important topic in many engineering problems. Some methods are known to support this process, but there is no general method which can deal with a broad class of systems. In this study, a method based on differential algebra is shown to be an efficient solution to the external validation problem. Differential algebra can be used to prove that a set of nonlinear polynomial differential equations are able to represent the input and output behaviour of a set of measurement or simulation data. Estimating any parameters of the system for this proof is not necessary. The advantage of this method is the decoupling of the validation of the mathematical model structure from the validation of the numerical, maybe physical, parameters of the system. This decoupling makes the external validation process much simpler. In addition, differential algebra can be used to show that different sets of measurement data were generated by the same system with the same identifiable set of parameters. In other words, the proposed solution can prove that the system structure and its numerical parameters did not change during several measurements.
2. Problem Formulation
The external validation of mathematical models is one of the major challenges in simulation technology. Model validation is substantiating that the model, within its domain of applicability, behaves with satisfactory accuracy consistent with the study objectives. Model validation deals with building the right model [1]. After the equations have been created, the model is simulated. The first simulation is almost always unsatisfactory. The equations are then modified and the simulations repeated until time runs out or satisfactory behaviour is achieved [2]. This statement from practical applications is an example for the importance and the need for efficient methods for the validation of simulation models.
A large number of model validation methods are known, as well addressed by Balci [3] and Banks [1] in their work, with a listing of 77 different methods. A majority of these methods are related to the field of software development and cannot be used for the validation of models of dynamic technical systems. For continuous dynamic systems, Murray-Smith [4] presented a comprehensive review on the available methods. He mentioned that strong links between system identification, fault detection, fault diagnosis, and model validation exist. In particular, the concept of structural identifiability is shown to be important for validity determination of models.
Forrester and Senge proposed to conduct a model on the basis of the model-builder’s personal knowledge and then extend the structural verification test by including criticisms from others with direct experience of the real system [5]. This empirical test can also be performed as a theoretical structure test by comparing the model equations with general knowledge in the literature [6]. These definitions of structural correctness of a model are directly addressing the complete internal structure of the system. The method which is proposed in this publication also deals with the structural correctness of a model, from a perspective from which the model structure is able to represent the input and output behaviour of a system.
Currently, it can be concluded from a thorough review of the literature that no general automatic method for the model validation problem is available. The reasons for this may be attributed to the fact that several promising methods available for use are only applicable to systems with special properties such as linear systems or to academic examples. Other broad ranges of methods have only limited benefits compared with the well-known and simple methods, including the direct comparison of simulation results with measurement data.
Given the importance of validation methods, it is surprising how only few model validation methods are implemented in standard software packages. One example of a successful implementation is the “Reality Check” in the simulation software “Vensim.” The idea is based on the definition of cases the model has to fulfil and the automatic check, if the model behaves as suggested. Each “Reality Check” test consists of a test input coupled to an expected behaviour. It is important to notice that these tests refer only to the behaviour and not to the structure [2].
In this study, an interesting method from the field of differential algebra is introduced, which has the potential to overcome some of the current model validation problems.
2.1. Outline of the Paper
Section 3 contains a short introduction to the basic concepts of differential algebra which are necessary to understand the proposed model validation methods. For a more detailed description of these concepts, the reader is referred to [7, 8]. Section 4 is focused on the structural validation of models, which is a known application of differential algebra. With this structural validation, it can be proven that the structure of a model is capable enough of representing the measured input and output signals of a system. In Section 5, new methods are proposed which enable not only the validation of the model structure but also the measurement data used for these validations. With this method, it is possible to decide if two or more sets of measurement data were obtained from the same system with the same structure and the same numerical parameters. To accomplish this task, it is not necessary to know any of these numerical parameters. In addition, Section 5 shows how different parameters belonging to different sets of measurement data can be isolated. The fact that this is only possible for identifiable parameters is also shown at the end of Section 5. Sections 3 to 5 of this paper provide extensive coverage of the successful and unsuccessful validation of the models by using differential algebra.
3. Model Validation by Utilizing Differential Algebra
A promising approach to model validation was shown by Ljung and Glad [9]. It is based on differential algebra and has several advantages over the methods which are currently widely used. For a better understanding of these methods, some basic concepts and definitions from differential algebra are first described in the next section.
3.1. Background on Differential Algebra
Differential algebra originated in 1950 based on the work of mathematician Ritt [7, 8]. The main idea behind differential algebra was to extend the well-known concepts of classical algebra to systems of differential equations [10]. In addition to classical algebra, differential algebra defines a differential ring as an algebraic ring endowed with a derivation [11]:A subset of the differential ring is an ideal if it satisfies the following three conditions. The first condition is that . The second condition is that if , then . The last condition is that if and , then [12]. The variables , , and are all differential polynomials.
Starting from a set of differential polynomials,the differential ideal generated by is denoted by . It consists of all differential polynomials which could be formed from the elements of by multiplication with arbitrary polynomials, addition, and differentiation. A differential ideal is called prime, when implies that either or . It is called radical or perfect if implies . The smallest radical ideal including a given set of differential polynomials is denoted by [13]. Since the differential polynomials in the radical ideal are vanishing to zero, if the zeros or solutions of the original system are inserted, this ideal could be used to study the solution sets of the original system [12].
A key concept of differential algebra is the algorithmic reduction of the differential equations [12]. The main idea is to transform the original system of differential polynomials into a new form wherein the analysis of the system is much simpler than the analysis of the original system. One algorithm which is able to perform such a transformation is Ritt’s algorithm of differential algebra [7]. Ritt’s algorithm constructs a finite number of autoreduced sets [13]:where each is a characteristic set of prime differential ideals such thatRitt’s algorithm takes a set of differential polynomials and reduces each one by a pseudo-division which is very similar to the well-known division of polynomials with multiple variables.
Two sets of differential polynomials which are reduced with respect to each other are called autoreduced sets, where the sets with the lowest possible rank are called the characteristic sets [14]. This approach is very interesting for model validation problems because the reduced sets form some kind of a triangular representation of the original system. This implies that some of these reduced sets have fewer parameters than the original system, but they vanish if the solutions of the original systems are inserted into them. For a detailed mathematical description of the abstract concepts of differential algebra, the reader is referred to Ritt [7] and Kolchin [15].
On the basis of these concepts, Ljung and Glad [9] showed that the characteristic set can be used to prove the uniqueness of model parameters as well as the general usefulness of these models.
3.2. Applications of Differential Algebra for Model Validation
In particular, in large and complex models with a high number of degrees of freedom, probing the reason for deviations between the simulations and measurements is very difficult. Moreover, proving that two or more sets of measurement data were generated by the same system with the same parameterization is advantageous. In general, the deviations may be due to structural or parametric problems of the simulation model. Structural problems are problems with the mathematical structure of the equations. Parametric problems are problems with the numerical values of the (possibly physical) parameters of the model. In other words, structural problems are those where the equations of the simulation model are not able to represent the dynamics of the system under measurement. This is the case if the model assumptions are not correct, for example, due to unmodelled dynamics. In contrast to structural problems, parametric problems are caused by insufficient numerical values of the model parameters. In the case of mechanical systems, these values might be the mass or spring constants.
To improve the simulation model, it is advantageous to distinguish between structural and parametric errors, which is difficult in practice. One widely accepted method for the improvement of models is the identification of the parameters of the system. The identified parameters may differ from the real system parameters, for example, due to unmodelled dynamic effects or measurement noise. Unfortunately, the parameter identification algorithms may not determine perfect parameter values. Furthermore, the reason can be a structural problem of the model, which implies that no optimal set of parameters exists for this model. Moreover, other reasons for failure in parameter identification could be, for example, numerical problems or inappropriate start values for the iterative parameter identification. If the parameter identification succeeds and the numerical parameters are close to the expected and physically meaningful parameters, it is possible to conclude that there are no structural problems inside the simulation model. However, unsuccessful parameter identification is not a helpful indicator to find the reason for model deviations, because no conclusion as to the source of the deviation can be derived. If the model has a unique set of parameters and the parameter identification is initiated with meaningful values close to the real values, then this could serve as a useful method for the improvement of the model.
One general requirement for the estimation of a unique set of parameters is the global identifiability of the parameters. Following the work of Ljung and Glad [9], a linear parameterized transfer function is globally identifiable at , if the equations in that arise from the equivalencehave the only solution atThere are two phenomena which could account for the parameters to be nonidentifiable: structural nonidentifiability and practical nonidentifiability [16]. The structural identifiability is related to the structure of the equations in the model and is independent of the measurement data. It was first considered by Bellman and Åström [17] for linear systems. Structural identifiable parameters can also be practically nonidentifiable because of low quality of the measurement data. This is the case if, for example, the measurement noise is too high, the excitation provided to the nonlinear system is insufficient or when the length of the measures is too short.
Since structural identifiability is only a property of the system structure and independent of the measurement data, it is a very important property for a guaranteed validation of the model. It is known that the parameters of structural unidentifiable models can only be estimated in combination with other parameters. For example, if the product of two parameters is the only identifiable one, a perfect accordance of the input and output behaviour with the estimated parameters can mask significant errors in the individual parameters. A high value of one parameter can be compensated by a low value of the second parameter which leads to an inconclusive result of the validation exercise [4]. Several methods for structural identifiability analysis are known [18]. Unfortunately, all of them present limitations related to the nonlinearity and the size of the system under consideration. By size, we refer to the number of state variables, the number of parameters, and the number of observables [19].
One elegant algorithmic method for identifiability analysis based on differential algebra was presented by Ljung and Glad [9]. Ljung and Glad showed that the parameters of a nonlinear system are structural identifiable ones if the system can be rewritten as a linear regression in its parameters. This can be achieved through the use of Ritt’s algorithm [7].
Starting from a very general description, the systemcan be transformed into a triangular form. The word triangular here refers to the (physical) parameters of the system where every set of polynomials to has more parameters and states than the previous setsThe variables and represent the inputs and outputs of the system, is a time variant variable representing, for example, the state of the system, and is the set of constant parameters of the system. The characteristic set includes only the input and output variables and and their derivatives. It can therefore be used for testing the usefulness of models without identifying any parameter [9].
To compute the characteristic set, it is necessary to define a ranking for the variables and the inputs and outputs of the system. A ranking is a total ordering of variables, which determines the sequence of variables to be eliminated. To obtain a parameter-free characteristic set , the rankingis used [9], where , , and are the derivative orders of the inputs, outputs, and the states of the system (if they exist).
It was shown by different authors in [20, 21] that the elimination of variables to obtain an input and output description of a system can be successfully used in fault detection. Therefore, it is obvious to consider its applicability for the external validation of simulation models.
A characteristic set generates the same differential ideal as generated by the differential polynomials of the model to be validated but in a special form, which is easier to validate. With the knowledge of a characteristic set in the base ring of the model, finding an input and output relation without any states and without any parameters of the model is possible [14].
It is to be mentioned that the differential algebraic elimination algorithm requires equations which are polynomial in , and their derivatives. This is not as restrictive as it may seem. For example, the equationcan also been written aswhich fulfils the requirements for applying the elimination algorithm [9].
A general algorithm for transforming nonlinear systems with linear control inputs (12) into systems of differential polynomials is shown in [22]
4. State-of-the-Art Applications in External Validation
The characteristic set (8) consists only of functions of the inputs and outputs of the model and their derivatives; therefore, it is possible to check if a model is able to represent the measured or simulated dynamics of a system. The interesting property of this solution is that it is not necessary to estimate any parameter or to numerically solve complex equations. After the numerical calculation of the derivatives of the input and output signals of the system, the equations in have to be evaluated only at discrete time steps, which is a simple task. This approach represents a structural validation of the model equations. To demonstrate the advantages of the proposed method, a linear quarter car model was considered (Figure 1). The road for the model was generated by a power spectral density model [23] in accordance with ISO 8606. For the examples section, the Rosenfeld-Gröbner algorithm [24] has been used to generate the characteristic set.
4.1. Example 1: Structural Validation of Models
The first example shows the application of the characteristic set for the structural validation of a model. All parameters of the model are shown in Table 1.
The displacement of the wheel relative to the chassis can be measured and denoted asIt is also assumed that the acceleration of the chassis can be measured. The complete equation of motion for the chassis is thenApplying the elimination ranking approach presented in (9) for the structural validation,the following characteristic set, consisting of an equation,and an inequality,is obtained.
After the discretization of (16) by approximating the derivatives using difference quotients, the left-hand side of the equation can be evaluated, which represents the residual of the structural accordance. The residual in Figure 2 is very small compared with the excitation signals, which reflects the functionality of this approach.
4.2. Example 2
To show that the proposed approach is able to detect deviations in the structure of the model, the linear system equation, (14), is extended by a nonlinear spring:where the nonlinear spring stiffness is set to a value of Nm−3. Equation (18) is then used to generate the input and output signals for the validation equation, (16). Since the structure of the system in (18), which generates the signals, differs significantly from the structure of the equation from which the validation equation is derived in (14), the validation equation results in a considerable residual (Figure 3).
5. New Approaches for External Validation
Due to the high potential of the structural validation of models, this approach is extended and further developed in the current section. We will show how differential algebra can be used to prove that two sets of measurement data can only be generated by a model with the same structure and the same parameters. For this proof, the knowledge of these numerical parameters is again not necessary.
5.1. An Automatic Proof of Equal System Parametrization by Using Different Measurement Data Sets
The question on whether two measures are related to only one system becomes interesting when we want to know whether the parameters change during one or several tests. This is very useful to prove, for example, if a system is time-invariant or if the different sets of data belong truly to one and the same system. This becomes all the more important if the simulation model answers perfectly to one excitation signal and poorly to another excitation signal.
By assuming for simplicity a single-input single-output system (SISO) and two sets of measurement data, the following ranking can be obtained:where the variables , , , , , and are different orders of derivative.
The main idea behind this new ranking is to assume that two systems with the same structure of the underlying differential equations have the same physical parameters but different input and output signals. After the elimination of these equal and constant parameters, by using differential algebra, the resulting characteristic set would vanish, if the correct input and output signals are inserted into the characteristic set. The phrase “correct signals” here implies that the assumption of equal equations and equal parameters of both systems is correct.
5.1.1. Example 3
The third example below shows how the characteristic set can be used to prove the equality of the parameters of one model and the different sets of measurement data. Again, the quarter car model with the same parameters as in Example 1 is used for the generation of two different input and output signals. The only difference is another street profile for the second set of data, all parameters being equal. The characteristic set is generated by modelling the same model of (14) twice with the same parameters but with different inputs and outputs.
The equations of the system are therefore equal, except for the variables of the inputs and outputs:With the elimination ranking for the equality test of both parameterizations from (19),the following validation equation and inequality are obtained:With exactly the same parameters in both models, the residual in Figure 5 is very small, which again proves the functionality of this approach.
5.1.2. Example 4
As a counterexample, the chassis mass in (20) is now increased by 5 kg for the generation of the first set of input and output signals, , , and . The residual with this wrong parameterization is much higher (as shown in Figure 6).
This simulation shows that the approach is very useful to detect changes in parameters between different sets of measurement data, without any knowledge of their numerical values.
5.2. Isolation of Changed Parameters
If the structural validation shows a perfect accordance of the structure with all signals and additionally a change of the parameters has been detected with the proposed method, inferring which parameters have changed is possible. In order to accomplish this, the elimination ranking from (19) is extended by an explicit modelling of parameters which can change during different data sets. The idea is to generate a characteristic set which will vanish if the inserted input and output data were generated by the same system with mainly the same parameters, except for those parameters which were modelled as variable parameters. The phrase “variable parameters” means that the parameters are constant within each data set, but these constant values might change in each measurement data set.
Suppose that the first parameter of the system is the only one that has changed; the following ranking can be used to generate such a characteristic set:where denotes the first parameter of the model for the first set of measurement data and is the first parameter of the model for the second set of measurement data. It is to be noted that proving the equality of parameters only holds for identifiable parameters. Unidentifiable parameters can compensate deviations within the other unidentifiable parameters. This is shown in more detail later.
5.2.1. Example 5
If a test of the equality of parameters shows changes between different sets of measurement data, as in Example 4, it is necessary to isolate the changed parameters. The isolation can be performed by introducing different variables for the parameters of the two models.
In this example, the chassis mass is assumed to have changed between two sets of measurement data, which is shown in Figure 7.
Equations (20) and (21) are extended here with the different chassis mass parameters and :as well as the elimination ranking, which can now be expressed as follows:The following characteristic set with one equation and three inequalities can then be obtained:In Figure 8, both models (26) and (27) are excited with the same street profile as in Example 3 and the chassis mass of the second quarter car model is increased by 5 kg.
As expected, the residual is very small compared with the input and output signals of the model. This shows that the residual of the validation equation, that is, (29), is not sensitive against any change of the chassis mass because a derivation of this parameter is explicitly modelled. The proposed procedure can now be repeated for all parameters and parameter combinations until the parameter(s), which had changed between the two sets of measurement data, are found. It will be shown later that this is only possible for identifiable parameters.
5.2.2. Example 6
As a counterexample, the damping in (27) is increased by 10%, in addition to the increased mass of 5 kg of the parameter . The residual in Figure 9 is now much higher than without any change in the damping. It can be concluded that the residual of validation equation (29) is sensitive to the unmodelled change of the damping constant as it should rightly be.
5.2.3. Example 7
It was already mentioned earlier that unambiguous conclusions about changed parameters are only possible for identifiable parameters or identifiable parameter combinations. The reason is the possible equality of the input and output behaviour of the systems with different unidentifiable parameters.
By deriving the transfer function from (27) using the Laplace transform, it is easy to see that any change of the mass leads to the same input and output behaviour as a common change of the spring stiffness and damping constant by the same factor would causeAs a numerical example, let us consider the chassis mass of (27) to increase by 5 kg and the spring and damping constants and to increase by 10%. As expected, the amplitude of the residual in Figure 10 is similar to the residual in Figure 8, where only the chassis mass was increased.
The previous example showed that for a clear assessment about changed parameters it is necessary only to consider the identifiable parameters of the system or to substitute the nonidentifiable parameters by parameter combinations which are identifiable. This result is only restrictive if it is assumed that a few parameters may change because different combinations of parameters can be nonidentifiable. In the last example, in the event of an error in the displacement sensor, with which the relative velocity is also obtained by numerical differentiation, a multiplicative fault of this sensor cannot be distinguished from a changed chassis mass. Both changes can lead to the same input and output behaviour. If it is only necessary to check whether any of the parameters are equal, like in Example 3, this problem is not as restrictive as it seems to be. The reason is that a change of one parameter has to be compensated exactly by another parameter which is improbable. For example, a multiplicative fault of the sensor, like in Example 7, can be detected with our proposed approach in this study.
6. Conclusion
We have demonstrated that methods from differential algebra can be very useful for external validation of simulation models. Because the examples presented were linear in their parameters, the validation problem with a common parameterization could have been solved by simple linear algebra as well. However, the advantage of the proposed differential algebraic approach is that it can also handle nonlinear systems. In addition, these methods have an algorithmic nature, which allows for ease of automation. With algorithms from differential algebra, the complex external validation problem can be divided into much simpler problems and we can test whether the equations of a model are generally able to fulfil a set of measurement data. Our study has shown examples to prove that two or more sets of data are generated by the same model parameterized by the same set of identifiable combinations of parameters. It was also shown that if two or more sets of data were generated by the same structure of a model, differential algebra can answer which parameters changed between different sets of measurement data. However, for usage with industrial relevant systems, a residual which is less sensitive against measurement noise and ideally linear in terms of the structural accordance has to be defined. Moreover, there is a need to enhance the efficiency of the algorithms for the computation of the characteristic sets. Nevertheless, to conclude, differential algebra shows great potential for the external validation of simulation models.
It has to be mentioned here that there are several other applications for this method, which have to be considered further in future. This method may be used to check the exact approximability of systems over time and to prove that requirements of systems can be fulfilled by at least one parameterization of the considered system, during a technical development process. As an example, this might be used to prove that there is at least one set of parameters for a selected controller with which the system has exactly the desired input and output behaviour.
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.