Abstract
This paper investigates a type of STGM (system testability growth model) based on the nonhomogeneous Poisson process which incorporates TGEF (testability growth effort function). First, we analyze the process of TGT (testability growth test) for equipment, which shows that the TGT can be divided into two committed steps: make the unit under test be in broken condition to identify TDL (testability design limitation) and remove the TDL. We consider that the amount of TGF (testability growth effort) spent on identifying TDL is a crucial issue which decides the shape of testability growth curve and that the TGF increases firstly and then decreases at different rates in the whole life cycle. Furthermore, we incorporate five TGEFs: an Exponential curve, a Rayleigh curve, a logistic curve, a delayed S-shape curve or an inflected S-shaped curve which are collectively referred to as Bell-shaped TGEFs into STGM. Results from applications to a real data set of a stable tracking platform are analyzed and evaluated in testability prediction capability and show that the Bell-shaped function can be expressed as a TGF curve and that the logistic TGEF dependent STGM gives better predictions based on the real data set.
1. Introduction
Generally, system testability is defined as the probability of fault detection or isolation for a specific period of time in a specified environment, which is quantified by various testability indexes, such as FDR, FIR, and FAR [1–3]. Over the last several decades, many testability test technologies, especially testability demonstration test for equipment, have been researched [4–15]. However, some research work indicates that the result of testability demonstration test departures from the actual value of testability greatly and makes the result of testability demonstration test unauthentic [4]. The root cause for this is that the vacancy of TGT in tracking and measuring the growth of testability as equipment is being developed.
In general, any fault diagnosis system similar to reliability for large-scale and complicated equipment like a missile may be premature when the fault diagnosis system has been developed, in which many TDLs hide, such as nonexpectant failure, test vacancy, ambiguity group, fuzzy point, improper resistance tolerance, or threshold. And then, the designers analyze and judge the root cause of TDL and further track to the design of equipment testability, like UUT, redesign of test equipment or interface equipment, circuit of BIT, fault diagnosis software, and test program of ATE. Thus, almost all the fault diagnosis systems for large-scale and complicated equipment need a certain extent of time period to develop TGT to identify and remove the TDLs to attain the expected value under contract.
The TGT is an important and expensive part of equipment testability development. A TDL in UUT leads to an output that differs from specifications and requirements. The aim of TGT is to identify and remove the TDL and further to increase the fault detection/isolation probability that a designed testability will work as intended in the hands of the designers. The testability growth test phase aims at identifying and removing the TDLs. Typically, TGT is more constructive and has high confidence level for system testability rather than the testability demonstration test at the acceptance stage only.
A common approach for measuring system testability is by using an analytic model whose parameters are generally estimated from available data on fault detection/isolation. In this paper, the analytic model is referred to by us as STGM. A STGM provides a mathematical relationship between the number of TDLs removed and the test time during the whole life cycle and generally is used as a tool to estimate and predict the progress of system testability. At the same time, the STGM can be used for planning and controlling all test resources during development and can assure us about the testability of equipment. There is an extensive body of literature emphasizing the importance of TGT, but no depth of research is done on STGM [3, 8].
In the context of TGT, the basic objective is to identify the TDLs and remove them one by one. Therefore, the process of TGT contains not only a TDL identification process but also a TDL removal process and that the key issues are the TGE which are consumed to identify the TDL and the effectiveness of TDL removal, respectively. In general, the TGE can be represented as man-hour, TGT cost, the times of fault injection, and so forth. The functions that describe how a TGE is distributed over the TGT phase are referred to by us as TGEF. The shape of the observed testability growth curve depends strongly on the time distribution of the TGE. Due to the randomness of TGT and consider the designer’s capability and familiarity to UUT synchronously, the other key issue (i.e., the effectiveness of TDL removal) can be referred to by us as a constant TDL identification rate and a constant TDL removal rate, respectively.
There is an extensive body of literature [16–40] on software reliability growth model describing the relationship between the test time and the amount of test-effort expended during that time, in which the test effort was often described by the traditional Exponential [19, 21], Rayleigh [16, 19, 21], logistic [29, 30, 37, 38], delayed-shaped [19, 39] or inflected S-shaped [18, 39] curves. Thus, to address the issue of TGEF, this paper will use the above five test effort curves which are collectively referred to as Bell-shaped TGEFs to describe the relationship between the test time and the amount of TGE expended during that time by analyzing the consumption rule of TGE. Sometimes the TGE can be represented as the number of faults injected or occurred naturally instead of man-hours or TGT cost. Similarly, to address the issue of effectiveness of TDL removal, we assume that the TDL identification rate and the TDL removal rate are constant.
In this paper, the process of TGT is decomposed into fault injection or occurrence, TDL identification, and TDL removal processes which are all based on NHPP. We show how to integrate time dependent TGEF, constant TDL identification rate, and constant TDL removal rate into STGM to find the framework of STGM. We further pay main attention on the consumption pattern of TGE based on fault injection collection of fault occurring naturally and then discuss how to integrate five Bell-shaped TGEFs into STGM. A method to estimate the model parameter is provided. Experimental results from a stable tracking platform are analyzed and the five STGMs based on the above five Bell-shaped TGEFs are compared with each other to show which STGM can give better prediction.
The remainder of the paper is organized as follows. In Section 2, we found a framework of STGM considering TGEF. The fault occurrence, TDL identification and TDL removal process based on NHPP are also described and analyzed in this section. In Section 3, we pay main attention to analyzing the consumption rule of TGEF which is represented by fault injection or collection of fault occurred naturally at design & development stage and trial & in-service stage, respectively. At the same time, we show how to incorporate the five Bell-shaped TGEFs into STGM. Parameters of the proposed STGMs as estimated by the method of LSE are discussed in Section 4. Finally, Section 5 discusses the goodness of the above five STGMs on prediction with the application of these models to a real data set.
2. A General Framework of STGM Based on NHPP
In general, the precondition for testability growth test is that the UUT is in failed condition. That is to say, there is one or more faults occurred in the UUT. Under the precondition, TGT is the process of running a testability diagnostic program and capturing the fault detection and isolation data to identify the TDLs and remove them. In other words, the TGT aims at identifying the TDLs and removing them one by one under the precondition of fault occurred. Therefore, the process of removing TDLs can roughly be divided into three steps: fault occurrence, TDL identification, and TDL removal. At the same time, each of the above three steps can be described effectively as a counting process based on NHPP.
In practice, the most important factor which affects the STGM’s evaluation and predication accuracy is TGE. TGE is the cumulative testability growth effort consumed in TGT which can be measured by the number of fault injection, test cost, staff, and so on. The consumed TGE indicates how TDLs are identified effectively in the UUT and can be modeled by different curves considering the consumption pattern in TGT. In this section, a framework for foundation of STGM with TGEF is proposed.
2.1. Fault Occurrence, TDL Identification, and Removal
TDL identification may be realized under the condition that the UUT must be broken. Once the UUT is in broken condition which usually can be realized by fault injection or collection information of fault occurred naturally, the design of testability begins to run to identify the TDL and analyze the root cause of the TDL. When the specific root cause is recognized, the designers can remove the TDL accordingly.
We can use the number of residual TDLs as a measure of system testability. The great the number of TDLs, the lower of testability level would be. So, the process of TGT is the process to identify and remove the TDLs existing in the UUT. In other words, the process of TGT is to identify and remove the TDL one by one. So, the process of TGT can be described by Poisson process considering the randomness of fault injection, fault detection and isolation, TDL identification, and removal etc. Here, we give an apt shown in Figure 1 to describe the process of TGT.
The fives processes given by Figure 1 about TGT and fault detection and isolation are (a) the process of fault occurred naturally or injected of the UUT, (c) the process of identifying that TDL exists in the UUT, (d) the process of removing TDL exist in the UUT, (b) and (e) depict the fault detection and isolation before and after the TGT, respectively.
In (a), is the failure mode set which is obtained by FMEA and is the total number of failures in the UUT. With the progress of TGT, the time-dependent failure phenomena which can be realized by fault injection or collection of fault information occurred naturally at random time serial .
In (b), When some failures occurred at random time serial , the system testability is activated to detect and isolate the failures accordingly. If a failure cannot be detected and isolated successfully, a TDL is identified. Practically, some failures may be easy to detect and isolate, and some others are not. For example, and cannot be detected and isolated correctly, so two TDLs are identified and are noted as and , respectively.
With the progress of TDL identification, increasing TDLs are identified gradually; thus we have the process (c) which describes the counting process of TDL identified.
In (d), the designers analyze the root cause of the TDL and try their best to modify the design of testability and remove the TDL accordingly. But, not all the TDLs identified can be removed successfully because of designer’s familiarity to UUT. For example, the presented in process (c) still cannot be removed successfully after an infinite amount of test time and test effort consumed. All the TDLs which can be identified but cannot be removed successfully form the process (d). In TGT, we should pay more attention to the expected average number of TDLs identified and removed successfully.
By comparison with process (b), process (e) gives the growth effect of fault detection and isolation after TGT apparently.
2.2. NHPP in TGT
There is an extensive body of literature [12, 32] on NHPP used for description of counting process effectively, such as the probabilistic failure process in software and equipment. In this paper, the NHPP is introduced to describe the TDL detection and removal process. is a nonnegative integer and a time-dependent nondecreasing function, which describes the cumulative number of TDLs identified and removed up to time . If , is the counts of the TDL identified and removed in the interval .
The following characters of can be derived from the assumption that “the probability that two or more faults occurred synchronously is wee and can be neglected in testability engineering” [3].(1).(2)The process has independent increment.(3).(4).
Thus, the process of follows a Poisson process with a parameter , where is the TDL intensity of (i.e., number of TDLs identified and removed correctly per unit time). The counting process with a constant has a smooth increment, which is named homogeneous Poisson process. Otherwise, the counting process with a time-dependent has a fluctuant increment, which is named nonhomogeneous Poisson process. HPP can be considered as a special case of NHPP, and NHPP is an extend case of HPP.
Make ; then is the expected mean number of TDLs identified and removed successfully in time , so can be expressed asSo
To any , , follows the Poisson distribution with parameters , so we have
We can use either the number of TDLs identified and removed or the number of remaining TDLs as a measure of testability quality. Here, we use the number of TDLs identified and removed successfully (i.e., ) as a measure of system testability quality. Parallel to the analysis of Figure 1, is critical for both the mean and variance of testability estimation and prediction. Hence, calculation of is our main focus in the following parts of this paper.
2.3. A General Framework of STGM Considering TGEF
A STGM provides a mathematical relationship between the number of TDLs identified and removed successfully () from the UUT and the test time. STGM can be used as a tool to estimate and predict the progress of system testability. A TDL inherent in the UUT leads to an output that the fault occurred or injected cannot be detected and isolated correctly. The TGT aims at identifying these TDLs and removing them. At any time during the TGT phase, the basic process of TGT includes four steps which are shown in Figure 2.
In Figure 2, is the target testability level which has been fixed at the beginning of TGT. But it is frequently realized that this target may not be achievable for a number of reasons, like inadequacy of TGE or inefficiency of the test team. So, TGT is a repetitive work of “test-identification-correction-test” generally until the conditional expression is satisfied. is the estimation value of testability at time .
Firstly, we need to estimate at time . From the above analysis, the main cause leading to is the TDL existing in the UUT. In this paper, we assume that the removal of a TDL from the UUT includes two phases. In the first phase the TDL identification team, primarily consisting of test personnel, identifies a TDL by making the UUT be in failed condition. Concretely, the identification team can either collect the fault occurred naturally or injected to verify the level of testability. During the TGT phase, much testability growth effort is consumed. The consumed TGE indicates how the TDLs are identified effectively in the UUT and can be modeled by different curves. Actually, the system testability is highly related to the amount of testability growth effort expenditures spent on identifying TDLs. In this step, the work in the process of TDL identification is calculated by , such as man-hour, TGT cost, and the times of fault injection. signifies the cumulative TGE consumed in identifying the TDLs up to the given time . Then another team, primarily consisting of designers, analyzes the reason for the TDL and modifies the design of testability to remove the TDL by redesigning of UUT or ATE and the corresponding interface, optimizing the circuit of BIT, debugging the diagnostic software and rewriting the diagnostic program of ATE, etc. All the work about removing the TDLs will be done in the third step of Figure 2 by the designers of system testability, which is described by a constant . Finally, all the teams evaluate the value of system testability after TGT noted as and give a test conclusion that if , the TGT can be stopped, else the TGT will be continued until the condition expression is satisfied. Apparently, testability can be enhanced if TGT is done for a prolonged period, that is, .
The STGM developed below is based on the following assumptions.(1)The TDL identification and its removal processes all follow NHPP.(2)The failure of fault detection/isolation at random times is caused by the TDLs which are inherent in the UUT.(3)All TDLs are mutually independent and have the same contribution to the failure of fault detection/isolation occurred in the system.(4)There is a one-to-one correspondence between a failure and the corresponding test.(5)Two phases can be observed within the TGT process: TDL identification and TDL removal. There is no time lag between the TDL identification and its removal. Whenever a TDL is identified, the goal of TDL removal is to analyze the root cause of the TDL and to remove it. At the same time, no new TDLs are introduced.(6)The mean number of TDL identified in the time interval by the current TGE is proportional to the mean number of TDLs unidentified in the UUT. The mean number of TDL identified and removed in the time interval by the current testability growth effort is proportional to the mean number of TDLs uncorrected in the UUT. The proportions are constants and expressed by and , respectively.(7)The consumption of TGE is modeled by .
Let be the MVF of the expected number of TDL identified and let be the MVF of the expected number of TDLs identified and removed in time . Then, according to the above assumptions, we describe the STGM based on TGEF as follows:Note that here we assume . Solving (4) under the boundary condition we have
Due to the space limitations, here we only propose the testability growth model of FDR in general. FDR is defined as the capability to detect fault occurred in the UUT, and FDR is also used to describe the average detection probability to the failure mode set which is gained from FMEA. The mathematical model of FDR can be formulated as [1–3]where is the total number of failures. is the number of failures which can be detected accurately by the system testability design.
According to the above assumption (4), we can have that the number of failures which can be detected accurately by the system testability design isSubstituting it into (8), we obtainSubstitute (in (6)) into (10); the can be rewritten as
Depending on how elaborate a model one wishes to obtain is, one can use to yield more complex or less complex analytic solutions for . Different reflect various assumption patterns of TGE in TGT.
Equation (11) is the modeling framework which considers TGEF into the NHPP STGM. At the same time, (11) can provide designers or testers with an estimation of the time needed to reach a given level of FDR. And that, it may also be used to determine the appropriate ending time for TGT, and can provide useful information for making decision on turn phase.
3. STGM with Bell-Shaped TGEF
Actually, the system testability is highly related to the amount of testability growth effort expenditures spent on identifying the TDLs. In this section we develop some STGMs in which five Bell-shaped TGEFs: an Exponential TGEF, a Rayleigh TGEF, a logistic TGEF, a delayed S-shaped TGEF, and an inflected S-shaped TGEF are taken into account. The forms of TGEF are all discussed in the literature about software reliability growth model [16–39]. Here, we choose appropriate forms of TGEF in view of the TGT’s specialty.
3.1. Bell-Shaped TGEF
In TGT, one of the key factors is the TGE which can be represented as the number of man-hours, the times of fault injection, and the cost of TGT, and so forth. The functions that describe how an effort is distributed over the testability growth phase are referred to by us as TGEF. TGEF describes the relationship between the test time and the amount of TGE expended in the test time. Actually, fault injection is one of the best effective measures to make the UUT be in broken condition and to identify the inherent TDLs in the UUT further. There is an extensive body of literature on fault injection used for identifying TDL in testability test [1–3, 41–43]. Therefore, we will use the times of fault injection as a measurement of TGE, which has an advantage that it is very intuitive and that testability growth effort can be quantified exactly at the same time.
In general, the whole life-cycle of equipment can be divided into three stages: demonstration stage, design & development stage, and trial & in-service stage. But the empirical analysis shows that the effective and credible stages to conduct TGT are design & development stage and trial & in-service stage considering the cost effectiveness in which fault injection and collection information for naturally occurred faults are done, respectively [3, 6]. At the same time, TGE are calculated by times of fault injection and the number of naturally occurred faults at design & development stage and trial & in-service stage, respectively.
Actually, the more faults we inject or collect, the more confidence we obtain in the estimation of system testability. Unfortunately, TGT with redundant fault injections or a large number of failure data may lead to excessive cost and too much time consumption. It is impracticable with the constraint of test cycle and total cost.
To give an accurate and reasonable description of TGE, we need to know not only how many faults need to inject at design & development stage but also how many failure data need to collect at trial & in-service stage. That is to say, we need to analyze the TGE’s consumption pattern in TGT by combining Figure 3 which gives an abridged general view of testability growth process in the whole life-cycle.
In general, the TGT time can be represented by the serial number of testability growth test stage instead of the actual execution time. The assumptions which are shown in Figure 3 and will be used for description of TGEF are as follows.(1)The TGT stage is expressed as the time span and is composed by the design & development stage time span and the trail & in-service stage time span , respectively.(2)The times of TGT based on fault injection at design & development stage are , and the maximum number of fault injection at the design & development stage is marked as . At the design & development stage, we can observe data pairs in the form , , is the number of fault injection at the th TGT stage, and is the number of TDLs identified successfully at the th TGT stage.(3)After a period of time especially the th TGT at design & development stage, the UUT is put into the trail & in-service stage at which the number of TGT based on fault occurred naturally is and the maximum number of failures collecting is marked as . In a similar way, the same data pairs in the form , are observed by collecting the failure data occurred naturally. is the number of failures by collecting the naturally occurred fault and is the number of TDLs identified successfully at the th TGT stage.(4)With the progress of TGT, , is the true value of FDR at the th TGT stage, and that . The TGT can be stopped only when the is achieved, in which is the FDR’s desired value of TGT shown in Figure 3.
Based on the above assumptions, we will analyze the TGE consumption pattern from two perspectives; one is fault injection based TGE consumption at design & development stage; the other is naturally occurred fault based TGE consumption at trail & in-service stage.
At the beginning of the design & development stage, FDR level is relatively lower; hence, a certain number of TDLs will be identified as long as we inject less fault. In order to identify a certain number of TDLs at the design & development stage, the number of faults need to inject grow up with the level of FDR, which can be formulated as [1–3]
For example, with reference to (12), the number of faults which need to be injected with the change of FDR level and number of TDLs, respectively, are shown in Figure 4. As seen from Figure 4, we find that at the same number of TDLs, the FDR level needs to achieve increase as time goes on, and the number of faults needs to inject increases accordingly. At the same level of FDR, the greater number of TDLs needs to identify, the greater number of faults needs to inject. Consequently, TGEF is a time-dependent increasing function at design & development stage.
Collection of failure which occurred naturally is also an effective measure to identify TDL When the UUT is put into the trail & in-service stage. Actually, the number of faults at trail & in-service stage is “small sample” under several constraints such as high reliability requirements or specified test cycle. Considering the growth of reliability, the expected number of faults at regular intervals will decrease gradually. That is to say, at trail & in-service stage, TGEF is a time-dependent decreasing function which has the similar variation tendency shown in Figure 5.
In conclusion, at the whole TGT phase, TGEF increases firstly and then decreases at different rate. Based on the increase-decrease characteristic, we will use a kind of Bell-shaped function which increases firstly and then decreases to fit the practical growth rate of TGEF. Bell-shaped function, just as its name implies, draws a time-dependent curve like a bell which has a similar shape to the curve shown in Figure 6.
It should be noted that Figure 6 is just a schematic used to depict the Bell-shaped function; the concrete functional expression depends on the actual TGE data set. Because actual testability growth effort data represent various expenditure patterns, sometimes the testability growth effort expenditures are difficult to describe only by a Bell-shaped curve. In this paper, the Exponential TGEF, the Rayleigh TGEF, the logistic TGEF, the delayed S-shaped TGEF, and the inflected S-shaped TGEF have been used to explain the testability growth effort, which can be derived from the assumption that “the efforts that govern the pace of identifying the TDL for the UUT increase firstly and then decrease gradually.”
3.1.1. The Exponential TGEF
Yamada et al. [21] found that the TEF in software reliability growth could be described by a Weibull-type distribution with two cases: the Exponential curve and the Rayleigh curve.
The Exponential TGEF over time period can be expressed as
The current TGE expenditure rate at time iswhere .
The Exponential curve is used for process that declines monotonically to an asymptote.
Substitute the in (13) into (7); we obtain
3.1.2. The Rayleigh TGEF [16, 19, 21, 37]
Another alternative of the Weibull-type distribution is the Rayleigh curve. It has been empirically observed that software development projects follow a life-cycle pattern described by the Rayleigh curve.
The Rayleigh TGEF over time period can be expressed as
The current TGE expenditure rate at time iswhere .
Therefore, is a smooth Bell-shaped curve and reaches its maximum value at time
The Rayleigh curve first increases to a peak, and then decreases at a decelerating rate.
Substitute the in (16) into (7), we obtain
3.1.3. The Logistic TGEF
Logistic TEF was originally proposed by Parr [16] about software reliability growth model. It exhibits similar behavior to the Rayleigh curve, except during the early part of the project. Huang et al. [28, 30, 37] proposed that a logistic testing effort function can be used instead of the Weibull-type curve to describe the test effort patterns during the software development process. In some two dozen projects studied in the Yourdon 1978–1980 project survey, the Logistic TEF appeared to be fairly accurate in describing the expended test efforts [44].
The logistic TGEF over time period can be expressed as
The current TGE expenditure rate at time iswhere .
Therefore, is a smooth Bell-shaped function, and reaches its maximum value at time
Substitute the in (20) into (7), we obtain
Ohba [17, 18] found that the test effort function in software reliability growth could be described by S-shaped distribution with two cases: the delayed S-shaped curve and the inflected S-shaped curve.
3.1.4. The Delayed S-Shaped TGEF [17, 40]
The delayed S-shaped TGEF over time period can be expressed as
The current TGE expenditure rate at time iswhere .
Therefore, is a smooth Bell-shaped function and reaches its maximum value at time
Substitute the in (24) into (7); we obtain
3.1.5. The Inflected S-Shaped TGEF [18, 40]
An inflected S-shaped TGEF over time period can be expressed as
The current TGE expenditure rate at time iswhere .
Therefore, is a smooth Bell-shaped function and reaches its maximum value at time
Substitute the in (28) into (7); we obtain
With reference to the above five Bell-shaped TGEFs, the instantaneous TGEs which are shown in Figures 7, 8, 9, 10, and 11 decrease ultimately during the testability growth life-cycle because the cumulative TGEs approach a finite limit. As shown from (15), (19), (23), (27), and (31), we find that not all the inherent TDLs in the UUT can be fully removed, even after a long TGT period because the total amount of TGE to be consumed during the TGT phase is limited to . This assumption is reasonable because no system testability design company will spend infinite resources on TGT considering the restricted cost & development cycle.
3.2. Bell-Shaped TGEF Dependent STGM
In this section, we will use the above five TGEFs in the foundation of STGM. One of these can be substituted into (10) to have five kinds of STGM which are formulated as follows.
3.2.1. EX-STGM
Substitute (13) into (11); the EX-STGM can be formulated as
3.2.2. RA-STGM
Substitute (16) into (11); the RA-STGM can be formulated as
3.2.3. LO-STGM
Substitute (20) into (11); the LO-STGM can be formulated as
3.2.4. DS-STGM
Substitute (24) into (11); the DS-STGM can be formulated as
3.2.5. IS-STGM
Substitute (28) into (11); the IS-STGM can be formulated as
4. Estimation of STGM Parameters
Fitting a proposed model to actual TGT data involves estimating the model parameters from the real TGT data set. Two popular estimation techniques are MLE and LSE [45]. The MLE estimates parameters by solving a set of simultaneous equations. However, the equation set may be very complex and usually must be solved numerically. The LSE minimizes the sum of squares of the deviations between what we actually observe and what we expect. In order to avoid the solution of complex simultaneous equations of MLE, in this section, we employ the method of LSE to estimate the parameters of the above five TGEFs. Using the estimated TGEFs, the other parameters , , in (5), (6) can also be estimated by LSE.
Due to the limitations of paper size, reference to the above five TGEFs, only the parameters , of the Exponential TGEF in (13) are estimated by the method of LSE. At the same time, the parameters , given in (5) and given in (6) can all be estimated by LSE. For the method of LSE, the evaluation formula , and are as follows:
Differentiating , , with respect to , , and , respectively, setting the partial derivatives to zero, and rearranging these terms, we can solve this type of nonlinear least square problems. For a simple illustration, we only consider the generalized Exponential TGEF.
Take the partial derivatives of with respect to , ; we get
Thus, the least squares estimator is given by solving the above equation, yieldingNext, we have
The parameter can also be obtained by substituting the least squares estimator into the above equation.
Similarly, the parameters , of the Rayleigh TGEF in (16), , , of the logistic TGEF in (20), , of the delayed S-shaped TGEF in (24), , of the inflected S-shaped TGEF in (28) can also be estimated by LSE.
5. Data Analysis and STGM Comparisons
5.1. Data Description
To validate the proposed STGMs with the above five Bell-shaped TGEFs, TGT on a stable tracking platform have been performed. The TGT data set employed (listed in Table 1) was from the testability laboratory of National University of Defense Technology for a stable tracking platform which can isolate the movement of moving vehicle, such as car, ship, and aircraft. FMEA of the stable tracking platform had been done and had gained that the stable tracking platform consisted of approximately 350 functional circuit level failures. Over the course of 12 weeks at the design & development stage, 72 TDLs were identified by injecting 203 functional circuits level failures. Failures were injected by 1553B fault injection equipment, ARINC 429 fault injection equipment, RS232/422 fault injection equipment, CAN bus fault injection equipment, and the like. Further, testability designers analyzed the root cause of the TDLs and had tried their best to modify the design of testability and removed 46 TDLs successfully. On the other hand, over the course of 12 weeks at the trial & in-service stage, 110 functional circuit level failures had occurred naturally. 14 TDLs were identified, in which 7 TDLs were removed successfully.
5.2. Criteria for Model Comparison
A STGM can generally be analyzed according to its estimation capability, fitness capability, and predictive capability. That is to say, a STGM can be analyzed according to its ability to reproduce the identified TDL and behavior of the UUT and to predict the future behavior of the UUT from the observed TDL data. In this paper, the STGMs are compared with each other based on the following three criteria.
5.2.1. The Accuracy of Estimation Criterion [39]
For practical purposes, we will use AE to calculate the accuracy of estimation. AE is defined as
is the actual cumulative number of identified TDL after the TGT, and is the estimated number of initial TDLs. is obtained from system testability TDL tracking after TGT.
5.2.2. The Goodness-of-Fit Criteria
To quantitatively compare long-term predictions, we use MSE because it provides a well-understood measure of the difference between actual and predicted values. The MSE is defined as [30, 37, 45]
A smaller MSE indicates a smaller fitting error and better performance.
5.2.3. The Predictive Validity Criterion
The capability of the model to predict TDL identification and removal behavior from present & past TDL behavior is called predictive validity. This approach which was proposed by Musa et al. [45] can be represented by computing RE for a data set
Assuming that we have identified and removed TDLs by the end of TGT time , we employ the TDL data up to time () to estimate the parameters of . Substituting the estimates of these parameters in the MVF yields the estimate of the number of TDLs by time . The estimate is compared with the actual number . The procedure is repeated for various values of . We can check the predictive validity by plotting the relative error for different values of . Numbers closer to zero imply more accurate prediction. Positive values of error indicate overestimation and negative values indicate underestimation.
5.3. STGM Performance Analysis
In this section, we present our evaluation of the performance of the proposed STGMs when applied to DS listed in Table 1.
All the parameters of the Exponential TGEF, the Rayleigh TGEF, the logistic TGEF, the delayed S-shaped TGEF, and the inflected S-shaped TGEF are also estimated by LSE. Firstly, the two unknown parameters , of the Exponential TGEF are solved by LSE, giving the estimated values (fault times) and /week. Correspondingly, the estimated parameters of the Rayleigh TGEF are (fault times), and /week. In similarly, the estimated parameters of the logistic TGEF, the delayed S-shaped TGEF, the inflected S-shaped TGEF are (fault times), , /week, (fault times), /week, (fault times), , and /week. In order to clearly show the comparative effectiveness of the fitness for the observed TGE data, the comparisons between the observed five current TGE data and the estimated five current TGE data are illustrated graphically in Figures 12 and 13, respectively. Similarly, the comparisons between the observed five cumulative TGE data and the estimated five cumulative TGE data are illustrated graphically in Figures 14 and 15, respectively.
In order to check the performance of the five above TGEFs and make comparisons with each other, here we select some comparison criterions for the estimation of TGEFs [30, 37, 45]
The PE, Bias, and Variation for the above five TGEFs are listed in Table 2. From Table 2, we see that the inflected S-shaped TGEF has lower values of PE, Bias, and Variation than the other four TGEFs. On average, the inflected S-shaped TGEF yields a better fit for this data set, which can also be drown from Figures 12 to 15 approximately.
Table 3 lists the estimated values of parameters of different STGMs. We also give the values of AE, RE, and MSE in Table 3. It is observed that the STGM with inflected S-shaped TGEF (i.e., IS-STGM) has the smallest value of MSE when compared with other STGMs.
From Table 3 we see that the LO-STGM has lower values of AE and RE than the other four Bell-shaped TGEF dependent STGMs and that the RA-STGM has lower value of MSE than the other four Bell-shaped TGEF dependent STGMs. On average, the logistic TGEF dependent STGM yields a better fit for this data set.
Finally, Figures 16, 17, 18, 19, and 20 depict the RE curve for different selected STGMs.
Substitute the estimated value of , , listed in Table 3 into (32), (33), (34), (35), (36), we can find five TGEF dependent STGMs of FDR. Figures 21, 22, 23, 24, and 25 depict the growth curve of FDR at the whole TGT stage.
Finally, the performance of STGM strongly depends on the kind of data set. If a system testability designer plans to employ STGM for estimation of testability growth of UUT during system development processes, the testability designers need to select several representative models and apply them at the same time. Although models sometimes give good results, there is no single model that can be trusted to give accurate results in all circumstances.
From our results, we can draw the following conclusions.(1)The Bell-shaped TGEF may be a good approach to providing a more accurate description of resource consumption during TGT phase. Particularly, the inflected S-shaped TGEF has the smallest value of Bias, Variation, and PE compared with the other four TGEFs when applied to DS for a stable tracking platform.(2)By incorporating the Bell-shaped TGEF into the structure of STGM, the STGMs with different TGEFs are very powerful and flexible for various test environments. The RA-STGM has the smallest value of MSE compared with the other four STGMs when applied to DS for a stable tracking platform. Similarly, the LO-STGM has the smallest value of AE and RE compared with the other four STGMs.
Acronyms
| UUT: | Unit under test |
| ATE: | Automatic test equipment |
| BIT: | Built-in test |
| FMEA: | Failure mode and effect analysis |
| FDR: | Fault detection rate |
| FIR: | Fault isolation rate |
| FAR: | False alarm rate |
| HPP: | Homogeneous Poisson process |
| NHPP: | Nonhomogeneous Poisson process |
| STGM: | System testability growth model |
| TGT: | Testability growth test |
| TDL: | Testability design limitation |
| TGE: | Testability growth effort |
| TGEF: | Testability growth effort function |
| MVF: | Mean value function |
| EX-STGM: | A STGM with an Exponential TGEF |
| RA-STGM: | A STGM with a Rayleigh TGEF |
| LO-STGM: | A STGM with a logistic TGEF |
| DS-STGM: | A STGM with a delayed S-shaped TGEF |
| IS-STGM: | A STGM with an inflected S-shaped TGEF |
| LSE: | Least square estimation |
| MLE: | Maximum likelihood estimation |
| MSE: | Mean square of fitting error |
| RE: | Relative error |
| PE: | Prediction error |
| MRE: | Magnitude of relative error. |
| : | Counting process for the total number of TDLs identified and removed in |
| : | Total number of failures in the UUT, which can be acquired by FMEA |
| : | Number of initial TDLs present in UUT before TGT |
| : | Time-dependent TDLs content function, which will decrease with the identification and removal of TDLs |
| Number of TDLs remaining in UUT when the TGT has been stopped | |
| : | MVF of TDL identified and removed in time |
| : | Cumulative number of TDL identified up to |
| : | Cumulative number of TDL removed up to |
| : | TDL intensity for |
| : | Constant TDL identification rate, which is the TDL identification intensity for |
| : | Constant TDL removal rate, which is the TDL removal intensity for |
| : | Cumulative TGE consumption up to time , which can be measured by the number of fault injected or occurred naturally, test cost, man-hour, and so on |
| : | Current TGE consumption at time |
| : | Time-dependent FDR function in the whole life cycle |
| : | |
| : | Total amount of TGE eventually consumed |
| : | The Exponential TGEF |
| : | Total amount of TGE eventually consumed in an Exponential TGEF |
| : | TGE consumption rate of an Exponential TGEF |
| : | |
| : | The Rayleigh TGEF |
| : | Total TGE eventually consumed of a Rayleigh TGEF |
| : | TGE consumption rate of a Rayleigh TGEF |
| : | |
| : | The logistic TGEF |
| : | Total TGE eventually consumed of a logistic TGEF |
| : | Constant parameter in the logistic TGEF |
| : | TGE consumption rate of the logistic TGEF |
| : | |
| : | The delayed S-shaped TGEF |
| : | Total TGE eventually consumed of a delayed S-shaped TGEF |
| : | TGE consumption rate of the delayed S-shaped TGEF |
| : | |
| : | The inflected S-shaped TGEF |
| : | Total TGE eventually consumed of an inflected S-shaped TGEF |
| : | Constant parameter in the inflected S-shaped TGEF |
| : | TGE consumption rate of the inflected S-shaped TGEF |
| : | |
| : | Expected number of TDLs by time estimated by a model |
| : | Actual number of TDLs by time . |
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.
Acknowledgments
The authors would like to acknowledge the support and constructive comments from the editor and the anonymous reviewers. Particularly, we appreciate Dr. Hong-Dong Fan for his constructive and insightful suggestions for improving the details of this paper. The research reported here is partially supported by the NSFC under Grants 61304103.