Abstract
Validation is critical to the success of software trustworthiness measurement. A large number of software trustworthiness measures are proposed; however, most of them are not validated from a theory perspective. Therefore, they lack theoretical foundation and will induce unnecessary cost and useless calculation. In this paper, we bring measurement theory into software trustworthiness measurement, construct a source codes oriented software trustworthiness measure based on extensive structure in the measurement theory, and validate the developed measure by use of axiomatic approaches. Compared with some software trustworthiness measures that are already presented, this measure can evaluate software trustworthiness better from a theory perspective.
1. Introduction
Because of the outstanding problems with software trustworthiness, people pay more and more attention to the research into trustworthy software. One of the core scientific problems in this research is the software trustworthiness measurement [1]. However, few research works carry out theoretical validation for their measures. It is important that the measures we use are valid. That is, measures must measure what they are supposed to measure. Theoretical validation is a required activity for using or defining measures that make sense and is a necessary step for the empirical validation of measures [2]. On the other side, many researches do not present the methods to translate the measure results back to the empirical world. Then it is difficult to use the measure results to provide guidance for the improvement of software trustworthiness. Meanwhile, most researchers in software trustworthiness measurement area do not address scale types. Scale types are very important for meaningful statistical operations. Because many empirical and numerical conditions are not covered by a certain scale type, for example, comparisons of arithmetic means are not a meaningful statistic [3, 4].
There are two main approaches in theoretical validation and they are measurement theory and axiomatic approaches [5, 6]. In measurement theory, the empirical understanding of an attribute is formalized through the definition of an empirical relational system and a numerical relational system is defined to provide measure values for that attribute and relations among these values [7]. Measurement theory presents a clear definition of a measure under the assumption of a homomorphism from the empirical world into the numerical world and gives the conditions for the use of measures on certain scale levels. The major advantage of measurement theory is the representational condition, which attests that the properties of the attributes in the empirical world should be preserved by the measures in the numerical world and vice versa. Part of the empirical understanding of a software attribute may be formally defined as desirable properties of the measures for that attribute. Axiomatic approaches are used with a number of properties for theoretical validation [8, 9]. A property is a condition or a basic assumption of reality. The measure’s properties in axiomatic approaches must be seen as properties that characterize the numerical world. However, they indirectly affect the empirical world. Therefore, properties in axiomatic approaches can be used as guidelines for the definition of a measure.
Software trustworthiness measures based on measurement theory and axiomatic approaches are appropriate methods to deal with problems, like theoretical validation of measures, measurement scales, the empirical interpretation of numerical results, and properties of measures. However, measurement theory and axiomatic approaches are mostly applied to measure internal attributes such as size, complexity, and cohesion or used to validate the software internal attributes measures from theory [2, 9–13]. The fundamental problem of the application of measurement theory or axiomatic approaches in software external attributes measurement is that, for many software external attributes, it is not known what the empirical relational system looks like. In order to make software trustworthiness measurement more rigorous, we used axiomatic approaches to measure software trustworthiness and presented some desirable properties of the measures for software trustworthiness in the view of trustworthy attributes [8], including monotonicity, acceleration, sensitivity, and substitutivity. In this paper, we construct a source codes oriented software trustworthiness measure based on extensive structure in the measurement theory, present four desirable properties of the measures for software trustworthiness from the standpoint of module, and carry out theoretical validation for the developed measure by use of axiomatic approaches.
The rest of this paper is organized as follows. In Section 2 we review some related works. We describe some basic definitions and notations from measurement theory and present extensive structure based software trustworthiness measures in Section 3. In Section 4 we introduce four desirable properties of software trustworthiness measures from the standpoint of module and validate the nonadditive software trustworthiness measure based on extensive structure established in Section 3 by use of axiomatic approaches. The measurement procedure based on the measures built in Section 3 is presented in Section 5. We make a case study in Section 6. Section 7 contains the comparative study. The conclusion and future work come in the last section.
2. Related Works
There are three kinds of software trustworthiness measurements: attributebased software trustworthiness measurement, behaviorbased software trustworthiness measurement, and processbased software trustworthiness measurement [14].
The general framework for the attributebased software trustworthiness measurement is as follows. The attributes that affect software trustworthiness are firstly selected. Then, measurement models are built to measure these selected attributes. Finally, models and methods are presented to measure software trustworthiness based on measurement results of the selected attributes. Typical models and methods include axiomatic approaches [8, 15–20], fuzzy comprehensive evaluation method [21, 22], rough set theory [23], Bayesian Networks [24–26], evidence theory [27, 28], weakness analysis [29], questionnaire survey and statistical analysis [30–33], and dynamic system [34–36].
The software normal behavior model is firstly built under trustworthy environment in the behaviorbased software trustworthiness measurement. Then, the software actual behavior is extracted based on the monitored actual operating statuses of the software at the checkpoint. Finally, the actual behavior is compared with the normal behavior to evaluate software trustworthiness. Typical comparison methods include program slicing [37, 38], data mining [39, 40], and concurrency theory [41–46].
Processbased software trustworthiness measurement uses the collected data describing the behaviors and attributes of the process as evidence to measure software trustworthiness. Classic methods contain Trusted Software Methodology [47, 48], Trustworthy Process Management Framework [49, 50], and so on.
3. Software Trustworthiness Measure Based on Extensive Structure
We first present some basic concepts from measurement theory.
Definition 1 (empirical relational system [7]). Given an attribute, an empirical relational system is an ordered tuple , where (1)E denotes a nonempty set of empirical objects for which we want to measure the attribute;(2) are ary empirical relations on capturing our intuitive knowledge on the attribute;(3) are empirical binary operations on , i.e., .
The empirical relational system is used to model our empirical understanding of the objects’ attributes that we want to measure.
Definition 2 (numerical relational system [7]). Given an attribute, a numerical relational system is an ordered tuple , where (1) is the set of values that we use to measure the attribute;(2) denote ary relations on ;(3) denote numerical binary operations on , i.e., .
The numerical relational system is purposefully defined to mirror the empirical relational system in the realm of values. The link between the empirical relational system and the numerical relational system is made via measure.
Definition 3 (measure [7]). A function is said to be a measure.
However, with the exception of empirical objects, the empirical relational system also presents information about our empirical understanding of an attribute of a set of empirical objects. If this information is not considered, any function that maps into is a measure according to Definition 3. In order to discard all of those measures that are at variance with our intuition, measurement theory introduces the definition of scale.
Definition 4 (scale [7]). Let be an empirical relational system, be a numerical relational system, and be a measure. The triple is a scale if and only if for all and for , the following holds:
Extensive structure is one of the most important measurement structures. It is essential in physics and is very important in the area of software measurement too. They are not brought into software external attributes measurement and software trustworthiness measurement. The reason is that it is difficult to build the extensive structure about software external attributes or software trustworthiness. The following is the definition of extensive structure.
Definition 5 (extensive structure [7]). Let be a nonempty set, a binary relation on , and a closed binary operation on . The empirical relational system is an extensive structure if the following axioms are satisfied: (1)Weak order: is a weak order; i.e., it satisfies transitivity and strong completeness.(2)Weak associativity: (3)Monotonicity: (4)Archimedean: : if , then for any there exists a natural number n, such that . One has
The most practical use of measurement is the representation theorem of extensive structure.
Theorem 6 (representation theorem of extensive structure [7]). Suppose is a nonempty set, a binary relation on , a closed binary operation on , and the set of all real numbers. Then there is a realvalued function satisfying if and only if is an extensive structure. If another function fulfills these properties, then there exists a positive real value that holds.
In order to apply extensive structure in software trustworthiness measurement, the first step is to define an empirical relational system about software trustworthiness. Therefore, the software entity must be represented in one way or another. In this paper, we use the software abstraction presented in [12] and define software as a system. However, we modify the definition of module given in [12].
Definition 7 (system [12]). A system is represented as a pair , where is the set of elements of and represents a binary relation on expressing the relationships between ’s elements.
Definition 8 (module). For a given system , a module is a system such that As an example, can be defined as the set of code statements and as the set of data flows from one statement to another. A module may be a code segment or a subprogram.
Given a system , let be the set of all the modules in . Then is the set that are to be measured. Suppose , and let Then, we define the binary operation on as where is a multiset composed of the elements in and . From the definition of module, we know that .
Now we need to define the empirical ordering : equally or more trustworthy than. Software trustworthiness is the ability of software to satisfy user expectation with its behaviors and results. Both the software behaviors and the results are decided by the implementations of software. Then the software trustworthiness can be obtained by computing the quantification of the conformance of the implementations adopted by software with the implementations expected by users. The implementations expected by users are referenced as the properties and functions that the program elements and program unit should carry if a program is trustworthy; for example, a variable should be initialized before used [51]. Moreover, software is composed of the modules through some relations and the module is the collection of program elements. So we let be determined by the following observation criterion: If , then is observed if and only if when the ratio of the number of program elements with the implementations expected by users in to the number of program elements in is greater or equal than the ratio of the number of program elements with the implementations expected by users in to the number of program elements in .
Claim 1. is an extensive structure.
Proof. Transitivity: let , . Suppose the ratios of the number of program elements with the implementations expected by users in to the number of program elements in are , and , respectively.
Then, from the definition of , we know that implies and implies . As the relation is transitive for real numbers, then it holds that , leading to the observation that .
Strong completeness: since any pair of real numbers can be compared using , then strong completeness is shown by the definition of .
From and , we can get is a weak order.
Weak associativity: because therefore Similarly, since then Notice that associativity is satisfied by , therefore (4) Monotonicity: we first prove that ,
The number of program elements with the implementations expected by users in is the sum of the number of program elements with the implementations expected by users in and that in . Because implies the number of program elements with the implementations expected by users in is equal or more than the number of program elements with the implementations expected by users in , therefore, the number of program elements with the implementations expected by users in is equal or more than the number of program elements with the implementations expected by users in ; then it follows that Likewise, if , we can deserve that .
In a similar way, we can prove that
(5) Archimedean: since satisfies monotonicity, then it is easy to get that Archimedean holds for .
In summary, is an extensive structure.
Definition 9 (an additive module trustworthiness measure based on extensive structure). For a given software system , let be the set of all the modules in and be the set of all positive real number. Suppose . Denote the number of program elements with the implementations expected by users in by and the number of program elements in by . Then, the additive module trustworthiness measure based on extensive structure is defined as , where
According to the representation theorem of extensive structure and the definition of , it is easy to obtain the following result.
Claim 2. (1) assumes the extensive structure .
(2) For , i.e., is an additive homomorphism from to .
Definition 10 (a nonadditive module trustworthiness measure based on extensive structure). For a given software system , let be the set of all the modules in and be the set of all positive real numbers. Suppose , , . Then, the nonadditive module trustworthiness measure based on extensive structure is defined as , where
Claim 3. (1) assumes the extensive structure , and for , i.e., is a nonadditive homomorphism from to .
(2) can be used as a ratio scale.
Proof. Let Because then and From Claim 2, we know that Then we can get Because , therefore is a nonadditive measure and can be modified to the additive measure by a strictly monotonic function. From the result in [18], it follows that assumes the extensive structure too.
(2) Noticing that it follows that can be used as a ratio scale.
To sum up, the conclusion can be verified.
In the following, on the basis of the module trustworthiness measures described in Definitions 9 and 10, we present an additive software trustworthiness measure based on extensive structure and a nonadditive software trustworthiness measure based on extensive structure. For a given software system , let be a collection of modules of such that and i.e, the set of elements of is partitioned into the sets of elements of the modules in . Suppose the number of the module in is and denote them by separately.
Definition 11 (an additive software trustworthiness measure based on extensive structure). For a given software system , an additive software trustworthiness measure based on extensive structure is defined as
Definition 12 (a nonadditive software trustworthiness measure based on extensive structure). For a given software system , a nonadditive software trustworthiness measure based on extensive structure is defined as
4. Numerical Relational System Based on Axiomatic Approaches
There may be many nonadditive software trustworthiness measures that assume the extensive structure and can be used as a ratio scale. Now the problem is how to prove is a proper software trustworthiness measure. Axiomatic approaches have already been used to check if the existing measures comply with a specific set of properties for the attribute that they purport to measure. We once used axiomatic approaches to measure software trustworthiness and presented some desirable properties of software trustworthiness measures from the standpoint of trustworthy attributes [8]. Consulting the properties given in [8], in this section we give four desirable properties of software trustworthiness measures in the view of module and prove that complies with these four properties.
For generality, let
where(i): a parameter related to the substitutivity between module trustworthiness(ii): the module trustworthiness obtained through with
The desirable properties of software trustworthiness measures in the view of module are depicted below.
(1) Monotonicity where . It means that the increment of a module trustworthiness leads to software trustworthiness increase.
(2) Acceleration where . It states that the increase of the module trustworthiness leads to their utilization efficiency to decrease.
(3) Sensitivity Sensitivity is used to describe the percentage changes of software trustworthiness caused by the percentage changes of module trustworthiness. They should be positive.
(4) Substitutivitywhere are used to express the difficulty of the substitution between module trustworthiness. Substitutivity implies that the module trustworthiness can substitute each other to some extent.
Claim 4. Monotonicity holds for .
Proof. From the definition of , we know that is a continuously differentiable function. By solving the partial derivatives of with respect to , we can obtain Since Then Therefore, the conclusion can be verified.
Claim 5. satisfies the acceleration.
Proof. By computing the second order partial derivatives of with respect to , we get that Because and , therefore Then we can obtain the conclusion that the acceleration is satisfied by .
Claim 6. Sensitivity is satisfied by .
Proof. For , by calculating it is easy to get that Due to , then In summary, is sensitive to module trustworthiness.
Claim 7. complies with substitutivity.
Proof. Since is a continuously differentiable function, and Then, for , we can get According to (33), it follows that Since Then substitutivity holds for .
5. The Measurement Procedure Based on the Measures Established in Section 3
The measures given in this paper are used to measure source codes trustworthiness in the software implementation phase, and the source codes can be written in an imperative language or in an objectoriented language. We have presented specifications for the implementations expected by users of the program elements involved in the imperative languages [51]; if you want to use the measures built in this paper to measure the trustworthiness of source codes written in an objectoriented language, you need to propose the specifications for the implementations expected by users of the program elements involved in the corresponding objectoriented language.
The measurement procedure based on the proposed measures in Section 3 consists of four steps as shown in Figure 1. For a given software system , we first find a partition of , where is the set of all modules in . Then, for , we compute according to Definition 9 in Step 2. For , based on the results of computation of , are calculated according to Definition 10 in Step 3. In the final step, on the basis of the results of Step 3, the software trustworthiness is obtained by utilizing Definition 12.
6. Case Study
To demonstrate the effectiveness of our measure, we apply to measure the trustworthiness of quick sort program written in the C language in Algorithm 1. For simplicity, we just consider the program element: variable. Here we use the trustworthy properties of variable given in [51] as variables’ implementations expected by users. Let , where(i) is composed of the variables in line , i.e., (ii) consists of the variables in line , i.e., (iii) is made up of the variables in line , i.e.,

Then . From the trustworthy properties of variable described in [51], we know that do not have one and only one purpose, the same identifiers are still used in main function, and then they are not expected by users. Moreover, and do not have meaningful names; therefore, they are not expected by users. So, we can get , , and . It follows that , , and .
Suppose ; then , , , and
7. Comparative Study in Terms of Measurement Theory and the Properties Given in Section 4
In this section, we compare with the decomposition of trustworthy attributes based software trustworthiness measure (DTABSTM) [16], axiomatic approaches based software trustworthiness measure (AABSTM) [17], fuzzy theory based software trustworthiness measure (FTBSTM) [22], and evidence theory based software trustworthiness measure (ERBSTM) [28] through the measurement theory and the properties presented in Section 4. The comparative results are given in Table 1, where represents that the measure holds for corresponding property and expresses that the measure does not comply with the corresponding property. DTABSTM, AABSTM, ERBSTM, and FTBSTM do not build measurement structures in the measurement theory; therefore they do not comply with measurement theory. References [16, 17] have proved that monotonicity, acceleration, sensitivity, and substitutivity hold for DTABSTM and AABSTM separately. ERBSTM and FTBSTM do not satisfy acceleration and substitutivity. The reason is that both of them do not consider the efficiency of using attributes and the quantitative relations between trustworthy attributes, while correlations between trustworthy attributes are the cause that trustworthy attributes can substitute each other to some extent. From Table 1, we can get that is better than all the four measures in terms of measurement theory and the properties introduced in Section 4.
8. Conclusion and Future Work
In this paper, we apply measurement theory to measure the source codes trustworthiness, present an extensive structure based software trustworthiness measure , and validate with axiomatic approaches from a theory perspective. Compared with some popular software trustworthiness measures, the measure can evaluate software trustworthiness better in terms of measurement theory and the properties described in Section 4.
However, only a small case is given to demonstrate the effectiveness of , in order to support ’s practical usefulness, we will experiment using the real cases and develop a tool based on . The specifications for the implementations expected by users of the program elements involved in the imperative languages that are given by us in [51] are not complete, and the specifications are only suitable for imperative languages. In the future, we will expand the specifications proposed in [51] and establish specifications for the implementations expected by users of the program elements involved in the objectoriented languages. We do not give methods for computing the values of and ; how to determine the values of and is an important future work. The set of properties presented in Section 4 should be considered sets of necessary properties that need to be satisfied by a software trustworthiness measure. It is possible that some important properties are omitted due to oversight; extending and refining the set of properties are the future works too.
Data Availability
The data used to support the findings of this study are available from the corresponding author upon request.
Disclosure
A preliminary version of this work was presented at the 2016 International Symposium on System and Software Reliability (ISSSR2016) [Research of Software Trustworthiness Measure Based on Validation [52]].
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This work is supported by the Innovation Group Project of National Natural Science Foundation (61321064), National Key R&D Plan InterGovernmental R&T Cooperation Subproject (2016YFE0100600, 2016YFE0100300), National Natural Science Foundation of China (61370100), Doctoral Research Fund of Zhengzhou University of Light Industry (13501050045), and Science and Technology Project of Henan Province (182102210617).