A Quantitative Assessment Approach to COTS Component Security

Abstract

The vulnerability of software components hinders the development of component technology. An effective assessment approach to component security level can promote the development of component technology. Thus, the current paper proposes a quantitative assessment approach to COTS (commercial-off-the-shelf) component security. The steps of interface fault injection and the assessment framework are given based on the internal factors of the tested component. The quantitative assessment algorithm and formula of component security level are also presented. The experiment results show that the approach not only can detect component security vulnerabilities effectively but also quantitatively assess the component security level. The score of component security can be accurately calculated, which represents the security level of the tested component.

1. Introduction

In the area of software engineering, component-based software engineering (CBSE) has currently become a research point. The various development technologies of components further improve the development efficiency and performance of components, but the problems on the reliability and security of components have not yet been resolved. Current component testing approaches have focused on component functionality testing, which, to some extent, guarantees the correctness and integrity of component functionality. However, component security testing is rarely researched as a special subject. Component security testing mainly involved in testing the vulnerabilities of component which can be exploited to crack software component. There are some general component vulnerabilities such as memory leakage, stack overflow, access beyond boundary, and access protected page. In addition, the special difficulties for the security testing of COTS component are as follows. Firstly, the source codes are unavailable. Secondly, The general security pattern is difficult to attain by analyzing the general classification model of software vulnerability. Finally, The COTS component is very independent, which results in some problems such as gaining the effective component information and running the independent component. On the other hand, the assessment of component security has not been solved. Research on the quantitative assessment of component security level is rather rare. In the field of software testing, quantitatively assessing the security level of the component system not only guarantees and enhances the system’s reliability and security but also conducts a quantitative assessment level to software developers and facilitates users in selecting software products based on its security to reduce the risk of attack.

Nevertheless, component security vulnerabilities are usually studied, but the quantitative assessment of component security is not common. Most scholars only research on the theoretical description of security assessment, but practical implementation methods are not presented. A basic assessment approach was given in the literature [1]. Based on the knowledge of information security, the main approach is dividing the component security vulnerability into several ranks [2]. The security level of components is assessed using the software testing technology. The relevant experiments were conducted but were not studied in depth.

Enough internal information should be collected before assessing the security level for the software component. But, it is usually very difficult to obtain the internal information for COTS (commercial-off-the-shelf) components, because its source codes are unavailable. In order to obtain more information about COTS component, it is necessary that some data are inputted the tested component to drive the component running. When the tested component is running, some internal information including some security information can be collected by monitoring the running state of the tested component. The data-driven method can obtain internal security information for COTS component. In the current research, we propose a feasible approach to assess the component security level quantitatively through using data-driven method and learning from relevant statistical and probabilistic methods. The exceptional component information monitored by the dynamic monitoring system can be obtained by the testing system of component vulnerability [3]. Relevant security factors are assigned to appropriate security exceptions according to the classification of the vulnerability in the Common Vulnerabilities and Exposures (CVE) vulnerability library. Each of the exceptional vulnerability factor values can be obtained from the triggering causes of software security vulnerabilities, which is a harmful degree combined with the characteristics of the component security vulnerabilities. Finally, six commercial components, in which security vulnerabilities exist, are collected for experimental analysis in the CVE Web site and are assessed by the proposed approach. The number of each interface and methods used for each interface can be obtained by dynamically executing the tested components. The executed probabilities of each interface and methods used for each interface can be determined by statistical method. At the same time, fault injection operators and the logical prediction rules of exceptional database (PRED) are proposed according to the component security exceptions commonly used. This approach not only can assess the component security level but also effectively assess the security level of each interface, thus providing better assessment capabilities.

2. Related Work

At present, commercial-off-the-shelf (COTS) components, such as military, mechanization, financial components, and so on, are used by more and more commercial software including some crucial security software. COTS security takes an important part in component systems because the security of the whole software system is determined by it. Therefore, assessing the security level of the component is used to test the security of the entire software system, which has become an essential part in component testing. To the best of our knowledge, research on assessing the software component security mainly focuses on component security requirements and assessment approaches, vulnerability assessment, and quantitative risk assessment.

Han and Zheng [4] and Jabeen and Jaffar-Ur Rehman [5] studied the security requirements and security assessment methods of components but did not give the specific relevant assessment methods. Md. Khan et al. examined the security testing description and component assessment [6–8]. Models of the user data security, component security framework, and security assessment model of software were proposed. Khan and Han developed an assessment scheme for the security properties of software components. The assessment scheme provided a numeric score that indicates the relative strength of the security properties of the component [8]. However, the approach consists of two prerequisites: (1) a security requirement specification of the candidate component and (2) a component-specific security rating. The approach requires the source codes of candidate component to be available. The security property descriptions of the components based on the security assessment standards of information technology were proposed by Khan et al. [9]. The main attributes of resource allocation, user data protection, and communications were included. The model was based on the active interface, which could provide the basis for reasoning and assessing a component’s security to meet certain security requirements of a particular application. Common criteria (CC) provide a schema for evaluating information system and enumerate the specific security requirements for such systems [10]. The functional requirements in CC describe the security behavior or functions expected of an information system to counter threats, but not all the requirements in CC may be applicable directly to software components because of the distributed nature of software components and their compositional complexity.

Alhazmi and Malaiya proposed a time-based model for the total vulnerabilities and an alternative model which is analogous to the software reliability growth models [11]. However, this model cannot quantitatively assess the security level of software. Zhang et al. presented a vulnerability risk assessment method based on the attack tree model and Bayesian network model [12]. The analysis method can assess the risk probability of the software system but not its security level. The security risk assessment was introduced to the classical MDA (Model-Driven Architecture) framework [13]. The extension offers early assessment and early validation of security requirements. However, the improved framework still has not provided the quantitative mechanism of the software security. A systematic methodology was proposed for some software vulnerability assessment and security function verification [14]. A scalable and adaptable automatic test system was implemented to test hundreds of software releases, but the security assessment of software releases was not implemented. The state of the architecture-based approach to reliability assessment of component-based software was presented [15]. The common requirements of the architecture-based models were identified, and the classification was proposed. However, the approach only gave a framework for the reliability assessment of component-based software.

Idongesit proposed a quantitative risk assessment model based on annualized loss expectancy and the exposure factor, which is the percentage of asset loss due to the potential threat [3]. The model can assess the potential threats to a development effort and rank these threats based on the risk metric calculation. The Common Vulnerability Scoring System (CVSS) [16] provides an open framework for communicating the characteristics and effects of IT vulnerabilities. Its quantitative model ensures repeatable accurate measurement while enabling users to see the underlying vulnerability characteristics used to generate the scores. However, CVSS measures software vulnerability based on the external factors of software system. Occasionally, the result cannot precisely represent the software vulnerability.

In summary, we propose a new quantitative assessment of component security (QACS) approach to represent more accurately the security level of the COTS component based on the internal factors of software component. The internal factors of software component are some important information inside the component instead of the outside enviornment. There are some main internal factors such as the component objects, component interfaces, component methods, and component attributes. These internal factors reflect more accurate security features for the tested component.

3. A Quantitative Assessment Framework for Component Security

With the further development of component-based software engineering, COTS components have become frequently used in software systems. The source codes of the COTS component are unavailable and highly independent. The traditional testing approach of interface mutation is suitable for system integration testing. That is, mutation operators are only effective at the interface between the module and the subsystem. Therefore, the traditional method is unsuitable for testing the COTS component system. Interface mutation testing, which is suitable for the COTS component, can inject faults at the component interface level by extending the mutation operators in the program mutation and increasing the mutation operators at the component interface level. The defects of components can be identified through observation and analysis of the testing result of the system. Figure 1 shows the quantitative assessment framework of the component security level.

Figure 1 

A quantitative assessment framework.

The interface fault injection operators of the COTS component include irregular input parameter operators, mutation operators based on program language, mutation operators based on interface specifications [17], and mutation operators based on interface definition language (IDL) [18]. The fault injection operators, which are commonly used, are as follows: boundary value parameter, parameter beyond the length, parameter attribute replaced, parameter swap, parameter change, parameter assignment, parameter set null, and so on.

The test cases of fault injection can be generated through the selection of a suitable fault injection operator based on parameter types and number of component methods in the interface information. The test scripts can be generated according to the test cases. The component can be run with the test scripts. The running state can be monitored through the dynamic monitoring mechanism. Component vulnerability of the interface methods can be determined by the running state of the component, security prediction mechanism, prediction rules, and vulnerability factor in various methods. The vulnerable score can be obtained under certain running environments in terms of the vulnerability assessment method of component 𝐶. The vulnerability score, which can assess the security of component 𝐶, is called the security level of component.

3.1. Fault Injection Operator of the Component Vulnerability

Definition 1. Fault injection operator ∏ is the rule that generates a number of fault injection test cases, namely, ∏→{Ψ₁,Ψ₂,…,Ψ_𝑛}. Ψ_𝑖 denotes a fault injection test case. A class of fault injection test cases can be implemented by one fault injection operator.

Definition 2. Interface fault injection operation 𝑂 is a binary relation 𝑂(∏,𝑃), where 𝑃 is a class of interface parameters, 𝑂(∏,𝑃)={(Ψ_𝑖,𝑝_𝑖)∈{Ψ₁,Ψ₂,…,Ψ_𝑛}×𝑃. Fault injection test case, which is implemented by fault injection operator ∏, acts on the interface parameter 𝑝_𝑖, where 𝑂(∏,𝑃) can be short for 𝑂 only if it does not cause confusion.

Whether the generated test cases of fault injection are representative is the key to designing the fault injection operator, that is, whether the generated test cases of fault injection can trigger the component’s security vulnerability as greatly as possible. Two errors, namely, environmental errors and interface parameter errors, can trigger component security vulnerabilities in a component system. The first error, which is discussed in the previous paper [1], does not repeat it again. The focus of the current paper is the interface parameter fault injection. The errors in the interface parameter include three types: IDL description errors, boundary value errors, and irregular input errors. In detail, they are the method errors defined by the IDL description, all boundary value, and irregular input value errors, respectively. Based on the cause of triggering software security vulnerabilities in practical experience, 18 fault injection operators of component vulnerability are specifically designed by comprehensively analyzing the interface mutation operators and the operators of parameter fault injection. These operators are PAS (Parameter Attribute Swap), PSN (Parameter Set Null), IPO (Insert Parameter Operator), PFB (Parameter Flip Bit), PRI (Parameter Reverse-probability Input), IIV (integer Irregular Value), FIV (Float Irregular Value), CIV (Char Irregular Value), BIV (Boolean Irregular Value), RSV (Random String Value), LSV (Long String Value), FSV (Format String Value), DSV (Directory String Value), USV (URL and string value of file directory), CSV (System Command String Value), SSV (SQL string injection), and CSS (Cascading Style Sheet). The new security vulnerabilities found in components can also be added.

In addition, the reverse distribution function of the parameter input domain is generated using the input operator PRI of the parameters of reverse probability distribution. The parameters of reverse probability distribution are then generated. The experimental result shows that component security exception can be effectively triggered through the irregular value of input domain. The input distribution function, which is usually difficult to obtain, is a probability density function. The one dimensional and multidimensional functions are the two commonly used kinds. 𝑄 is assumed the input distribution function, of which the value is in the interval [0,1], and the value 0 indicates the probability of selection input field is 0. The algorithm of reverse probability distribution 𝑄^′ is given in the literature [19].

A number of models have been designed to facilitate the automatic management of the fault injection operator, such as the automatic learning model of the fault injection operator, adaptive model, topic model, intelligent vulnerable trees, among others. They can automatically increase and manage the operator libraries of fault injection.

3.2. Exceptional Predication Rules and Vulnerability Factors

Definition 3. The component exceptional predication rule is a set of all the possible exceptions that may arise under the exceptional circumstances.

Definition 4. The vulnerability factor of the component represents the degree of the component security vulnerabilities resulting from exceptions. The maximum value is 1, and the minimum value is 0. Table 1 shows the component exceptional predication rules and relative vulnerability factors used to assess the component security level. Each of the factor value of exceptional security vulnerability, which can objectively describe the component security level, is obtained from the triggering causes of software security vulnerabilities, which is a harmful degree combined with the characteristics of component security vulnerabilities based on experience software engineering.

Table 1 

Exceptional predication rules and vulnerability factors.

4. Component Security Assessment Algorithm

Suppose 𝐶 is a component, and 𝐼 is a set of interface in 𝐶. Let 𝑝_𝑖 represent the execution probability of interface 𝑖, and 𝑀_𝑖 is a method properties set of interface 𝑖. Meanwhile, 𝑝_𝑖𝑚 is the executable probability in method 𝑚 of interface 𝑖. Suppose |𝐼|=𝑁, |𝑀_𝑖|=𝑆. 𝑥 is the input parameter of the method attribute in 𝐶. 𝑇 is vulnerable error test cases set and 𝑇={𝑡₁,𝑡₂,𝑡₃,…,𝑡_𝑛}. Δ is all the input set of component 𝐶. 𝑄^′ is the reverse probability distribution of Δ. PRED is the logical prediction rules of exceptional database. VC is the vulnerability factor of exceptional function. Before the vulnerability assessment method of component is given, the following definitions should be introduced first.

Definition 5. Let 𝑃 represent an 𝑁×𝑆 probability matrix that can be described as 𝑃=(𝑝_𝑖𝑗)_𝑛×𝑠, row 𝑖 represents the interface number of component 𝐶, and column 𝑗 represents the method number of interface 𝑖. The element 𝑝_𝑖𝑗 states the executable probability for function 𝑗 at interface 𝑖 of component 𝐶, where 𝑝_𝑖𝑗=𝑝_𝑖×𝑝_𝑗.

Definition 6. Interface method coverage (MC), where MC= (the number of methods executed at least once/the total number methods 𝑆) ×100%, is used to measure the sufficient degree of the interface method during component testing.

Definition 7. Interface coverage (IC), where IC= (the number of interfaces executed at least once/the total number interface 𝑁) ×100%, is used to measure the sufficient degree of the interface test during component testing.

Definition 8. Component test adequacy CA can be measured by a two-tuple (i.e., MC and IC). Generally, the value is determined by the total score of MC and IC, that is, CA=[∑_𝑖=1,𝑁(MC_𝑖)]/𝑁+IC. A greater value of CA indicates a more comprehensive test; the relationship between MC and IC is parallel.

Suppose Sum 𝑀 represents the number of test methods, cnt represents the number of vulnerability methods in component 𝐶, and VC_𝑖 represents the vulnerability factors of exceptional method. The value of vulnerability factor VC_𝑖 is assigned to 0 in case the interface method has no security exceptions. The execution probability of the vulnerability method 𝑖 is labeled 𝑃_𝑖. The component vulnerability assessment formula is given as follows: Vul_𝑐=1−^cnt_𝑖=11−𝑃_𝑖×VC_𝑖. (1)

The value 1−Vul_𝑐, the security level of the component 𝐶, can be derived from the Formula (1). The vulnerability assessment formula of the component has the following properties.

Property 9. (0≤Vul_𝑐≤1).

Proof. For any vulnerability method, 1≥𝑃_𝑖≥0 and 1≥VC_𝑖≥0 can be determined according to the definition of VC. Therefore, 1≥𝑃_𝑖×VC_𝑖≥0, 1≥1−𝑃_𝑖×VC_𝑖≥0 and 0≤∏^cnt_𝑖=1(1−𝑃_𝑖×VC_𝑖)≤1 are derived, that is, 0≤Vul_𝑐≤1.

Property 9 shows that the value of any component’s vulnerability index is between 0 and 1, with a larger value indicating a higher vulnerability index.

Property 10.  (Vul_𝑐≥max(𝑃_𝑖×VC_𝑖)≥0).

Proof. For any vulnerability method, 1≥𝑃_𝑖≥0 and 1≥VC_𝑖≥0 can be determined according to the definition of VC. Therefore, max(𝑃_𝑖×VC_𝑖)≥0. In addition, 1≥1−𝑃_𝑖×VC_𝑖≥0 and 1−Vul_𝑐=∏^cnt_𝑖=1(1−𝑃_𝑖×VC_𝑖) can be derived from Formula (1). ∏^cnt_𝑖=1(1−𝑃_𝑖×VC_𝑖) = (1−𝑃₁×VC₁)×(1−𝑃₂×VC₂)×⋯×(1−max(𝑃_𝑖×VC_𝑖))×⋯×(1−𝑃_cnt×VC_cnt); therefore, ∏^cnt_𝑖=1(1−𝑃_𝑖×VC_𝑖)≤(1−max(𝑃_𝑖×VC_𝑖)) and 1−Vul_𝑐≤(1−max(𝑃_𝑖×VC_𝑖)) are derived, that is, Vul_𝑐≥max(𝑃_𝑖×VC_𝑖). Thus, Vul_𝑐≥max(𝑃_𝑖×VC_𝑖)≥0 can proved correct.

Property 10 shows that the vulnerability index of any one component is not less than the peak value of the methods’ vulnerability indexes of the component. All situations of the vulnerability methods of component are synthetically considered by the index of the vulnerability of component.

Based on the component vulnerable assessment method based on the analysis of the interface, the algorithm of the component’s security assessment level is given as follows.

The input probability matrix 𝑃 in Algorithm 1 is not only derived from the component requirement specifications or equal probability but is also statistically calculated after the component is practically executed. The vulnerability factor VC_𝑖𝑗 is assigned to 0 given that the component has no security exceptional interface method. We usually set the value of 𝛿 and 𝜃 from 0 to 1 according to the complexity of tested component and the testing requirement. If the number of interfaces and methods of tested component is no more than 200, we usually let the value of 𝛿 and 𝜃 to be 1 to ensure a complete testing of COTS. In addition, the test cases of interface parameters are generated using the TGSM [20, 21] algorithm. First, the test cases of the interface parameter fault-injection, expressed in matrix form, are generated in terms of fault injection operator. The 𝐾-factor coverage solution matrix is generated by the TGSM algorithm, which is based on the parameter set of the matrix form. The fault injection test cases are composed of row data of the solution matrix. Test cases generated by the TGSM algorithm not only greatly reduce the scale of the test cases but also have certain coverage. The algorithm can trigger most of the security exceptions with fewer test cases.

Input: Interface information XML file; Fault injection operator; Prediction rules PRED; Probability matrix 𝑃; MC’s threshold and IC’s threshold value 𝛿 and 𝜃. Output: The security level of the whole component 01 { 02  Read XML file; 03  MC = 0, IC = 0, 𝑖 = 0, cnt = 0; // cnt is the number of component vulnerability methods 04  While IC < 𝛿 do  // For each interface 𝑖 in 𝐼 05  { 06   𝑖=𝑖+1, 𝑗=0; 07  While MC < θ do  // For each method 𝑚 in 𝑀𝑖 08  { 09   𝑗=𝑗+1; // 𝑗 is the number of the testing methods 10  Generate method parameter values set according to fault injection operators; 11  Call testing cases generating algorithm TGSM(); //call the generation algorithm of the    minimum 𝐾 factors combined cover test case based on solution matrix 12  Running 𝐶 and 𝑀𝑖𝑗; 13  If (the output after running 𝐶 and 𝑀𝑖𝑗 satisfies PRED) 14   { 15      increment cnt; 16      Vul𝑖𝑗=VC𝑖𝑗 × 𝑃𝑖𝑗;// the vulnerability level of the method 𝑗 in the interface 𝑖 17   } 18  MC = 𝑗/𝑆; 19  } 20  IC = 𝑖/𝑁; 21  } 22   Vul𝑐=1−∏cnt𝑖=1(1−𝑃𝑖×VC𝑖); 23 (Output) 1−Vul𝑐; 24 }

Algorithm 1 

QACS (quantitative assessment of component security) algorithm.

Cnt is the number of vulnerability methods of the component 𝐶 in the algorithm results. Vul_𝑖𝑗 is the vulnerability ratio of the method 𝑗 at the interface 𝑖 of the component 𝐶. Vul_𝑐 represents the vulnerability level of the component 𝐶. 1−Vul_𝑖𝑗, the security level of the component 𝐶, represents the security of the method 𝑗 in the interface 𝑖 of component 𝐶.

5. Experiment Analysis

To validate the effectiveness of the proposed approach, the approach was implemented in our component security testing system (CSTS) platform [22]. The component security testing and assessing system based on interface fault injection (CSTASboIFT) was implemented using C # language in the Microsoft. NET platform. It is used as a subsystem of the CSTS platform. Figure 2 shows the function flow chart of CSTASboIFT.

Figure 2 

The function flow chart of CSTASboIFT.

In the experiment, six commercial components (Table 2), in which security vulnerabilities exist, were collected for experimental analysis in the CVE Web site. Several group experiments were conducted for these components. A component composed of six interfaces and each interface consisting of eight methods were assumed so that the component could be regarded as a 6×8 matrix. The first group of experiments tested the condition of 𝑃_𝑖𝑗 assigned to a fixed matrix and VC_𝑖𝑗 divided into interval descending. As VC_𝑖𝑗 changed, the 1−Vul_𝑐 was affected. That is, only the level of vulnerability factors affected the component security level. The second group of experiments tested the condition of VC_𝑖𝑗 assigned to a fixed matrix and 𝑃_𝑖𝑗 divided into interval descending. As 𝑃_𝑖𝑗 changed, 1−Vul_𝑐 was affected. That is, only the executed probability affected the component security level. The last group of experiments considered the effect on 1−Vul_𝑐, as both vulnerability factor and executed probability simultaneously changed.

Table 2 

The six typical tested components.

5.1. Experiment One

The vulnerability factor value is divided into intervals based on the knowledge of security vulnerability provided by the CVE Web site. 𝑃_𝑖𝑗 represents the executable probability of the method 𝑗 at the interface 𝑖, VC_𝑖𝑗 represents the vulnerability factors of the method 𝑗 at the interface 𝑖, and Vul_𝑐 represents the vulnerability level of component 𝐶. 1−Vul_𝑐 is the security level of component 𝐶.

The first experiment assumes that 𝑃_𝑖𝑗 is a 6×8 fixed matrix, which is made up of random values. For every interval, 𝑃_𝑖𝑗 is the same matrix. VC_𝑖𝑗 is divided into 10 intervals: [0.00,0.09], [0.10,0.19], [0.20,0.29], [0.30,0.39], [0.40,0.49], [0.50,0.59], [0.60,0.69], [0.70,0.79], [0.80,0.89], and [0.90,1.00]. The tests were done about ten times with different random values of 𝑃_𝑖𝑗 and VC_𝑖𝑗, respectively, in the experiment, and then the value 1−Vul_𝑐 was got by calculating the average result. Under the condition that 𝑃_𝑖𝑗 is regarded as a 6×8 matrix at every interval of VC_𝑖𝑗, VC_𝑖𝑗 affects the value 1−Vul_𝑐.

When Vul_𝑖𝑗 is equal to 𝑃_𝑖𝑗×VC_𝑖𝑗, as 𝑃_𝑖𝑗 is a fixed value, the values of Vul_𝑐 and Vul_𝑖𝑗 are proportional to the value of VC_𝑖𝑗. The value of 1−Vul_𝑐 is reversely proportional to VC_𝑖𝑗. A larger VC_𝑖𝑗 value indicates a lower security level. As the vulnerability number increases, the security level decreases slowly. The results are shown in Figure 3. The testing results show that the proposed assessment Formula (1) is effective.

(a) cnt=5

(b) cnt=10

(c) cnt=15

(d) cnt=20

Figure 3 

The comparison for different VC𝑖 and cnt.

5.2. Experiment Two

VC_𝑖𝑗 is assumed a 6×8 fixed matrix made up of random values in the second experiment. Then, 𝑃_𝑖𝑗 is divided into 10 intervals: [0.00,0.09], [0.10,0.19], [0.20,0.29], [0.30,0.39], [0.40,0.49], [0.50,0.59], [0.60,0.69], [0.70,0.79], [0.80,0.89], and [0.90,1.00]. The tests were done about ten times with different random values of 𝑃_𝑖𝑗 and VC_𝑖𝑗, respectively, in the experiment, and then the value 1−Vul_𝑐 was got by calculating the average result. Under the condition that VC_𝑖𝑗 is regarded as a 6×8 matrix at every interval of 𝑃_𝑖𝑗,  𝑃_𝑖𝑗 affects the value of 1−Vul_𝑐.

When VuL_𝑖𝑗 is equal to 𝑃_𝑖𝑗×VC_𝑖𝑗, as the value of VC_𝑖𝑗 is fixed, the values of Vul_𝑐 and VuL_𝑖𝑗 are proportional to the value of 𝑃_𝑖𝑗. The value of 1−Vul_𝑐 is reversely proportional to VC_𝑖𝑗. A larger 𝑃_𝑖𝑗 value indicates a lower security level. That is, the component is more insecure. As the number of cnt increases, the decline rate of the security level is initially fast and then becomes slow. The results, as presented in Figure 4, show that the proposed assessment Formula (1) is effective.

(a) cnt=5

(b) cnt=10

(c) cnt=15

(d) cnt=20

Figure 4 

The comparison for different 𝑃𝑖 and cnt.

5.3. Experiment Three

Both parameters 𝑃_𝑖 and VC_𝑖 are assumed to influence the value of 1−Vul_𝑐 in Experiment three. The two parameters are randomly divided into 10 intervals. The effect on the value of 1−VuL_𝑐 when the two parameters changed simultaneously was observed. The results show that when 𝑃_𝑖 is proportional to VC_𝑖, the value of 1−Vul_𝑐 changes significantly as shown in (a) and (b) of Figure 5. When 𝑃_𝑖 is inversely proportional to VC_𝑖, the value of 1−Vul_𝑐 changes indistinctly as shown in (c) and (d) of Figure 5. As the vulnerability number cnt increases, the security level decreases. Results are shown in Figure 5. The values of 𝑃_𝑖 and VC_𝑖 are based on the main 𝑦-axis values (the left axis), and the value of the 1−Vul_𝑐 is based on the secondary 𝑦-axis values (the right axis). The testing results show that the proposed assessment Formula (1) is suitable for assessing the COTS component.

(a) cnt=5

(b) cnt=10

(c) cnt=15

(d) cnt=20

Figure 5 

The comparison for different 𝑃𝑖, VC𝑖, and cnt.

5.4. Experiment Four

The components were assessed in Table 3 based on Algorithm 1. The value of IC and MC is set to 1. 𝑃_𝑖 can be determined by executing the tested component 100 times or more.

Table 3 

Testing results for six tested components.

During the system testing process, the test cases of fault injection can be implemented by testing the component information and fault injection operators. The testing state can be recorded through the dynamic monitoring mechanism after running the tested component. The vulnerability type of the component method and vulnerability factor can be derived from comparing the testing state with the component information record in PRED. The execution probability 𝑃_𝑖 of each method in the component can be determined by running the tested component 100 times, respectively. Then, the component security level can be determined using Formula (1). The testing results are shown in Table 3. Because different components have different method parameters which have different parameter types such as integer, string, and boolean, we select suitable fault injection factors to test the component security based on the above presented 18 fault injection operators of component in Table 3.

A higher level of security indicates better relative security of the component. The component is secure if and only if the component security level is 1. Otherwise, it has security vulnerability. In contrast to the test results in Table 3, the level of security is closer to the vulnerability information published in the CVE Web site. The values ​​of execution probability and vulnerability factors have been proved to affect the component security. The results agree with the previous experimental results, further confirming the feasibility and effectiveness of this approach.

5.5. Experimental Comparison

Table 4 shows the comparison with other assessment approaches. The related assessment approaches have various merits as shown in the analysis of Table 4. Our QACS approach mainly focuses on quantitatively assessing the security level of the COTS component. The source codes of the COTS component are generally unknown. The internal factors of the component are considered in QACS. The QACS approach is implemented in the CSTASboIFT that is integrated into our CSTS [22] system developed to test the security of the COTS component.

Table 4 

The comparison with related assessment approaches.

Aside from the above-mentioned approaches, CVSS is a widely used quantitative assessment approach for software component. Khan and Han developed an assessment scheme for the security properties of software components. The assessment scheme provided a numeric score that indicates the relative strength of the security properties of the component [8]. Results of the comparison with the CVSS approach and Khaled approach are shown in Table 5. In addition, the security level of six tested components is also listed according to the reported empirical data of security Web sites in Table 5. The initial CVSS Base Score is calculated by the CVSS formula [16].

Table 5 

The comparison with CVSS and Khaled approaches.

Analyzing Table 5, the QACS approach is found to be close to the empirical data reported on some security Web sites regarding the security of the tested COTS components. The security score of the QACS approach is more accurate than that of others because the internal factors of the tested component are considered. The assessment process of the QACS approach is objective, without human intervention. Nevertheless, the CVSS approach is mainly focused on assessing the vulnerability of the general software component. If an assessor is not familiar with the assessed software, the vulnerability score becomes inaccurate because of the assessor’s subjectivity in the assessment process. On the other hand, the Khaled approach is an approximate assessment approach based on the assumed component environment. In addition, presently there are no more effective assessment approaches for the security level of software component according to our studies.

6. Conclusion

In component-based software development and maintenance activities, the assessment of the component reliability and security level is essential in the third-party component. Research on an effective quantitative assessment method for component security level is valuable in the CBSE. The current paper proposes the QACS approach for quantitatively assessing component security, which is based on the interface fault injection of the component. The fault injection operators of the component vulnerability and PRED are defined and discussed. The assessment framework and algorithm for component security are proposed as well. The testing effect proves excellent in practical application. QACS has the following main advantages: (1) the security exceptional assessment rules and vulnerability factors of the component are given, providing the basis for security assessment; (2) QACS can assess the security level of the third-party component, and the score of component security can be accurately calculated; (3) QACS has better operability.

Acknowledgments

We would like to thank Professor T. Y. Chen for his constructive suggestions and valuable discussion in this paper writing. In addition, this work was supported partly by the National Natural Science Foundation of China (NSFC) under Grants no. 61202110 and no. 61063013, Natural Science Foundation of Jiangsu Province under Grant no. BK2012284, and the Research Fund for the Doctoral Program of Higher Education of China under Grant no. 20103227120005.