Abstract
Wireless communication protocols are indispensable in Internet of Things (IoT), which refer to rules and conventions that must be followed by both entities to complete wireless communication or service. Wireless protocol conformance testing concerns an effective way to judge whether a wireless protocol is carried out as expected. Starting from existing test sequence generation methods in conformance testing, an improved method based on overlapping by invertibility and multiple unique input/output (UIO) sequences is proposed in this paper. The method is accomplished in two steps: first, maximum-length invertibility-dependent overlapping sequences (IDOSs) are constructed, then a minimum-length rural postman tour covering the just constructed set of maximum-length IDOSs is generated and a test sequence is extracted from the tour. The soundness and effectiveness of the method are analyzed. Theory and experiment show that desirable test sequences can be yielded by the proposed method to reveal violations of wireless communication protocols in IoT.
1. Introduction
Wireless communication is essential and critical to Internet of Things (IoT) [1, 2]. The reliability of wireless communication transmission largely depends on whether the wireless communication protocol is implemented as specified. Conformance testing [3, 4] is widely used to check whether an implementation conforms to its specification in areas such as traditional communication protocols and reactive systems; i.e., there must be the same behavior in the implementation for any I/O behavior observed in the specification. In FSM-based conformance testing, a protocol is called a specification and is expressed as a Finite State Machine (FSM), while an implementation under test is considered to be a “black box”, the I/O behavior of which can only be observed. A test sequence is an I/O sequence such that whether an implementation conforms to the specification may be concluded by delivering the input sequence of the test sequence to the implementation and comparing the resulting output sequence with the output sequence in the test sequence.
Test techniques based on state identification [5–10] are well-known in FSM-based conformance testing. Test techniques based on state identification are supposed to identify every state and verify every transition of the specification in the implementation. A state is said to be identified when a state identifier is delivered to the state. A transition is said to be verified when the end state of the transition is identified. A state identifier of a state is a nonempty set of input sequences for the state such that the set of corresponding output sequences can characterize the state. Unique input/output (UIO) sequence is a popular state identifier such that UIO-based techniques are commonly used for test sequence generation.
Since wireless communication protocols can also be modeled by FSMs, FSM-based conformance testing is also applicable to wireless protocol conformance testing in IoT. Zigbee is a typical wireless communication protocol and is divided into four layers: physical layer (PHY), media access control lay (MAC), network layer (NWK), and application layer (APL). The connection and release process of nodes in the MAC layer is modeled by the FSM in Figure 1 and UIO sequences of every state are listed in Table 1. On this basis, test sequences based on FSMs can be constructed using UIO sequences as state identifiers. There is a test sequence / / of the FSM in Figure 1 based on UIO sequences in Table 1. When the input sequence of this test sequence is applied to an implementation, every state of the FSM in Figure 1 can be identified and every transition of the FSM in Figure 1 can be verified in the implementation. Meanwhile, an output sequence can be obtained. It can be concluded whether the connection and release process of nodes is executed as the specification specifies by comparing the obtained output sequence with the expected one in the test sequence.
Test sequence reduction has long been an active research topic in FSM-based conformance testing. One approach is to convert the problem into the Rural Chinese Postman Problem from which a test sequence is extracted. For the purpose of transition verification every transition of an FSM is followed by an appended UIO sequence in the test sequence. Bo Yang et al. reduced test sequences by overlapping and multiple UIO sequences [11]. Benefiting from overlapping, more than one transition may be verified by a single appended UIO sequence. Hierons improved the method of UIO sequences through the use of an invertibility criterion, thereby achieving more overlapping [12], i.e., verifying even more transitions of an FSM with a single appended UIO sequence. However, multiple UIO sequences which are conducive to test sequence reduction are not considered by Hierons.
In order to reduce the cost of testing while assuring the effectiveness, this paper presents an improved method to generate reduced test sequences for wireless protocol conformance testing of IoT. Transitions in a test sequence are of three kinds: a copy of transitions in an FSM, UIO sequences which have been appended to the test sequence for the purpose of transition verification for the FSM, and the transitions which have been appended to the test sequence. In the improved method, invertibility is taken into account to leverage as much overlapping as possible such that all the transitions of an FSM can be verified with as few appended UIO sequences as possible. Rural symmetric augmentation is the main measure for test sequence generation. The replicated transitions during the rural symmetric augmentation are exactly the transitions for concatenation of test subsequences in the test sequence. More options of rural symmetric augmentation can be supplied by multiple UIO sequences than those supplied by single UIO sequences; i.e., a sensible choice of UIO sequences can lead to a minimum number of replicated transitions during a rural symmetric augmentation, and a minimum number of transitions for concatenation are achieved in the test sequence. In this way, further reduced test sequences are obtained by the proposed method.
The rest of the paper is organized as follows. In Section 2, the basic concepts and assumptions used in this paper are presented. The improved method is described with simple examples in Section 3, and the soundness and effectiveness of the method are discussed. Experimental evaluation is set out in Section 4. Then, the related work about the testing of IoT is reviewed in Section 5 and the paper is concluded briefly in Section 6.
2. Preliminaries
In this section, we introduce the definitions related to FSMs and graphs, together with the assumptions necessary for FSM-based conformance testing.
2.1. Definitions
An FSM is formally defined as a 6-tuple where(i) and are finite and nonempty sets of input symbols and output symbols, respectively;(ii) is a finite and nonempty set of states;(iii) represents the initial state of ;(iv) denotes the state transition function;(v) is the output function.
According to this definition, when an input symbol is delivered to the current state , moves to the state with an output produced by . The transition function and output function can be extended to finite input sequences, i.e., for an input symbol , an input sequence (where is the set of finite sequences of input symbols), and a state , , and where concatenation is denoted by juxtaposition.
A transition is defined by a tuple where is the start state, is an input of , is the associated output, and is the end state. A transition is invertible if it is the unique transition ending at with input and output .
A UIO sequence of a state is an input/output sequence such that the input/output behavior exhibited by the state is unique, i.e., given a UIO sequence of a state , and the input sequence of is denoted by ; for any state , it is always true that . A UIO sequence is irreducible if it is not a UIO sequence anymore when the end symbol of the sequence is deleted. UIO sequences in this paper will be supposed irreducible unless otherwise specified. There may be more than one UIO sequence for a state and the UIO sequences may be of different length.
An example FSM is described in Figure 2 where , , and is the initial state of . There are four invertible transitions , , , and . The UIO sequences for every state are shown in Table 2.
An FSM can be perceived as a labeled, directed graph . A state of is represented as a node of , and there is an edge labeled with from node to in if and only if and in , where and are the start and end node of the edge. The numbers of incoming and outgoing edges of a node are called the in-degree and out-degree of the node, respectively. If the in-degree equals out-degree at each node then the directed graph is symmetric. Suppose that is an asymmetric directed graph; if is a symmetric directed graph generated from by making copies of the edges then is a symmetric augmentation of . When the total cost of copies of edges is minimized, is said to be a minimal symmetric augmentation of . Suppose that is the set of edges of and , if is a symmetric directed graph generated from by making copies of the edges such that each edge in is included in at least once and the total cost of copies of edges is minimized then is called a rural symmetric augmentation of . A sequence of contiguous edges forms a path of where and are the start and end node of the path, respectively. A path that starts and ends at the same node forms a tour, furthermore, if the tour traverses every edge of exactly once then it is an Euler tour. A rural postman tour is a tour that traverses a given set of edges at least once. The Rural Chinese Postman Problem is to find a minimum-length rural postman tour for a given set of edges. FSMs and their directed graph representations are used interchangeably throughout this paper.
2.2. Assumptions
Given a specification FSM with states, the fault domain of is the set of all possible implementations of over the input alphabet of . refers to the implementations with up to states in . This paper only focuses on implementations in .
An FSM is strongly connected if for any two distinct states and there is an input sequence that takes from to ; i.e., , , , .
Two states and are equivalent if for any input sequence there are always the same output sequences from and . An FSM without equivalent states is said to be minimal or reduced; otherwise is reducible by joining equivalent states.
An FSM is deterministic if at any state for any input there is at most one transition leading to the next state. Otherwise, is nondeterministic.
Only strongly connected, minimal and deterministic FSMs are considered in this paper. In addition, it is assumed that UIO sequences for each state of an FSM are available and are derived from successor trees in advance. It is noted that only state transition functions in forms of are considered; i.e., state transitions without any input in IoT are out of the scope of this paper.
3. Test Sequence Reduction
3.1. Key Properties of the Method
3.1.1. Overlapping by Invertibility
Definition 1 (invertibility-dependent UIO sequence). If a transition (, , /) is invertible and is a UIO sequence of , then (/) is an invertibility-dependent UIO sequence of .
The transition of in Figure 2 is invertible and is a UIO sequence of , then is an invertibility-dependent UIO sequence of .
Definition 2 (invertibility-dependent overlapping sequence). Given a transition sequence / and a UIO sequence of , if every transition is verified by an invertibility-dependent UIO sequence when is verified by , then is an invertibility-dependent overlapping sequence (IDOS).
There is a transition sequence of in Figure 2 and a UIO sequence of . When the last transition (, , ) of is verified by the UIO sequence of , by working backward is verified by , is verified by , and is verified by ; i.e., all the other transitions except the last one are verified by invertibility-dependent UIO sequences. As a result, is an IDOS.
For to be invertibility-dependent UIO sequences, indispensable requirements for transitions in are put forward.
Theorem 3. Given a transition sequence and a UIO sequence of , is an IDOS if and only if every transition (, , /) is invertible.
Proof. The sufficiency of the condition is proved inductively by working backward. In the base case when , (, , /) is sure to be verified by an invertibility-dependent UIO sequence since (, , /) is invertible. In the inductive step, assume that () is verified by an invertibility-dependent UIO sequence , then ( is definitely verified by an invertibility-dependent UIO sequence ( since (, , /) is invertible. In this way, every transition () is backward verified inductively by an invertibility-dependent UIO sequence when (, , /) is verified by . Thus, it is a sufficient condition for a transition sequence to be an IDOS that every transition (, , /) is invertible.
Next, the necessity of the condition is proved by contradiction. As shown in Figure 3, suppose that is an IDOS with a noninvertible transition (, , /) ; i.e., there is another transition (, , /) ending at with the same input and output. Obviously, cannot be identified by ; i.e., (, , /) cannot be verified by . This contradicts with the assumption that is an IDOS. Accordingly, it is a necessary condition for a transition sequence to be an IDOS that every transition (, , /) is invertible.
Definition 4 (set of IDOSs). Given a set in which every sequence is an IDOS of an FSM , if every transition of is included in one and only one IDOS of then is a set of IDOSs of .
Of all the IDOSs from a state, a maximum-length IDOS is the one contains no fewer transitions than any other ones. There is no doubt that the longer IDOSs are, the more overlapping can be achieved, and the shorter test sequences will be obtained. For maximum overlapping, this paper is only interested in maximum-length IDOSs.
The set is a set of IDOSs of in Figure 2. There is only one IDOS in the set since the only IDOS already covers all the transitions of . Obviously, is also a maximum-length IDOS starting from .
3.1.2. Multiple UIO Sequences
In the improved method, for any maximum-length IDOS , , the associated test subsequence is expressed as where is a UIO sequence of . A minimum-length rural postman tour covering all the test subsequences is subsequently constructed by a rural symmetric augmentation and a test sequence is obtained from the tour. Accordingly, transitions for concatenation of test subsequences in the test sequence are derived from the transition replications during the rural symmetric augmentation. It is noted that different choice of UIO sequences may result in different rural symmetric augmentations; i.e., a minimum number of transition replications can be achieved by a sensible choice of UIO sequences during the rural symmetric augmentation. The minimum number of transition replications during the rural symmetric augmentation indicates the minimum number of transitions for concatenation of test subsequences in the test sequence, leading to a reduced test sequence. In other words, the result of using single UIO sequences can only in best-case scenarios obtain the same length of minimum-length rural postman tours as that of using multiple UIO sequences.
3.2. Design of the Method
It is known from Theorem 3 that noninvertible transitions restrict the generation of IDOSs. From this point of view, FSMs can be partitioned into two subsets. One is FSMs with only invertible transitions and the other is FSMs with noninvertible transitions. For FSMs with only invertible transitions, if the FSMs are symmetric test sequences can be obtained directly. Otherwise, the FSMs should be augmented firstly. So FSMs with only invertible transitions can also be partitioned into two subsets. One is symmetric FSMs with only invertible transitions and the other is asymmetric FSMs with only invertible transitions. Generally, FSMs are classified into three categories: symmetric FSMs with only invertible transitions, asymmetric FSMs with only invertible transitions and FSMs with noninvertible transitions. The improved method is described in two steps for each type of FSMs and the detail varies for different types.
Step 1. Construct the set of maximum-length IDOSs.
Step 2. With the consideration of multiple UIO sequences, generate a minimum-length rural postman tour covering the set of maximum-length IDOSs and extract a test sequence from the tour.
3.2.1. Symmetric FSMs with Only Invertible Transitions
Step 1 (maximum-length IDOSs generation). There is an Euler tour in a directed graph if and only if the directed graph is strongly connected and symmetric. Under the assumption that all the FSMs are strongly connected, there must be Euler tours for symmetric FSMs with only invertible transitions. An Euler tour starting from and ending at the initial state is definitely an IDOS since there are only invertible transitions; furthermore, it is a maximum-length IDOS from the initial state since there is no other one containing more transitions. In other words, an Euler tour of a symmetric FSM with only invertible transitions is the only sequence in the set of maximum-length IDOSs.
Note that an Euler tour starting from and ending at the initial state may not be unique; i.e., there may be nonunique sets of maximum-length IDOSs. Nonetheless, all the sets of maximum-length IDOSs hold the following properties.
(i) There is only one maximum-length IDOS covering all the transitions of the FSM in every set of maximum-length IDOSs.
(ii) The start state of the maximum-length IDOS is the initial state of the FSM in every set of maximum-length IDOSs.
(iii) The end state of the maximum-length IDOS is the initial state of the FSM in every set of maximum-length IDOSs.
Step 2 (test sequence generation). For symmetric FSMs with only invertible transitions, test sequences are denoted by where is an Euler tour staring from and ending at and is a minimum-length UIO sequence of . It can be inferred that test sequences from different sets of maximum-length IDOSs are always of the same length when there is more than one set of maximum-length IDOSs. As a result, a randomly generated set of maximum-length IDOSs will do for test sequence generation of symmetric FSMs with only invertible transitions.
in Figure 2 is a symmetric FSM with only invertible transitions and is an Euler tour starting from and ending at . It is known from Step 1 that is a set of maximum-length IDOSs of . A test sequence is resulted from Step 2.
3.2.2. Asymmetric FSMs with Only Invertible Transitions
Step 1 (maximum-length IDOSs generation). An FSM is asymmetric which refers to the fact that there are nodes whose out-degree does not equal in-degree. It is known that where and denote the out-degree and in-degree of a node, respectively [13]. The notation refers to the number of edges in a directed graph. It is concluded from that where and are the sets of nodes whose out-degree outnumbers in-degree and in-degree outnumbers out-degree, respectively. Correspondingly, and represent the amount of out-degree over in-degree for a node in and the amount of in-degree over out-degree for a node in , respectively. The relation implies that if a path passes through as many edges as possible exactly once then the path consumes as many out-degree and in-degree as possible. Every path through a node takes up one incoming edge as well as one outgoing edge, so if the out-degree outnumbers in-degree at a node, its outgoing edges cannot be traversed completely by paths passing through the node; i.e., some of its outgoing edges must instead be traversed by paths starting from the node. Thus, it is advisable to start the paths from nodes whose out-degree outnumbers in-degree if all the edges should be traversed exactly once with as few paths as possible. Moreover, it can be proved by contradiction that the paths from nodes whose out-degree outnumbers in-degree must end at nodes whose in-degree outnumbers out-degree otherwise the paths will continue to extend.
According to the above analysis, maximum-length IDOSs generation for asymmetric FSMs with only invertible transitions is performed as follows: start from a node and go down along an outgoing edge until a node is reached and all the outgoing edges of have been traversed. Thus a maximum-length IDOS from to is obtained. Add the maximum-length IDOS to the set of maximum-length IDOSs and delete the corresponding edges in the directed graph. Repeat the above process until all the nodes of the directed graph are isolated which implies that the set of maximum-length IDOSs is obtained.
Similarly, there may be nonunique sets of maximum-length IDOSs for asymmetric FSMs with only invertible transitions and all the sets of maximum-length IDOSs hold the following properties:
(i) For any set of maximum-length IDOSs, is the set of start states for maximum-length IDOSs.
(ii) For any set of maximum-length IDOSs, is the set of end states for maximum-length IDOSs.
(iii) For any node , the number of maximum-length IDOSs starting from is in every set of maximum-length IDOSs.
(iv) For any node , the number of maximum-length IDOSs ending at is in every set of maximum-length IDOSs.
(v) The total number of maximum-length IDOSs is which equals in each set of maximum-length IDOSs.
Step 2 (test sequence generation). With the set of maximum-length IDOSs, the same method as that of Bo Yang et al. is used to construct test sequences and the detail of the method is described in Algorithm 1. The core of the method is the rural symmetric augmentation over an FSM augmented by maximum-length IDOSs and the multiple UIO sequences for the end states of maximum-length IDOSs. The core of the symmetric augmentation is the states involved. According to the above properties, for any set of maximum-length IDOSs, there is no difference about the states involved in the symmetric augmentation such that a random set of maximum-length IDOSs will do for asymmetric FSMs with only invertible transitions when there is more than one set of maximum-length IDOSs.
in Figure 4 is an asymmetric FSM with only invertible transitions and its UIO sequences are shown in Table 3. The nonunique sets of maximum-length IDOSs for are listed as follows:
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
, ,
, ,
, ,
, .
Clearly, all the sets of maximum-length IDOSs satisfy the properties in this section. The augmentation of using a random set of maximum-length IDOSs is shown in Figure 5 and the associated test sequence of is / .
3.2.3. FSMs with Noninvertible Transitions
Step 1 (maximum-length IDOSs generation). To address an FSM with noninvertible transitions in a similar way to the first two types of FSMs, noninvertible transitions are removed from the FSM and saved to a set. The remainder excluding noninvertible transitions is either an FSM or more than one connected component with only invertible transitions. And thus maximum-length IDOSs of the remainder excluding noninvertible transitions are generated in the same way as symmetric or asymmetric FSMs with only invertible transitions. Naturally, it comes to the same conclusion as the first two types of FSMs that a random set of maximum-length IDOSs will do when there are nonunique sets of maximum-length IDOSs. Next, a union of maximum-length IDOSs of the remainder excluding noninvertible transitions and all the noninvertible transitions is derived; moreover, whenever the start state of a maximum-length IDOS is the end state of a noninvertible transition, the maximum-length IDOS is concatenated with the noninvertible transition. If a maximum-length IDOS is an Euler tour then a node which is the end state of a noninvertible transition is preferred to be the start state of the tour. The set after all possible concatenations is a set of maximum-length IDOSs of the FSM with noninvertible transitions.
Given a union of maximum-length IDOSs of the remainder excluding noninvertible transitions as well as all the noninvertible transitions, there may be nonunique sets of maximum-length IDOSs for the FSM with noninvertible transitions because of different concatenation. The properties of nonunique sets of maximum-length IDOSs for FSMs with noninvertible transitions are described as follows:
(i) In any set of maximum-length IDOSs, for any state which is the start state of a maximum-length IDOS, if there are maximum-length IDOSs starting from then there must be maximum-length IDOSs starting from in every other set of maximum-length IDOSs.
(ii) In any set of maximum-length IDOSs, for any state which is the end state of a maximum-length IDOS, if there are maximum-length IDOSs ending at then there must be maximum-length IDOSs ending at in every other set of maximum-length IDOSs.
Step 2 (test sequence generation). For FSMs with noninvertible transitions, test sequence generation is in the same way as asymmetric FSMs with only invertible transitions, i.e., by means of rural symmetric augmentation over an FSM augmented by maximum-length IDOSs and the multiple UIO sequences for the end states of maximum-length IDOSs. The core of rural symmetric augmentation is still the states involved. It is known from the above properties that for any set of maximum-length IDOSs the states involved in the rural symmetric augmentation are the same such that a random concatenation will do.
Considering in Figure 6 with UIO sequences in Table 4, the following nonunique union of maximum-length IDOSs of the remainder excluding noninvertible transitions and noninvertible transitions confirm that a random union will do. / , / , / , / .Take a random union of maximum-length IDOSs of the remainder excluding noninvertible transitions and noninvertible transitions , , , ; nonunique sets of maximum-length IDOSs from different concatenations , , and , , confirm that a random concatenation will do. The augmentation of using a randomly concatenated set of maximum-length IDOSs , , is shown in Figure 7 and the resulting test sequence is .
Algorithm 1 describes the detail of the improved method for all types of FSMs. Note that self-loops are always given priority to traverse in the process of maximum-length IDOSs generation. When creating the new state set in , for the end state of every maximum-length IDOS, there is a state in . When creating the new transition set in , for every maximum-length IDOS, there is a transition from the start state of the maximum-length IDOS to state labeled with the corresponding label of the maximum-length IDOS. When creating a new transition set in , for every UIO sequence of the end state of every maximum-length IDOS, there is a transition from to the end state of every UIO sequence labeled with the corresponding UIO sequence.
Theorem 5. Given an FSM , suppose that is a set of maximum-length IDOSs and is a sequence over generated from Algorithm 1, then is a reduced test sequence of .
Proof. The soundness of is first proved; i.e., is a test sequence of . Then the effectiveness of is assessed in terms of test generation and test execution cost, respectively. From a general standpoint, the notion of cost in the context of testing is complex and can be related to many factors. In our context, the effort required for generating test sequences is measured in terms of test sequence computational complexity. Test sequence execution cost is measured by the length of test sequences. Although these are clearly approximation methods, for practical reasons such methods have been commonly used in a number of testing studies [14–16].
(1) Soundness Analysis
(i) Check whether every transition defined in is verified in the implementation
All the maximum-length IDOSs in are included in since is a sequence over . Algorithm 1 indicates that every maximum-length IDOS in is followed by a UIO sequence that verifies the last transition of the sequence. According to Definition 2, i.e., for any maximum-length IDOS , if (, , /) is verified then every transition (, , /) is verified, it is inferred that transitions of all the maximum-length IDOSs in are verified in . According to Definition 4, i.e., every transition in is included in one and only one IDOS of , it is concluded that every transition defined in is verified in the implementation by .
(ii) Check whether every state in is defined in the implementation
is supposed to be strongly connected such that for any state of there is at least one transition ending at . It is proved that every transition defined in is verified in by identifying the end state of the transition; i.e., every state of is checked in the implementation by .
(2) Effectiveness Analysis
(i) Analysis of Computational Complexity. The test sequence has the same computational complexity as those of Aho et al. and Hierons since all the three test sequence generation methods are based on the max flow/min cost problem and the networks used in every method are of the same order.
(ii) Analysis of Length Reduction. As mentioned above, transitions in a test sequence are of three kinds and the cost of a test sequence comes from the UIO sequences which have been appended for the purpose of transition verification and the transitions which have been appended for the purpose of concatenation of test subsequences to generate a minimum-length rural postman tour.
For symmetric FSMs with only invertible transitions, is in the form of where (, , /)(, , /)(, , /) is an Euler tour staring from and ending at and is a minimum-length UIO sequence of ; i.e., the cost of only comes from one appended minimum-length UIO sequence of .
For asymmetric FSMs with only invertible transitions, all the transitions are divided into a least number of maximum-length IDOSs; i.e., all the transitions are verified by a least number of appended UIO sequences such that the cost of from the appended UIO sequences is minimal. For any maximum-length IDOS (, , )(, , )…(, , ), there is a test subsequence (, , )(, , )…(, , ) where is a UIO sequence of . is obtained from a minimum-length rural postman tour covering all the test subsequences. The minimum-length rural postman tour is generated by the rural symmetric augmentation and thus the cost of for concatenation comes from the replicated transitions during the rural symmetric augmentation. Benefiting from multiple UIO sequences, a minimum number of transition replications is reached by a sensible choice of UIO sequences during the rural symmetric augmentation; i.e., the cost of from the transitions for concatenation is minimal.
For FSMs with noninvertible transitions, transitions excluding noninvertible ones are divided into a least number of IDOSs. According to Theorem 3, the acquired IDOSs concatenate with noninvertible transitions as much as possible such that a set of maximum-length IDOSs with a least number of maximum-length IDOSs is obtained; i.e., all the transitions of an FSM with noninvertible transitions are verified by a least number of appended UIO sequences such that the cost of from the appended UIO sequences is minimal. Similarly, a minimum number of transition replications during the rural symmetric augmentation is reached by a sensible choice of UIO sequences; i.e., the cost of from the transitions for concatenation is minimal.
In short, for all types of FSMs, the execution cost is reduced effectively without increasing the generation cost such that is a reduced test sequence of .
4. Case Study
We experiment with the aforementioned , , and which are random generation of different types of FSMs. and are also randomly generated FSMs; moreover, they are example FSMs used by Bo Yang et al. and Hierons, respectively. While the experimental results shed some light on how the improved method behaves with randomly generated FSMs, they bring no insight on the test sequence reduction of FSMs that are produced by software designers. For this reason, experiments on FSMs modeling INRES protocol [17], GUI for password modification in a property management system [18], page function of Gmail system [19] and the connection and release process of MAC layer in Zigbee protocol are carried out, and the FSM modeling page function of Gmail system is adjusted to satisfy the strong connectivity and UIO availability assumptions.
The FSMs used in the experiment are admittedly small. However, it is important to note that FSMs are mostly used to model the behavior of complex classes or class clusters, particularly complex control classes in protocols or reactive systems. They are rarely used to model entire systems which will result in large and unmanageable models for software engineers and testers. Completeness degree is a general factor to reveal the complexity of both large and small FSMs such that test sequence reduction of large FSMs can to some extent be learned through relatively small FSMs with the same distribution of completeness degree. Given an FSM with inputs, transitions and states, completeness degree of is denoted by ; moreover, the higher the completeness degree is, the more complex the FSM is. In this section, completeness degrees of FSMs range from 0.24 to 1.
The associated data of the experiment is illustrated in Table 5, , and denote the number of inputs, states and transitions of every FSM. Completeness degree of every FSM is calculated and listed. , , and are test sequences resulting from Bo Yang et al., Hierons, and the improved method, respectively. , , and represent the length of the corresponding test sequences.
For symmetric FSMs with only invertible transitions, and are both Euler tours starting from the initial state followed by a minimum-length UIO sequence of the initial state. Thus, and are always the same for symmetric FSMs with only invertible transitions. When the Euler tour conforms to the definition of fully overlapping transition sequences (FOTSs) [11], is the same as and ; otherwise tends to be longer because of more appended UIO sequences as well as the possible extra transitions for concatenation.
For asymmetric FSMs with only invertible transitions, and are the same if the maximum-length IDOSs comply with the definition of FOTS. Otherwise, tends to be longer because of more appended UIO sequences as well as the possible extra transitions for concatenation. is an Euler tour from a rural symmetric augmentation of an asymmetric FSM with only invertible transitions followed by a minimum-length UIO sequence of the initial state. The number of appended transitions in the rural symmetric augmentation is a deciding factor of .
For FSMs with noninvertible transitions, is longer than since every noninvertible transition is verified individually by an appended UIO sequence. is the same as at best otherwise tends to be longer than .
As shown in Table 5, test sequences generated from different test methods conform to the above theoretical analysis. In this section, two-sample t-test is performed to compare test methods in terms of the length of the associated test sequences. Single sample K-S check in SPSS is used to verify the normality of the data studied since t-test is a parametric test and requires data to be normally distributed. Normality results are reported whenever samples deviate significantly from the normal distribution and the result of single sample K-S check in Table 6 shows that all the data in this experiment follow normal distribution. The paired samples statistics in Table 7 indicates that the average length of test sequences from the improved method is superior to those of the other two methods.
Vargha-Delaney effect size measure is also calculated to get more credible conclusions. When comparing two methods, measures the probability that one method would perform better than the other method. A value of 0.5 would mean that the two methods have equal probability of performing better than the other. The Vargha-Delaney effect size measure from comparing to and is shown in Table 8. The results show that is statistically and of the time significantly shorter than and , respectively.
5. Related Work
There are many works on IoT testing. Xiaoping Che et al. presented a logic-based approach to test the conformance and performance of XMPP protocol which is gaining momentum in IoT through real execution traces and formally specified properties [20]. Dimitrios Serpanos et al. introduced testing for security for IoT systems and especially fuzz testing, which is a successful technique to identify vulnerabilities in systems and network protocols [21]. Martin Tappler et al. presented a model-based approach to test IoT communication via active automata learning [22]. Combining Model-Based Testing (MBT) and a service-oriented solution, Abbas Ahmad et al. presented Model-Based Testing As A Service (MBTAAS) for testing data and IoT platforms [23]. Hiun Kim et al. introduced IoT testing as a Service-IoT-TaaS which is composed of remote distributed interoperability testing, scalable automated conformance testing, and semantics validation testing components adequate for testing IoT devices [24]. John Esquiagola et al. used the current version of their IoT platform to perform performance testing [25]. Daniel Kuemper et al. described how concepts for semantically described web services can be transferred into the IoT domain [26]. Philipp Rosenkranz et al. propose a testing framework which supports continuous integration techniques and allows for the integration of project contributors to volunteer hardware and software resources to the test system [27]. A connective and semantic similarity clustering algorithm (CSSCA) and a hierarchical combinatorial test model based on FSM are proposed by Kai Cui et al. [28].
Test sequence generation and reduction has long been an active research topic. Porto et al. used identification sets which are subsets of a characterizing set to identify states and obtained reduced test sequences [29]. Locating sequences are used to make sure that every element of a characterizing set is applied to the same state. Jourdan et al. generated shorter test sequences by means of reducing the number of locating sequences [30]. Baumgartner et al. proposed a mixed integer nonlinear programming (MINLP) model to formalize how the total cost of testing depends on the sequence and the parameters of the elementary test steps [31]. To provide an efficient formalization of the scheduling problem and avoid difficulties due to the evaluation of an objective function during the relaxation of the integer variables, the MINLP was formulated as a process network synthesis problem. Hierons et al. affirmed the importance of invertibility in test sequence reduction and considered three optimisation problems associated with invertible sequences [32]. Petrenko et al. addressed the problem of extending the checking experiment theory to cover a class of FSMs with symbolic extensions [33]. They also reported the results that further lift the theory of checking experiments for Mealy machines with symbolic inputs and symbolic outputs. Hierons et al. described an efficient parallel algorithm that uses many-core GPUs for automatically deriving UIOs from Finite State Machines [34]. The proposed algorithm uses the global scope of the GPU’s global memory through coalesced memory access and minimizes the transfer between CPU and GPU memory. Song et al. introduced a practical conformance testing tool that generates high-coverage test input packets using a conformance test suite and symbolic execution. This approach can be viewed as the combination of conformance testing and symbolic execution [35]. Bokil et al. presented an automated black box test suite generation technique for reactive systems [36]. The technique is based on dynamic mining of specifications in form of an FSM from initial runs. The set of test cases thus produced contain several redundant test cases, many of which are eliminated by a simple greedy test suite reduction algorithm to give the final test suite.
6. Conclusions
Taking Zigbee protocol as an example, this paper introduces how FSM-based conformance testing works in wireless protocol conformance testing of IoT. An improved method in which both overlapping by invertibility and multiple UIO sequences are considered is proposed to achieve test sequence reduction for wireless protocol conformance testing of IoT. Based on invertibility, transitions of all types of FSMs are verified with as few appended UIO sequences as possible. Multiple UIO sequences contribute to generate a shorter test sequence by means of reducing transitions for concatenation of test subsequences in the test sequence. Moreover, test sequences can be further reduced by removing the transition sequence which follows the UIO sequence of the maximum-length IDOS that is verified last in the tour. Theory and experiment indicate that the execution cost is reduced effectively by the improved method under the premise of not increasing the generation cost. The improved method is also applicable to traditional protocol conformance testing as well as reactive systems. Numerous experimental data and practical examples about wireless protocols of IoT will be gathered in the future work to analyze the effectiveness of the method.
Data Availability
The data used to support the findings of this study are included within the article.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.
Acknowledgments
This work is supported by National Natural Science Foundation of China (Grant no. 61572306 and Grant no. 61502294), the IIOT Innovation and Development Special Foundation of Shanghai (Grant no. 2017-GYHLW-01037), and the CERNET Innovation Project (NGII20170513 and NGII20170206).