Mathematical Problems in Engineering

Mathematical Problems in Engineering / 2014 / Article
Special Issue

Stochastic Systems: Modeling, Optimization, and Applications

View this Special Issue

Research Article | Open Access

Volume 2014 |Article ID 764819 |

Kemin Wang, Yongbin Wang, Zhengtao Jiang, Wenlong Fu, "A Multistep Extending Truncation Method towards Model Construction of Infinite-State Markov Chains", Mathematical Problems in Engineering, vol. 2014, Article ID 764819, 6 pages, 2014.

A Multistep Extending Truncation Method towards Model Construction of Infinite-State Markov Chains

Academic Editor: Shuping He
Received09 Oct 2013
Accepted14 Dec 2013
Published12 Jan 2014


The model checking of Infinite-State Continuous Time Markov Chains will inevitably encounter the state explosion problem when constructing the CTMCs model; our method is to get a truncated model of the infinite one; to get a sufficient truncated model to meet the model checking of Continuous Stochastic Logic based system properties, we propose a multistep extending advanced truncation method towards model construction of CTMCs and implement it in the INFAMY model checker; the experiment results show that our method is effective.

1. Introduction

Continuous Time Markov Chains (CTMCs) have been used in various areas of research as a formalism; so far, the model checking of CTMCs has been a hot research topic in computer science research communities. Some algorithms and implementations have been shown in several papers and tools [17]. However, our research aims to the model checking problem of the Infinite-State CTMCs, which means that the states of the CTMCs in our interest can be infinite. The papers [811] and the tool INFAMY [7] are with the same interest, which are also our research basis. Due to the explosion of states of CTMCs, our approach is based on a truncated CTMC model, which is determined by the exploration on the fly [9, 10], which means that the depth of model is computed dynamically by the exploration of states.

The truncation process is involved with reachability analysis [9, 10]; that is, the transient probability is computed with the exploration of the model. The transient probability is carried out by uniformization method [10]. Together with the transient analysis when constructing the model, computation is very heavy, so to get a sufficient truncated model to meet the requirement of related Continuous Stochastic Logic (CSL) property or certain precision as fast as possible is our destination. We introduce a multistep extending advanced truncation method to meet this end, and the experimental results show our method is effective.

The main contents of the paper are organized as follows. Section 2 introduces the truncation based reachability analysis; Section 3 introduces an advanced truncation algorithm and do experiments. Section 4 proposes some multistep extending solutions and experiments. Section 5 talks about our result and next job.

2. Truncation Based Reachability Analysis

2.1. Finite Truncations

Let be a CTMC, where is a countable set of states, is the rate matrix, and is a labeling function.

First, we introduce some paths and probabilistic measures from [9]. A (timed) infinite path is an infinite sequence satisfying , and for all . For the path and , let denote the th state, and let denote the time spent in . For , let denote such that is the smallest index with . For , let denote the set of all paths, and let denote the set of all paths starting from . For state , a probability measure, denoted by , on the set can be defined. A finite path is a finite sequence for satisfying and for . Let denote the length of the path, denote the first state, and denote the last state of the path. Let denote the set of all finite paths. We omit the superscript if it is clear from context.

Next, we introduce the notion of depth. Let be a finite subset of with depth ; that is, for . For now, one may think of as being equal to the support of the initial distribution . However, can be an arbitrary finite set. This will allow us not only to deal with the initial distribution, but also to compute truncation depths for nested CSL formulas. The depth of state corresponds to the minimal distance from the set .

Definition 1. For and , the depth function is defined by .

Observe that for all . The subscript is omitted if is clear from the context. Intuitively, corresponds to the minimal length of any finite path starting from and ending in .

We consider a partition of the state space , where is the set of states with depth . We say that the set is the layer with depth and call its elements layer- states. Assume that is a special state not in and furthermore is also an atomic proposition not in . For , let denote the set of states with depth greater than ; that is, .

Further, we write . We define the -truncated CTMC as follows.

Definition 2. Let be a CTMC and let be the layer-zero states. For , we define the -truncated CTMC of by , where the states are . The labeling function is defined by if , and . The rate matrix is defined by the following: equals if , equals if = , and equals otherwise.

The -truncation of an infinite CTMC is illustrated in Figure 1. Intuitively, the transition matrix is restricted to the truncated state space , and is the distinguished absorbing state, which, by construction, is only reachable from states with depth . In state only the atomic proposition holds, which indicates that the system is in state . Since we consider finitely branching CTMCs, not surprisingly, the -truncated CTMC is always finite. The absorbing state has been introduced to abstract . We assume that for any state formula . We consider the probability of reaching the absorbing state in the -truncated CTMC , that is, . For mere notational convenience, we extend to states of with depth higher than for all with . For a fixed , we define the forward rate of a state within . For , it is the sum of the rates that go into the next layer , and, for , it is the sum of the rates entering states in .

2.2. The Logic CSL

The logic we consider is CSL without steady-state operator and unbounded until operator [10]. Let be an interval with and . Let and . The syntax of state formula and path formulas is

The semantics of the state formulas and path formula are precisely defined in [10]. In this paper, the state formula like the kind of will be more focused on; it means that starting some state, the probability of the system that satisfies a path formula would meet the rational relation or not, so the first solution to this problem is to compute this probability, so this is reduced to transient analysis, which is our main interest, to improve the efficiency of transient analysis for the whole model while extending the border states layer by layer.

2.3. Transient Analysis

Transient analysis means that, starting at state , the transient probability vector at time , it can be denoted as . If , we have if and 0 otherwise. The uniformization method based solution of was given at [10]. Let and , where is the uniformization rate satisfying and denotes the th Poisson probability with parameter and vector is transient probability of the uniformized DTMC at step .

The truncated model in our method is given by dynamically exploring from the initial state(s); once we want to add a layer on it, we need to compute the reachability probability from the initial(s) to the current layer states which we want to add. As shown in Figure 1, is the initial states set, and each of is explored dynamically, so we get a -depth truncated model of the infinite one. is the absorbing state, which is the border states set, for which the sum of the reachability probabilities from the initial state(s) is less than ; is the precision of the result, which can be , , or , and so forth, which can be set under INFAMY model checker as a circumstance variable for some certain need.

The reachability probability is carried out by the uniformization method to the CTMC, which is a relatively fast method to get the transient probability at a certain bound time at some state, which is denoted as . However, if the state explosion situation is very serious, the time to construct the model layer by layer would be of much cost. So we introduce an advanced truncation algorithm to explore the states, this method can improve the efficiency of the model constructing and model checking, and then at Section 4, we further propose some multistep extending solutions, which are implemented based on INFAMY model checker, the experiment results show that these solutions can help to improve the efficiency.

3. An Advanced Truncation Algorithm and Experiments

The truncation process is implemented by extending the states layer by layer from the initial states, and all the new states need to be transient analyzed and then to be extended further no matter how small the probability is. For that the precision of the result is under some certain value, so some states with relatively nearly no contribution to the result can be omitted when extending; thus, an advanced truncation algorithm is introduced. It is different from the finite state projection (FSP) [9, 10] and layered chain and uniform chain method [9, 10]. The algorithm is shown in Algorithm 1.

(4)   compute
(5)   while     do
(7)    compute
(8)    while     do
(9)      such that
(10)     and
(12)    compute
(13)   end while
(14)   compute
(15)  end while
(16) end procedure

The algorithm aims to stop the extending of less important (small probability) states and proceed to the extending of much important (large probability) states; the less or more is determined by the state reduction policy; as shown in Figure 2, line 7, and . The policy means that the states in the border states set, which has been sorted upward, , will be excluded from the extending states set, for the sum of them is just exactly less than the precision. So with this policy, the number of states to be transient analyzed will be smaller to the FSP and forward-layered based model.

We consider the dependability of a fault-tolerant workstation cluster which is directly taken from case studies of [7]. Figure 2 depicts a dependable cluster of workstations. The cluster consists of two subclusters, which, in turn, contain workstations connected via a central switch. The two switches are connected via a backbone. Each component of the system can break down and is then fixed by a single repair unit responsible for the entire system. Hereby, the quality of service (QoS) constraint minimum requires at least    workstations to be operational, where . Workstations have to be connected via switches. If in each subcluster the number of operational workstations is smaller than , the backbone is required to be operational to provide the required service. We consider the property.

. This probability means that the QoS drops below minimum quality within time unit.

For the property, we compare PRISM [6], FSP method and layered method of INFAMY. The results are given in Tables 1, 2, and 3. Because the resulting probabilities are very small in some cases, we use a precision of here, for the computation of the truncation point. Results for INFAMY are given for the layered chain, FSP, and advanced configurations, respectively. The uniform chain configuration is omitted, as it is always dominated by the layered chain configuration. PRISM implements three different engines: a sparse-matrix and two symbolic engines. We used the sparse-matrix engine as it was the fastest one. The results are shown in Tables 1, 2, and 3.

Time bound Forward-layered
DepthTime (s) Prob

5.0 316 4501668363 [ , ]
10.0 517 661.52373652 [ , ]
20.0 517 12932373652 [ , ]
30.0 517 14532373652 [ ,
50.0 517 20962373652 [ , ]

Time bound Advanced
Depth Time (s) Prob

5.0 18 6.6 4071 [ , ]
10.0 25 23.3 7118 [ , ]
20.0 38 105.2 14434 [ , ]
30.0 50 271.6 22426 [ ,
50.0 69 1491.5 38448 [ , ]

Time bound PRISM (sparse engine)FSP
Time (s)
ProbDepthTime (s) Prob

5.0 16 5.7 4405 [ , ]
10.0 22 20.7 8425 [ , ]
20.0 33 113.9 19161 [ , ]
30.0 42 353.6 31185 [ ,
50.0 57 1219.7 57705 [ , ]

The experiment conditions are shown as follows.

Host Machine: it includes Mac Book, OS: Mac OS X 10.6.8, Processor: 2.2 GHz Intel Core 2 Duo T7500, and Storage: 2 GB 667 MHz DDR3 SDRAM.

Guest Machine: it includes Virtual Machine Software: VirtualBox for mac; Virtual OS: Linux ubuntu12.04 LTS 32Bit; Processor: 2.2 GHz Intel Core 2 Duo T7500; Storage: 512 MB

From Tables 1, 2, and 3, we can see that, for , FSP based INFAMY is faster, but for , , INFAMY model checker needs more time; this is because the transient analysis when constructing the model needs more computing. This is also the result of [11]. For the advanced method of the context, as in Tables 1, 2, and 3, comparing advanced with FSP, we can get that, under advanced based method, the depth of model is much deeper, and the states number is smaller; for the time costing, for , , the costing is less reduced, but for , the costing is greater; this result is reasonable, for the exploration policy is essentially undeterministically efficient for different models; for the current case, when , the current states number is very large, even with the reduction policy, with no contribution for it any more. We need to take other techniques to tackle this situation. Thus, we propose a multistep extending solution. See Section 4.

4. Multistep Extending Solutions and Experiments

The mutistep extending solution aims to reduce the extending of less important (small probability) states and enhance the extending of more important (large probability) states; for the latter states, we can, for example, extend two or more steps per extending, and for the former states, we can, for example, extend one step per extending, as we know that transient probabilities will be computed once again before the states were added to the border states, so, if we extend two or more steps, transient analysis at the intermediate states will be omitted; thus, time on the model construction will be reduced; then we can make the model much faster to converge to the absorbing state.

We continue with the upper case study. As shown in Algorithm 1, Line 11, is to extend the model from set and get the border states set . This means that the advanced method in the upper section is extending one step per extending. Now, we design some multistep extending solutions:(1)two-step extending solution, that is, ;(2)three-step extending solution, that is, ;(3)synthesis solution 1: a synthesis extending solution separates set to three parts, as set is a sorted set, which is sorted upward by the probabilities, so we can separate as , , and , and the sizes of each part are the same. Then we can get(i);(ii);(iii);(iv);(4)synthesis solution 2 separates set to two parts, so we can separate to and , and the sizes of the two parts are the same. Then we can get(i);(ii);(iii).

The experiments data are shown in Table 4. From the data in Table 4, we can see that under different extending solutions, the times needed are different, synthesis solution 2 performs better, when , and the time (including model constructing and model checking) has been reduced to 1000 s. This means that for this case, the step per extending should be relatively small; for synthesis solution 2, half small states extend one step per-extending, and other half large states extend two steps per extending. And for this, we continue to propose a solution to revise synthesis solution 2, named Solution D the policy is as follows: For the states in set , the gap between the largest and smallest probabilities: . We can separate set to two parts and , such that.(i) and ;(ii) .And then we can get that(i) ;(ii) ;(iii) .

Time bound Advanced (one step per extending)Two-step per extending Three-step per extending
Depth Time (s) Depth Time (s) Depth Time (s)

5.0 18 6.6 4071 20 8.9 4756 33 15.0 7895
10.0 25 23.3 7118 29 26.5 8027 38 41.3 11849
20.0 38 105.2 14434 42 115.3 16315 52 174.0 20816
30.0 50 271.6 22426 54 327.6 25308 64 456.0 30785
50.0 69 1491.5 38448 72 1006.5 43824 82 1572.6 50942

Synthesis solution 1 Synthesis solution 2 Solution D

5.0 35 11.9 6539 20 9.6 4564 20 8.6 4459
10.0 42 35.4 9772 26 29.7 7646 26 24.9 7383
20.0 53 131.2 17777 41 111.7 15621 39 91.1 15170
30.0 61 367.4 26362 53 279.7 24239 50 250.9 23063
50.0 74 1349.6 45170 71 989.1 42703 71 791 40403

The experiment results are as shown in Table 4. From Table 4, we can see that current solution D performs better on this case.

5. Conclusion

The multistep extending advanced truncation method can improve the efficiency of model construction of Infinite-State CTMCs; this is because the transient probabilities of states which have been jumped have not been computed, so to some extent this method is effective; however, which solution performs better needs to be experimented; there is no general solution that fits well for all cases. The efficiency is determined by the iterations when computing the transient probability. Less iteration is more efficient. However, this approach is essentially a linear approach to improve the efficiency; when the outsider state gets explosion, this approach will be less effective; just as in our case study, this approach can be used effectively to improve the model checking efficiency at a relatively small time bound . For future work, we need to consider other techniques to tackle the state explosion problem on model checking of CTMCs. And other works like [1215] can also be considered.

Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.


The authors would like to thank Lijun Zhang and Ernst Moritz Hahn for their original work on INFAMY model checker and the free contribution of the tool codes; Definitions 1 and 2 are directly taken from [10]. This paper is partially funded by the Communication University of China, 2013 Planned Research Program, and Research Funding Foundation of School of Computer Science, and the scholarship from China Scholarship Council for the Excellent University Teachers Study Abroad Program and supported by the National Natural Science Foundation of China (61103199), Beijing Municipal Natural Science Foundation (4112052); Supported by Engineering Program Project of CUC (3132013XNG1326, 3132013XNG1321), and the National Key Science and Technology Pillar Programs of China (2012BAH51F02, 2012BAH38F05, and 2013BAH66F02).


  1. A. Aziz, K. Sanwal, V. Singhal, and R. Brayton, Verifying Continuous Time Markov Chains, pp. 269–276, Springer, 1996.
  2. C. Baier, J. P. Katoen, and H. Hermanns, “Approximative symbolic model checking of continuous-time markov chains,” in CONCUR ’99 Concurrency Theory, vol. 1664 of Lecture Notes in Computer Science, pp. 146–161, Springer, 1999. View at: Google Scholar
  3. H. Hermanns, J. Meyer-Kayser, and M. Siegle, “Multi terminal binary decision diagrams to represent and analyse continuous time markov chains,” in Proceedings of the 3rd International Workshop on the Numerical Solution of Markov Chains, pp. 188–207, Citeseer, 1999. View at: Google Scholar
  4. J. P. Katoen, M. Kwiatkowska, G. Norman, and D. Parker, Faster and Symbolic CTMC Model Checking, Springer, 2001.
  5. C. Baier, B. Haverkort, H. Hermanns, and J. Katoen, “Model-checking algorithms for continuous-time Markov chains,” IEEE Transactions on Software Engineering, vol. 29, no. 6, pp. 524–541, 2003. View at: Publisher Site | Google Scholar
  6. Prism model checker,
  7. Infamy model checker,
  8. A. Remke, B. R. Haverkort, and L. Cloth, “Model checking infinite-state Markov chains,” in Proceedings of the 11th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS '05), vol. 3440 of Lecture Notes in Computer Science, pp. 237–252, Springer, April 2005. View at: Google Scholar
  9. E. M. Hahn, H. Hermanns, B. Wachter, and L. Zhang, “Time-bounded model checking of infinite-state continuous-time markov chains,” Fundamenta Informaticae, vol. 95, no. 1, pp. 129–155, 2009. View at: Publisher Site | Google Scholar
  10. L. Zhang, H. Hermanns, E. M. Hahn, and B. Wachter, “Time-bounded model checking of infinite-state continuous-time markov chains,” in Proceedings of the 8th International Conference on Application of Concurrency to System Design (ACSD '08), pp. 98–107, June 2008. View at: Publisher Site | Google Scholar
  11. E. M. Hahn, H. Hermanns, B. Wachter, and L. Zhang, “Infamy: an infinite-state markov model checker,” in Computer Aided Verification, Springer, pp. 641–647, 2009. View at: Google Scholar
  12. S. He and F. Liu, “Robust stabilization of stochastic Markovian jumping systems via proportional-integral control,” Signal Processing, vol. 91, no. 11, pp. 2478–2486, 2011. View at: Publisher Site | Google Scholar
  13. S. He and F. Liu, “Robust l2-l filtering of time-delay jump systems with respect to the finite-time interval,” Mathematical Problems in Engineering, vol. 2011, Article ID 839648, 17 pages, 2011. View at: Publisher Site | Google Scholar
  14. S. He, Z. Ding, and F. Liu, “Output regulation of a class of continuous-time markovian jumping systems,” Signal Processing, vol. 93, no. 2, pp. 411–419, 2013. View at: Google Scholar
  15. H. Shen, S. Xu, J. Lu, and J. Zhou, “Passivity-based control for uncertain stochastic jumping systems with mode-dependent round-trip time delays,” Journal of the Franklin Institute, vol. 349, no. 5, pp. 1665–1680, 2012. View at: Publisher Site | Google Scholar

Copyright © 2014 Kemin Wang et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

More related articles

 PDF Download Citation Citation
 Download other formatsMore
 Order printed copiesOrder

Related articles

We are committed to sharing findings related to COVID-19 as quickly as possible. We will be providing unlimited waivers of publication charges for accepted research articles as well as case reports and case series related to COVID-19. Review articles are excluded from this waiver policy. Sign up here as a reviewer to help fast-track new submissions.