Abstract

Resources available for operation in cognitive radio networks (CRN) are generally limited, making it imperative for efficient resource allocation (RA) models to be designed for them. However, in most RA designs, a significant limiting factor to the RA’s productivity has hitherto been mostly ignored, the fact that different users or user categories do have different delay tolerance profiles. To address this, in this paper, an appropriate RA model for heterogeneous CRN with delay considerations is developed and analysed. In the model, the demands of users are first categorised and then, based on the distances of users from the controlling secondary user base station and with the assumption that the users are mobile, the user demands are placed in different queues having different service capacities and the resulting network is analysed using queueing theory. Furthermore, to achieve optimality in the RA process, an important concept is introduced whereby some demands from one queue are moved to another queue where they have a better chance of enhanced service, thereby giving rise to the possibility of an improvement in the overall performance of the network. The performance results obtained from the analysis, particularly the blocking probability and network throughput, show that the queueing model incorporated into the RA process can help in achieving optimality for the heterogeneous CRN with buffered data.

1. Introduction

In recent times, there has been a deservedly growing interest in cognitive radio networks (CRN) as a possible driver for next generation (xG) wireless communications. This special interest in CRN hinges on its promise of much higher resourcefulness, particularly considering the spectrum usage. In its design, CRN enables an allotted spectrum space to be used by two different networks, a primary and a secondary one, under certain preconditions agreed upon by both networks [1]. With this kind of arrangement, much better utilisation of the rather scarce spectrum becomes inevitable. This promise has triggered considerable research effort to develop and describe the CRN, as well as to address possible challenges to its introduction and implementation.

In CRN, primary users (PUs) generally take priority in the usage of the resources, especially spectrum, because they are the original or licensed owners of it. The secondary users (SUs) in the network must devise ways to achieve and maintain acceptable quality of service (QoS) despite the stringent conditions under which they have to operate [2]. In heterogeneous CRN especially, the demands of the SUs are usually different from one SU to another or from one category of users to another, and the CRN should be capable of meeting the different demands efficiently and timeously [3]. To make this possible, resource allocation (RA) models that capture the essential peculiarities and dynamics of the heterogeneous users in CRN and that can optimally assign the available resources fairly and favourably are required [4].

In developing RA models for CRN, an important and realistic criterion for categorising demands of the heterogeneous users is their level of delay tolerance that is permissible for an acceptable QoS. Depending on the kind of service being provided, different users may have differing delay tolerance characteristics. Usually, because resources for SUs’ transmission are limited or sometimes even temporarily unavailable (e.g., because of PUs’ transmission), SUs, depending on the kind of service intended to be provided, may keep their data for transmission in a buffer (queue) and wait (usually for an acceptable time duration) for the requisite resources to be available for their transmission to be completed. Those delay instances and/or durations for SUs, and the properties of the queue developed as a result, are critical for the CRN’s QoS provisioning and applying queueing models in analysing the queue characteristics can help in improving their performance significantly. That is the main interest of this paper. The paper therefore develops and analyses an appropriate queueing model for heterogeneous CRN with users having different delay priorities or delay profiles. The analyses of the model show that, by investigating and exploiting the delay characteristics of users in CRN using queueing theory, a significant improvement in the optimal allocation of resources for the heterogeneous buffered CRN can be realised.

Developing appropriate RA models for CRN has been a recent research focus and significant progress has already been made in this regard. In earlier works of the authors, a comprehensive survey of various approaches that have been investigated and employed in achieving optimal (or near-optimal) solutions for RA in CRN has been carried out, with their pros and cons highlighted and discussed [5]. Optimal RA solutions for heterogeneous CRN were investigated in [6, 7] and the works were further strengthened in [8] to include QoS provisioning. The concept of cooperative diversity was introduced in [9] to address the problem of interference mitigation, thereby achieving even better results.

Similarly, other authors have also been (and are currently being) involved in investigating RA solutions for CRN. The authors in [1012] analysed RA problems in heterogeneous multiple-input multiple output CRN. In their works, the dual optimisation problems developed focused on optimising the transmit power and transmission time allocation of the SUs. To achieve their objectives, firstly, power allocation to each SU was optimised (on assumption of a constant transmission time), and secondly, optimal scheduling of the SUs was realised. In [13, 14], the authors studied RA in CRN, first with cooperative relays and then with imperfect spectrum sensing. The developed problems were nonconvex and nondeterministic polynomial-time-hard (NP-hard). By separating the problems into two, subchannel allocation and power allocation, a convex programming conversion was achieved and suboptimal solutions realised. All of these works and many more similar works all show that RA in CRN is an active research area and some interesting results have been achieved already. However, in the above-mentioned works and in most other related works on RA in CRN, no consideration has been given to the delay requirements of the SUs.

The above-mentioned works have all addressed RA problems in CRN but without any consideration of queueing capabilities in achieving better (and/or optimal) solutions. Works on RA in the CRN that have incorporated some kind of time delay or data queueing in their RA problem formulations are indeed very few. The available ones, as obtained by the authors in the process of this research work, are briefly reviewed. In [15], the authors, in developing their RA model, made provision for users that require real-time communication and gave them priority over non-real-time users. The real-time users were given priority in that they were admitted first into the network and given sufficient resources to transmit their data. Thereafter, an optimal number of non-real-time users were made to share the remaining resources of the network. Thus, the delay profiles of the users were used as a means of controlling the number of users admitted to the network. Authors in [16] classified the SUs into delay-sensitive and delay-tolerant SUs, with the delay-sensitive SUs having the higher priority in the allocation of resources, as their delay requirements must always be met first. In the analysis, the delay requirements of the delay-sensitive SUs were simply transformed into a constant rate requirement using very elementary ideas of queueing theory.

A study on band allocation in CRN when both PUs and SUs have data queues was carried out in [17]. In order to satisfy the required QoS for the SUs, the authors proposed that each SU be probabilistically assigned to a PU band and showed that such arrangement gives a better performance than either random or fixed allocations for the SUs. In [18], the authors developed a model that separates the users by the amount of data backlog they have in their queues waiting to be transmitted. The allocation of resources was carried out based on the size of the backlog; users with small backlogs were given just sufficient resources to transmit their data, while users with large backlogs shared the remaining resources among themselves in a manner that was fair and efficient. In the above-mentioned works, the significance of time delays and the effects on the overall performance of the RA problems had not been studied. Moreover, improving overall network capacity by developing appropriate queueing models to address the RA problems was never carried out, making this work uniquely different.

In this work therefore, the more realistic possibility of users having buffered data (queues) is incorporated into the RA problems of CRN and its effects are investigated. Particularly, the RA model developed in this paper addresses the very likely circumstance where heterogeneous SUs have data in their buffer waiting to be transmitted. With the SUs having different delay tolerance profiles, the model seeks to find a unique method of transmitting the SUs’ data in a manner in which QoS is guaranteed for each SU, while the overall productivity of the network is enhanced. By categorising the SUs and developing efficient queueing and RA mechanisms to best satisfy their requirements, a much greater capacity can be realised for the network. The contributions in this work are summarised as follows:(i)Developing and analysing a queueing model that captures delay characteristics of various categories of users in buffered heterogeneous CRN.(ii)Investigating optimality in the RA process for the heterogeneous CRN by studying the impact of varying the values of user demands into different queues and determining their effect on the overall performance of the network. The variation in arrivals is achieved by moving a factor of demands from users in a farther queue to a nearer one. itself is such that it can be changed continuously within a certain range, and an optimal value for that maximises the productivity of the network is realised.

3. System Model

The system model is shown in Figure 1. The model is a development on the work presented in [19]. The referenced work had considered an overlay network in which the SUs query a database to ascertain the PUs’ transmission and had designed the SUs’ connections according to the periodicity of the PUs’ traffic patterns. However, in this model, a centralised, underlay heterogeneous CRN is developed. The SUs are made to transmit on the entire PUs’ bandwidth within an acceptable interference power limit. This makes it possible to concentrate on analysing the SU network of the CRN separately and intently, so as to achieve the utmost for the SUs, and this without causing significant harm to the PUs.

The SUs are classified based on their delay profiles as either delay-sensitive (DS) or delay-tolerant (DT) users, the classes differentiated by the duration of delay time for acceptable service. The SUs are assumed to be both mobile and capable of performing adaptive modulation and coding (AMC) to dynamically change their modulation and coding schemes. Moreover, the SUs, depending on their distance to the secondary user base station (SUBS), are placed in virtual rings. The nearest ring to the SUBS operates with the highest AMC scheme, while the farthest ring operates with the lowest AMC scheme. Data transmission requests of users within a ring are placed in a queue of that ring and service (transmission of data) is carried out using the available subchannels. The queues therefore act as a buffer, should there be a possible delay in immediate transmission due to insufficiency in resource availability. There are subchannels, which automatically correspond to the number of parallel servers in each ring (or queue). Queues are finite with a maximum length . As a result of the mobility of SUs, arrival rates into queues can be adjusted so that the maximum productivity of the network can be realised. To accomplish this, a fraction of the demands of users, particularly the DT demands, which have a high delay profile, is moved from a farther ring (queue) to a closer ring (queue), so that the demands can possibly be transmitted at a higher rate. The intent is that, by such an arrangement, a likely reduction in both the time and energy consumed in transmitting the data can be achieved, making it possible for significant improvement in the capacity of the SUs of the CRN.

3.1. Queueing Model

The queueing model is set up to determine the capacity of the system. The queueing analysis shows the significance of the fraction of demands that is moved between queues. To make the model manageable and easier to analyse, only two concentric rings are considered, meaning that there are two parallel queues, each served by multiple servers (subchannels). The analysis can however be extended to three or more rings. In the initial analysis, the closest ring to the SUBS is assigned to transmit at a modulation scheme of -QAM ( bits per symbol) and the farthest ring is assigned to transmit with a modulation scheme of -QAM ( bits per symbol). Arrivals into each of the queues follow a Poisson distribution with arrival rates for queue and for queue . Service is exponential with rates for queue and for queue . and correspond to the data rate of the AMC scheme operated in each ring, meaning bits per symbol and bits per symbol for queue and queue , respectively. Service per unit time in queue is therefore significantly faster than in queue , since it operates at a higher service rate. Some of the arrivals into each queue are DT demands and users can move from one ring to another. The essence is to determine whether the CRN’s productivity can be improved by moving the DT demands of the farther ring to the ring nearer to the SUBS. In the model therefore, a fraction , which represents the DT demands of queue , is moved to queue where the demand is capable of being transmitted at a higher data rate. The challenge is to be able to find the value , the fraction of the demands from the farther queue to be moved to the nearer queue that will not be counterproductive but will rather optimise the total productivity of the network.

4. Analysis of Model

For the analysis carried out in this work, the model is restricted to two rings (queues) for simplicity. To analyse the queueing model developed, the following parameters are defined:(i)Total number of subchannels (which is also equivalent to the number of multiple servers in each queue) = (ii)Arrival rate into queue 1 = (iii)Arrival rate into queue 2 = (iv)Fraction of queue 2 (DT demands) moved to queue 1 = (v)Total arrival into queue (vi)Total arrival into queue (vii)Total arrival into the network , with the traffic intensities given as ; ; (viii)Service rate of queue 1 = (ix)Service rate of queue 2 =

The queueing model is a continuous-time Markov chain (CTMC) queue with a finite buffer. The model is shown in Figure 2. From Figure 2, it can be observed that the total arrival into queue is the addition of the original arrival and the fraction of arrival to queue that is redirected to queue . Similarly, the total arrival to queue becomes what is left of the original arrival after the fraction has been taken away. If there are no arrivals into queue or no part of queue is moved to queue , then and arrival to queue is simply limited to .

Let be the state space of the combined queues, where represents the number of data packets in the system from queue . Hence, . The state space diagram is shown in Figure 3. (for simplicity, this is subsequently written as ) is the probability that at time there are data packets in the system, implying that there are packets in queue and packets in queue available for transmission (including the packets in service). The system can now be studied at steady state using the stationary distribution conditions. If = is the steady state vector, the conditions and for CTMCs hold, where is the generator matrix and is the identity matrix.

From the state space diagram as shown in Figure 3 and using the steady state conditions, the equilibrium balance equations are obtained as

The set of linear equations (1), being finite, can be solved simultaneously in a number of ways. Three different approaches for solving these types of equations and obtaining solutions are discussed.

4.1. Standard Approach

The set of linear equations (1) can be solved using any of the well-established standard numerical analysis methods for solving simultaneous linear equations. These methods are broadly classified into two which are the direct approach (such as by using backward substitution or Gaussian elimination) and the iterative approach (such as Jacobi or Gauss-Seidel methods). Using any of these approaches, state probabilities are obtained, which can then be used to obtain the various performance measures of interest. These approaches have been used extensively in obtaining state probabilities, most especially for networks with manageable queue sizes. Detailed analysis on these methods has been well presented in [20] and it is therefore not necessary to repeat them in this work. References [21, 22] are classic examples of the use of these approaches in analysing queueing models and obtaining state probabilities for CRN. The major challenge with this approach is the high complexity and computational demands of the analysis, especially for large networks. Due to the complex nature of CRN, it is arguable whether the approach is best suited for its RA optimisation determination, especially when heterogeneous considerations are made, as developed in this work.

4.2. Equivalent Two-Class System

Although the developed queueing problem of RA in CRN may be solved using the standard approach, other (presumably better) methods for obtaining solutions can as well be investigated. The second approach being explored here is referred to as an equivalent two-class, first-come first-served system. By a careful consideration of the developed model, it is observed that the overall system can be closely comparable to a two-class, nonpriority queueing model in which customers are served using the first-come first-served discipline with different arrival rates and service rates [23]. According to [23], such a system can be regarded as an M/G/1 queue, where G is a mixture of two exponential distributions (a hyperexponential distribution) and packet arrivals are grouped into a single arrival stream. The analysis of this system has already been carried out in works like [23] and gives the following expressions for the queue lengths , , and representing the queue length of queue 1, queue 2, and the overall queue, respectively:

4.3. State Reduction

The final solution approach to be investigated is the use of state reduction. This approach is the most significant of the three approaches considered in this work. State reduction is a technique for obtaining steady state probabilities for finite (and even infinite) state Markov chains. The approach is developed by Grassmann, Taksar, and Heyman and is generally referred to as the GTH algorithm [24]. This approach is very important in that it is employed to study the effect of and to find optimal values for the parameter for different possible CRN scenarios. By varying , the arrival rates into each queue can be adjusted and the effects on the overall performance of the network investigated. After obtaining the equilibrium probabilities of the Markov chain through state reduction, the optimum value of the parameter is then obtained through Newton’s method. It is important to note that the definition of is not necessarily limited to the fraction of the DT demands being moved from one queue to the other alone, as is being considered in this work. It could be defined to be any other factor, for instance, a fraction of the higher priority demands. The important question to be addressed is how to determine the value of that will maximise the overall productivity of the network. The derivations and analysis of how the state reduction approach is used in obtaining state probabilities and the optimal value of are provided in Appendices A and B.

4.4. Performance Metrics

Performance measures employed to demonstrate the effect of on the network, as considered in this paper, are the blocking probability and system throughput. Both of these measures are obtainable using the steady state probabilities. Other performance measures, such as the average number of packets in the queue or in the system and the average waiting time of a packet in the queue or before the packet’s transmission is completed, are easily obtainable from the two measures evaluated but for the sake of brevity, they are not considered in this paper. The blocking probability is the probability that a packet that arrives in the network is blocked or dropped because of meeting a full system (i.e., all servers are fully engaged and the waiting queue is full) and is therefore not served. In steady state, the blocking probability is defined asBlocking probability, = , The system throughput is defined as the number of total arrivals that are eventually served, that is, the number of total arrivals that are not dropped or blocked. The throughput is in effect the effective arrival rate, represented as . The throughput is given as follows: Throughput = effective arrival rate, . That is:Throughput = total arrival rate, , and hence,

4.5. Scalability of the Model

It is important to discuss the scalability of the model. If the rings (and invariably, the queues) are extended beyond two, the network becomes larger and indeed, it becomes more difficult to analyse such networks. However, analysing queueing models with more than two queues, though difficult, is not impossible to achieve. Two propositions on how this can be achieved are provided. First, if the queues are within a limited number, say three or four, then it is possible to analyse the network using the exact same ideas presented in this section, that is, by using stationary distribution conditions to obtain equilibrium balance equations which are then solved using any of the approaches already discussed to obtain state probabilities. It has to be stated, however, that the number of equations to be solved grows exponentially with the number of queues being considered, meaning that the computational complexity can be significantly increased when the networks are large. The second proposition, which may be more appropriate for larger networks with higher number of rings (and queues), is to use the matrix-analytic approach developed by Neuts and Miller [25, 26]. This approach has been proven to be useful in solving both continuous-time and discrete-time queueing models with a considerably high level of accuracy and reliability. The computational complexity is also manageable, even for large networks.

5. Results and Discussion

In this section, numerical results of the developed model are presented. The results are obtained using the analysis from the state reduction method, which, as earlier argued for the developed model, is the best of the methods presented for obtaining state probabilities and studying the effect of the fraction of packets moved from queue 2 to queue 1, . For the purpose of validation, however, Gaussian elimination (an example of direct method) was also used to obtain state probabilities and the values obtained correspond exactly to those obtained using state reduction. Generally, values of state probabilities are usually typically very close, irrespective of the methods employed in obtaining them. The major difference is always in the level of complexity in the computations when different methods are employed.

For ease of analysis, the network model, comprising two separate queues, is limited to servers (subchannels) in each queue. A queue length of is equally employed in each queue. The model, parameters used (number of servers, queue length, etc.), and the results obtained are comparable to, and validated by, the work in [27]. The model is however easily scalable, and the results obtained are fair representations of larger networks. For the first consideration (as presented in the first set of plots), SU demands in queue 1 are served at a data rate of bits per symbol (64-QAM modulation) for each unit of time, while demands in queue 2 are served at a rate of bits per symbol (4-QAM modulation) for each unit of time. In the second consideration, the service rates at both queues are reduced to study the effect of such reduction on the overall performance of the network. SU demands in queue 1 are therefore served at bits per symbol (4-QAM modulation) for each time unit, while demands in queue 2 are served at a rate of bit per symbol (BPSK modulation) for each time unit. In all analyses, the arrival rate to each of the queues is gradually increased from to bits per symbol per unit of time. The value of is varied between 0 and 100% of . Performance results of blocking probability and throughput of the heterogeneous CRN are presented and discussed.

Figures 4 and 5 present the blocking probability and throughput performance measures, respectively, when the service rate is at bits per symbol per unit time for queue 1 and bits per symbol per unit time for queue 2. From the results, it can be observed that, at low arrival rates, the blocking probability, which is the probability of finding the system full, is very low, implying that the system can effectively service almost all arrivals. The effective arrival rate, or throughput, is thus very high, in fact, close to the total number of arrivals in the system. Again, it can be observed that, by gradually increasing arrivals, the effective arrival steadily increases, although at an increasing blocking probability. It therefore implies that, by increasing the arrival rate, the system throughput can indeed be increased, albeit at a decreasing rate. Eventually, a highest possible value of the throughput is obtained because of the obvious limitation in server capacity. Also, the results show that, by increasing , the blocking probability decreases while the throughput increases, signifying an improvement in the overall performance of the network.

In Figures 6 and 7, the results of both blocking probability and throughput are likewise presented; only in this instance, the service rates of the servers are bits per symbol per unit time in queue 1 and bit per symbol per unit time in queue 2, respectively. The results show similar trends to those presented in Figures 4 and 5, except that, at some point, the improvement in performance due to an increase in is completely eliminated and rather, a gradual reduction in performance is observed. This is because, as more and more of the demands of queue 2 are moved to queue 1 in the hope of being served more quickly, a tipping point (which invariably corresponds to the optimum value) is reached after which an additional increase in the value of queue 2 demands moved to queue 1 no longer improves the overall performance. Rather, the blocking probability increases more significantly, owing to the continuous increase in queue 1, resulting in a decrease in the network throughput as well as a poorer overall network.

It is very significant to observe from a comparison of the results presented in Figures 4 and 5 and Figures 6 and 7 that if all the demands of queue 2 are DT and queue 1 has a very high service rate (high enough to conveniently accommodate demands from both queues 1 and 2, as observable in Figures 4 and 5), it would be practically unnecessary to have any service at all in queue 2. This is because, in Figures 4 and 5, even at = 100%, that is, after moving the entirety of the demands in queue 2 to queue 1 (with the assumption that all the demands are DT), the throughput performance had still not declined. In comparison with Figures 6 and 7, where the service rate in queue 1 is relatively lower, after some optimum value of , an increase in its value results in a decline in the overall performance of the system because the capacity of queue 1 has been overstretched and therefore both the blocking probability and the throughput performances begin to depreciate. It is therefore important for any given problem formulation always to find the optimum value of and to allow only that fraction of queue 2 demands to be moved to queue 1 in order to achieve an overall best (optimum) performance for the network.

Figures 8 and 9 describe the performance of the blocking probability and the throughput as a function of for different values of arrival. From the plots, it is possible to determine the value of the total rewards and to observe its effects on the overall network performance. According to Figure 9, the total rewards for any given value of are simply the difference between the throughput value at the given and the throughput value at , for any value of the arrival rates. Similarly, the optimal value of for each of the arrival rates considered can easily be observed. At the total arrival rate of 8 bits per symbol per unit time, an optimal value for was never realised because, even at = 100%, improvement in performance was still being observed. However, for total arrival rates of 12 and 16 bits per symbol per unit time, an optimal value of is realised at the peaks of the plots, after which the performance of the network begins to depreciate. Finally, it should be noted that a change in either the arrival or service rates of the queues will shift the optimum value of , either to the right or to left of the plots. Therefore, a universal optimum value of cannot necessarily be realised; only specific values for specifically developed problem formulations and network parameters (such as arrival rates, service rates) are feasible.

6. Conclusions

In this paper, a queueing model has been developed and studied for RA optimisation in heterogeneous CRN with different users having buffered data in queues waiting for transmission. In the model, by leveraging the different delay tolerance profiles of user demands and the mobility of users, a queueing system is developed that can achieve a greater overall capacity in the RA of the network while still satisfying the varying demands of each of the users or user categories. In the RA model developed, the user demands are classified as either DS or DT, while a fraction of the DT demands in the queue farther from the SUBS is moved to the queue nearer to the SUBS for possible faster transmission. This is achievable because the ring (or equivalently, the queue) nearer to the SUBS transmits at a higher rate, which implies that its service rate (and thus its capacity) is higher. The ring can therefore accommodate a larger number of SUs (and their demands) than the ring farther from the SUBS. By changing over a given range, its effect on the overall performance of the network is analysed and optimal values for , depending on prevalent arrival or/and service conditions, are realised. The results of the various analyses presented show that, with the developed model, an improvement in blocking probability and an optimal throughput can be effectively achieved for the heterogeneous CRN with buffered data.

Appendix

A. Obtaining the Steady State Probabilities and Achieving Optimal Values through State Reduction

For easier depiction, let represent the state . Also, for each , let be the additional data rate (reward) achieved through the movement of data packets from queue 1 to queue 2. From any state , the rate of going to another state is , where ’s form the elements of the generator matrix . Implicitly, both and depend on the parameter . Hence, to achieve an optimal total data rate (maximum throughput), an optimal value of has to be obtained such that the total additional data rates (total rewards) are maximised when the system is in equilibrium (steady state). This therefore defines the problem as that of determining the optimal value of when the network is in steady state. Let be the total expected rewards (total additional data rate due to ); the objective of the optimisation problem is given as and the constraints are the limitations in both arrival and service rates of the network, as well as the time delays and the minimum data rate for an acceptable QoS.

Since is a variable entity, differential calculus can be employed in finding its optimal value. Let and be the respective first and second derivative of with respect to (for ease of reference, , , and are subsequently represented as , , and , resp.). is maximised by solving for . To solve using Newton’s method, has to be obtained. Also, since the problem is a maximisation problem, has to be negative. To obtain and , intermediate results from state reduction are used [27]. State reduction is first employed to obtain the steady state vector , the values of which are then used to obtain and its derivatives.

Using the state reduction method to obtain requires a relaxation of the steady state equation , so that the sum of is no longer . A different solution is obtained, say, and let its vector be . To differentiate from , is referred to as the chance that there are data packets in the network, while is the probability that there are data packets in the network at any given time. It can immediately be observed that the sum of is not . Rather, a certain state, say state , is given a chance and the chances for all other states are calculated accordingly. Then, to find the probability of being in state , the chance is divided by the sum of all chances . Hence,or . Differentiating twice with respect to , this becomesand hence,

The values are obtained using the state reduction (or GTH) algorithm. The algorithm is provided in Appendix B. Once these values have been obtained, substituting into the set of equations above gives the corresponding . Also, after obtaining , obtaining the optimal value for is achieved by applying Newton’s method. This is carried out as follows.

Assume that there are states numbered from to and is finite. Recall . But . Substituting for gives Taking first and second derivatives with respect to becomes The above submissions yield the following equations for , and : To obtain the optimal value of , an approximate value of is first chosen, say , and a new value , presumably a better approximation, is obtained according to Newton’s method as follows: Substituting for and gives In the above formulation, maximising is subject to . If not, then the sign of has to be considered. If , the rewards can be increased by increasing . If , the rewards can be increased by decreasing .

B. The State Reduction and/or GTH Algorithm (See [27])

The values for , , and , required to obtain the steady state probabilities (and their derivatives), , , and , can be obtained through state reduction as explained below.

Recall that .

Let , and hence, where is the generator matrix. Taking first and second derivatives with respect to , we have The set of equations given above can be solved using state reduction to obtain , , and . The state reduction algorithm is as follows.

The states are numbered from to , giving a total of states. is finite. For each state, we find a steady state equation using .

For state , the steady state equation is given aswhere are the elements of the generator matrix . By using Gaussian elimination, equations can be used to eliminate from equations so that a new set of equations is obtained. The new set of equations is represented by Grassmann, Taksar, and Heyman (GTH) did show that can indeed be interpreted as the transition rates of a continuous-time Markov chain, meaning that it is unnecessary to calculate the diagonal elements . Specifically, if is defined as , then, We can use normal elimination methods to find all recursively, starting with and then calculating . This gives, for , Substituting for becomes Better still, a more recent approach to state reduction, modified by Grassmann [27], which gives a more convenient and quicker recursion, is obtained by making Hence, For , this gives For , this gives is now defined as The can be calculated by row, starting with row then continuing with row and so on. Once are calculated, normal back-substitution can be used to obtain thus.

Set and evaluate as follows: The equations for and do have the same structure as the equations for . This can be verified by using the fact that and writing as follows: Since , must be . Also, the th equation obtained from has the th element of as its constant term. If this constant term is denoted as , then, The last two equations are similar, except that all are replaced by . Consequently, if are the coefficients obtained by eliminating from , except for . From the algorithm given above, Again using the fact that , Similarly, if , can be obtained by a similar representation to and using back-substitution to find from the equation

Conflicts of Interest

The authors declare that there are no conflicts of interest regarding the publication of this paper.