Application of Reinforcement Learning in Cognitive Radio Networks: Models and Algorithms

Table 10

RL model for the channel auction scheme [37].

State	𝑠^𝑖𝑡=(𝑏^𝑖𝑡,𝐩^𝑖𝑡)∈𝑆; each state represents a two-tuple information composed of the fullness of the buffer state 𝑏^𝑖𝑡 and channel states 𝐩^𝑖𝑡=(𝑝^𝑖𝑡,1,𝑝^𝑖𝑡,2,…,𝑝^𝑖𝑡,𝑘), where 𝑝^𝑖𝑡,𝑘 represents the state of channel 𝑘 in terms of SNR
Action	𝑎^𝑖𝑡∈𝐴={𝑎^𝑖𝑡,1,𝑎^𝑖𝑡,2,…,𝑎^𝑖𝑡,𝑘}; each action represents the amount of a bid for white spaces in channel 𝑘. 𝐾 represents the number of available channels
Reward	𝑟^𝑖𝑡+1𝐬𝑡+1,𝐰𝑡+1,𝐚𝑡+1=𝑔^𝑖𝑡+1+𝑐^𝑖𝑡+1 represents the sum of the number of lost packets 𝑔^𝑖𝑡+1 and the channel cost 𝑐^𝑖𝑡+1 that SU 𝑖 must pay for using the channel. Note that the packet loss 𝑔^𝑖𝑡+1 and channel cost 𝑐^𝑖𝑡+1 depend on the global state 𝐬𝑡+1, available channels 𝐰𝑡+1, and bidding actions 𝐚𝑡+1of all competing SUs