Application of Reinforcement Learning in Cognitive Radio Networks: Models and Algorithms

Table 14

RL model for the channel hopping scheme [14].

State	𝐬^𝐢𝐤,𝐭=(𝑃^𝑖𝑘,𝑡,𝑔^𝑖𝑘,𝑡,𝑁^𝑖𝐶,𝑘,𝑡,𝑁^𝑖𝐷,𝑘,𝑡)∈𝑆1×𝑆2×𝑆3×𝑆4; substate 𝑃^𝑖𝑘,𝑡∈𝑆1={0,1} indicates an idle or busy channel; specifically, 𝑃^𝑖𝑘,𝑡=0 if PU activity does not exist, and 𝑃^𝑖𝑘,𝑡=1 if PU activity exists; substate 𝑔^𝑖𝑘,𝑡∈𝑆2={𝑞1,𝑞2,…,𝑞𝑁𝑔} represents gain, while 𝑁^𝑖𝐶,𝑘,𝑡∈𝑆3 and 𝑁^𝑖𝐷,𝑘,𝑡∈𝑆4 represent the numbers of control and data channels that get jammed, respectively
Action	𝐚^𝐢𝐭={𝑎^𝑖1,𝑡,𝑎^𝑖2,𝑡,…,𝑎^𝑖𝐾,𝑡}∈𝐴; subaction 𝑎^𝑖𝑘,𝑡=(𝑎^𝑖𝐶1,𝑘,𝑡,𝑎^𝑖𝐷1,𝑘,𝑡,𝑎^𝑖𝐶2,𝑘,𝑡,𝑎^𝑖𝐷2,𝑘,𝑡), where action 𝑎^𝑖𝐶1,𝑘,𝑡 (or 𝑎^𝑖𝐷1,𝑘,𝑡) indicates that the agent will transmit control (or data) packets in 𝑎^𝑖𝐶1,𝑘,𝑡 (or 𝑎^𝑖𝐷1,𝑘,𝑡) channels uniformly selected from the previously unjammed channels, while action 𝑎^𝑖𝐶2,𝑘,𝑡 (or 𝑎^𝑖𝐷2,𝑘,𝑡) indicates that the agent will transmit control (or data) packets in 𝑎^𝑖𝐶2,𝑘,𝑡 (or 𝑎^𝑖𝐷2,𝑘,𝑡) channels uniformly selected from the previously jammed channels
Reward	𝑟^𝑖𝑡+1𝐬^𝐢𝐤,𝐭+𝟏,𝐚^𝐢𝐭,𝐚^𝐦𝐭 represents the channel gain