Application of Reinforcement Learning in Cognitive Radio Networks: Models and Algorithms

Algorithm 5

RL algorithm for RL-DCS [27].

Repeat

(a) Choose action 𝐚^𝐢_𝐭=𝑎^𝑖_1,𝑡,𝑎^𝑖_2,𝑡,…,𝑎^𝑖_𝐾,𝑡∈𝐴₁×𝐴₂×⋯×𝐴_𝐾

(b) Receive delayed reward 𝑟^𝑖_𝑡+1𝐚^𝐢_𝐭 from ECE

(c) Update 𝑄-value:

   For each 𝑎^𝑖_𝑘,𝑡∈𝐚^𝐢_𝐭

     𝑄^𝑖_𝑡+1𝑎^𝑖_𝑘,𝑡=𝑄^𝑖_𝑡𝑎^𝑖_𝑘,𝑡+𝛼⋅𝑟^𝑖_𝑡+1𝑎^𝑖_𝑘,𝑡−𝑟^𝑖_𝑡𝑎^𝑖_𝑘,𝑡⋅𝑎^𝑖_𝑘,𝑡−𝑃^𝑖_𝑡𝑎^𝑖_𝑘,𝑡⋅𝑥

(d) Update probability 𝑃^𝑖_𝑡(𝑎^𝑖_𝑘,𝑡), which is the probability of taking action 𝑎^𝑖_𝑘,𝑡:

   For each 𝑎^𝑖_𝑘,𝑡∈𝐚^𝐢_𝐭

         𝑃^𝑖_𝑡(𝑎^𝑖_𝑘,𝑡1)=maxmin1+𝑒^{−𝑄𝑖𝑡+1(𝑎𝑖𝑘,𝑡)⋅𝑥},1−𝜀,𝜀