Application of Reinforcement Learning in Cognitive Radio Networks: Models and Algorithms

Algorithm 4

RL algorithm for the channel auction scheme [37].

Repeat

(a) Observe the current state 𝑠^𝑖_𝑡 and available channels 𝐰_𝑡

(b) Choose an action 𝑎^𝑖_𝑡 and submits it to the base station

(c) Receive channel allocation decision 𝑧^𝑖_𝑡 and the required channel cost 𝑐^𝑖_𝑡

(d) Estimate the representative state 𝑠̃_𝑡^−𝑖 and update the state transition probabilities𝑝_𝑡^−𝑖 of the other SUs

(e) Compute the estimated 𝑄-value 𝑄^𝑖_𝑡𝑠^𝑖_𝑡,𝑠̃_𝑡^−𝑖,𝐰_𝑡,𝐚_𝑡 as follows:

   𝑄^𝑖_𝑡𝑠^𝑖_𝑡,𝑠̃_𝑡^−𝑖,𝐰_𝑡,𝐚_𝑡≅𝑟^𝑖_𝑡+1𝐬_𝑡+1,𝐰_𝑡+1,𝐚_𝑡+1

           +𝛾∑_{(𝑠𝑖𝑡+1,̃𝑠−𝑖𝑡+1)∈(𝑆𝑖,𝑆−𝑖)𝐰𝑡+1∈𝐾}𝑋𝑝^𝑖_𝑡⋅𝑝_𝑡^−𝑖⋅𝑉^𝑖_𝑡+1𝑠^𝑖_𝑡+1,𝑠̃^−𝑖_𝑡+1,𝐰_𝑡+1

(f) Update 𝑄-table 𝑉^𝑖_𝑡+1𝑠^𝑖_𝑡+1,𝑠̃^−𝑖_𝑡+1,𝐰_𝑡+1 using learning rate 𝛼 as follows:

   𝑉^𝑖_𝑡+1𝑠^𝑖_𝑡+1,𝑠̃^−𝑖_𝑡+1,𝐰_𝑡+1=(1−𝛼)𝑉^𝑖_𝑡𝑠^𝑖_𝑡,𝑠̃_𝑡^−𝑖,𝐰_𝑡+𝛼𝑄^𝑖_𝑡𝑠^𝑖_𝑡,𝑠̃_𝑡^−𝑖,𝐰_𝑡,𝐚_𝑡