Application of Reinforcement Learning in Cognitive Radio Networks: Models and Algorithms

Algorithm 2

RL algorithm for joint DCS and channel sensing [11].

Repeat

(a) Choose action 𝑎^𝑖_𝑡

(b) Update 𝑄-value 𝑄^𝑖_𝑡𝑠^𝑖_𝑡,𝑎^𝑖_𝑡∈𝑄^𝑖_𝑡𝑠^𝑖_𝑡,𝑎^𝑖_se,𝑄^𝑖_𝑡𝑠^𝑖_𝑡,𝑎^𝑖_tx,𝑄^𝑖_𝑡𝑠^𝑖_𝑡,𝑎^𝑖_sw as follows:

𝑄^𝑖_𝑡+1𝑠^𝑖_𝑡,𝑎^𝑖_𝑡=𝑄^𝑖_𝑡+1𝑠^𝑖_𝑡,𝑎^𝑖_se=𝑄^𝑖_𝑡𝑠^𝑖_𝑡,𝑎^𝑖_se+𝛼⋅𝑒^𝑖𝑠^𝑖_𝑡⋅𝑇^𝑖_se−𝑄^𝑖_𝑡𝑠^𝑖_𝑡,𝑎^𝑖_seif𝑎^𝑖_𝑡=𝑎^𝑖_se𝑄^𝑖_𝑡+1𝑠^𝑖_𝑡,𝑎^𝑖_tx=𝑄^𝑖_𝑡𝑠^𝑖_𝑡,𝑎^𝑖_tx+𝛼⋅1−𝑒^𝑖𝑠^𝑖_𝑡⋅𝑇^𝑖_𝑢−𝑄^𝑖_𝑡𝑠^𝑖_𝑡,𝑎^𝑖_txif𝑎^𝑖_𝑡=𝑎^𝑖_txSwitchchannel(Changechannelfrom𝑠^𝑖_𝑡to𝑠^𝑖_𝑡+1)if𝑎^𝑖_𝑡=𝑎^𝑖_sw

(c) Update 𝑄-value:

        𝑄^𝑖_𝑡𝑠^𝑖_𝑡,𝑎^𝑖_sw=max_{1≤𝑠≤𝐾}𝑉^𝑖_𝑡(𝑠)−𝑉^𝑖_𝑡𝑠^𝑖_𝑡−𝜃

(d) Update policy:

       𝜋^𝑖𝑠^𝑖_𝑡+1,𝑎^𝑖_𝑡+1=𝑒^{𝑄𝑖𝑡(𝑠𝑖𝑡+1,𝑎𝑖𝑡+1)}𝑄^𝑖_𝑡𝑠^𝑖_𝑡+1,𝑎^𝑖_se+𝑄^𝑖_𝑡𝑠^𝑖_𝑡+1,𝑎^𝑖_tx+𝑄^𝑖_𝑡𝑠^𝑖_𝑡+1,𝑎^𝑖_sw