A Sarsa(λ) Algorithm Based on Double-Layer Fuzzy Reasoning

Algorithm 1

DFR-Sarsa(λ).

(1)  Initialize parameter vector 𝜽=0, eligibility trace vector 𝐞=0, discount factor 𝛾, step-size parameter 𝛼

(2) Repeat(for every episode):

(3) 𝑥← initial state

(4) According to (10), compute 𝜙_𝑟(𝐱), 𝑟=1,…,𝑁_𝑅

(5) According to 𝜀-greedy policy, select activation action 𝑢_𝑟, 𝑟=1,…,𝑁_𝑅

(6) According to (13), select action 𝑢 when state is 𝑥

(7) According to (16), compute 𝜇_{𝜈𝑟,𝑗}(𝑢), 𝑟=1,…,𝑁_𝑅, 𝑗=1,…,𝑀

(8) According to (17) and (18), compute 𝑄_𝑢

(9) Repeat(for each step of episode)

(10)  Update eligibility trace: 𝐞(𝑟,𝑗)←𝛾𝜆𝐞(𝑟,𝑗)+𝜙_𝑟(𝐱)𝜇_{𝜈𝑟,𝑗}(𝑢), 𝑟=1,…,𝑁_𝑅, 𝑗=1,…,𝑀

(11)  Take action 𝑢, receive next state 𝑥^′ and reward 𝑟

(12)  𝛿←𝑟−𝑄_𝑢

(13)  According to 𝜀-greedy policy, select activation action 𝑢_𝑟, 𝑟=1,…,𝑁_𝑅

(14)  According to (13), select action 𝑢^′ when state is 𝑥^′

(15)  According to (16), compute 𝜇_{𝜈𝑟,𝑗}(𝑢^′), 𝑟=1,…,𝑁_𝑅, 𝑗=1,…,𝑀

(16)  According to (10), compute 𝜙_𝑟(𝐱^′), 𝑟=1,…,𝑁_𝑅

(17)  According to (17) and (18), compute 𝑄_𝑢′

(18)  𝛿←𝛿+𝛾𝑄_𝑢′

(19)  𝜽←𝜽+𝛼𝛿𝐞

(20)  𝑢←𝑢^′

(21)  Until 𝑥^′ is the terminal state

(22) Until preset episode number or other terminal condition meets