Reinforcement Learning in an Environment Synthetically Augmented with Digital Pheromones

Algorithm 1

Temporal difference learning algorithm (adapted from [7]).

initialize all 𝑉(𝑠) arbitrarily

for all episodes

 initialize 𝑠

 repeat

    choose the next state 𝑠^′ using policy

    observe 𝑟

    𝑉(𝑠)←𝑉(𝑠)+𝛼𝑟+𝛾𝑉𝑠^′−𝑉(𝑠)

    𝑠←𝑠^′

 until 𝑠 is terminal state

  where 𝑉(𝑠) is the value of being in state 𝑠 and 𝑉(𝑠^′) is the value estimate

   of the resultant state 𝑠^′, 𝛾 is the discount rate, and 𝛼 is the learning rate.