Reinforcement Learning in an Environment Synthetically Augmented with Digital Pheromones

Table 3

States by time step.

(a) Neighborhood states at time 𝑡

	1	2	3
3	Aug (Raw): 0.5	Aug (Raw): 0.3	Aug (Raw): 0.1
	Aug (Level): 0	Aug (Level): 0	Aug (Level): 0
	Increase?: N	Increase?: N	Increase?: N
	Decrease?: Y	Decrease?: Y	Decrease?: Y
	Nbhd Max?: N	Nbhd Max?: N	Nbhd Max?: N
	Nbhd Min?: N	Nbhd Min?: N	Nbhd Min?: Y
	State: 4	State: 4	State: 5
2	Aug (Raw): 1.7	Aug (Raw): 1.1	Aug (Raw): 0.7
	Aug (Level): 1	Aug (Level): 1	Aug (Level): 0
	Increase?: N	Increase?: N	Increase?: N
	Decrease?: Y	Decrease?: Y	Decrease?: Y
	Nbhd Max?: N	Nbhd Max?: N	Nbhd Max?: N
	Nbhd Min?: N	Nbhd Min?: N	Nbhd Min?: N
	State: 20	State: 20	State: 4
1	Aug (Raw): 2.1	Aug (Raw): 1.9	Aug (Raw): 1.3
	Aug (Level): 2	Aug (Level): 1	Aug (Level): 1
	Increase?: N	Increase?: N	Increase?: N
	Decrease?: Y	Decrease?: Y	Decrease?: Y
	Nbhd Max?: Y	Nbhd Max?: N	Nbhd Max?: N
	Nbhd Min?: N	Nbhd Min?: N	Nbhd Min?: N
	State: 38	State: 20	State: 20

(b) Neighborhood states at time 𝑡+1

	1	2	3
3	Aug (Raw): 0.9	Aug (Raw): 0.6	Aug (Raw): 0.5
	Aug (Level): 0	Aug (Level): 0	Aug (Level): 0
	Increase?: Y	Increase?: Y	Increase?: Y
	Decrease?: N	Decrease?: N	Decrease?: N
	Nbhd Max?: N	Nbhd Max?: N	Nbhd Max?: N
	Nbhd Min?: N	Nbhd Min?: N	Nbhd Min?: Y
	State: 8	State: 8	State: 9
2	Aug (Raw): 1.7	Aug (Raw): 1.8	Aug (Raw): 1.4
	Aug (Level): 1	Aug (Level): 1	Aug (Level): 1
	Increase?: N	Increase?: Y	Increase?: Y
	Decrease?: N	Decrease?: N	Decrease?: N
	Nbhd Max?: N	Nbhd Max?: N	Nbhd Max?: N
	Nbhd Min?: N	Nbhd Min?: N	Nbhd Min?: N
	State: 16	State: 24	State: 24
1	Aug (Raw): 2.5	Aug (Raw): 2.6	Aug (Raw): 2.2
	Aug (Level): 2	Aug (Level): 2	Aug (Level): 2
	Increase?: Y	Increase?: Y	Increase?: Y
	Decrease?: N	Decrease?: N	Decrease?: N
	Nbhd Max?: N	Nbhd Max?: Y	Nbhd Max?: N
	Nbhd Min?: N	Nbhd Min?: N	Nbhd Min?: N
	State: 48	State: 56	State: 48