Constructing Temporally Extended Actions through Incremental Community Detection

Algorithm 2

Option learning with evolving communities.

(1)  Initial condition: learning agent with primitive options

(2)  Start option generation;

(3)  while true do

(4)   Sampling trajectories;

(5)   if condition for option discovery is met then

(6)    Use Louvain algorithm to detect communities;

(7)    Construct options from communities, having their and specified;

(8)    Take ER to form options’ internal policies ;

(9)    Add options to set;

(10)   break;

(11) Start learning;

(12) while not converged do

(13)  Perform Intra-Option Learning with the current option sets;

(14)  if reaches incremental processing cycle then

(15)    Call Algorithm 1 to update communities;

(16)    Reconstruct options from updated communities;

(17)    for reconstructed options, previous options do

(18)     if and then

(19)      

(20)    Take ER to improve options’ internal policies ;

(21)    Add updated options to set.