Dynamic Request Routing for Online Video-on-Demand Service: A Markov Decision Process Approach

Algorithm 2

Interval value iteration (IVI_↕) algorithm.

Input: a BMDP 𝑀_↕, a value function 𝑉_↕, and a place 𝜋 for holding the policy in current iteration

Output:  𝑉^′_↕opt and 𝜋_opt

(1) Create 𝑂_up, 𝑂_down; {𝑂_up and 𝑂_down hold order sequence of states in 𝑀_↕, i.e., 𝑠₁𝑠₂,…,𝑠_𝑛.}

(2) Create 𝑃^′_up, 𝑃^′_down; {𝑃^′_up and 𝑃^′_down hold the transition probabilities for the order-maximizing MDP with respect

  to order 𝑂_up and 𝑂_down, respectively.}

(3) Create 𝑟_up, 𝑟_down; {𝑟_up and 𝑟_down is the order-maximizing index for order sequences 𝑂_up and 𝑂_down, respectively.}

(4) Create 𝑖; {𝑖 is the index into an ordering 𝑂.}

(5)  𝑂_up=Sort_Decreasing_Order(𝑉_↑);

(6)  𝑂_down=Sort_Increasing_Order(𝑉_↓);

(7)  for all  𝑠∈𝑆  do

(8)  for all  𝑎∈𝐴  do

(9)   𝑟_up=Order_Maximizing_Ind(𝑀_↕,𝑂_up,𝑠,𝑎);

(10)   𝑟_down=Order_Maximizing_Ind(𝑀_↕,𝑂_down,𝑠,𝑎);

    {find order-maximizing index for transition probability in state 𝑠 under action 𝑎 according to (18).}

(11)   for  𝑖=1 to 𝑛  do

(12)    Update 𝑃^′_up(𝑠_{𝑂down(𝑖)}∣𝑠,𝑎) and 𝑃^′_down(𝑠_𝑂up(𝑖)∣𝑠,𝑎) according to (19);

(13)   end for

(14)  end for

(15)   𝑉^′_↑=max_{(𝑠)𝐚⊆𝐴}𝑅_↑(𝑠,𝑎)+𝛿∑_{𝑠′∈𝑆}𝑃^′_up𝑠^′𝑠,𝑎𝑉_↑𝑠^′;(*)

(16)  if  |𝐚|=1 and 𝐚={𝑎}  then

(17)   𝑉^′_↓=𝑅_↓(𝑠,𝑎)+𝛿∑_{𝑠′∈𝑆}𝑃^′_down𝑠^′𝑠,𝑎𝑉_↓𝑠^′;

(18)   𝜋(𝑠)=𝑎;

(19)  else

(20)    𝑉^′_↓=max_𝑎∈𝐚𝑅_↓(𝑠,𝑎)+𝛿∑_{𝑠′∈𝑆}𝑃^′_down𝑠^′𝑠,𝑎𝑉_↓𝑠^′;(**)

(21)   𝜋(𝑠)=𝑎;

(22)  end if

(23) end if