A Distributed Dynamic Super Peer Selection Method Based on Evolutionary Game for Heterogeneous P2P Streaming Systems

Abstract

Due to high efficiency and good scalability, hierarchical hybrid P2P architecture has drawn more and more attention in P2P streaming research and application fields recently. The problem about super peer selection, which is the key problem in hybrid heterogeneous P2P architecture, is becoming highly challenging because super peers must be selected from a huge and dynamically changing network. A distributed super peer selection (SPS) algorithm for hybrid heterogeneous P2P streaming system based on evolutionary game is proposed in this paper. The super peer selection procedure is modeled based on evolutionary game framework firstly, and its evolutionarily stable strategies are analyzed. Then a distributed Q-learning algorithm (ESS-SPS) according to the mixed strategies by analysis is proposed for the peers to converge to the ESSs based on its own payoff history. Compared to the traditional randomly super peer selection scheme, experiments results show that the proposed ESS-SPS algorithm achieves better performance in terms of social welfare and average upload rate of super peers and keeps the upload capacity of the P2P streaming system increasing steadily with the number of peers increasing.

1. Introduction

Future media internet will need to be able to distribute high quality video contents in an efficient, supple, and personalized way through dynamic and heterogeneous network environments. Multimedia streaming content distribution over internet has already become a major application due to users' growing demand of multimedia content and extraordinary growth of network technologies [1]. Streaming applications over heterogeneous P2P systems have gained an enormous popularity. However, providing streaming service over heterogeneous P2P networks is still a challenging task because of their inherent instability and unreliability. According to the characteristics of data dissemination structure, current P2P video streaming system deployed can be further divided into three types: tree-push based topology, mesh-pull based topology, and hierarchical hybrid topology. Among them, the hierarchical hybrid topology model, which divides the peers in P2P streaming systems into super peers and ordinary peers according to the performance differences of nodes, has become the focus of recent researches especially for heterogeneous P2P streaming applications. The hierarchical hybrid P2P model integrates the advantages of tree-based topology and mesh-based topology. It allows the nodes with stronger ability to participate fully in the P2P streaming system to get higher scalability and stability. However, the super peers selection problem, which is the key problem in hierarchical hybrid P2P streaming systems, is highly challenging because in the heterogeneous peer-to-peer environment a large number of super peers must be selected from a huge and dynamically changing network in which neither the node characteristics nor the network topology are known as a priori [2]. Furthermore, the super peers selection plays an important role in the efficiency and the Quality of Experience (QoE) of the whole P2P streaming system.

Currently, many approaches have been proposed to solve the super peer selection problem, but most of these researches focus on the capacity of peers [2–5], such as CPU cycle, memory space, and network bandwidth. And few studies focus on the behavioral attribute of peers. Super peers need to provide resources inquiry services and routing services for the other ordinary nodes within a group, and they also need to upload video streaming chunks for them. The cost of calculation and storage of super peers may lead to their system performances decline. An 𝐻₂𝑂 (hierarchical 2-level Overlay) protocol was proposed in [2] to select super peers in unstructured P2P networks which can take trust and secure paths, routing performance, and some other factors into account, but the notification messages between super peers are distributed through flooding mechanism, resulting in a lot of network overhead. Jin et al. proposed a super peer selection method based on proxy trust in [6]. They used proxy trust to identify the behaviors of super peers. With this metric a peer can select its own trusted super peers and isolate malicious super peers in the system. The method proposed in [6] reduced the overhead of network flooding in [2] to some extent, but it required a priori knowledge in the decision on whether to trust a node or not. Reference [7] proposed a super peer election algorithm based upon district partition which divided the whole P2P overlay into 𝑘 small districts and using distributed and parallel election algorithm among these small units to reduce message complexity as well as communication costs. A super-node selection protocol based on self-information theory was discussed in [8], which used a weighted mean mechanism based on the information vector of peers to promote the “best” nodes to be a super peer according to their capabilities and on-line time. Wolf et al. considered the super peer selection problem as a hub location optimization problem and proof it is NP hard in [9]. They also present a heuristic method based on evolutionary techniques and local search to solve this problem in [10]. Kratica et al. proposed two hybrid genetic algorithm approaches in [11] for solving this NP hard problem. The algorithms adopted two-segment encoding of peers and appropriate objective functions with modified genetic operators. They used local search heuristics to minimize the communication costs. But most of these researches studied general P2P networks and did not consider the characteristics of P2P streaming network. Wang et al. proposed a super peer selecting mechanism based on analytic hierarchy process in [12], which considered RTT, interest vector similarity, and bandwidth of the node synthetically. Being different from the above works, in this study, we present a super peer selection scheme based on evolutionary game in a hybrid heterogeneous P2P streaming networking scenario.

Game theory is a mature theory and a formal mathematic tool to model the strategy interaction between two or more decision makers, which has attracted more and more attention in computer science [13, 14] in recent years. The concepts of game theory provide a language to formulate structure, analyze, and understand strategic scenarios, which assumes that each participant involved in the game is “rational” to choose strategies to make his own benefit maximization. Roy et al. surveyed the existing game theoretic solutions in [14] which are designed to enhance network security, such as network intrusion detection, DDoS attacks, and internet worms. In traditional game theory, the Nash equilibrium is the most popular solution. It ensures that a player cannot improve its payoff if none of the other players in the game deviates from the solution. However, when the solution to a noncooperative game has multiple Nash equilibriums, a refined solution is required. Evolutionary equilibrium, which is based on the theory of evolutionary game theory, provides such a refined solution, and it ensures stability (i.e., group of players will not change their chosen strategies over time). An evolutionary game can explicitly capture the dynamics of interaction among the players in a population. In an evolutionary game, a player can observe the behavior of other players, learn from the observations, and make the best decision based on its knowledge. In addition, with replicator dynamics, the state of the game can be determined at a particular point in time, which is useful for investigating the trends of the strategies of the players while adapting their behavior to reach the solution.

The peers in P2P streaming systems are rational and selfish, and they can make decisions according to their needs and the behavior of other peers. In this paper, we present a distributed method for super peer selection dynamically in hybrid heterogeneous P2P streaming system. First we use evolutionary game theory framework to model the super peer selection procedure in hybrid heterogeneous P2P streaming system, and then we design a distributed super peer selection algorithm (ESS-SPS) based on Q-learning according to the Evolutionarily Stable Strategies (ESSs). We apply ESS-SPS algorithm to the real P2P streaming system LStream [15], which is deployed on the Henan Education and Research NETwork (HERNET). The experimental results show that compared with the traditional randomly super peer selection method, ESS-SPS method can provide the system with the higher streaming media upload rate stably and enabling the entire P2P streaming system upload capacity increasing steadily with the number of nodes increasing.

The rest of the paper is organized as follows. In Section 2, we describe the hybrid heterogeneous P2P streaming system model and utility functions of peer nodes. We model the super peer selection (SPS) procedure by evolutionary game and analyzes the Evolutionarily Stable Strategies for the SPS evolutionary game model by applying replicator dynamics equation in Section 3. In Section 4, we propose a distributed Q-learning algorithm for SPS evolutionary game model which can dynamically converge to the ESS. In Section 5, we show the experiment results to evaluate the performance of the proposed ESS-SPS algorithm. Last we conclude the paper in Section 6.

2. System Model and Utility Function

2.1. System Model

Peers in hybrid heterogeneous P2P streaming system are divided into peer groups according to their geographical location as shown in Figure 1 [16]. We expect that the video streaming chunks of a certain video channel can be shared within one peer group as much as possible to reduce unnecessary network traffic. There are a set of group peers who want to view a live video streaming channel simultaneously in the edge of the hybrid heterogeneous P2P live streaming system. Every peer can choose to act as a super peer (SP) or an ordinary peer (OP) in a group. If a peer choose to act as an SP, then he/she not only needs to act as a client to download video streaming chunks from the SPs in other peer groups but also needs to act as a server to upload video streaming chunks for both the SPs in other groups and the peers in the same group.

Figure 1 

Peer groups at the edge of hybrid heterogeneous P2P streaming system.

On the contrary, if a peer chooses to be an OP, he/she only needs to download or upload necessary streaming chunks in the same group. Assume that the upload and download bandwidth within a super group is larger than that cross groups. In such a case, peers tend to be an OP because of their selfish nature. However, from the other aspect, the OPs have a risk of receiving degraded streaming quality since there may not be sufficient SPs to download streaming chunks from other groups.

2.2. Utility Function

In the hybrid heterogeneous P2P streaming system, peers not only act as clients who download needed streaming chunks from other peers but also act as servers to upload video streaming chunks for the other peers. So, while a peer can benefit downloading video streaming chunks from the other peers, he/she also causes a cost in uploading video streaming chunks for the other peers, where the cost may be the resource consumed on uploading chunks, such as bandwidth, memory, and buffer size.

Assume that there are 𝑁 peers within the group, among which 𝑠 peers are willing to act as SPs to download video streaming chunks from the SPs of other groups. Suppose that the download rates of the 𝑠 SPs are 𝑟₁,𝑟₂,…,𝑟_𝑠; then the total download rate of the group peers is 𝑑_𝑠=∑^s_𝑖=1𝑟_𝑖.

As these 𝑠 SPs select peers outside the group for downloading streaming chunks randomly and independently, the download rate 𝑟_𝑖's are random variables. According to [17], the cumulative distribution function (CDF) of a peer's download bandwidth can be modeled as a linear function, which means that the probability density function (PDF) of a peer's download bandwidth can be viewed as a uniform distribution, which means 𝑟_𝑖's are uniformly distributed.

Obviously, if the total download rate 𝑑_𝑠 is no less than the source rate 𝑟, then the group peers can obtain the effect of real-time streaming, and all the group peers can obtain a certain profit 𝐺. Otherwise there will be some delay, and in this case we assume the gain of the peer is 0. Therefore, given the total download rate 𝑑_𝑠 and the source rate 𝑟, if peer 𝑖 chooses to be an SP, then its utility function is presented by 𝜋_SP,𝑖(𝑠)=Pr𝑑_𝑠≥𝑟𝐺−𝐶_𝑖,∀𝑠∈[1,𝑁], (1) where 𝐶_𝑖 is the cost of peer 𝑖 when he/she acts as an SP, and Pr(𝑑_𝑠≥𝑟) is the probability that the peer can obtain real-time streaming effect. If we assume that 𝑟₁,𝑟₂,…,𝑟_𝑠 are independent and identically distributed in [𝑟^𝐿,𝑟^𝑈], and 𝑠 is sufficiently large, then Pr(𝑑_𝑠≥𝑟) can be approximately computed as Pr(𝑑_𝑠≥𝑟)≈𝑄((𝑟̂−𝑠)/2)/(𝑠/12)), where 𝑟̂=(𝑟−𝑠𝑟^𝐿)/(𝑟^𝑈−𝑟^𝐿) and 𝑄(𝑥) is the Gaussian tail function ∫^∞_𝑥(1/√2𝜋)exp^−𝑥2/2𝑑𝑥 [18].

As the upload and download bandwidths within the group are large, it is assumed that the cost of uploading the streaming chunks to the other peers within the same group can be negligible. In this case, if a peer 𝑖 chooses to be an OP instead of an SP, then there is no cost for him/her, and the utility function is 𝜋_OP,𝑖(𝑠)=Pr𝑑_𝑠≥𝑟𝐺,if𝑠∈[1,𝑁−1];0,if𝑠=0. (2)

3. Super Peer Selection (SPS) Evolutionary Game Model

We adopt the concept of Evolutionarily Stable Strategy (ESS) [19, 20] to provide a robust equilibrium strategy for the selfish peers. In evolutionary game, Evolutionary Stable Strategy (ESS) is a solution to game theoretic problems which is equivalent to the Nash equilibrium but can be applied to the evolution of individuals behavior, which is defined as Definition 1. If all individuals are using ESSs, then they will do not change to a different strategy unless their expected payoff will be worse than the rest of the population in the system.

Definition 1. A strategy 𝑥^∗ is an ESS, if and only if, for all𝑥≠𝑥^∗, 𝑥^∗ satisfies(1)equilibrium condition: 𝐹(𝑥^∗,𝑥^∗)≥𝐹(𝑥,𝑥^∗), (2)stability condition: if 𝐹(𝑥,𝑥^∗)=𝐹(𝑥^∗,𝑥^∗), then 𝐹(𝑥^∗,𝑥)>𝐹(𝑥,𝑥), where 𝐹(𝑥₁,𝑥₂) is the utility obtained by a player when he/she uses strategy 𝑥₁ and the other player uses strategy 𝑥₂.

Since all peers are selfish and rational, they will cheat if their payoffs can be increased by cheating, which means that all peers are uncertain to the behavior and payoffs of the other peers. In such a case, peers will learn from strategy interaction in each round of game and try different strategies to improve their own utility. Hence, the percentage of peers who use a certain pure strategy may change during this process. Such kind of population evolution process can be modeled by replicator dynamics method, which is the basic dynamic mechanism of evolutionary game.

In a dynamic evolutionary game, an individual from a population, which means a player in the game, who is able to reproduce itself through the process of mutation and selection, is called a replicator. In such case, a replicator with a higher payoff can reproduce itself faster. When the reproduction process takes place over time, this can be modeled by using a set of ordinary differential equations called replicator dynamics equations. Replicator dynamics refers to the growth rate of the peers using certain pure strategy is proportional to the difference between the average payoffs obtained by using the pure strategy and the average payoffs of the peers within the group, which is important for an evolutionary game since it can provide information about the population. Given a particular point in time, it can accurately describe the dynamic relationship between individual behavior payoff and the group system evolution.

For clarity, we assume that the cost of all peers serving as an SP is same firstly, and heterogeneous situation under hybrid P2P streaming systems will be discussed in the next section. In replicator dynamics, it is assumed that a peer chooses pure strategy 𝑖 from a finite set of strategies 𝐵={SP,OP} including acting as an SP or acting as an OP. Let 𝑛_𝑖 denote the number of individuals choosing strategy 𝑖, and let the total population size be 𝑁=∑²_𝑖=1𝑛_𝑖. The proportion of individuals choosing strategy 𝑖 is 𝑥_𝑖=𝑛_𝑖/𝑁, and it is referred to as the population share. The population state can be denoted by the vector 𝑋={𝑥₁,𝑥₂}. The replicator dynamics can be described as follows: 𝑑𝑥_𝑖𝑑𝑡=𝑥_𝑖(𝑡)𝜋_𝑖(𝑡)−𝜋(𝑡), (3) where 𝜋_𝑖(𝑡) is the average payoff obtained by the peers using pure strategy 𝑖, and 𝜋(𝑡) is the average payoff of all peers within the group.

It can be seen from (3) that if using pure strategy 𝑖 can get a higher payoff than the average level, the probability of a peer using strategy 𝑖 will grow, and the growth rate is in proportion to the difference between the average payoff of using pure strategy 𝑖 and the average payoff of all peers.

3.1. Analysis of the SPS Evolutionary Game

According to (1) and (2), the average payoff of a peer if he/she chooses to be an SP can be computed by 𝜋_SP(𝑥)=^𝑁−1_𝑖=0𝑖𝑥𝑁−1_𝑖(1−𝑥)^{𝑁−1−𝑖}×Pr𝑑_𝑖+1≥𝑟𝐺−𝐶_𝑖, (4) where 𝑥 is the probability of a peer being an SP, and _𝑖^𝑁−1𝑥^𝑖(1−𝑥)^{𝑁−1−𝑖} is the probability that there are 𝑖 SPs out of other 𝑁−1 peers. Similarly, the average payoff of a peer if he/she chooses to be an OP is given by 𝜋_OP(𝑥)=^𝑁−1_𝑖=1𝑖𝑥𝑁−1_𝑖(1−𝑥)^{𝑁−1−𝑖}Pr𝑑_𝑖≥𝑟𝐺. (5)

According to (4) and (5), the average payoff of a peer is 𝜋(𝑥)=𝑥𝜋_SP(𝑥)+(1−𝑥)𝜋_OP(𝑥). (6)

Substituting (6) back to replicator dynamics differential equation (3), for 𝑖=SP, we have 𝑑𝑥(𝑑𝑡=𝑥1−𝑥)𝜋_SP(𝑥)−𝜋_OP(𝑥). (7)

At equilibrium point 𝑥^∗, no player will deviate from the optimal strategy, which means 𝑑𝑥/𝑑𝑡|_𝑥∗=0, and we can get 𝑥^∗=0, 𝑥^∗=1, or 𝑥^∗ is the solutions to 𝜋_SP(𝑥)−𝜋_OP(𝑥)=0. However, since 𝑑𝑥/𝑑𝑡|_𝑥∗=0 is only the necessary condition for 𝑥^∗ to be ESS, we examine the sufficient condition for each ESS candidate and draw the following conclusions with the proofs shown in Theorems 3–5:(1)𝑥^∗=0 is an ESS only when Pr(𝑑₁≥𝑟)𝐺−𝐶≤0;(2)𝑥^∗=1 is an ESS only when Pr(𝑑_𝑁≥𝑟)𝐺−Pr(𝑑_𝑁−1≥𝑟)𝐺≥𝐶; (3)let 𝑥^∗ be the solution to 𝜋_SP(𝑥)=𝜋_OP(𝑥), and 𝑥^∗∈(0,1); then 𝑥^∗ is an ESS.

Lemma 2. Let 𝑔(𝑥)=𝜋_𝑆𝑃(𝑥)−𝜋_𝑂𝑃(𝑥); then 𝑔^′(𝑥)<0, for all 𝑥∈[0,1].

Proof. According to (4) and (5), we have 𝑔(𝑥)=^𝑁−1_𝑖=0𝑖𝑥𝑁−1_𝑖(1−𝑥)^{𝑁−1−𝑖}𝑣_𝑖−𝐶, (8) where 𝑣_𝑖=[Pr(𝑑_𝑖+1≥𝑟)−Pr(𝑑_𝑖≥𝑟)]𝐺 is the additional gain by introducing one more SP into the i-SPs P2P streaming system.
For any 𝑥∈[0,1], by taking the derivative of 𝑔(𝑥) over 𝑥, we have 𝑔^′(𝑥)=^𝑁−1_𝑖=0𝑖𝑥𝑁−1_𝑖(1−𝑥)^{𝑁−1−𝑖}𝑣_𝑖=−𝐶_{𝑁−1𝑖=0}𝑖𝑁−1𝑖𝑥^𝑖−1(1−𝑥)^{𝑁−1−𝑖}−𝑥^𝑖(𝑁−1−𝑖)(1−𝑥)^{𝑁−2−𝑖}𝑣_𝑖=^𝑁−1_𝑖=0𝑖𝑥𝑁−1_𝑖−1(1−𝑥)^{𝑁−2−𝑖}×(𝑖(1−𝑥)−𝑥(𝑁−1−𝑖))𝑣_𝑖=^𝑁−1_𝑖=0𝑖𝑥𝑁−1_𝑖−1(1−𝑥)^{𝑁−2−𝑖}[𝑖−𝑥(𝑁−1)]𝑣_𝑖. (9)
By introducing an integer 𝑖₁ that satisfies 𝑖₁≤(𝑁−1)𝑥<𝑖₁+1, we can get that 𝑔^′(𝑥)=^𝑖1_𝑖=0𝑖𝑥𝑁−1_𝑖−1(1−𝑥)^{𝑁−2−𝑖}[𝑖−𝑥(𝑁−1)]𝑣_𝑖+^𝑁−1_𝑖=𝑖1+1𝑖𝑥𝑁−1_𝑖−1(1−𝑥)^{𝑁−2−𝑖}[𝑖−𝑥(𝑁−1)]𝑣_𝑖. (10)
Since 𝑣_𝑖 is a decreasing function in terms of 𝑖, which means that 𝑣_𝑖≥𝑣_𝑖1 when for all 𝑖≤𝑖₁, and 𝑣_𝑖<𝑣_𝑖1 when for all 𝑖>𝑖₁. Therefore, according to (10), we have 𝑔^′(𝑥)<^𝑖1_𝑖=0𝑖𝑥𝑁−1_𝑖−1(1−𝑥)^{𝑁−2−𝑖}[𝑖−𝑥(𝑁−1)]𝑣_𝑖1+^𝑁−1_𝑖=𝑖1+1𝑖𝑥𝑁−1_𝑖−1(1−𝑥)^{𝑁−2−𝑖}[𝑖−𝑥(𝑁−1)]𝑣_𝑖1=𝑣_𝑖1^𝑁−1_𝑖=0𝑖𝑥𝑁−1_𝑖−1(1−𝑥)^{𝑁−2−𝑖}[𝑖−𝑥(𝑁−1)]=𝑣_𝑖1𝑑∑^𝑁−1_𝑖=0𝑖^𝑁−1𝑥^𝑖(1−𝑥)^{𝑁−1−𝑖}𝑑𝑥=0. (11) And therefore, 𝑔^′(𝑥)<0, for all 𝑥∈[0,1].

Theorem 3. The condition for 𝑥^∗=0 to be an ESS is 𝑃𝑟(𝑑₁≥𝑟)𝐺−𝐶≤0.

Proof. According to (4)–(6), the average payoff can be written as 𝜋(𝑥,0)=𝜋_OP(0)+𝑥𝜋_SP(0)−𝜋_OP(0), (12) where 𝜋_SP(0)=Pr(𝑑₁≥𝑟)𝐺−𝐶, 𝜋_OP(0)=0.(1)If Pr(𝑑₁≥𝑟)𝐺−𝐶>0, that is, 𝜋_SP(0)>𝜋_OP(0), every peer will deviate to 𝑥=1 to get 𝜋_SP(0) rather than 𝜋_OP(0). (2)If Pr(𝑑₁≥𝑟)𝐺−𝐶<0, that is, 𝜋_SP(0)<𝜋_OP(0), every peer will select to stay at 𝑥=0 to obtain 𝜋_OP(0) rather than 𝜋_SP(0). (3)If Pr(𝑑₁≥𝑟)𝐺−𝐶=0, that is, 𝜋_SP(0)=𝜋_OP(0), then 𝑔(0)=𝜋_SP(0)−𝜋_OP(0)=0. According to Lemma 2, we know that 𝑔^′(𝑥)<0 is true for any 𝑥∈[0,1]; so 𝑔(𝑥)=𝜋_SP(𝑥)−𝜋_OP(𝑥)<𝑔(0)=0 holds for any 𝑥∈[0,1]. In such a case, we have 𝜋(0,𝑥)=𝜋_OP(𝑥)>𝜋(𝑥,𝑥)=𝜋_OP(𝑥)+𝑥(𝜋_SP(𝑥)−𝜋_OP(𝑥)), which means 𝑥^∗=0 is an ESS according to Definition 1.
Therefore, 𝑥^∗=0 is an ESS only when Pr(𝑑₁≥𝑟)𝐺−𝐶≤0.

Theorem 4. The condition for 𝑥^∗=1 to be an ESS is 𝑃𝑟(𝑑_𝑁≥𝑟)𝐺−𝑃𝑟(𝑑_𝑁−1≥𝑟)𝐺≥𝐶.

Proof. According to (4)–(6), the average payoff that a peer uses mixed strategy 𝑥 and the other peer use mixed strategy 𝑥^∗=0 can be written as 𝜋(𝑥,1)=𝜋_OP(1)+𝑥𝜋_SP(1)−𝜋_OP(1), (13) where 𝜋_SP(1)=Pr(𝑑_𝑁≥𝑟)𝐺−𝐶, 𝜋_OP(1)=Pr(𝑑_𝑁−1≥𝑟)𝐺. (1)If Pr(𝑑_𝑁≥𝑟)𝐺−Pr(𝑑_𝑁−1≥𝑟)𝐺<𝐶, that is, 𝜋_OP(1)>𝜋_SP(1), every peer will deviate to 𝑥=0 to get 𝜋_OP(1) rather than 𝜋_SP(1). (2)If Pr(𝑑_𝑁≥𝑟)𝐺−Pr(𝑑_𝑁−1≥𝑟)𝐺>𝐶, that is, 𝜋_OP(1)<𝜋_SP(1), every peer will select to stay at 𝑥=1 to obtain 𝜋_SP(1) rather than 𝜋_OP(1). (3)If Pr(𝑑_𝑁≥𝑟)𝐺−Pr(𝑑_𝑁−1≥𝑟)𝐺=𝐶, that is, 𝜋_OP(1)=𝜋_SP(1), then 𝑔(1)=𝜋_SP(1)−𝜋_OP(1)=0. According to Lemma 2, we know that 𝑔^′(𝑥)<0, for all 𝑥∈[0,1]. In such a case, we have 𝜋(1,𝑥)=𝜋_OP(𝑥)+1×(𝜋_SP(𝑥)−𝜋_OP(𝑥))>𝜋(𝑥,𝑥)=𝜋_OP(𝑥)+𝑥(𝜋_SP(𝑥)−𝜋_OP(𝑥)), which means 𝑥^∗=1 is an ESS according to Definition 1.
Therefore, 𝑥^∗=1 is an ESS only when Pr(𝑑_𝑁≥𝑟)𝐺−Pr(𝑑_𝑁−1≥𝑟)𝐺≥𝐶.

Theorem 5. If 𝑥^∗∈(0,1) is a solution to equation 𝜋_𝑆𝑃(𝑥)=𝜋_𝑂𝑃(𝑥), then 𝑥^∗ is an ESS.

Proof. Let 𝜋_𝑖(𝑥,𝑥^∗) be the average payoff of player 𝑖 that uses mixed strategy 𝑥 and the other peers use mixed strategy 𝑥^∗. Then we have 𝜋_𝑖𝑥,𝑥^∗=𝑥𝜋_SP𝑥^∗+(1−𝑥)𝜋_OP𝑥^∗. (14)
Since 𝑥^∗ is a solution to 𝜋_SP(𝑥)=𝜋_OP(𝑥), we have 𝜋_SP(𝑥^∗)=𝜋_OP(𝑥^∗), and (14) becomes 𝜋_𝑖𝑥,𝑥^∗=𝜋_SP𝑥^∗=𝜋_𝑖𝑥^∗,𝑥^∗, (15) which means that 𝑥^∗ satisfies the equilibrium condition defined in Definition 1.
Moreover, according to (6), we have 𝜋_𝑖(𝑥,𝑥)=𝜋_OP(𝑥)+𝑥(𝜋_SP(𝑥)−𝜋_OP(𝑥)) and 𝜋_𝑖(𝑥^∗,𝑥)=𝜋_OP(𝑥)+𝑥^∗(𝜋_SP(𝑥)−𝜋_OP(𝑥)). Therefore, we have 𝜋_𝑖(𝑥^∗,𝑥)−𝜋_𝑖(𝑥,𝑥)=(𝑥^∗−𝑥)(𝜋_SP(𝑥)−𝜋_OP(𝑥)).
From Lemma 2, we know that 𝑔(𝑥)=𝜋_SP(𝑥)−𝜋_OP(𝑥) is monotonically decreasing. Since 𝜋_SP(𝑥^∗)=𝜋_OP(𝑥^∗), so when 𝑥<𝑥^∗, we have 𝜋_SP(𝑥)−𝜋_OP(𝑥)>0; and when 𝑥>𝑥^∗, we have 𝜋_SP(𝑥)−𝜋_OP(𝑥)<0. Therefore, for for all 𝑥≠𝑥^∗, we have (𝑥^∗−𝑥)(𝜋_SP(𝑥)−𝜋_OP(𝑥))>0, that is, 𝜋_𝑖(𝑥^∗,𝑥)>𝜋_𝑖(𝑥,𝑥), for all 𝑥≠𝑥^∗, which means 𝑥^∗ satisfies the stability condition defined in Definition 1. So we know that 𝑥^∗ is an ESS.

3.2. SPS Model for Hybrid Heterogeneous P2P Streaming System

In this section, we will further discuss how to choose super peers in heterogeneous peer groups for hybrid P2P streaming systems, where peer nodes may be different in performance when they choose to be a super group peer.

Let 𝑥_{𝑖,𝑏𝑖} be the probability of peer 𝑖 use pure strategy 𝑏_𝑖∈𝐵. According to the replicator dynamics equation, which refers to the growth rate of the proportion of the peers using certain pure strategy is proportional to the difference between the payoffs obtained by using the pure strategy and the average payoffs of the peers within the group; the evolutionary dynamics of 𝑥_{𝑖,𝑏𝑖} is given by the differential equation as follows: 𝑑𝑥_{𝑖,𝑏𝑖}𝑑𝑡=𝑥_{𝑖,𝑏𝑖}𝜋_𝑖𝑏_𝑖,𝑥_−𝑖−𝜋_𝑖𝑥_𝑖, (16) where 𝜋_𝑖(𝑏_𝑖,𝑥_−𝑖) is the average payoff of peer 𝑖 using pure strategy 𝑏_𝑖, and 𝜋_𝑖(𝑥_𝑖) is the average payoff of peer 𝑖 using mixed strategy.

To simplify the analysis, we start from the two-player game, and then the results will be extended to general multiplayer game.

Let 𝑥₁, 𝑥₂ be the probability of peer 1 and peer 2 who choose to act as an SP, respectively. Let 𝐵₁=Pr(𝑑₁≥𝑟)𝐺   and   𝐵₂=Pr(𝑑₂≥𝑟)𝐺; then the payoff matrix of peer 1 and peer 2 can be presented in Table 1. And then, the average payoff 𝜋₁(SP,𝑥₂) can be calculated by 𝜋₁SP,𝑥₂=𝑥₂𝐵₂−𝐶₁+1−𝑥₂𝐵₁−𝐶₁, (17) and the average payoff   𝜋₁(𝑥₁) can be calculated by 𝜋₁𝑥₁=𝑥₁𝑥₂𝐵₂−𝐶₁+𝑥₁1−𝑥₂𝐵₁−𝐶₁+𝑥₂1−𝑥₁𝐵₁. (18)

Table 1 

Payoff matrix of the two-player super peer selection evolutionary game.

From (16), the replicator dynamics equation of peer 1 can be calculated by 𝑑𝑥₁𝑑𝑡=𝑥₁1−𝑥₁𝐵₁−𝐶₁−2𝐵₁−𝐵₂𝑥₂. (19)

And the replicator dynamics equation of peer 2 can be calculated by 𝑑𝑥₂𝑑𝑡=𝑥₂1−𝑥₂𝐵₁−𝐶₂−2𝐵₁−𝐵₂𝑥₁. (20)

At the equilibrium point, we know that (𝑑𝑥₁/𝑑𝑡)=0 and (𝑑𝑥₂/𝑑𝑡)=0. According to (19) and (20), we can get five equilibriums. They are (0,0),(0,1),(1,0),(1,1), and the mixed strategy equilibrium ((𝐵₁−𝐶₂)/(2𝐵₁−𝐵₂),(𝐵₁−𝐶₁)/(2𝐵₁−𝐵₂)).

According to [20], in a evolutionary dynamic system, if an equilibrium of the replicator dynamics equations is a locally asymptotically stable point, it is an ESS. Equations (19) and (20) can be regarded as a nonlinear dynamic system and we can examine whether the five equilibriums are ESSs by analyzing the Jacobian matrix of the nonlinear dynamic system. By taking partial derivatives of (19) and (20), we can get the Jacobian matrix as 𝜕𝑥̇𝐽=¹𝜕𝑥₁𝜕𝑥̇₁𝜕𝑥₂𝜕𝑥̇₂𝜕𝑥₁𝜕𝑥̇₂𝜕𝑥₂=𝐽₁₁𝐽₁₂𝐽₂₁𝐽₂₂, (21) where   𝐽₁₁=(1−2𝑥₁)(𝐵₁−𝐶₁−(2𝐵₁−𝐵₂)𝑥₂), 𝐽₁₂=−𝑥₁(1−𝑥₁)(2𝐵₁−𝐵₂), 𝐽₂₁=−𝑥₂(1−𝑥₂)(2𝐵₁−𝐵₂), and 𝐽₂₂=(1−2𝑥₂)(𝐵₁−𝐶₂−(2𝐵₁−𝐵₂)𝑥₁). At the locally asymptotically stable point, we have that det(𝐽)>0, and tr(𝐽)<0 [20]. By substituting the five equilibriums into (21), we can get that (1)if 𝐵₂−𝐵₁−𝐶₁>0 and 𝐵₂−𝐵₁−𝐶₂>0, there is a unique ESS (1,1), which means that both peer 1 and peer 2 will converge to be SPs; (2)else if 𝐵₂−𝐵₁−𝐶₁>0 while 𝐵₂−𝐵₁−𝐶₂<0, there is a unique ESS (1,0), which means that peer 1 will converge to be an SP and peer 2 tends to be an ordinary peer;(3)else if 𝐵₂−𝐵₁−𝐶₁<0 while 𝐵₂−𝐵₁−𝐶₂>0, there is a unique ESS (0,1), which means that peer 2 will converge to be an SP and peer 1 tends to be an ordinary peer;(4)else there are two ESSs (0,1) and (1,0), where the converged strategy profiles depend on the initial strategy profiles.

From the above analysis, we can see that when the gain of being an SP (𝐵₂−𝐵₁) is greater than the cost of being an SP (𝐶₁ or 𝐶₂), the peer tends to be an SP. And the peer with a higher cost tends to be a free rider and rely on the peer with a lower cost.

After analyzing the two-player Super Peer Selection game, we can deduce from it that the peer with a higher cost (𝐶_𝑖) tends to rely on the peer with a lower cost, which can be extended to multiplayer game. If there are multiple peers in the Super Peer selection game, the strategy of the peers whose cost is higher will converge to “OP” with greater probabilities. Conversely, the peers with lower costs tend to be SPs who will suffer relatively severer losses if no one serves as an SP.

4. A Distributed Q-Learning Algorithm for ESS

From the previous section, we can see that the ESS can be found by solving the replicator dynamics equations. However, solving the replicator dynamics equations requires the exchange of strategies adopted by other peers and their private information. In such a case, a Q-learning approach [21] which is a type of reinforcement learning (i.e., learning by interaction) is used. With this ability to learn, complete payoff information of other users in the same group is no longer required for SP selection. In this section, we will present a distributed Q-learning algorithm that can gradually converge to ESS without information exchange.

We first discretize the replicator dynamics equation as 𝑥_𝑖(𝑡+1)=𝑥_𝑖(𝑡)+𝜋_𝑖(𝑡)−𝜋(𝑡)𝑥_𝑖(𝑡), (22) where 𝑡 is the slot index and 𝑥_𝑖(𝑡) is the probability of peer 𝑖 being an SP during slot 𝑡. Here, we assume that each slot can be further divided into 𝑀 subslots and each peer can choose to be an SP or an OP at the beginning of each subslot.

From (22), we can see that in order to update 𝑥_𝑖(𝑡+1), we need to first compute 𝜋_𝑖(𝐵,𝑥_−𝑖(𝑡)) and 𝜋_𝑖(𝑥_𝑖(𝑡)). Let us define an indicator function 𝑜_𝑖(𝑡,𝑘) whose values equal to 1 if player 𝑖 chooses to be an SP at the beginning of 𝑘th subslot in time slot 𝑡. Otherwise, the function value is 0.

The directly payoff of player 𝑖 at subslot 𝑘 in slot 𝑡 can be computed by 𝜋_𝑖()=𝑡,𝑘𝐺−𝐶_𝑖,ifpeer𝑖choosestobeanSP,𝑟^𝑡≥𝑟,−𝐶_𝑖,ifpeer𝑖choosestobeanSP,𝑟^𝑡<𝑟,𝐺,ifpeer𝑖choosestobeanOP,𝑟^𝑡≥𝑟,0,ifpeer𝑖choosestobeanOP,𝑟^𝑡<𝑟, (23) where 𝑟^𝑡 is the total download rate of the SPs and 𝑟 is the video source rate.

Then, 𝜋_𝑖(𝐵,𝑥_−𝑖(𝑡)) can be approximated as 𝜋_𝑖𝐵,𝑥_−𝑖(𝑡)=∑^𝑀_𝑘=1𝜋_𝑖(𝑡,𝑘)𝑜_𝑖(𝑡,𝑘)∑^𝑀_𝑘=1𝑜_𝑖(𝑡,𝑘). (24)

Similarly, 𝜋_𝑖(𝑥_𝑖(𝑡)) can be approximated as 𝜋_𝑖𝑥_𝑖(𝑡)=1𝑀^𝑀_𝑘=1𝜋_𝑖(𝑡,𝑘). (25)

In the distributed Q-learning algorithm, 𝑄-value (i.e.,  𝑄(𝑡)) is used to maintain the knowledge about each peer, and the decision can be made based on this knowledge. The Super Peer Selection algorithm is described in Algorithm 1.

Step 1: Given the slot index 𝑡=0, each peer initializes 𝑥𝑖(𝑡)=𝑥𝑖(0) with a random between [0,1]. Step 2: Initialize 𝑄-value associated with 𝑄(𝑡)=0 for all peers in super groups. Step 3: loop       During slot 𝑡, for k = [1 : M]       if  𝑥𝑖(𝑡)≤𝛾  then      peer 𝑖 randomly choose to be a SP to serve as a super peer to download streaming chunks     from the peers outside the group with dowpnload rate 𝑟𝑖(𝑡,𝑘) or to be an OP to download     streaming chunks from the SP.      else     Choose strategy      𝑏∗=argmax𝑏𝑄(𝑘)      end if     peer 𝑖 computes the indicator function 𝑜𝑖(𝑡,𝑘) and his/her directly payoff  𝜋𝑖(𝑡,𝑘) using (23). Step 4: peer 𝑖 computes average payoff  𝜋𝑖𝐵,𝑥−𝑖(𝑡) and 𝜋𝑖𝑥𝑖(𝑡). Step 5: Updates the probability of being a SP 𝑥𝑖(𝑡+1) using (22). Step 6: Update 𝑄(𝑘+1)=(1−𝜆)𝑄(𝑘)+𝜆𝜋𝑖(𝑥𝑖(𝑡))+𝛽max𝑏𝑄(𝑘) Step 7: endloop for all peers in super group.

Algorithm 1 

A distributed 𝑄-learning. Super Peer Selection algorithm for ESS (ESS-SPS).

In Algorithm 1, a peer performs the exploration step with probability 𝛾, and 𝜆 denotes the learning rate that is used to control the speed of adjustment of the 𝑄-value. A new 𝑄-value 𝑄(𝑘+1), which is the expected payoff for the future iterations, is obtained based on the previous value 𝑄(𝑘) along with the new observed payoff 𝜋_𝑖(𝑥_𝑖(𝑡)). Here, the new observed payoff is biased by the outcome of choosing the best action based on the available knowledge (i.e., max_𝑏𝑄(𝑘)).

5. Experimental Results

For evaluation purposes, we consider the traditional randomly super peers selection method, which is denoted as Random SPS for performance comparison. In Random SPS, each peer acts as an individual and randomly selects some peers from its partner list for downloading video chunks. Such a protocol has been widely used in the existing P2P systems, such as PPLive [22] and Cool Streaming [23]. For convenience, in the rest of this paper, we denote the proposed ESS-based approach as ESS-SPS.

In this section, we describe the experiment results which are conducted on the LStream P2P live streaming platform deployed on the Henan Education and Research NETwork (HERNET). In our experiments, there are about 2000 users in the hybrid heterogeneous P2P live streaming system. The video is initially stored at an original video server with upload bandwidth 𝑟=3Mbps. The request round is 1 second and the relay buffer length is 30 seconds and the streaming fragment size is 𝐿=1KB.

In the first experiment, assume that there are 20 heterogeneous peers in one super group, their gain is 𝐺=1, and their cost 𝐶_𝑖 is a random between [0,0.3]. The first experiment compares the differences between ESS-SPS method and traditional Random-SPS method in the aspect of the obtained social welfare, which is defined by the sum of the utility obtained by all peers. As it is shown in Figure 2, ESS-SPS method obtains a stable Nash equilibrium and chooses suitable numbers of SPs through the collaboration among the peers in the same super group. It maintains a relatively higher social welfare level under the condition of relatively higher video source rate 𝑟. But the social welfare level of traditional Random-SPS method decreases linearly and rapidly with the source rate increasing.

Figure 2 

The social welfare comparison between ESS SPS and Random SPS.

In the second experiment, we evaluate the convergence property of the ESS-SPS. In Figure 3, we show that the replicator dynamic of the SPS game model with heterogeneous peers. We can see that, starting from high initial values, all peers gradually reduce their probabilities of being an SP since being an OP more often can bring a higher payoff. However, because too low probability of being an SP may increase the chance of having no peer be an SP, the probability of being an SP will finally converge to a certain value which is determined by the number of peers in the group.

Figure 3 

Behavior dynamics of peers in a heterogeneous super group.

The third experiment compares the upload capacity of the P2P streaming network using ESS-SPS algorithm and traditional Random-SPS method. As it is shown in Figure 4, when the nodes number up to about 2100, the load of the super peers close to saturation, and when the system scale continues to increase, the increase in the upload capacity of the P2P streaming system is not obvious. On the contrary, ESS-SPS algorithm eliminates the system bottlenecks and the upload capacity of the P2P streaming system increased steadily with the number of peers increasing.

Figure 4 

The upload capacity of P2P streaming system comparison between ESS-SPS and Random SPS.

The fourth experiment compares the average upload rates of super peers using ESS-SPS algorithm and traditional Random-SPS method, respectively. Average upload rates of super peers is the ratio of the total sum of upload rate of all the super peers to the number of them, which reflects the upload capability for the P2P streaming system provided by the super peers. As it is shown in Figure 5, the SPs generated by ESS-SPS provide higher upload capacity for the P2P streaming system and the average upload rates of super peers increase steadily with system uptime increasing.

Figure 5 

The average upload rates of SPs comparison between ESS-SPS and Random SPS.

6. Conclusion

In this paper, we propose a super peer selection game model based on evolutionary game, which is used in hybrid heterogeneous P2P streaming system to address the network inefficiency problem encountered by the traditional randomly super peer selection scheme. By deriving the ESS for every peer, we further propose a distributed Q-learning algorithm for each peer to converge to the ESS by learning from his/her own past payoff history. The experiment results indicate that, compared with the traditional Random-SPS scheme, the proposed method achieves much better social welfare and enables the upload capacity of the whole P2P streaming system to increase stably with the peer number increasing.

Acknowledgments

This research was supported by the National High Technology Research and Development Program of China 2009AA012201. The authors are grateful to the anonymous reviewers for a careful checking of the details and for the helpful comments that improved this paper.