#### Abstract

A new model which allows both the effect of partial selfing selection and an exponential function of the expected payoff is considered. This combines ideas from genetics and evolutionary game theory. The aim of this work is to study the effects of partial selfing selection on the discrete dynamics of population evolution. It is shown that the system undergoes period doubling bifurcation, saddle-node bifurcation, and Neimark-Sacker bifurcation by using center manifold theorem and bifurcation theory. Numerical simulations are presented not only to illustrate our results with the theoretical analysis, but also to exhibit the complex dynamical behaviors, such as the period-3, 6 orbits, cascade of period-doubling bifurcation in period-2, 4, 8, and the chaotic sets. These results reveal richer dynamics of the discrete model compared with the model in Tao et al., 1999. The analysis and results in this paper are interesting in mathematics and biology.

#### 1. Introduction

Evolutionary game theory was extended to the genetic model. The notion of an evolutionary stable strategy (ESS) which was proposed by Smith [1] (1982) is important. And the basic static solution concept of an ESS has been quite successful in predicting the equilibrium behavior of individuals in a population. A series of papers were devoted to the genetic matrix game models. But previous studies on the matrix game models all assumed that genetic mating was random. For example, Lessard [2] (1984) analyzed a frequency-dependent two-phenotype selection model of single-locus with multiallele. Cressman [3] (1996) studied a two-species evolutionary dynamics. Cressman et al. [4] (2003) discussed evolutionarily stable sets in the genetic model of natural selection. Tao et al. [5] (1999) investigated the discrete frequency dynamics of two phenotype diploid models. In fact, in genetic mating, there exists partial selfing selection (Rocheleau and Lessard [6]). From [6], each individual can reproduce by selfing or random outcrossing with constant probabilities, respectively, in a population. After considering partial selfing selection, we establish a nonlinear frequency dynamics, which becomes more realistic but more complicated.

We consider a one-locus two-allele model where genotypic fitness is an exponential function of the expected payoff and the frequency. Suppose that each individual of the population can reproduce by selfing with constant probability . If , then there is no partial selfing selection, which was discussed in [5]. If , then one-dimensional discrete-time dynamical systems are extended to two-dimensional systems, which lead to great effect on the genetic model. It was shown in [7] that a stable polymorphic equilibrium is not an ESS under some condition. In this paper, we focus on the bifurcation analysis of the genetic model. We find that the model admits the unstable period doubling bifurcation, the saddle-node bifurcation, and the Neimark-Sacker bifurcation, which are new interesting phenomena when considering partial selfing selection.

#### 2. Model

Consider a diploid population with nonoverlapping generations and establish a genetic model with one locus and two alleles and . The fitness of population individual is frequency-dependent (i.e., it depends on the genetic makeup of the population concerned). To show the results, the following assumptions will be made: (i)Mendelian segregation; (ii)sex ratio is independent of genotype; (iii)no gametic selection; (iv)the fecundity of offspring is equal; (v)no mutation or migration.

According to [6], each individual can reproduce by selfing or random outcrossing with probability or , respectively, in the population. and are the frequencies of genotypes in the current generation and in the next generation, respectively. The genotypic frequency of genotype among adults in the current generation is . The corresponding selection values of genotypes are . and are the frequencies of allele in the current generation and in the next generation, respectively. and are the frequencies of allele in the current generation and in the next generation, respectively. and are genotypic frequencies of allele and allele among adults in the current generation, respectively. (or ) is the fraction of individuals of genotype which play pure strategy (or ). After selection but before mating, we have After mating and reproduction, there are

In our paper, the fitness is taken to be an exponential function of the expected payoff as [6]. Assume that an individual in the population can use one of two strategies and ; each individual plays the same pure strategy throughout its lifetime. And the payoff matrix is Let be the proportion of individuals using strategy ; then, we can get that the expected payoffs to and are Furthermore, we take the fitness function Since system (2) is on the simplex , we can get the following system: where From (1) and (2), we obtain Obviously, it is difficult to use and to denote the genotypic frequency directly. If the population is polymorphic (i.e., ), we can use the fixation index (wright 1949 [8]) according to [6]. The genotypic frequencies can be written in the form where . The value of varies from one generation to the next.

Equations (1) to (9) can deduce thatwhere

Throughout the paper, we discard the degenerate situations where all are identical or where and suppose that is not the genotype in the parental generation.

#### 3. Model Analysis and Basic Definitions

Recall Definition 1 in [7], which is a similar definition of phenotypic and genotypic equilibria under partial selfing selection according to [2]. The two types of the equilibria include all situations in which the population is in equilibrium.

*Definition 1. *A phenotypic equilibrium is an equilibrium where all pure strategies in current use have equal expected payoff. A genotypic equilibrium is a nonphenotypic equilibrium where the effective fitnesses of all alleles present in the current population are equal. The effective fitness of allele is defined to be . (In addition, in the case , the effective fitness of allele is . In the case , the effective fitness of allele is .)

According to Definition 1, a phenotypic equilibrium is an equilibrium where all pure strategies in current use have equal expected payoff. So, a polymorphic phenotypic equilibrium exists if and only if . Then, we have where , and .

Let denote polymorphic phenotypic equilibrium; then, we have the Jacobian matrix at of system (10) where

We studied the system (10) and obtained Theorem 3.1 (i) and (ii) in [7]. In [7], the authors showed that a stable polymorphic equilibrium is not an ESS under some condition. In this paper, we focus on the bifurcation of system (10). For convenience, we present some lemmas and theorems of the stability of polymorphic phenotypic equilibrium in [7] at first.

Lemma 2. * is an interior ESS if and only if . *

Lemma 3. *If , then . If , then . *

Theorem 4. *Suppose that is a polymorphic phenotypic equilibrium.*(i)*Suppose that . If is not an ESS, then is unstable. If is an ESS, then is stable for . *(ii)*Suppose that . If is an ESS, then is unstable. If is not an ESS, then is stable for .*(iii)*Suppose that . If , then is unstable for . If , then is stable for , unstable for . *

* Proof. *Consider only the case since the proof of the case is analogous. From the Jacobian matrix at , we have
If is not an ESS, then from Lemma 2. The inequality implies that . So, is unstable. If is an ESS, then . By the discussion on the monotone function, we can obtain the following inequalities:
So, is stable.

Now, we discuss the stability of when . If , then . If , we have the following.*Case* *1*. , where and .*Case* *2*. , where and .

We only prove Case 1 since Case 2 is analogous.

Write
and let . Using the translation
system (10) becomes
By center manifold theory, we can obtain the following reduced system which is locally homeomorphic with system (10):
where . If , then . So, we have . If , we can calculate the reduced system by similar analysis,
where . It is easy to obtain the results. This completes the proof.

#### 4. Bifurcation Analysis

Based on the analysis in Section 3, we discuss the period doubling bifurcations, the saddle-node bifurcation, and the Neimark-Sacker bifurcation of the positive fixed point () in this section. We choose parameter as a bifurcation parameter to study the period doubling bifurcations and the Neimark-Sacker bifurcation and parameter as a bifurcation parameter to study the saddle-node bifurcation by using center manifold theorem and bifurcation theory in [9, 10]. Suppose that is the same as that in (14).

##### 4.1. Period Doubling Bifurcation

In the analysis of period doubling bifurcations, we take as the bifurcation parameter and prove that there is period doubling bifurcation at the fixed point for . When , the characteristic polynomial of Jacobian matrix at is We have the following characteristic values of (22): Let And let denote . Then,

Denote the right-hand side of (10) by , and use the translation system (10) becomes where It is easy to obtain that According to Lemma 3, there are So, we have In order for system (27) to undergo period doubling bifurcation, we require that the following is not zero [10]: If , then for , for .

From the previous analysis and the theorem in [10], we have the following result.

Proposition 5. * Suppose that is a polymorphic phenotypic equilibrium. If and , then system (10) undergoes a period doubling bifurcation at for . *

Moreover, we have the following.(i)If and (), then period doubling bifurcation happens for (). And 2-periodic points are saddle points (asymptotically stable nodes).(ii)If and (), then period doubling bifurcation happens for (). And 2-periodic points are unstable nodes (saddle points).

*Example 6. * Let , , , , and . There are two polymorphic phenotypic equilibria and . For , we have and . Then, period doubling bifurcation happens for , and 2-periodic points are asymptotically stable nodes. For , we have and . Then, period doubling bifurcation happens for , and 2-periodic points are unstable nodes.

The effects of partial selfing selection on the dynamics of population evolution are shown further in Figures 1, 2, and 3. More complex dynamical behaviors of the genetic system are exhibited by numerical simulations. In Figure 1, the partial selfing selection leads period doubling bifurcation to emerge earlier, and leads chaos to emerge earlier. In Figure 3, the model exhibits the complex dynamical behaviors, such as the period-3, 6 orbits, cascade of period-doubling bifurcation in period-2, 4, 8, and the chaotic sets. By choosing as a bifurcation parameter, we show that the complex dynamical behaviors such as the period-3, 4, 6 orbits, cascade of period-doubling bifurcation in period-2, 4, 8, and the chaotic sets can occur as crosses some critical values in Figure 2.

**(a)**

**(b)**

**(a)**

**(b)**

**(c)**

**(d)**

##### 4.2. Saddle-Node Bifurcation

We take as the bifurcation parameter for studying the saddle-node bifurcation by using center manifold theorem.

If , then there is unique polymorphic phenotypic equilibrium, and . The Jacobian matrix of is where and .

The characteristic values of are where can make sure that .

Using the translation we have where

By the center manifold theory, we know that the stability of near can be determined by a one-parameter family of equations on a center manifold, which can be represented as follows: for and sufficiently small. Assume that a center manifold has the form The center manifold must satisfy So, we obtain the map restricted to the center manifold where , and .

By condition (41) and the saddle-node bifurcation theorem in [9], we can state the following result.

Proposition 7. * Suppose that is a polymorphic phenotypic equilibrium. If , , , and , then system (10) undergoes a saddle-node bifurcation at for where , , and . *

Moreover, if , then two new polymorphic phenotypic equilibria are created for and disappear for . If , then two new polymorphic phenotypic equilibria are created for and disappear for . (see Figure 4).

In Proposition 7, the conditions and guarantee that . If there is no partial selfing selection (i.e., ), then the saddle-node bifurcation does not occur. This means that the parameter of partial selfing selection leads to more complex dynamical behavior of the genetic system.

##### 4.3. Neimark-Sacker Bifurcation

We take as the bifurcation parameter and prove the existence of Neimark-Sacker bifurcation. The characteristic polynomial of Jacobian matrix at is where If the following condition is satisfied then the eigenvalues of the characteristic equation are complex conjugate. When , (44) is equivalent to If condition (45) is satisfied, we can calculate that

Next, we study the normal form of (10) when .

Use the translation system (10) becomes where ,

In order for system (48) to undergo Neimark-Sacker bifurcation, we require that the following discriminatory quantity is not zero [9]: where From previous analysis and the theorem in [9], we have the following theorem.

Proposition 8. * Suppose that is a polymorphic phenotypic equilibrium of system (10). If and , then system (10) undergoes a Neimark-Sacker bifurcation at for . *

Moreover, if and , then an attracting invariant closed curve bifurcates from for . If and , then a repelling invariant closed curve bifurcates from for .

*Example 9. *Let , , , , and . We have that is a polymorphic phenotypic equilibrium which satisfies . According to , and , we have that the invariant closed curve bifurcates from for being attracting. Let , , , , and . We have that is a polymorphic phenotypic equilibrium which satisfies . According to and , the invariant closed curve bifurcates from for being repelling.

#### 5. Conclusion

A discrete population genetics model with partial selfing selection is investigated. We assume that each individual can reproduce by selfing or random outcrossing with probability or , respectively, in the population. Some population, such as human population, does not mate at random. So, the partial selfing selection model is reasonable.

In this paper, the conditions for the stability of polymorphic phenotypic equilibria are obtained by the Jury conditions and the center manifold theorem. ESS is not the necessary condition of the stability of the polymorphic phenotypic equilibria. The theoretical analysis and numerical simulations present the existence of stable and unstable period doubling bifurcations, saddle-node bifurcation, and Neimark-Sacker bifurcation. Numerical simulations exhibit more complex dynamical behavior of the genetic system under partial selfing selection.

#### Acknowledgments

The author is thankful to the referees for helpful suggestions for the improvement of this paper. The research of the first author was supported by the Scientific Research Fund of Sichuan Provincial Education Department under Grant 12ZB082.