#### Abstract

A quadratic stochastic operator (in short QSO) is usually used to present the time evolution of differing species in biology. Some quadratic stochastic operators have been studied by Lotka and Volterra. The general problem in the nonlinear operator theory is to study the behavior of operators. This problem was not fully finished even for quadratic stochastic operators which are the simplest nonlinear operators. To study this problem, several classes of QSO were investigated. We study -QSO defined on 2D simplex. We first classify -QSO into 20 nonconjugate classes. Further, we investigate the dynamics of three classes of such operators.

#### 1. Introduction

The history of quadratic stochastic operators can be traced back to Bernstein’s work [1]. The quadratic stochastic operator was considered an important source of analysis for the study of dynamical properties and modelings in various fields such as biology [1–7], physics [8, 9], economics, and mathematics [3, 6, 10, 11].

One of such systems which relates to the population genetics is given by a quadratic stochastic operator [1]. A quadratic stochastic operator (in short QSO) is usually used to present the time evolution of species in biology, which arises as follows. Consider a population consisting of species (or traits) . We denote a set of all species (traits) by . Let be a probability distribution of species at an initial state and let be a probability that individuals in the th and th species (traits) interbreed to produce an individual from th species (trait). Then, a probability distribution of the species (traits) in the first generation can be found as a total probability; that is,
This means that the association defines a mapping called *the evolution operator*. The population evolves by starting from an arbitrary state , then passing to the state (the first generation), then to the state (the second generation), and so on. Therefore, the evolution states of the population system are described by the following discrete dynamical system:

In other words, a QSO describes a distribution of the next generation if the distribution of the current generation was given. The fascinating applications of QSO to population genetics were given in [6].

In [12], it was given a long self-contained exposition of the recent achievements and open problems in the theory of the QSO. The main problem in the nonlinear operator theory is to study the behavior of nonlinear operators. This problem was not fully finished even in the class of QSO (the QSO is the simplest nonlinear operator). The difficulty of the problem depends on the given cubic matrix . An asymptotic behavior of the QSO even on the small dimensional simplex is complicated [11, 13–16]. In order to solve this problem, many researchers always introduced a certain class of QSO and studied their behavior, for example, Volterra-QSO [11, 17–20], permutated Volterra-QSO [21, 22], Quasi-Volterra-QSO [23], -Volterra-QSO [24, 25], non-Volterra-QSO [13, 15], strictly non-Volterra-QSO [26], F-QSO [27], and non-Volterra operators generated by product measure [28–30]. However, all these classes together would not cover a set of all QSO. Therefore, there are many classes of QSO which were not studied yet. Recently, in the papers [31, 32], a new class of QSO was introduced. This class was called a -QSO. In this paper, we are going to continue the study of -QSO. This class of operators depends on a partition of the coupled index set (the coupled trait set) . In case of two-dimensional simplex , the coupled index set (the coupled trait set) has five possible partitions. The dynamics of -QSO corresponding to the point partition (the maximal partition) of have been investigated in [31, 32]. In the present paper, we are going to describe and classify such operators generated by other three partitions. Further, we also investigate the dynamics of three classes of such operators.

The paper is organized as follows. In Section 2, we give some preliminary definitions. In Section 3, we discuss the classification of -QSO related to . It turns out that some obtained operators are -Volterra-QSO (see [24, 25]) and permuted -Volterra-QSO. The dynamics of -Volterra-QSO are not fully studied yet. In [24, 25], some particular cases have been investigated, which do not cover our operators. Therefore, in further sections, we study dynamics of -Volterra-QSO and permuted -Volterra-QSO. In Section 4, we study the behavior of -Volterra-QSO taken from class . In Section 5, we study the behavior of a permuted -Volterra-QSO taken from class . Note that is a permutation of . In Section 6, we study the behavior of a permuted Volterra-QSO taken from class . In the last section, we just highlight the dynamics of Volterra-QSO taken from class which was already studied in [17–19].

#### 2. Preliminaries

Recall that a quadratic stochastic operator (QSO) is a mapping of the simplex: into itself, of the form: where and is a coefficient of heredity, which satisfies the following conditions: Thus, each quadratic stochastic operator can be uniquely defined by a cubic matrix with conditions (5).

We denote sets of fixed points and -periodic points of by and , respectively. Due to Brouwer’s fixed point theorem, one always has that for any QSO . For a given point , a trajectory of starting from is defined by . By , we denote a set of omega limiting points of the trajectory . Since and is compact, one has that . Obviously, if consists of a single point, then the trajectory converges and a limiting point is a fixed point of .

Recall that a Volterra-QSO is defined by (4) and (5) and the additional assumption:

The biological treatment of condition (6) is clear: *the offspring repeats the genotype (trait) of one of its parents*.

One can see that a Volterra-QSO has the following form: where Moreover,

This kind of operators is intensively studied in [11, 17–20, 33]. Note that this operator is a discretization of the Lotka-Volterra model [5, 7] which models an interacting competing species in the population system. Such a model has received considerable attention in the fields of biology, ecology, and mathematics (see, e.g., [2, 3, 7, 8]).

In [24], a notion of -Volterra-QSO, which generalizes a notion of Volterra-QSO, has been introduced. Let us recall it here.

In order to introduce a new class of QSO, we need some auxiliary notations.

We fix and assume that elements of the matrix satisfy

For any fixed , the QSO defined by (4), (5), and (10) is called *-Volterra-QSO*.

*Remark 1. *Here, we stress the following points. (1)Note that an -Volterra-QSO is a Volterra-QSO if and only if .(2)It is known [17] that there is not a periodic trajectory for Volterra-QSO. However, there are such trajectories for -Volterra-QSO [24].

By following [25], take ; then, for and By using and denoting , , , one then gets

-Volterra-QSO.

Note that

We call that an operator is * permuted **-Volterra-QSO* if there is a permutation of the set and an -Volterra-QSO such that for any . In other words, can be represented as follows:

We remark that if , then a permuted -Volterra-QSO becomes a permuted Volterra-QSO. Some properties of such operators were studied in [19, 34]. The dynamics of certain class of permuted Volterra-QSO have been investigated in [32]. Note that, in [24, 25], a class of -Volterra-QSO has been studied. An asymptotic behavior of permuted -Volterra-QSO has not been investigated yet. Some particular cases have been considered in [31, 32].

In this paper, we are going to introduce a new class of QSO which contain -Volterra-QSO and permuted -Volterra-QSO as a particular case.

Note that each element is a probability distribution of the set . Let and be vectors taken from . We say that * is equivalent to * if . We denote this relation by .

Let be a support of . We say that * is singular to * and denote by if . Note that if , then if and only if ; here, stands for a standard inner product in .

We denote sets of coupled indexes by For a given pair , we set a vector . It is clear due to the condition (5) that .

Let and be some fixed partitions of and , respectively; that is, , , and , , where .

*Definition 2. *A quadratic stochastic operator given by (4) and (5) is called a -QSO with respect to the partitions (where the letter “” stands for absolutely continuous-singular) if the following conditions are satisfied: (i)for each and any , , one has that ;(ii)for any , and any and , one has that ;(iii)for each and any , , one has that ;(iv)for any , and any , and one has that .

*Remark 3. *If is the point partition, that is, , then we call the corresponding QSO by -QSO (where the letter “” stands for singularity) since in this case every two different vectors and are singular. If is the trivial, that is, , then we call the corresponding QSO by -QSO (where the letter “” stands for absolute continuous) since in this case every two vectors and are equivalent. We note that some classes of -QSO have been studied in [35]. In the present paper, we restrict ourselves to the -case. Note that, in general, the class of -QSO will be studied elsewhere in the future.

*Remark 4. *For the -QSO, that is, in the case , the condition (iii) of Definition 2 is trivial and the condition (iv) means that there is a permutation of the set such that for any where , , are vertices of the simplex .

*A Biological Interpretation of a **-QSO*. We treat as a set of all possible traits of the population system. A coefficient is a probability that parents in the th and th traits interbreed to produce a child from the th trait. The condition means that the gender of parents do not influence having a child from the th trait. In this sense, is a set of all possible coupled traits of parents. A vector is a possible distribution of children in a family while parents are carrying traits from the th and th types. A biological meaning of a -QSO is as follows: a set of all differently coupled traits of parents is splitted into groups (here is less than the number of traits) such that the chance (probability) of having a child from any trait in two different families whose parents’ coupled traits belong to the same group is simultaneously either positive or zero (the condition (i) of Definition 2); meanwhile, two families whose parents’ coupled traits belong to two different groups and cannot have a child from the same trait, simultaneously (condition (ii) of Definition 2). Moreover, the parents who are sharing the same type of traits can have a child from only one type of traits (condition (iv) of Definition 2 and Remark 4).

#### 3. Classification of -QSO on 2D Simplex

In this section, we are going to study -QSO in two-dimensional simplex; that is, . In this case, we have the following possible partitions of :

We note that, in [31, 32], -QSO related to the partition which is the maximal partition of has been investigated. In this paper, we are aiming to study -QSO related to the partitions , , and . We shall show that these three classes of -QSO are conjugate to each other. Therefore, it is enough to study -QSO related to the partition . A class of -QSO related to the partition will be studied in elsewhere in the future.

Let us recall that two operators and are called * (topologically or linearly) conjugate* if there is a permutation matrix such that . Let be a permutation of the set . For any vector , we define . It is easy to check that if is a permutation of the set corresponding to the given permutation matrix , then one has that . Therefore, two operators and are conjugate if and only if for some permutation . Throughout this paper, we shall consider “conjugate operators” in this sense. We say that two classes and of operators are conjugate if every operator taken from is conjugate to some operator taken from and vice versa.

Proposition 5. * A class of all -QSO corresponding to partition (or ) is conjugate to a class of all -QSO corresponding to partition .*

*Proof. *We show that two classes of all -QSO corresponding to partitions and are conjugate to each other. Analogously, one can show that two classes of all -QSO corresponding to partitions and are conjugate to each other as well.

Assume that an operator given by
is a -QSO corresponding to partition . This means that the coefficients of satisfy the following three conditions: (i) , (ii) , and (iii) where .

We consider the following operator: , where . It is clear that is conjugate to , where
such that for any ; equivalently, (in a vector form) for any . Now, we are going to show that is a -QSO corresponding to . In order to show it we have to check three conditions.(i). Indeed, since , , one has .(ii). Indeed, since , , and , we obtain that . In the same manner, we can get that .(iii). Indeed, since , , , and , we have that .

This shows that any -QSO taken from the class corresponding to partition is conjugate to some -QSO taken from the class corresponding to partition . Analogously, we can show that any -QSO taken from the class corresponding to partition is conjugate to a -QSO which belongs to the class corresponding to partition , where is the same permutation as given above. This completes the proof.

Therefore, it is enough to study a class of all -QSO corresponding to the partition . Now, we shall consider some subclass of a class of all -QSO corresponding to partition by choosing coefficients in special forms where (see Table 1).

The choices of the cases , where , will give 36 operators from the class of -QSO corresponding to partition . Finally, we obtain 36 parametric operators which are defined as follows:

Theorem 6. *All 36 operators from the class of -QSO corresponding to partition defined as above are classified into 20 nonconjugate classes:
*

*Proof. *It is easy to check that partition is invariant under only one permutation . The proof of the theorem can be easily insured with respect to this permutation. It is straightforward.

The main problem is to investigate the dynamics of these classes of operators. In what follows, we are going to study three classes , , and . From the list, one can conclude that these three classes of operators are either -Volterra-QSO or permuted -Volterra-QSO. The class was already studied in [17, 18]. The rest of the classes of operators would be studied in elsewhere in the future.

#### 4. Dynamics of -QSO from Class

In this section we are going to study dynamics of -QSO from class .

We need some auxiliary facts about properties of the function given by where . If , then the function becomes the identity mapping. Therefore, we shall consider only the case of .

Proposition 7. * Let be a function given by (21) where . Then, the following statements hold true.*(i)*One has that **.*(ii)*The function ** is increasing.*(iii)*One has that ** for any **.*(iv)*One has that ** for any **.*

*Proof. * Let be a function given by (21) where .(i)In order to find fixed points of the function , we should solve the following equation: . It follows from the last equation that . Since , we get that or . Therefore, .(ii)Since , the function is increasing. (iii)Since , we may get that . (iv)Let and . Due to (iii), we have that . Since is increasing, we obtain that for any . This means that is a bounded decreasing sequence. Consequently, it converges to some point , and should be a fixed point; that is, . This means that . Similarly, one can show that if , then for any . This completes the proof.

Now, we are going to study dynamics of a -QSO taken from : where . This operator is an -Volterra-QSO, and its behavior was not studied in [24, 25].

Let , , and be the vertices of the simplex and let be an edge of the simplex , where . Let , , and .

Theorem 8. * Let be a -QSO given by (22) and let be an initial point. Then the following statements hold true. ** One has that
** If , then
** If , then
** If , then
*

*Proof. *Let be a -QSO given by (22), let be an initial point, and let be a trajectory of starting from the point .

In order to find fixed points of (22), we should solve the following system of equations:
We shall separately consider two cases: and .

Let . From the first equation of (27), we get that and (see Proposition 7(i)). It follows from the second equation of (27) that if , then , or , and if , then . This means that .

Let . The first equation of (27) takes the form . From the second equation of (27), we get that . This yields that or . In both cases, the third equation of (27) holds true. Therefore, we have that .

Let . It is clear that . Therefore, due to Proposition 7(iv), we have that . Hence, . Now, we shall separately consider two cases: and .

Let . In this case, and . Hence, .

Let . We need the following result.*Claim*. One has that . Moreover, for any , there exists (depending on ) such that .*Proof of Claim*. If , then . This means that .

Let , and suppose that all elements of the trajectory belong to set ; that is, . It follows from and that . This with implies that . On the other hand, we have that . It yields that is decreasing; hence, it converges to some point . This is a contradiction. This completes the proof of Claim.

Due to Claim, there exists such that for all . Therefore, , and is an increasing sequence which converges to . This yields that converges to , where . We know that should be a fixed point. Consequently, and .

Let . Due to Proposition 7(iv), we have that , whenever . Therefore, if , then . Since is not empty, we obtain that . Let , then for all . Moreover, we have that and . This means that . Therefore, if , then .

Let and . Then for any . Since , one gets that . This implies that is decreasing, and hence it converges to . Consequently, . We know that should be a fixed point. Since , we find that and . This means that .

Let . Then for any . Since , we have that and . If , then . This yields that is decreasing, and hence it converges to . Therefore, converges to . Since is a fixed point, we have that and . In the similar manner, one may have that if , then .

This completes the proof.

#### 5. Dynamics of -QSO from Class

We are going to study dynamics of a -QSO taken from : where . One can immediately see that this operator is a permuted -Volterra-QSO. As we mentioned, the behavior of such kinds of operators is not studied yet. It is worth mentioning that is a permutation of .

Let

It is clear that for any and .

Theorem 9. * Let be a -QSO given by (28) and let be an initial point. Then, the following statements hold true.** One has that
** One has that
** If , then
** If , then
*

*Proof. * Let be an initial point and let be a trajectory of starting from the point .

In order to find fixed points of , we have to solve the following system:

Let . From the first equation of the system (35), one can find that or (see Proposition 7(i)). If , then . If , then the second equation of the system (35) becomes as follows:
So, the solutions of this quadratic equation are . We can verify that the only solution belongs to . Therefore, one has . Hence, whenever .

Let . The system (35) then takes the following form
So, by letting (any ), the second equation of the system (37) can be written as follows
The solutions of the last equation are . One can check that the only solution belongs to . Therefore, one has that whenever .

(ii) Let . Now, we are going to show that the operator given by (28) does not have any order periodic points in the set , where . In fact, since the function is increasing (due to Proposition 7(ii)), the first coordinate of increases along the iteration of in the set . This means that does not have any order periodic points in set . Therefore, it is enough to find periodic points of in . In this case, in order to find 2-periodic points, we have to solve the following system of equations: