#### Abstract

We shall explore a nonlinear discrete dynamical system that naturally occurs in population systems to describe a transmission of a trait from parents to their offspring. We consider a Mendelian inheritance for a single gene with three alleles and assume that to form a new generation, each gene has a possibility to mutate, that is, to change into a gene of the other kind. We investigate the derived models and observe chaotic behaviors of such models.

#### 1. Introduction

Recently, chaotic dynamical systems become very popular in science and engineering. Besides the original definition of the Li-Yorke chaos [1], there have been various definitions for “chaos” in the literature, and the most often used one is given by Devaney [2]. Although there is no universal definition for chaos, the essential feature of chaos is sensitive dependence on initial conditions so that the eventual behavior of the dynamics is unpredictable. The theory and methods of chaotic dynamical systems have been of fundamental importance not only in mathematical sciences, but also in physical, engineering, biological, and even economic sciences. In this paper, a chaos would be understood in the sense of Li-Yorke [3, 4] (the precise definition will be given in the next section).

In this paper, we introduce and examine a family of nonlinear discrete dynamical systems that naturally occurs to describe a transmission of a trait from parents to their offspring. Here, we shall present some essential analytic and numerical results on dynamics of such models.

In [5], it was presented an approach to the dynamics at the cellular scale in which cells can progress, namely, modify their biological expression and mutate within Darwinian-type selective processes, out of the interaction with other cells. A heterogeneous distribution among cells produces mutations and selections generated by net destructive and/or proliferative events [5]. In this event, all living systems are evolutionary: birth processes can generate individuals that fit better the outer environment, which in turn generates new ones better and better fitted [5]. One can refer to [5–8] for the general information about mathematical models of complex systems (including mutations and selections). In this paper, we are presenting a mathematical model of the evolution of the percentage of different alleles of a given trait after the mutation process.

As the first example, we consider a Mendelian inheritance of a single gene with two alleles and (see [9]). Let an element represent a gene pool for a population; that is, , are the percentage of the population which carries the alleles and , respectively. For the convenience, we express it as a linear combination of the alleles and where, and . The rules of the Mendelian inheritance indicate that the next generation has the following form: where

Here, (resp., ) is the probability that the child receives the allele (resp., ) from parents with the allele ; (resp., ) is the probability that the child receives the allele (resp., ) from parents with the alleles and , respectively; and (resp., ) is the probability that the child receives the allele (resp., ) from parents with allele . It is evident that

Thus, the evolution (3) is a nonlinear dynamical system acting on the one dimensional symplex which describes the distribution of the next generation which carries the alleles and , respectively, if the distribution of the current generation is known.

Recall that in the simple Mendelian inheritance case, that is, and , the dynamical system (3) has the following form:

We assume that prior to a formation of a new generation each gene has a possibility to mutate, that is, to change into a gene of the other kind. Specifically, we suppose that for each gene the mutation occurs, with probability , and occurs with probability . Moreover, we assume that “*the mutation occurs if and only if both parents have the same allele*.” Then, we have that , and the dynamical system (3) has the following form:

An operator given by (7) is called* a quadratic stochastic operator* [10]. The name “*stochastic*” can be justified if we consider the simplex as a set of all probability distributions of the finite set, so that, the operator (7) maps a probability distribution to a probability distribution.

We introduce some standard terms in the theory of a discrete dynamical system . A sequence , where , is called a trajectory of starting from an initial point . Recall that a point is called a fixed point of if . We denote a set of all fixed points by . A dynamical system is called regular if a trajectory converges for any initial point . Note that if is regular, then limiting points of are fixed points of . Thus, in a regular system, the fixed point of dynamical system describes a long run behavior of the trajectory of starting from any initial point. The biological treatment of the regularity of dynamical system is rather clear: in a long run time, the distribution of species in the next generation coincide with distribution of species in the current generation, that is, stable.

A fixed point set and an omega limiting set of quadratic stochastic operators (QSO) were deeply studied in [11–16] and quadratic stochastic operators (QSO) play an important role in many applied problems [17, 18]. In [10], it was given a long self-contained exposition of recent achievements and open problems in the theory of quadratic stochastic operators.

*Definition 1. *A dynamical system is said to be ergodic if the limit
exists for any .

Based on some numerical calculations, Ulam has conjectured [19] that any QSO acting on the finite dimensional simplex is ergodic. However, Zakharevich showed [20] that, in general, Ulam’s conjecture is false. Namely, Zakharevich showed that the following QSO is not ergodic:

In [21], Zakharevich’s result was generalized in the class of Volterra QSO.

We define the th order Cesaro mean by the following formula: where and . It is clear that the first order Cesaro mean is nothing but . Based on these notations, Zakharevich’s result says that the first order Cesaro mean of the trajectory of the operator given by (9) diverges for any initial point except fixed points. Surprisingly, in [22], it was proven that any order Cesaro mean , for any , of the trajectory of the operator diverges for any initial point except fixed points. This leads to a conclusion that the operator might have unpredictable behavior. In fact, in [23], it was proven that the operator exhibits the Li-Yorke chaos. It is worth pointing out that some strange properties of Volterra QSO were studied in [24, 25].

In the literature, all examples of nonergodic QSO have been found in the class of Volterra QSO (see [10, 20, 21]). Based on these examples, the Ulam conjecture was modified as follows:* any non Volterra QSO acting on the finite dimensional simplex is ergodic, that is, operators having chaotic behavior can be only found among Volterra QSO.* However, in this paper, we are aiming to present the continual family of nonergodic and chaotic QSO which are non Volterra QSO.

Note that if QSO is regular, then it is ergodic. However, the reverse implication is not always true. It is known that the dynamical system (7) is either regular or converges to a periodic-2 point [26]. Therefore, in 1D simplex, any QSO is ergodic. In other words, the evolution of a mutation in population system having* a single gene with two alleles* always exhibits an ergodic behavior (or almost regular or almost stable). It is of independent interest to study the evolution of a mutation in population system having* a single gene with three alleles*. In the next section, we consider an inheritance of a single gene with three alleles , , and and show that a nonlinear dynamical system corresponding to the mutation exhibits a nonergodic and Li-Yorke chaotic behavior.

#### 2. Inheritance for a Single Gene with Three Alleles

In this section, we shall derive a mathematical model of an inheritance of a single gene with three alleles.

As it was in the previous section, an element represents a linear combination of the alleles , , and in which the following conditions are satisfied and , that is, are the percentages of the population which carry the alleles , , and respectively.

We assume that prior to a formation of a new generation each gene has a possibility to mutate, that is, to change into a gene of the other kind. We assume that the mutation occurs if* both parents have the same alleles*. Specifically, we will consider two types of the simplest mutations; assume that(1)mutations , , and occur with probability ;(2)mutations , , and occur with probability .

In this case, the corresponding dynamical systems are defined on the two-dimensional simplex In the first mutation, we have In the second mutation, we have

Let us first recall the definition of the Li-Yorke chaos [1, 3, 4].

*Definition 2. *Let be a metric space. A continuous map is called* Li-Yorke chaotic* if there exists an uncountable subset such that for every pair of distinct points, we have that
In this case, is* a scrambled set* and is* a Li-Yorke pair*.

Let us turn to the discussion of operators , given by (12) and (13), respectively. In both cases, if , that is, if a mutation does not occur, then dynamical systems (12) and (13) coincide with Zakharevich’s operator (9). As we already mentioned that Zakharevich’s operator exhibits the Li-Yorke chaos [23].

Let . In the first case, the operator is a permutation of Zakharevich’s operator (9). Therefore, the operator is nonergodic and does exhibit the Li-Yorke chaotic behavior [22, 23, 27]. In the second case, the operator is a permutation of the regular operator which was studied in [11]. By applying the same method which was used in [11], we may easily show that the operator is also regular.

It is easy to check that and .

This means that, in the first case, the evolution operator is a convex combination of two Li-Yorke chaotic operators , , meanwhile, in the second case, the evolution operator is a convex combination of the Li-Yorke chaotic and regular operators , . These operators , were not studied in [11, 27]. It is of independent interest to study the dynamics of operators and . The reason is that, in the first case, the convex combination presents a transition from one chaotic biological system to another chaotic biological system (we shall see in the next section that, in some sense, their dynamics are opposite each other); meanwhile, in the second case, the convex combination presents a transition from the ordered biological system to the chaotic biological system. In the next section, we are going to present some essential analytic and numerical results on dynamics of the operators and given by (12) and (13), respectively.

#### 3. Attractors: Analytic and Numerical Results

##### 3.1. Analytic Results on Dynamics of

We are aiming to present some analytic results on dynamics of : where and . As we already mentioned, this operator can be written in the following form: for any , where

Let be a permutation matrix. The proofs of the following results are straightforward.

Proposition 3. *Let be the evolution operator given by (15), where . Let and be sets of fixed points and omega limiting points of , respectively. Then the following statements hold true.*(i)*Operators and are commutative, that is, .*(ii)*If then .*(iii)*If is a finite set then .*(iv)*One has that , for any .*

We are aiming to study the fixed point set , where . It is worth mentioning that and , where , , and are vertices of the simplex and is a center of the simplex .

Recall that a fixed point is nondegenerate [18] if and only if the following determinant is nonzero at the fixed point :

Proposition 4. *Let be the evolution operator given by (15), where . Let be a center of the simplex . Then the following statements hold true. *(i)*All fixed points are nondegenerate.*(ii)*One has that for any .*

*Proof. *(i) Let be a fixed point. One can easily check that

If , then the above expression is positive. Therefore, all fixed points are nondegenerate.

Let . In this case, the above expression is equal to zero if and only if . Since , we have that .

Without loss of generality, we may assume that (See Proposition 3(i)). Let . Since , we have that . We then obtain that
This is a contradiction. In a similar way, one can have a contradiction whenever . This shows that, in the case , all fixed points are nondegenerate.

(ii) We shall show that . The simple calculation shows that . It is clear that . This means that the operator does not have any fixed point on the boundary of the simplex , that is, . Moreover, all fixed points are nondegenerate. Due to Theorem in [18], should be odd. On the other hand, due to Corollary in [18], one has that . In Proposition 3, (iii) yields that . Therefore, we get that .

A local behavior of the fixed point is as follows.

Proposition 5. *Let be the evolution operator given by (15), where . Then the following statements hold true.*(i)*If , then the fixed point is repelling.*(ii)*If , then the fixed point is nonhyperbolic.*

*Proof. *It is worth mentioning that, since , the spectrum of the Jacobian matrix of the operator at the fixed point must be calculated as follows:
After simple algebra, we have that Spec. It is clear that . Consequently, if , then the fixed point is repelling and if , then the fixed point is nonhyperbolic. This completes the proof.

We shall separately study two cases and .

Theorem 6. *Let be the evolution operator given by (15), where . Then is an infinite compact subset, for any .*

*Proof. *Let . Since is continuous and , an omega limiting set is a nonempty compact set and , for any . We want to show that is infinite, for any . Since is repelling, we have that . Let us pick up any point from the set . Since the operator does not have any periodic point, the trajectory of the point is infinite. Since is continuous, we have that . This shows that is infinite for any .

*Remark 7. *It is worth mentioning that the sets of omega limiting points and of the operators and are infinite. However, unlike the operator , we have inclusions and . Moreover, both operators and are nonergodic [20, 27].

Numerically, we shall see in the next section that the evolution operator given by (15), where , has the following properties.(i)The operator is nonergodic.(ii)The operator exhibits the Li-Yorke chaos.

Now, we shall study the case . The operator takes the following form

In this case, the fixed point is nonhyperbolic and the spectrum of the Jacobian matrix of the operator at the fixed point , calculated by (18), is .

Let us define the following sets:

Proposition 8. *We have the following cycles: *(i)*;*(ii)*;*

*Proof. *Let be an operator given by (22). One can easily check that
The proof the proposition follows from the above equality.

Theorem 9. *Let be the evolution operator given by (22). The following statements hold true. *(i)* is a Lyapunov function.*(ii)*Every trajectory converges to the fixed point .*

*Proof. *(i) Let be an operator given by (22). It follows from (24) that
On the other hand, we have that
Therefore, one has that , for any . This means that is decreasing a long the trajectory of . Consequently, is a Lyapunov function.

(ii) We know that is a decreasing bounded sequence. Therefore, the limit exists. We want to show that . Suppose that . It means that . Since , we get that

On the other hand, since , there exists such that for any one has that

This is a contradiction. It shows that .

Therefore, . We want to show that .

We know that . It follows from (24) that
This means that . This completes the proof.

##### 3.2. Numerical Results on Dynamics of

We are going to present some pictures of attractors (an omega limiting set) of the operator given by (15).

In the cases and , the corresponding operators , have similar spiral behaviors which reel along the boundary of the simplex [16, 20]. However, one of them moves clockwise and another one moves anticlockwise. In these cases, we have that and .

We are interested in the dynamics of the evolution operator while approaches to from both left and right sides. In order to see some antisymmetry, we shall provide attractors of and at the same time.

If is an enough small number, then we can see that the omega limiting sets of operators and are separated from the boundary (see Figure 1).

**(a)**

**(b)**

If becomes close to , then we can see some chaotic pictures. We observe from the pictures (see Figures 2 and 3) that, in the cases and , the attractors are the same but different from each other by orientations. There are some pictures for the values of (see Figures 2, 3, 4, and 5). For the evolution operator ,* the bifurcation point* is and the influence of the chaotic operators , would be dismissed. Therefore, the operator becomes regular.

**(a)**

**(b)**

**(a)**

**(b)**

**(a)**

**(b)**

**(a)**

**(b)**

##### 3.3. Analytic Results on Dynamics of

We are aiming to present some analytic results on dynamics of : As we already mentioned, this operator can be written in the following form: , for any , where It is clear that is Zakharevich’s operator (9) and the operator is a permutation of the operator which was studied in [11]. By means of methods which were used in [11], we can easily prove the following result.

Proposition 10. *Let be the evolution operator given as above. Then the following statements hold true. *(i)*The operator has a unique fixed point which is attracting.*(ii)*The vertexes of the simplex are 3-periodic points.*(iii)* is a Lyupanov function.*(iv)*The operator is regular in the set .*

By means of the same methods and techniques which are used for the operator , we can prove the following results.

Proposition 11. *Let be the evolution operator given by (30). Then it has a unique fixed point ; that is, . Moreover, one has that*(i)*if , then the fixed point is repelling;*(ii)*if , then the fixed point is attracting;*(iii)*if , then the fixed point is non-hyperbolic.*

Theorem 12. *Let be the evolution operator given by (30). Then the following statements hold true.*(i)*If , then is an infinite compact set, for any .*(ii)*If , then , for any .*

Numerically, we shall see in the next section that the evolution operator given by (30), where , has the following properties.(i)The operator is nonergodic.(ii)The operator exhibits the Li-Yorke chaos.

##### 3.4. Numerical Results on Dynamics of

We are going to present some pictures of attractors (an omega limiting set) of the operator given by (30).

In the cases and , the operator is chaotic and the operator is regular. Since , the evolution operator gives the transition from the regular behavior to the chaotic behavior. Consequently, we are aiming to find* the bifurcation point* in which* we can see the transition from the regular behavior to the chaotic behavior*.

If is a very small number then attractors of the operator are separated from the boundary of the simplex (see Figure 6). However, the influence of the operator is still higher and the operator is nonergodic and chaotic.

**(a)**

**(b)**

If becomes close to (from the left side), then we can see some interesting pictures (see Figure 7). If we continue to increase , then the evolution operator becomes regular (for any ). This means that* the bifurcation point* is . Therefore, in order to have a transition from the regular behavior to the chaotic behavior, we need one bifurcation point .

**(a)**

**(b)**

#### 4. Conclusions

In this paper, we present the mathematical model of the evolution of traits having 3 alleles by mutating the biological environment. We have presented two types of mutations. We have shown that a mutation (a mixing) in the system can be considered as a transition between two different types of systems having Mendelian inheritances. Namely, the first mutation presents the transition between two chaotic biological systems; meanwhile the second mutation presents the transition between regular and chaotic systems.

In the first mutation, we have presented some pictures of attractors of the operator given by (15). In the cases and , the corresponding operators , have similar spiral behaviors which reel along the boundary of the simplex. However, one of them moves clockwise and another one moves anticlockwise. In these cases, we had that and . If is an enough small number then we observed that the omega limiting sets of operators and are separated from the boundary (see Figure 1). If becomes close to then we had some chaotic pictures. We observed from the pictures (see Figures 2 and 3) that, in the cases and , the attractors are the same but different from each other by orientations. There are some pictures for the values of (see Figures 2–5). For the evolution operator , the bifurcation point is and the influence of the chaotic operators , would be dismissed. Therefore, the operator becomes regular. This means that during the transition between two (in some sense, opposite each other) chaotic systems, at some point of the time, the system should become stable.

In the second mutation, we have presented some pictures of attractors of the operator given by (30). In the cases and , is chaotic and is regular. The evolution operator gives the transition from the regular behavior to the chaotic behavior. If is a very small number then attractors of the operator are separated from the boundary of the simplex (see Figure 6). However, the influence of the operator is still higher and the operator is nonergodic and chaotic. If becomes close to (from the left side), then we can see some interesting pictures (see Figures 7 and 8). If we continue to increase , then the evolution operator becomes regular (for any ). This means that the bifurcation point is . Therefore, in order to have a transition from the regular behavior to the chaotic behavior, we need one bifurcation point . Since the operator is the convex combination of chaotic (nonergodic) and regular transformations, it is natural to expect the bifurcation scenarios in this evolution. Namely, in order to have a transition from regular to chaotic behavior we have to cross from the bifurcation point. Numerical result also confirms the theoretical result about the exact value of bifurcation point. However, the biological plausiblity of this value is unknown for the authors.

**(a)**

**(b)**

In this paper, we have considered two types of mutations of three alleles which occurred with the same probability. It is natural to consider mutations with different probabilities among alleles. In this case, it is expected to have more complicated dynamics in the biological system. The future research is to study the dynamics of the mutated biological system having a single gene with a finite number of alleles.

#### Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.