#### Abstract

In this work, we establish a new model of one prey and two predators with antipredator behavior. The basic properties on the positivity and boundedness of solutions and the existence of equilibria are established. Through analyzing the global dynamics, we find that there exist some values of the parameters such that one of the predators can be driven into being extinct by another. Furthermore, the coexistence of the three species is investigated which shows that the antipredator behavior makes the species coexist by periodic oscillation. The results give a new insight into the influence of antipredator behavior in nature selection.

#### 1. Introduction

Antipredator behavior is very common in the nature. This behavior usually developed by the evolutionary adaptation of the prey and predator. And the prey with the antipredator behavior can help itself to struggle against its predators. The antipredator behavior has already been widely studied by the ecologist [1–6]. Many literatures have been done to analyze the dynamics and bifurcation of the prey and predator model with the antipredator behavior [7–10]. In the early work [7] a basic model with antipredator behavior was considered and the numerical study of this model shows that more efficient antipredator behavior leads to increase in the density of the prey population, reduction in the ratio of predator-to-prey densities, and damped oscillations. Then in the following work [9] they considered the model where the antipredator behavior does not directly benefit the prey population but reduces the growth of the predator population and they found that the antipredator behavior helps the prey population to resist predator aggression. And in the paper [8] they considered a predator-prey system with prey population that guided antipredator behavior and achieved a great antipredator rate that induces the prey population to persist.

All of the model mentioned above assumed that the antipredator behavior only happened in two species; one is the prey and the other the predator. However, the antipredator behavior always influences more than one predator and can also make the predators’ competition become more complex [2]. There are already some works [11, 12] that considered the prey and predator model with two predators and one resource. In [11], they considered a model about two predators competing for the same resource and showed that there are only two coexistence styles for all three species: periodic oscillation or steady states, which depends on the resource for prey. In [12] they consider a parasitism-competition model with one host and multiple parasites; it shows the principle of competitive exclusion and demonstrates a mechanism by which abiotic factors lead to persistence of nectar robbing.

To the best of our knowledge, the model that consists of one prey and two predators with antipredator behavior is rarely discussed. Thus following the idea of [8], we consider an antipredator behavior model with two predators, and we use the Holing type function to illustrate the antipredator behavior, where is the prey, is the predator, and and are two constants about the antipredator behavior. Then the model takes the following form:where are defined as the above discussion and the subscript is about the two predators. The constant is the growth rate of prey, is the effect of the th predation on the prey, is the is the natural death rate of the th predator in the absence of prey, and is the efficiency and propagation rate of the th predator in the presence of prey. And all of the parameters in system (2) are positive.

The coordinate transformation takes system (2) toThis system is the main system that we analyze in this article.

The remainder of this paper is organized as follows. In the next section we analyze the basic properties of of system (4), including the existences and local stabilities of the equilibria, the boundedness, and the positivity of the solution of the system. In Section 3, the global dynamics of the system is investigated and the competitive exclusion condition is obtained. The coexistence of solutions on the curved surface is carried out in Section 4. A brief discussion is at the end of the paper.

#### 2. Basic Properties

We first analyze the positivity and boundedness of system (4).

Theorem 1. *All orbits of system (4) with positive initial conditions are positive and if and then the orbits of system (4) with positive initial conditions are also bounded.*

*Proof. *Since the three coordinate planes of are invariant, thus all the trajectories of system (4) are positive invariant in . Next, we show the boundedness of the system (4).

All of the orbits with positive initial conditions are positive, which means . Moreover, the coordinate change takes system (4) to the following systemThe boundedness will be proved by using the Poincaré Compactification method [11, 13, 14].

In the local chart the transformationtakes system (5) toThe invariant plane is the infinity of in the local chart . And the invariant plane with and is the infinity of in the local chart . Thus system (7) has unique equilibrium at infinity . Make a small perturbation around and then we can get the perturbation system of (7) as It follows from and that the equilibrium is unstable.

In the local chart the transformation takes system (5) toSystem (10) has infinitely many equilibria with at infinity if restricted at the infinity of . At each equilibrium system (10) has a double zero eigenvalue and a negative eigenvalue with the eigenvector in its infinity direction.

Then we continue to analyze the stability of equilibrium on its center manifold. Similar to the discussion in Proposition 3 in [11] we have the center manifold of to be the plane As and , it is obvious that all the solution in plane will go to . So has two one dimensional unstable manifold on its center manifold, which means is unstable.

For the local chart using the transformation and then by the same method we can obtain that the equilibrium is also unstable.

Then we use the local charts , and covered all equilibria at the infinity of system (4) in and find that all equilibria are unstable. Thus all equilibria at infinity of system (4) in its finite direction are repeller. So all trajectories of system (4) with positive initial and the condition will not reach the infinity which means every trajectory is bounded.

*Remark 2. *It is easy to check that the boundedness conditions and do not hold and the parameters satisfy and then every trajector starting from will first go to the -axis and then go to infinite.

Then we analyze the boundary equilibria of system (4).

Proposition 3. *The following statements hold.**(1) If and then system (4) has one boundary equilibrium in .**(2) If and then system (4) has two boundary equilibria and in , where .**(3) If and then system (4) has two boundary equilibria and in , where .**(4) If and then system (4) has three boundary equilibria , and in .*

The Jacobin matrix of system (4) at is given byand then we can obtain the following local phase portraits for the boundary equilibria through the standard calculation.

Proposition 4. *The following statements hold.**(1) is a saddle with a two dimensional stable manifold and a one dimensional unstable manifold.**(2) has the following local phase portraits.**(2.1) is saddle with a two dimensional stable manifold and a one dimensional unstable manifold if and .**(2.2) is saddle with a one dimensional stable manifold and a two dimensional unstable manifold if and or and .**(2.3) is an unstable node if and .**(3) has the following local phase portraits.**(3.1) is saddle with a two dimensional stable manifold and a one dimensional unstable manifold if and .**(3.2) is saddle with a one dimensional stable manifold and a two dimensional unstable manifold if and or and .**(3.3) is an unstable node if and .*

Next, we analyze the positive equilibria. Solving the following equations

If and then the infinitely many positive positive equilibria fill up the the segment in with endpoints at boundary equilibria and , where If then from the (14) we can get , which implies that there exist a constant such that , , and . If then we can get the infinitely many positive positive equilibria fill up the the set in with endpoints at boundary equilibria and , whereThen we can get the following proposition.

Proposition 5. *Each positive equilibrium of or is unstable.*

*Proof. *Let be the positive value of positive equilibrium in or then we can get the Jacobin Matrix of system (4) at which isIt follows that eigenvalues and of this matrix satisfySo , and have at least one positive value which implies that each positive equilibrium of or is unstable.

#### 3. Global Dynamics

In this section we mainly talk about the competitive exclusion conditions of system (4).

Theorem 6. *Assuming the boundedness conditions and hold and is a constant, then if one of the following conditions*(1)*, and ,*(2)*, and ,*(3)*, and ,** hold and all of the equal signs are not established at the same time then all orbits of system (4) in are asymptotic to the orbits on the plane in forward time.*

*Proof. *It follows from Theorem 1 that there exist a large enough compact bounded set which does not include and is positive invariant set of system (4).

Let Clearly this function is a continuous differential positive function in . And we haveIt is obvious that any of the conditions that were mentioned in this theorem imply thatThus, we can get Hence, the solutions of system (4) with the initial point in satisfyLet Then is an invariant set of system (4). It follows from Lasalle Principle that all solutions of system (4) in are asymptotic to the orbits on the plane in forward time. This completes the proof.

Using the same method as above theorem and taking the functionthen we can conclude the following theorem.

Theorem 7. *Assuming the boundedness condition and hold and is a constant, then if one of the following conditions*(1)*, and ,*(2)*, and ,*(3)*, and ,** hold and all of the equal signs are not established at the same time then all orbits of system (4) in are asymptotic to the orbits on the plane in forward time.*

Note that the existence condition of the equilibria is . The we have Therefore, it follows from the proof of Theorems 6 and 7 that if equilibria exit and the boundedness condition holds then if satisfy an extra condition which can make or , where , then all the orbits of system (4) in are asymptotic to the orbits on the or plane in forward time.

#### 4. Dynamics on Curved Surface

In this section we mainly consider the dynamics when the existence condition of holds.

Theorem 8. *For system (4) denote the solution of system (4) with as its initial value,. If , , , where is a constant. Then for we have .*

*Proof. *Since , , , thus we havewhich impliesIntegrating both sides of the above equation we haveThen using sole existence theorem of solution to ordinary differential equations we can get this conclusion.

From the above theorem we know all the trajectories of system (4) will stay in a curved surface which is based on the initial condition. So for each trajectory we can consider the restricted system on the curved surface where Since , , and , thus the equation has only one positive root. So system (30) has only one positive equilibrium when , where and satisfies . And from the formula we can also get .

Furthermore, we can obtain the following results for the restricted system (30).

Theorem 9. *Assuming , and If the bifurcation parameter and it is small enough then there exists a stable limit cycle bifurcate from the unique positive equilibrium of system (30).*

*Proof. *We use the Hopf bifurcation method that is mentioned in the [15] to prove the result. We first verify the transversal condition of the system at the unique positive equilibrium.

It is easy to get the Jacobin matrix evaluated at the equilibrium Denote that eigenvalues of the matrix as , and thenwhen We have , and the eigenvalues are , where Moreover, we get And this implies that the bifurcation parameter satisfies the transversal condition and can bifurcate at least one limit cycle. Then it is sufficient to calculate the first Lyapunov coefficient to prove the stability and uniqueness of the limit cycle.

We fix the parameter at its critical value , and then the equilibrium has the -coordinate Transforming this equilibrium to the origin, we take the following transformation: Then system (30) becomes where and furthermore using the transformationThen, we obtain the system where Then using the following formula in [15] we can compute the first Lyapunov coefficient where , and others can be deduced by analogy, and we get It is clear that and . Thus if and , then the first Lyapunov coefficient . Therefore, a unique and stable limit cycle bifurcates from the unique positive equilibrium via the Hopf bifurcation for .

This result implies that on the plane system where the of system (30), since the first Lyapunov coefficient , there will be more than one limit cycle bifurcated from the unique equilibrium. But on the curved surface with and there will be only one stable limit cycle bifurcated from the unique equilibrium, which means when the antipredator coefficient is small enough, the system with one prey and one predator can coexistence at more than one possible way, and the oscillations are based on the initial condition. But when there are two predators that coexistence on the curved surface , there is only one possible way for the two predators and one prey to coexist on that curved surface.

Furthermore, we can obtain a weaker condition which allows to become bigger. The result is given as follows.

Theorem 10. *For system (30), if and then there will be at least one limit cycle in the phase plane of system (30).*

*Proof. *It can be proved by utilizing the Poincaré Bendixson theorem [16, 17].

It follows from Theorems 1 and 8 that there exist a large enough positive bounded invariant set for system (30) which does not include and and the unique positive equilibrium is included.

It follows from (32) that the unique positive equilibrium in is an unstable focal point, so we can find a small enough number and make the set such that any trajectory that starts from the will go out of the set in finite time. Hence we can find a positive bounded invariant set which includes no equilibrium.

Therefore, it follows from the Poincaré Bendixson theorem that there must exist at least one limit cycle in . This completes the proof.

*Note.* The above theorem implies that the coexistence and oscillation always exit with any and for system (30).

#### 5. Numerical Simulation

Numerical simulations are done to illustrate the dynamic behaviors of systems (4) and (30). Figure 1 shows that when there is only one limit cycle in the phase diagram of system (30) and Figure 2 shows that when the number of limit cycle will be greater than . From Figures 3 and 4 we can see that when birth rate and death rate are the same, the predator that is less easily affected by the antipredator behavior survives easily.

#### 6. Discussion

In this work we consider a new model of one prey and two predators with the antipredator behavior. To analyze the model, we first prove the positivity and boundedness of system (4) and give the existence and stability conditions of those equilibria. Then we give the competitive exclusion conditions of system (4), which implies that when the condition holds one of the predators is driven into extinction by the other. We also analyzed the coexistence of the three species of system (4) and found that those three species can coexist in periodic oscillations.

Our results lead to a new insight into natural selection mechanism, especially how the antipredator behavior influences the natural selection. From Theorems 6 and 7, we know that when the parameters and are the same, the predator with smaller is more easily to be driven into extinction, which means the predator that is less easily affected by the antipredator behavior survives more easily. And Theorems 9 and 10 imply that the antipredator effect can lead to complex periodic coexistence of these three species. Theorem 9 and numerical simulation especially imply that when the antipredator effect is weak, then there will be one possible way for these three species to coexist on the curved surface but the one prey and one predator system can coexist at more than one possible way which is also induced by the antipredator effect.

In the second section we have talked about the dynamic when the equilibria exist, but it is under a very strict condition. If we consider a more general condition, there might be more interesting results of the coexistence that what we can analyze in future studies.

#### Data Availability

No data were used to support this study.

#### Conflicts of Interest

The authors declare that they have no conflicts of interest.