#### Abstract

In this paper, a class of predator-prey systems with fear effect is investigated, the integrability conditions of the origin and the positive equilibrium are obtained, and the fact that three limit cycles can be bifurcated from the positive equilibrium is proved, so bistable phenomenon can occur for this system.

#### 1. Introduction

Predator-prey systems which were considered to be among the most important models in ecological systems have been investigated intensively and extensively since they were proposed; there have been many good results for these systems. The first model for predator-prey problem was Lotka-Volterra model; this system was improved gradually year by year. Many new models were constructed with the development of biological system.

Functional response which was introduced by Holling has a great effect on predator-prey system. In 1965, the famous Holling type II functional response of predators [1] was proposed by Holling. As application, the population dynamics of predator-prey systems with the Holling type II functional response have been studied by many authors; for such a model, the result about the existence of a unique stable limit cycle has also been proved. Considering the reality, many more complicated functional responses were suggested one by one, and predator-prey systems have also been modelled gradually. For example, some monotone response functions dependent on prey have been investigated by May [2], Seo and DeAngelis [3], and Huang et al. [4]. Furthermore, Zhu et al. [5], Ruan and Xiao [6], Freedman and Wolkowicz [7], and Wolkowicz [8] studied some nonmonotone response functions dependent on prey. Some interesting bifurcation phenomenon was found for those systems. Functional responses dependent on prey only have been considered intensively; some functional responses dependent on both prey and predators were also considered in recent years, and the famous functional responses dependent on both prey and predators are the Beddington-DeAngelis functional responses, see [9–13]), and ratio dependent functional response [14, 15].

Although the functional responses become more and more complicated for predator-prey systems, they can only reflect the direct killing. There is profound lack of the functional responses and many realities which cannot be concluded in those systems. For example, in 2016, Zou and Wang considered a predator-prey model incorporating the cost of fear to explore the impact that fear can have on population dynamics in predator-prey systems, dynamical behaviors of this system was discussed, and a limit cycle which can be bifurcated from the positive equilibrium by Hopf bifurcation was proved [16]. But the number of limit cycles is not considered in their paper. In fact there can be many limit cycles in predator-prey system which can lead to multiple stable phenomenons. For example, the system of Leslie-type predator-prey schemes with a nonmonotonic functional response and Allee effect allows the existence of three limit cycles under certain conditions over the parameters; by Hopf bifurcation [17], the first two cycles are infinitesimal ones generated, and then the third one can be bifurcated from a homoclinic loop. The singular perturbation theory which was developed by Dumortier and Roussarie can be used to study the canard phenomenon. The authors studied the canard phenomenon for predator-prey systems with response functions of Holling types [18] by using this method. For a Gause type predator-prey system with Holling type III functional response and Allee effect on prey, multiple limit cycles were considered in [19] by multiple Hopf bifurcations. In [20], a criterion to calculate the multiplicity of a multiple focus was developed by the authors for general predator-prey systems. Recently, the number of limit cycles in a Leslie-Gower-type predator-prey system with weak Allee effect on prey was considered in [21].

The number of limit cycles is closely related not only to the Hilbert 16th problem but also to bistable phenomenon or multiple stable phenomenon. For instance, in [21], the existence conditions for which three limit cycles appear and surround different equilibrium points were proved. Above all, the three limit cycles have different stability which means the phenomenon of multistability. For finitely smooth planar autonomous differential systems, in [22], the author considered the bifurcation theory. Theory of rotated equations is discussed and applied to a population model in [23]. For a family of real planar polynomial ODE systems depending on parameters, the problem of how to determine the systems in the family which become time-reversible after some affine transformation was considered in [24]. Recently, an improvement on the number of limit cycles bifurcating from a nondegenerate center of homogeneous polynomial systems was obtained in [25] by normal form method. Some new perturbation methods were given in [26–28]. The integrability of equivalent systems was also considered in [29, 30]. Therefore, in this paper, we reinvestigated the predator-prey system with fear effect which is written as follows:

By transformation and time scale system (1) can be reduced toAnd system (4) has three singular points where

The classification of the three singular points and stability have been discussed in [16], but the integrability and center problem remained. The rest of the paper is organized as follows. In the next section, we prove that the origin of system (4) is integral. In Section 3, the first four Lyapunov constants will be computed; bifurcation of limit cycles and center conditions of (4) are investigated. Section 4 is devoted to discussing the kind of the boundary equilibrium. At last, a conclusion is drawn to illustrate that bistable phenomenon can occur in this system.

#### 2. Integrability Conditions of

For the integrability of origin of system (4), the authors have proved that it is a saddle. In this section, we will compute the saddle values; furthermore, the integrability conditions can be obtained at the same time. We only consider a special case; namely, let ; we study the integrability conditions theoretically. The system (4) becomes

Theorem 1. *The first five saddle values at the origin of system (7) are If (1) (2) (3) If , (4) (5) Here are given in the appendix.*

Thus, we can get the following integrability conditions of system (7) easily.

Theorem 2. *The origin of system (7) is integrability if and only if one of the following conditions holds: *

*Proof. *When condition (I) holds, system (7) becomes which has an integrating factorWhen condition (II) holds, system (7) can be rewritten aswhich admits an integrating factorWhen condition (III) holds, system (7) can be changed intowhich admits an integrating factorWhen condition (IV) holds, system (7) can be transformed intowhich has an integrating factor

#### 3. Hopf Bifurcation at

According to results in [16], there exists a positive singular point . Let ; system (4) can be transformed into Here Suppose there are two characteristic values ; then namely, and we have Without loss of generality, let and the singular point becomes . From the first equation , we getMeanwhile, the second equation means thatwhich yields

Accordingly, system (7) can be changed into

The Jacobin matrix at the of system (34) can be written as follows: and by linear transformation system can be shifted toThe Jacobin matrix at the origin of system (37) can be written as follows:

and, with the help of computer algebraic system Maple, we can compute the first five Lyapunov constants for system (37) by using formal series method which was given in the following.

Theorem 3. *For system (37), the first five Lyapunov constants at the origin can be given as follows: where are given in the appendix.*

Theorem 3 yields the following.

Proposition 4. *For system (37), the first five Lyapunov constants at the origin are zero if and only if one of the following conditions holds:*

Furthermore, we have the following.

Theorem 5. *The origin of system (37) is a center if and only if one of the conditions in Proposition 4 holds.*

*Proof. *When condition (40) holds, system (37) becomeswhich has four invariant algebraic curves and admits an integrating factorWhen condition (41) holds, system is simplified intowhich has three invariant algebraic curves and admits an integrating factor

Two examples of phase plane of system (42) when condition (40) or (41) is satisfied are given in Figures 1 and 2.

Based on Theorem 3, we can conclude the following.

Theorem 6. *The origin of system (37) is fifth order weak foci if and only if one of the following conditions holds: *

*Proof. *Let , and we obtain submitting into , we have Solve and , and we can obtain the results.

Considering the reality, all parameters are positive in a predator-prey system, so we cannot get five limit cycles for this system.

In fact, means that and, submitting it into , we get Figure 3 which shows that cannot be larger than zero simultaneously.

We can adjust the parameters to satisfy , and all parameters are positive. For example, Then three limit cycles can be bifurcated from the origin; namely, bistable phenomenon can occur for this system; this is not discussed in previous discussion.

#### 4. Boundary Singular Point

In this section, we consider the boundary singular point of system (4). The boundary singular point is and the Jacobin matrix at of system (4) is It is easy to testify that when , system (4) has two eigenvalues with different signs which means is a saddle. When , system (4) has two eigenvalues with negative signs which means is a stable node. When , namely, , system (4) becomes and one eigenvalue is zero. System (4) becomesShifting the of system (4) to origin, system (4) becomesthe Jacobin matrix at origin system (56) is By transformation and time scale system (56) becomesBased on the theorem in [31], we have the following.

Theorem 7. *The origin of system (56) is a saddle-node singular point.*

*Proof. *From , we get a solution by implicit function theorem; submitting it into , we have Based on the theorem in [31], the origin of system (56) is a saddle-node singular point.

An numerical simulation of phase portrait of system (56) when is given in Figure 4 which shows that the origin of system (56) is a saddle-node singular point.

#### 5. Conclusion

In this paper, a class of predator-prey systems with fear effect is investigated; the integrability conditions of and positive equilibrium are obtained and proved by finding their integrating factors. Furthermore, we compute the first five focal values at the positive equilibrium; considering the reality, when we choose all parameters to be positive, the fact that at least three limit cycles can be bifurcated from the positive equilibrium is proved. Consequently, bistable phenomenon can occur for this system.