#### Abstract

We investigate positive solutions of a prey-predator model with predator saturation and competition under homogeneous Dirichlet boundary conditions. First, the existence of positive solutions and some sufficient and necessary conditions is established by using the standard fixed point index theory in cones. Second, the changes of solution branches, multiplicity, uniqueness, and stability of positive solutions are obtained by virtue of bifurcation theory, perturbation theory of eigenvalues, and the fixed point index theory. Finally, the exact number and type of positive solutions are proved when or converges to infinity.

#### 1. Introduction

Considering the destabilizing force of predator saturation and the stabilizing force of competition for prey, Bazykin [1] proposed the function response in the prey-predator model instead of the classical Holling-type II functional response. For this functional response, the prey-predator model is taken as the following form:

In this paper, we are concerned with the positive solution of the boundary value problem of the following elliptic system corresponding to the system (1.1): where is a bounded domain in with smooth boundary ;â€‰â€‰ are positive constants; is constant; and are nonnegative constants.

If , then (1.2) is reduced to the classical Lotka-Volterra prey-predator model which has received extensive study in the last decade, see [2â€“8]. In particular, the existence of positive solutions for this case was completely understood, see Dancer [8]. It has been conjectured that there is at most one positive solution, but this was shown only for the case the space dimension is one, see [9]. For space dimension greater than one, this is still an open problem; we also refer to [10, 11] for some partial results on uniqueness. The stability of positive solutions was studied in [10, 11], but the results are still far from being complete.

The case when and was first studied by Blat and Brown [12]. In this case, the term is known as the Holling-Tanner interaction term, and we refer to [5, 12â€“17] for more discussion on this model. In [12], Blat and Brown studied the existence of positive solutions to (1.2) by making use of both local and global bifurcation theories. The case when goes to infinity was extensively studied by Du and Lou in [13, 14, 18]. They gave a good understanding of the existence, stability, and number of positive solutions for large .

However, the case when and was first studied by Bazykin in the paper [1], more detailed background on this case, we can refer to [1]. And more works can refer to [19], Wang studies the existence, multiplicity, and stability of positive solutions of (1.2). However, Our work is more specific and meticulous than theirs. In particular. Firstly, the changes of solution branches, uniqueness, and stability of positive solutions are obtained by virtue of bifurcation theory, perturbation theory of eigenvalues, and the fixed point index theory. Secondly, the exact number and type of positive solutions are proved when or is large.

This paper is organized as follows: in Section 2, we give sufficient and necessary conditions for the existence of coexistence states of (1.2) by using index theory. In Section 3, by using as a main bifurcation parameter, the multiplicity of coexistence stats to (1.2) is investigated in the gap between the sufficient and necessary conditions for the existence of coexistence states which are found in Section 2. In Section 4, the multiplicity, uniqueness, and stability of coexistence states of (1.2) are investigated when or converges to infinity.

#### 2. Existence and Nonexistence of Coexistence States

In this section, we will obtain existence and nonexistence of coexistence states. Firstly, we present some basic results which will be used in this paper.

Let be all eigenvalues of the following problem: where . It is easy to see that is simple and is strictly increasing in the sense that and imply . When , we denote by . Moreover, we denote by the eigenfunction corresponding to with normalization and positive in .

Define . It is well known that for any , the problem has a unique positive solution which we denote by . It is well known that the mapping is strictly increasing, continuously differentiable from to and that uniformly on as . Moreover, we have in . It follows that (1.2) has two semitrivial solutions and if .

Next, we give an a priori estimate based on maximum principle. Its proof will be omitted.

Lemma 2.1. *Any coexistence state of (1.2) has an a priori boundary, that is,
*

In the following, we set up the fixed point index theory for later use. Let be a Banach space. is called a wedge if is a closed convex set and for all . For , we define , we always assume that . Let be a compact linear operator on . We say that has property on if there exists and , such that .

For any and , we denote . Assume that is a compact operator and is an isolated fixed point of . If is FrÃ©chet differentiable at , then the derivative has the property that . We denote by the fixed point index of at relative to .

We state a general result of Dancer [20] on the fixed point index with respect to the positive cone (see also [6]).

Lemma 2.2. *Suppose that is invertible on .*(i)* If has property on , then .*(ii)* If does not have property on , then , where is the sum of algebra multiplicities of the eigenvalues of which are greater than .*

We introduce some notations as follows:â€‰, where ,â€‰, where ,â€‰, .

Define by where and . It follows from Maximum Principle that is a compact positive operator, is complete continuous and FrÃ©chet differentiable. Denote , observe that (1.2) has a positive solution in if and only if has a positive fixed point in .

If and , then , , and are the only nonnegative fixed points of . Then , , and are well defined. We calculate the FrÃ©chet operator of as follows:

We can obtain the following lemmas by similar methods to those in the proofs of Lemmas 1 and 2 in [19].

Lemma 2.3. *Suppose that , one has*(i)*, where is the degree of in relative to *(ii)* if , then ;*(iii)* if , then ;*(iv)* if , then .*

Lemma 2.4. *Suppose that , one has*(i)* if , then ;*(ii)* if , then .*

Next, we will show some results of existence and nonexistence of positive solutions of (1.2).

Theorem 2.5. * If , then (1.2) has no positive solution; if and , then (1.2) has no nonnegative nonzero solution.** If and (1.2) has a positive solution, then .** If and (1.2) has a positive solution, then .*

* Proof. *(i) First assume that is a positive solution of (1.2), then satisfies
and so by the eigenvalue problem. Due to the comparison principle for eigenvalues, we have , a contradiction. Next, assume that is a nonnegative nonzero solution of (1.2). If and , then by the previous proof. We can also similarly derive when and , which is a contradiction again.

(ii) Assume that is a positive solution of (1.2). Then by (i), and so the positive semitrivial solution exists. Since
is a lower solution of (1.2). By the uniqueness of , . Furthermore, since satisfies the equation
one has , which implies the result.

(iii) Let be a positive solution of (1.2); then exists with as in (ii). Similarly, the given assumption implies the existence of positive solution of (1.2) with . Similar to the proof of (i), we have . This follows since the function has a minimum at and (for and ).

Theorem 2.6. * If and . Then (1.2) has at least a positive solution.** Suppose that . Then (1.2) has positive solution if and only if and .*

* Proof. *(i) By Lemmas 2.3 and 2.4, we have
So (1.2) has at least one positive solution.

(ii) We first prove the sufficiency. Since , (1.2) has no solution taking the form with . If and , note that ; from Lemma 2.4, we have
Hence (1.2) has at least one positive solution.

Conversely, suppose that is a positive solution of (1.2). Then , and . Since satisfies
It follows that .

Theorem 2.7. *If one of the following conditions holds, then (1.2) has no positive solutions:*(i)* and ,*(ii)* and .*

* Proof. *Since the proof of Theorem 2.7 is similar to the proof of Theorem 3 of [19], we omit it.

#### 3. Global Bifurcation and Stability of Positive Solution

In this section, we consider a positive solution bifurcates from the semitrivial nonnegative branch by taking as a bifurcation parameter and fixing . Furthermore, we show that the existence of global bifurcation of (1.2) with respect to parameter and its stability. Moreover, the multiplicity, uniqueness, and stability of positive solutions are obtained by means of perturbation theory of eigenvalues and the fixed point theory.

Let be the principal eigenvalue of the following problem: and is the corresponding eigenfunction with .

Let ; then , and satisfies where

Clearly, is continuous, , and the FrÃ©chet derivative . Let be the inverse of with Dirichlet boundary condition. Then, we have

Define the operator as follows: then is a compact operator on . Let ; then is continuous, and . with if and only if is a nonnegative solution of (1.2).

Lemma 3.1. *Assume that . Then is a bifurcation point of (3.2), and there exist positive solutions of (3.2) in the neighborhood of , where .*

*Remark 3.2. *The proof of Lemma 3.1 is similar to the proof of Theorem 9 in [19]. The proof of Lemma 3.1 shows that there exist and continuous curve such that , and satisfies , where . Hence is a bifurcation solution of (3.2), where , .

If we take , then the nontrivial nonnegative solutions of (1.2) close to are either on the branch or the branch .

Let be a compact continuously differentiable operator such that . Suppose that we can write as , where is a linear compact operator and the FrÃ©chet derivative . If is an isolated fixed point of , then we can define the index if at as , where is a ball with center at such that is the only fixed point of in . If is invertible, then is an isolated fixed point of and . If , then it is well known that the Leray-Schauder degree , where is equal to the algebraic multiplicities of the eigenvalue of which is greater than one.

Next, we will extend the local bifurcation solution given by Lemma 3.1 to the global bifurcation.

Let .

Theorem 3.3. *Suppose that ; then the global bifurcation of bifurcating branch of positive solutions of (1.2) becomes unbounded by going to infinity in .*

* Proof. *Let
Suppose that is an eigenvalue of . Then we have
Clearly , otherwise, , since all eigenvalue of the operator is greater than 0, so , a contradiction. Therefore, for some such that is the eigenvalue of the following problem:
It is well known that is increasing with respect to on and can be ordered as
On the other hand, if , then all eigenvalues of are greater than 0; furthermore, . Thus, is the eigenvalue of if and only if there exists some , such that .

Suppose that . Then for any . Hence, has no eigenvalue greater than 1, and as .

Suppose that . Then for any . Since , , and is increasing with respect to . Hence, there exists a unique , such that . So , where is the principal eigenvalue of the following problem:
where .

In the following, we will prove that . In fact, if the assertion is false, we may assume that . Then there exists , such that , that is,
Multiplying the equation by , integrating over , and using Green's formula, we obtain
which leads to , a contradiction. This proves the assertion, so it is verified that the multiplicity of is one and for . According to global bifurcation theory [12], there exists a continuum of zeros of in bifurcating from , and all zeros of close to lie on the curve whose existence was proved by Lemma 3.1. Let be the maximal continuum defined by . Then, consists of the curve in the neighborhood of the bifurcation point . Let . Then is the solution branch of (1.2) which bifurcates from and remains positive in a small neighborhood of and . Thus the continuum must satisfy one of the following three alternatives.(i) contains in its closure points and , where is not invertible, and .(ii) is joining up from to in .(iii) is containing points of the form and , where .

Next, we prove that . Assume that . Then there exists and sequence such that when . It is easy to get that or . Suppose , then . Hence, we find either such that or such that . Since satisfies
It follows from the maximum principle that . Similarly, we can show that for .

Thus we only need the following three cases:

Suppose that . Then when . Let ; then satisfies
Thanks to estimates and Sobolev embedding theorem, there exists a convergent subsequence of , which we still denote by , such that in as , and because of . So taking the limit in (3.15) as , we get
It follows from the maximum principle that , which implies . This contradicts .

Suppose that . Then as . Let ; then satisfies
Similarly, By estimates and and Sobolev embedding theorem, there exists a convergent subsequence of , which we still denote by , such that in as , and because of . So taking limit in (3.17) as , we obtain
It follows from the maximum principle that . Hence . A contradiction with .

Suppose that . Similar to the previously mentioned, we can get contradiction.

Thus . By Lemma 2.1, we have . Thanks to estimates and and Sobolev embedding theorem, then there exists a constant such that ,. Hence the global bifurcation of positive solutions of (1.2) bifurcating at contains points with is arbitrarily large in .

In the following, we will study the stability of the bifurcation solution. Let , where is the inclusion mapping. Since is the linearized operator at for (1.2). By the proof of Lemma 3.1, we have span, Codim, and . Since , so it follows from [21] that is an -simple eigenvalue of .

Lemma 3.4. * is the eigenvalue of with the largest real part, and all the other eigenvalue of lie in the left half complex plane.*

* Proof. *We assume that is the eigenvalue of with the largest real part , and is the corresponding eigenfunction; then , equivalently,

Suppose that . Then is an eigenvalue of the operator , and then and , a contradiction. Thus . It follows that is an eigenvalue of the operator . Since is the principal eigenvalue of the operator ; furthermore, . This contradicts the assumption, so assumption does not hold. Which proves our conclusion.

We will use the linearized stability theory from [22]. Let , and be the linearized operators of (1.2) at , and , respectively. It follows that from Lemma 7, Corollary 1.13, and Theorem 1.16 [22, 23] that Lemma 3.5 holds.

Lemma 3.5. *There exists -function: , and , defined from the neighborhood of and into , respectively, such that and
**
where . Moreover , whence and have the same sign for . Where is the derivative of with respect to at , and is the derivative of with respect to .*

Lemma 3.6. *The derivative of with respect to at is positive.*

* Proof. * It follows from , that is,
Since , so . Clearly, , otherwise, , then , a contradiction. Hence is an eigenvalue of the operator . We consider , then , as . So is the principal eigenvalue of the operator , and is increasing with respect to as . Moreover, . Hence .

Lemma 3.7. *The derivative of with respect to at satisfies
*

*Proof. *By substituting into (1.2), differentiating with respect to , and then setting , we find that
where is the derivative of with respect to at .

Taking the inner product with , using Green's formula, and noting the definition of , we have

It follows from Lemmas 3.4â€“3.7 that we obtain the following Theorem.

Theorem 3.8. *Let . If , then bifurcation solution is stable; If , then bifurcation solution is unstable.*

In Section 2, from Theorems 2.5â€“2.6, we can obtain the sufficient condition and necessary condition on the existence of positive solution and find that there exists a gap between and when . Next, we will consider the multiplicity, stability, and uniqueness of positive solutions in the gap.

Theorem 3.9. *Assume and . Then there exist such that bifurcation positive solution is nondegenerate and unstable for and . Moreover, Problem (1.2) has at least two positive solutions.*

* Proof. *We first prove that bifurcation positive solution is nondegenerate and unstable. To this end, it suffices to show that there exists a sufficiently small such that for , any positive solution of (1.2) is nondegenerate and the linearized eigenvalue problem:
has a unique eigenvalue and with algebra multiplicity one.

Let and be sequences which approach as . Due to , we can set sequences and such that and as . It follows that is a solution of (1.2). Then the corresponding linearized problem (3.25) can become the following form:
where and
Observe that as ,â€‰â€‰ converges to
It is easy to get that is a simple eigenvalue of the operator with corresponding eigenfunction . Moreover, all the other eigenvalues of are positive and stand apart from . Therefore, using perturbation theory [24], we get that for large has a unique eigenvalue which is close to zero. In addition, all the other eigenvalues of have positive real parts and stand apart from 0. Note that is simple real eigenvalue which converges to zero, and we can take the corresponding eigenfunction such that as . If we show that for large , then the result follows. By multiplying to the first equation of and integrating over , we obtain
Multiplying the first equation of (1.2) with by and integrating, we have
Due to , the previous equation becomes
Using (3.29) and (3.31), we have
Recall that , here is defined in Remark 3.2, and so dividing the previous equation by and taking the limit, we have
which implies that for large . This proved our claim.

Next, we apply the method in [25] to show the remaining part of Theorem 3.9. A contradiction argument will be used; we assume that (1.2) has a unique positive solution , then this solution must be bifurcated from . Since there exists a positive solution near by the local bifurcation theory. So is nondegenerate, and the corresponding linearized eigenvalue problem has a unique eigenvalue with algebra multiplicity one such that . Due to these facts, it is easy to show that is invertible and does not have property on ; it follows that by Lemma 2.2 (ii). Finally, Using Lemmas 2.3â€“2.4 and the additivity property of the index, we obtain
which derives a contradiction. Hence the Proof is complete.

*Remark 3.10. *In Theorem 3.9, the multiplicity can be shown easily when . Note that for a sufficiently small , since , and so . Since there is no positive solution of (1.2) if by Theorem 2.5 (i) when and . Therefore we easily see that there must be at least two positive solutions for and some .

#### 4. Multiplicity, Uniqueness and Stability as or Is Large

In this section, taking or as a parameter, we investigate the multiplicity, stability, and uniqueness of positive solutions of (1.2) as or is large. In the following, we will always assume that , and let be fixed, unless otherwise specified.

Firstly, we consider the case that is large and is bounded away from . Hence the upper solution for and the lower solution for do not depend on when and .

Lemma 4.1. *For any small , there exists such that for , (1.2) has at least one positive solution which satisfies
*

*Proof. *Since the proof of Lemma 4.1 is similar to the proof of Lemma 3 in [19], we omit it.

Lemma 4.2. * If , then any positive solution of (1.2) approaches .** There exists a large such that any positive solution of (1.2) is nondegenerate and linearly stable when .*

* Proof. *(i) Let , we show that the compact operator converges to , where
It follows that any positive solution of (1.2) converges to the fixed point of in this case. It is easy to see that is a unique fixed point of , so the positive solutions of (1.2) could not converge to semitrivial solutions when . Thus the conclusion is complete.

(ii) We use a contradiction method. Assume that there exists , with and with such that
where is a positive solution of (1.2) with . By computing, we have
where are the complex conjugates of . From Lemma 2.1, we know that and . It follows that is bounded and is bounded from the following. Thus is bounded as we assume . Hence we can suppose that and . Thanks to estimate, we obtain and are bounded. Hence we may assume that and in strongly, here