Existence of Positive Solutions of Lotka-Volterra Competition Model with Nonlinear Boundary Conditions

Abstract

A Lotka-Volterra competition model with nonlinear boundary conditions is considered. First, by using upper and lower solutions method for nonlinear boundary problems, we investigate the existence of positive solutions in weak competition case. Next, we prove that −𝑑₁Δ𝑢=𝑢(𝑎₁−𝑏₁𝑢−𝑐₁𝑣), 𝑥∈Ω; −𝑑₂Δ𝑣=𝑣(𝑎₂−𝑏₂𝑢−𝑐₂𝑣), 𝑥∈Ω; 𝜕𝑢/𝜕𝜈+𝑓(𝑢)=0, 𝑥∈𝜕Ω; 𝜕𝑣/𝜕𝜈+𝑔(𝑣)=0, 𝑥∈𝜕Ω, has no positive solution when one of the diffusion coefficients is sufficiently large.

1. Introduction

In this paper, we study the existence of positive solutions to the following problem with nonlinear boundary conditions: −𝑑₁Δ𝑢=𝑢𝑎₁−𝑏₁𝑢−𝑐₁𝑣,𝑥∈Ω,−𝑑₂Δ𝑣=𝑣𝑎₂−𝑏₂𝑢−𝑐₂𝑣,𝑥∈Ω,𝜕𝑢(𝑢)𝜕𝜈+𝑓=0,𝑥∈𝜕Ω,𝜕𝑣(𝑣)𝜕𝜈+𝑔=0,𝑥∈𝜕Ω. (1)

Here,(H1)Ω⊂ℝ^𝑁(𝑁≥1) is an open bounded domain, and 𝜈 is the outward unit normal vector of the boundary 𝜕Ω;(H2)for 𝑖=1,2, 𝑎_𝑖, 𝑏_𝑖, 𝑐_𝑖, and 𝑑_𝑖 are positive constants;(H3)𝑓 and 𝑔 are strictly increasing 𝐶² functions in ℝ and 𝑓(0)=𝑔(0)=𝑓^′(0)=𝑔^′(0)=0;(H4)𝑏₁/𝑏₂>𝑎₁/𝑎₂>𝑐₁/𝑐₂.

In the usual interpretation of the competition model, 𝑢(𝑥) and 𝑣(𝑥) are population variables; it is natural to consider only nonnegative solutions of (1). There is clearly a trivial solution 𝑢=𝑣=0 for all values of the parameters. In addition, for some values of parameters, there exist two semitrivial solutions (𝑢,𝑣)=(𝑢,0) and (0,𝑣). More interesting are the so-called positive solutions or coexistence solutions, where both 𝑢(𝑥) and 𝑣(𝑥) are positive for all 𝑥∈Ω.

By using the positive operator theory, Ahn and Li [1] proved the existence of positive solutions to the following elliptic system: −𝑑₁Δ𝑢=𝑢𝑓(𝑢,𝑣),𝑥∈Ω,−𝑑₂Δ𝑣=𝑣𝑔(𝑢,𝑣),𝑥∈Ω,𝜕𝑢(𝑢)𝜕𝜈+𝛼=0,𝑥∈𝜕Ω,𝜕𝑣(𝑣)𝜕𝜈+𝛽=0,𝑥∈𝜕Ω, (2) where 𝑓(𝑢,𝑣), 𝑔(𝑢,𝑣) are 𝐶¹ functions in 𝑢 and 𝑣, 𝑓_𝑢≤0, 𝑓_𝑣≥0, 𝑔_𝑢≤0, and 𝑔_𝑣≥0 for (𝑢,𝑣)∈ℝ⁺×ℝ⁺; 𝛼 and 𝛽 are increasing functions in ℝ and 𝛼(0)=𝛽(0)=0.

The main project of our paper is to investigate the existence of positive solutions to the problem (1). In Section 2 we state some known results, which are useful throughout this paper. Section 3 is devoted to proving the existence of positive solutions by using the upper and lower solutions method. When the diffusion coefficient 𝑑₁ or 𝑑₂ is sufficiently large, we will prove that problem (1) has no positive solution.

For the homogeneous Neumann boundary conditions, that is, 𝑓(𝑢)≡0 and 𝑔(𝑣)≡0, problem (1) has been studied intensively by many authors. For the related results, please refer to, for instance, [2–8], [9, Section 4.3], and the references cited therein.

2. Preliminaries

In this section, we will introduce some notations and lemmas, which serve as the basic tools for the arguments to prove our results.

Throughout this paper, we will consider the solutions 𝑢,𝑣∈𝐶(Ω). For a given continuous function 𝑞∶Ω→𝑅, let 𝜎₁(𝑞,𝑑) be the principal eigenvalue of the following eigenvalue problem: −𝑑Δ𝑢+𝑞(𝑥)𝑢=𝜆𝑢,𝑥∈Ω,𝜕𝑢𝜕𝜈=0,𝑥∈𝜕Ω. (3) When the diffusion coefficient 𝑑=1, we denote the first eigenvalue for (3) by 𝜎₁(𝑞).

The variational characterization of 𝜎₁(𝑑,𝑞) is 𝜎₁𝑑,𝑞=inf_Ω𝑑∇𝜓²+𝑞(𝑥)𝜓²∶𝑑𝑥_Ω𝜓²𝑑𝑥=1,𝜓∈𝐻₁(Ω). (4) We are concerned with the relation between the sign of 𝜎₁(𝑑,𝑞) and the function 𝑞.

Lemma 1 (see [9]). The first eigenvalue 𝜎₁(𝑑,𝑞) of (3) has the following properties: _Ω𝑞(𝑥)𝑑𝑥≤0⟹𝜎₁𝑑,𝑞<0,∀𝑑>0. (5)

Lemma 2 (see [10]). Suppose that 𝑞(𝑥)∈𝐶(Ω) and 𝑀 is a positive constant satisfying 𝑀−𝑞(𝑥)>0. Then the following hold:(i)𝜎₁(𝑞)<0⇒𝑟[(𝑀−Δ)⁻¹(𝑀−𝑞(𝑥))]>1;(ii)𝜎₁(𝑞)>0⇒𝑟[(𝑀−Δ)⁻¹(𝑀−𝑞(𝑥))]<1;(iii)𝜎₁(𝑞)=0⇒𝑟[(𝑀−Δ)⁻¹(𝑀−𝑞(𝑥))]=1.

Next, we consider the nonlinear elliptic problem: −Δ𝑢=𝐸(𝑥,𝑢),𝑥∈Ω,𝜕𝑢(𝑢)𝜕𝜈+𝑓=0,𝑥∈𝜕Ω, (6) where 𝑓∈𝐶²(ℝ) is strictly increasing with 𝑓(0)=0.

Definition 3. Let 𝐸(𝑥,𝜉)∈𝐶(Ω×ℝ) be global Lipschitz continuous in 𝜉 for all 𝑥∈Ω. The functions 𝑢,𝑢∈𝐶²(Ω)∩𝐶¹(Ω) are called the upper and lower solutions of (6), if 𝑢 and 𝑢 satisfy −Δ𝑢≥𝐸(𝑥,𝑢)𝜕,𝑥∈Ω,𝑢(𝜕𝜈+𝑓𝑢)≥0,𝑥∈𝜕Ω,−Δ𝑢≤𝐸𝑥,𝑢,𝑥∈Ω,𝜕𝑢𝑢𝜕𝜈+𝑓≤0,𝑥∈𝜕Ω. (7)

By using the upper and lower solutions method, the following result was obtained by Ahn and Li [1].

Lemma 4. Suppose that 𝑢≥𝑢≥0 are upper and lower solutions of (6); then there exists a maximal solution 𝑢 of (6) such that 𝑢≥𝑢≥𝑢.

Lemma 5 (see [1]). Let 𝑃,𝑑 be positive constants and ℎ∈𝐶(Ω). Consider −𝑑Δ𝑢+𝑃𝑢=ℎ,𝑥∈Ω,𝜕𝑢(𝑢)𝜕𝜈+𝑓=0,𝑥∈𝜕Ω, (8) where 𝑓∈𝐶²(ℝ), 𝑓(0)=0, and 𝑓 is strictly increasing. Then the following hold:(i)problem (8) has a solution 𝑢∈𝑊^2,𝑚(Ω)∩𝐶^1,𝛼(Ω) for some 𝛼∈(0,1), and ‖𝑢‖_𝑊^2,𝑚≤𝐶₀‖ℎ‖_∞, (9) where 𝐶₀ is dependent on 𝑃;(ii)if 0≢ℎ≥0, then (8) has a unique positive solution.

Now we consider the following nonlinear boundary value problem: −Δ𝑢=𝑢𝐹(𝑥,𝑢),𝑥∈Ω,𝜕𝑢(𝑢)𝜕𝜈+𝑓=0,𝑥∈𝜕Ω. (10)

Lemma 6. Let 𝑓∈𝐶²(ℝ) be an increasing and convex function in ℝ⁺, and satisfy with 𝑓(0)=𝑓^′(0)=0. Assume that the function 𝐹 satisfies the following:(i)𝐹∈𝐶(Ω×ℝ⁺), 𝐹(𝑥,𝑢) is Lipschitz-continuous in 𝑢, and the Lipschitz constant 𝐶 is independent of (𝑥,𝑢)∈Ω×ℝ⁺;(ii)𝐹(𝑥,𝑢) is decreasing in 𝑢;(iii)𝐹(𝑥,0)>0 for 𝑥∈Ω, 𝐹(𝑥,𝑢)<0 in Ω×(𝑐₀,∞) for some constant 𝑐₀>0.
If 𝜎₁(−𝐹(𝑥,0))<0, then (10) has a unique positive solution. If 𝜎₁(−𝐹(𝑥,0))≥0, then 𝑢=0 is the only nonnegative solution of (10).

Proof. Let 𝜓>0 be the eigenfunction corresponding to the eigenvalue 𝜎₁(0)=0 of problem (3); we can obtain 𝐹(𝑥,𝜉)<𝜎₁ when 𝜉>𝑐₀, due to the hypothesis (iii). With the assumptions on the function 𝑓 in mind, for a large 𝑀>0, we have −Δ𝑀𝜓−𝑀𝜓𝐹𝑥,𝑀𝜓≥0,𝑥∈Ω,𝜕𝑀𝜓𝜕𝜈+𝑓𝑀𝜓≥0,𝑥∈𝜕Ω. (11) Therefore 𝑇(𝑀𝜓)≤𝑀𝜓, where the operator 𝑇𝑢∶=(−Δ⋅+𝑃⋅)(𝐹(𝑥,𝑢)𝑢+𝑃𝑢) is defined by the problem (8). 𝑇 is compact in the positive cone 𝐾⊂𝐶(Ω) by Lemma 5. The function 𝐹(𝑥,⋅)⋅+𝑃⋅ is monotone increasing on [0,‖𝑀𝜓‖_∞] for sufficiently large 𝑃>0. Therefore 𝑇 is increasing on the order interval (0,𝑀𝜓) with 𝑇(0)=0. Taking advantage of [1, Lemma 2.13], we know 𝑇^′(0)=(−Δ+𝑃)[𝐹(𝑥,0)+𝑃]. The spectral radius 𝑟(𝑇^′(0))>1 by Lemma 2. Then, the result of [11, Theorem 7.6] ensures our conclusion.
Next we show the uniqueness. Suppose 𝑢₁ is a positive solution of (10). Let 𝑢 be a maximal solution of (10). We claim that 𝑢=𝑢₁.
Suppose 𝑢≥𝑢₁≢𝑢. Then 0<_𝜕Ω𝑢₁𝑓(𝑢)−𝑢𝑓𝑢₁𝑑𝑆=_Ω−𝑢₁Δ𝑢+𝑢Δ𝑢₁=𝑑𝑥_Ω𝑢𝑢₁𝐹(𝑥,𝑢)−𝐹𝑥,𝑢₁𝑑𝑥≤0. (12) The first integral is positive, as 𝑓 is convex and 𝑓(0)=0. The last integral is nonpositive, since 𝐹(𝑥,𝑢) is decreasing in 𝑢. This contradiction demonstrates 𝑢≡𝑢₁. If 𝜆₁(−𝐹(𝑥,0))≥0, the proof is similar to [1, Lemma 2.17], and we omit it. This completes the proof.

By Lemma 5, we are able to conclude the following result.

Proposition 7. Suppose that 𝑓,𝑔 are convex functions in ℝ⁺. Then (1) has two semitrivial solutions (𝜃₁,0) and (0,𝜃₂), for some 0<𝛼<1, where 𝜃₁∈𝐶^1,𝛼(Ω) is the unique positive solution of −𝑑₁Δ𝑢=𝑢𝑎₁−𝑏₁𝑢,𝑥∈Ω,𝜕𝑢(𝑢)𝜕𝜈+𝑓=0,𝑥∈𝜕Ω, (13) and 𝜃₂∈𝐶^1,𝛼(Ω) is the unique positive solution of −𝑑₂Δ𝑣=𝑣𝑎₂−𝑐₂𝑣,𝑥∈Ω,𝜕𝑣(𝑣)𝜕𝜈+𝑔=0,𝑥∈𝜕Ω. (14)

Finally, we cite a strong maximum principle (see [5, Proposition 2.2]).

Lemma 8. Suppose that Ω is smooth and 𝑔∈𝐶(Ω×ℝ¹). Assume that 𝑧∈𝐶²(Ω)∩𝐶¹(Ω) and satisfies Δ𝑧(𝑥)+𝑔(𝑥,𝑧(𝑥))≥0,𝑥∈Ω,𝜕𝑧𝜕𝜈≤0,𝑥∈𝜕Ω. (15) If 𝑧(𝑥₀)=max_Ω𝑧(𝑥), then 𝑔(𝑥₀,𝑧(𝑥₀))≥0.

3. Existence and Nonexistence of Positive Solution

By using the upper-lower solutions argument for nonlinear boundary problems, we first study the existence of positive solutions to (1). Our method is technically and conceptually simple in the proof of existence results involving upper-lower solutions hypotheses and Leray-Schauder continuation argument. Next, we prove that problem (1) has no positive solution, if one of the diffusion coefficients is sufficiently large. Finally, we discuss the stability of semitrivial solutions.

We will show that the positive solutions of systems (1) have a priori bound.

Lemma 9. Suppose that (𝑢,𝑣) is a positive solution of (1). Then 𝑢≤𝑎₁/𝑏₁ and 𝑣≤𝑎₂/𝑐₂.

Proof. In view of the first equation of (1), 𝑢 satisfies −𝑑₁Δ𝑢−𝑢𝑎₁−𝑏₁𝑢−𝑐₁𝑣=0,𝑥∈Ω,𝜕𝑢(𝑢)𝜕𝜈+𝑓=0,𝑥∈𝜕Ω. (16) As 𝑢 is continuous on the compact set Ω and 𝑓(𝑢)≥0, thanks to Lemma 8, it is easy to know 𝑢≤𝑎₁/𝑏₁. In a similar manner, we obtain 𝑣≤𝑎₂/𝑐₂.
Next, we give the definitions of upper and lower solutions to (1).

Definition 10. Assume that (𝑢,𝑣),(𝑢,𝑣)∈𝐶¹(Ω)∩𝐶²(Ω). We called that (𝑢,𝑣) and (𝑢,𝑣) are the coupled upper and lower solutions of (1), if (𝑢,𝑣) and (𝑢,𝑣) satisfy −𝑑₁Δ𝑢−𝑢𝑎₁−𝑏₁𝑢−𝑐₁𝑣≥0,𝑥∈Ω,−𝑑₂Δ𝑣−𝑣𝑎₂−𝑏₂𝑢−𝑐₂𝑣𝜕≥0,𝑥∈Ω,𝑢(𝜕𝜈+𝑓𝑢)𝜕≥0,𝑥∈𝜕Ω,𝑣(𝜕𝜈+𝑔𝑣)≥0,𝑥∈𝜕Ω,−𝑑₁Δ𝑢−𝑢𝑎₁−𝑏₁𝑢−𝑐₁𝑣≤0,𝑥∈Ω,−𝑑₂Δ𝑣−𝑣𝑎₂−𝑏₂𝑢−𝑐₂𝑣≤0,𝑥∈Ω,𝜕𝑢𝑢𝜕𝜈+𝑓≤0,𝑥∈𝜕Ω,𝜕𝑣𝑣𝜕𝜈+𝑔≤0,𝑥∈𝜕Ω. (17)

Theorem 11. Suppose that (𝑢,𝑣) and (𝑢,𝑣) are the coupled upper and lower solutions of (1) and (𝑢,𝑣)≥(𝑢,𝑣). Then (1) has at least one solution (𝑢,𝑣) and (𝑢,𝑣)≥(𝑢,𝑣)≥(𝑢,𝑣).

Proof. For any given 𝑊∶=(𝑤₁,𝑤₂)∈[𝐶(Ω)]² and a sufficiently large positive constant 𝑀, let 𝐹¹(𝑥)=𝑤₁𝑎₁−𝑏₁𝑤₁−𝑐₁𝑤₂+𝑀𝑤₁,𝐹²(𝑥)=𝑤₂𝑎₂−𝑏₂𝑤₁−𝑐₂𝑤₂+𝑀𝑤₂. (18) Consider the following problem: −𝑑₁Δ𝑢+𝑀𝑢=𝑤₁𝑎₁−𝑏₁𝑤₁−𝑐₁𝑤₂+𝑀𝑤₁,𝑥∈Ω,𝜕𝑢(𝑢)𝜕𝜈+𝑓=0,𝑥∈𝜕Ω. (19) Since 𝐹¹(𝑥)∈𝐶(Ω), we see that (19) admits a unique solution 𝑢∈𝐶^1+𝛼(Ω) by Lemma 5. Similarly, the problem −𝑑₂Δ𝑣+𝑀𝑣=𝑤₂𝑎₂−𝑏₂𝑤₁−𝑐₂𝑤₂+𝑀𝑤₂,𝑥∈Ω,𝜕𝑣(𝑣)𝜕𝜈+𝑔=0,𝑥∈𝜕Ω (20) has a unique solution 𝑣∈𝐶^1+𝛼(Ω). Denote 𝑈=(𝑢,𝑣), 𝑈=(𝑢,𝑣), and 𝑈=(𝑢,𝑣). We define the ordered interval 𝐴=𝑈∈𝐶Ω²∶𝑈≤𝑈≤𝑈 (21) and an operator 𝑇∶𝐴→[𝐶(Ω)]² by 𝑈=𝑇𝑊. (22) Thanks to Lemma 5, we have ‖𝑢‖_𝐶^1,𝛼_(Ω)≤𝐶𝑤₁(𝑎₁−𝑏₁𝑤₁−𝑐₁𝑤₂)_∞+𝑤_1∞,‖𝑣‖_𝐶^1,𝛼_(Ω)≤𝐶𝑤₂(𝑎₂−𝑏₂𝑤₁−𝑐₂𝑤₂)_∞+𝑤_2∞. (23) Hence, 𝑇(𝐴) is bounded in [𝐶(Ω)]².
We claim that 𝑇∶𝐴→[𝐶(Ω)]² is a compact operator. To see this, it suffices to prove that the operator 𝑇 is continuous. Suppose that 𝑊_𝑛=(𝑤_1𝑛,𝑤_2𝑛)→𝑊=(𝑤₁,𝑤₂) in [𝐶(Ω)]². Denote 𝐹¹_𝑛(𝑥)=𝑤_1𝑛𝑎₁−𝑏₁𝑤_1𝑛−𝑐₁𝑤_2𝑛+𝑀𝑤_1𝑛,𝐹²_𝑛(𝑥)=𝑤_2𝑛𝑎₂−𝑏₂𝑤_1𝑛−𝑐₂𝑤_2𝑛+𝑀𝑤_2𝑛. (24) Then 𝐹_𝑛=(𝐹¹_𝑛,𝐹²_𝑛)→𝐹=(𝐹¹,𝐹²) in [𝐶(Ω)]². Let 𝑈_𝑛=𝑇𝑊_𝑛. By Lemma 5, we obtain 𝑢_{𝑛𝑊2,𝑝(Ω)}≤𝐶𝐹¹_𝑛(𝑥)−𝐹¹(𝑥)_∞+𝐹¹(𝑥)_∞,𝑣_{𝑛𝑊2,𝑝(Ω)}≤𝐶𝐹²_𝑛(𝑥)−𝐹²(𝑥)_∞+𝐹²(𝑥)_∞. (25) Therefore, 𝑈_𝑛⇀𝑈 in [𝑊^2,𝑝(Ω)]². Note that 𝑊^2,𝑝(Ω)↪𝐶^1+𝛼(Ω) is compact; it is deduced that 𝑈_𝑛→𝑈 in [𝐶^1+𝛼(Ω)]². It is obvious that 𝑈=(𝑢,𝑣) is the solution of −𝑑₁Δ𝑢+𝑀𝑢=𝑢𝑎₁−𝑏₁𝑤₁−𝑐₁𝑤₂+𝑀𝑤₁,𝑥∈Ω,−𝑑₂Δ𝑣+𝑀𝑣=𝑣𝑎₂−𝑏₂𝑤₁−𝑐₂𝑤₂+𝑀𝑤₂,𝑥∈Ω,𝜕𝑢(𝑢)𝜕𝜈+𝑓=0,𝑥∈𝜕Ω,𝜕𝑣(𝑣)𝜕𝜈+𝑔=0,𝑥∈𝜕Ω. (26) This shows that 𝑇∶𝐴→[𝐶(Ω)]² is continuous.
Now, we would like to prove 𝑇(𝐴)⊂𝐴. Suppose that 𝑊∈𝐴 and 𝑈=𝑇𝑊, where 𝑈=(𝑢,𝑣), 𝑊=(𝑤₁,𝑤₂). We first prove 𝑢≥𝑢. In virtue of (𝑤₁,𝑤₂)∈𝐴, we have that −𝑑₁Δ𝑢+𝑀𝑢≥𝑢𝑎₁−𝑏₁𝑢−𝑐₁𝑤₂+𝑀𝜕𝑢,𝑥∈Ω,𝑢(𝜕𝜈+𝑓𝑢)≥0,𝑥∈𝜕Ω. (27) Let 𝑧=𝑢−𝑢. Noting that 𝑢≥𝑤₁ and 𝑢 satisfies (19), it is obvious that −𝑑₁Δ𝑧+𝑀𝑧≥𝑢𝑎₁−𝑏₁𝑢−𝑐₁𝑤₂−𝑤₁𝑎₁−𝑏₁𝑤₁−𝑐₁𝑤₂+𝑀𝑢−𝑤₁≥0,𝑥∈Ω,𝜕𝑧(𝜕𝜈+𝑓𝑢)−𝑓(𝑢)≥0,𝑥∈𝜕Ω. (28) On the contrary, we assume that 𝑢≥𝑢 is not true. By the strong maximum theory (see [12]), there exists 𝑥₀∈𝜕Ω, such that min_𝑥∈Ω𝑧(𝑥)=𝑧(𝑥₀)=𝑢(𝑥₀)−𝑢(𝑥₀)<0. Thanks to the Hopf boundary lemma, we know 𝜕𝑧𝜕𝜈_𝑥=𝑥0<0. (29) In view of (28), we get 𝜕𝑧𝜕𝜈_𝑥=𝑥0≥−𝑓𝑢𝑥₀−𝑓𝑢𝑥₀>0. (30) This is a contradiction with (29). Thus 𝑧≥0; that is, 𝑢≥𝑢. Similarly, 𝑢≤𝑢, 𝑣≥𝑣, and 𝑣≤𝑣. By the Schauder fixed point theorem, 𝑇 has a fixed point 𝑈 in 𝐴. The proof is complete.

Theorem 12. Suppose that 𝑏₁/𝑏₂>𝑎₁/𝑎₂>𝑐₁/𝑐₂ and 𝑓,𝑔 are convex functions in ℝ⁺. Then the problem (1) has at least one positive solution.

Proof. By the assumption 𝑓^′(0)=𝑔^′(0)=0, there exists a constant 𝛽>0 such that lim_𝑠→0𝑓(𝑠)𝑠^1+𝛽=0,lim_𝑠→0𝑔(𝑠)𝑠^1+𝛽=0. (31)
Let 𝜃₁,𝜃₂∈𝐶^1,𝛼(Ω) be the unique positive solutions of (13) and (14), respectively. Set 𝑈=(𝑢,𝑣)=(𝜃₁,𝜃₂), 𝑈=(𝑢,𝑣)=(𝜀^1+𝛽𝜃₁+𝜀,𝜀^1+𝛽𝜃₂+𝜀), where 𝜀>0 is sufficiently small. Then 𝑈≤𝑈 as 𝜃₁ and 𝜃₂ are positive on Ω. To prove that 𝑈 and 𝑈 are the coupled upper and lower solutions of (1), it suffices to verify inequalities (17) in Definition 10. Consider the following.(i)Thanks to 𝜃₁,𝜃₂>0 on Ω, the following are obvious provided that 𝜀>0 is sufficiently small: −𝑑₁Δ𝑢=𝑢𝑎₁−𝑏₁𝑢≥𝑎₁−𝑏₁𝑢−𝑐₁𝑣,𝑥∈Ω,−𝑑₂Δ𝑣=𝑎₂−𝑐₂𝑣≥𝑎_𝑐−𝑐₂𝑣−𝑏₂𝑢𝜕,𝑥∈Ω,𝑢(𝜕𝜈+𝑓𝑢)𝜕=0,𝑣(𝜕𝜈+𝑔𝑣)=0,𝑥∈𝜕Ω. (32)(ii)By Lemma 8, we have that 𝜃₁(𝑥)≤𝑎₁/𝑏₁ and 𝜃(𝑥)≤𝑎₂/𝑐₂ on Ω. In virtue of 𝑎₁,𝑎₂>0 and 𝑏₁/𝑏₂>𝑎₁/𝑎₂>𝑐₁/𝑐₂, a direct computation gives −𝑑₁Δ𝑢−𝑎₁−𝑏₁𝑢−𝑐₁𝑣=−𝑑₁Δ𝜀^1+𝛽𝜃₁−𝜀+𝜀_1+𝛽𝜃₁+𝜀𝑎₁−𝑏₁𝜀^1+𝛽𝜃₁+𝜀−𝑐₁𝜃₂=𝜀^1+𝛽𝜃₁𝑎₁−𝑏₁𝜃₁−𝜀^1+𝛽𝜃₁+𝜀𝑎₁−𝑏₁𝜀^1+𝛽𝜃₁+𝜀−𝑐₁𝜃₂=−𝜀𝑎₁−𝑐₁𝜃₂(𝜀)𝑎+𝑜≤−𝜀¹−𝑐₁𝑎₂𝑐₂+𝑜(𝜀)<0,𝑥∈Ω (33)
provided 0<𝜀≪1. Since the function 𝑓 is convex in ℝ⁺, we know that, when 0<𝜀≪1, 𝜕𝑢𝑢𝜕𝜈+𝑓=𝜀^1+𝛽𝜕𝜃₁𝜀𝜕𝜈+𝑓^1+𝛽𝜃₁+𝜀−𝜀^1+𝛽𝑓𝜃₁+𝑓𝜀^1+𝛽𝜃₁=𝑓𝜀+𝜀_1+𝛽𝜃₁+𝜀𝜀^1+𝛽𝜃₁+𝜀^1+𝛽𝜀^1+𝛽𝜃₁+𝜀^1+𝛽𝜀^1+𝛽−𝑓𝜃₁≤0,𝑥∈𝜕Ω. (34) Similarly, −𝑑₁Δ𝑣−𝑎₁−𝑏₁𝑢−𝑐₁𝑣<0,𝑥∈Ω,𝜕𝑣𝑣𝜕𝜈+𝑓≤0,𝑥∈𝜕Ω. (35)
We have proved that 𝑈,𝑈 are the coupled upper and lower solutions of (1). Taking advantage of Theorem 11, (1) has at least one positive solution. The proof is complete.

Next, we show that (1) has no positive solution, when the diffusion coefficient 𝑑₁ or 𝑑₂ is sufficiently large.

Theorem 13. There exists a positive constant 𝑀=𝑀(𝑎_𝑖,𝑏_𝑖,𝑐_𝑖) such that when max{𝑑₁,𝑑₂}≥𝑀, problem (1) has no positive solution.

Proof. There exists a positive constant 𝐶, independent of 𝑢 and 𝑣, such that 𝑢−𝑢^∗_∞≤𝐶𝑑₁,𝑣−𝑣^∗_∞≤𝐶𝑑₂, (36) where 𝑢^∗=∫_𝜕Ω𝑓(𝑢)𝑢∫_𝜕Ω𝑓(𝑢),𝑣^∗=∫_𝜕Ω𝑔(𝑣)𝑣∫_𝜕Ω𝑔(𝑣). (37) Following the results of Lemma 9, we have max‖𝑢‖_∞,‖𝑣‖_∞𝑎≤𝐶≡max¹𝑏₁,𝑎₂𝑏₂. (38) Rewrite (1) as −𝑑₁Δ𝑢−𝑢^∗=𝑢𝑎₁−𝑏₁𝑢−𝑐₁𝑣,𝑥∈Ω,−𝑑₂Δ𝑣−𝑣^∗=𝑣𝑎₂−𝑏₂𝑢−𝑐₂𝑣𝜕,𝑥∈Ω,𝑢−𝑢^∗(𝑢)𝜕𝜕𝜈+𝑓=0,𝑥∈𝜕Ω,𝑣−𝑣^∗(𝑣)𝜕𝜈+𝑔=0,𝑥∈𝜕Ω. (39) Note that ‖𝑢(𝑎₁−𝑏₁𝑢−𝑐₁𝑣)‖_∞≤𝐶=max_{0≤𝑢,𝑣≤𝐶}|𝑢(𝑎₁−𝑏₁𝑢−𝑐₁𝑣)|. Multiplying the first equation of (39) by 𝑢−𝑢^∗, and integrating on Ω, we derive that, by Green’s identity, Holder’s inequality, and Poincare’s inequality, _Ω|∇𝑢|²≤𝑢𝑎₁−𝑏₁𝑢−𝑐₁𝑣𝑑_1Ω𝑢−𝑢^∗≤𝐶𝑑₁𝑢−𝑢^∗₂≤𝐶𝑑₁‖∇𝑢‖₂, (40) which implies that 𝑢−𝑢^∗₂≤𝐶𝑑₁. (41) By Lemma 5, (36), and (41), we obtain 𝑢−𝑢^∗_𝑊2,2(Ω)≤𝐶𝑢−𝑢^∗₂+𝑢𝑎₁−𝑏₁𝑢−𝑐₁𝑣_∞𝑑₁≤𝐶𝑑₁. (42) Thanks to the Sobolev embedding theorem (see [12]), for 0<𝛽<1, 𝑢−𝑢^∗_{𝐶1,𝛽(Ω)}≤𝐶𝑑₁. (43) Review to the third equation of (1), we obtain |𝐷𝑢||𝜈|≥|𝐷𝑢⋅𝜈|=𝑓(𝑢). (44) In view of (43) and (44), it is easy to see 𝐶𝑑₁≥𝑓(𝑢)>𝑓𝑢^∗−𝐶𝑑₁. (45) Note that 𝑓(𝑢) is increasing function and 𝑓(0)=0, as 𝑑₁→∞, and 0≥𝑓𝑢^∗>0. (46) This is a contradiction. The same proof as above works equally well for the case when 𝑑₂ is large. This completes the proof.

Finally, we discuss the stability of semitrivial solutions.

Theorem 14. Suppose that 𝑓,𝑔 are convex functions in ℝ⁺. Then the following hold:(a)the semitrivial solution (𝜃₁,0) is unstable;(b)the semitrivial solution (0,𝜃₂) is unstable.

Proof. For part (a), to prove the stability of (𝜃₁,0), we consider the corresponding elliptic system eigenvalue problem: −𝑑₁Δ𝑢−𝑎₁−2𝑏₁𝜃₁𝑢+𝑐₁𝜃₁𝑣=𝜆𝑢,𝑥∈Ω,−𝑑₂Δ𝑣−𝑎₂−𝑏₂𝜃₁𝑣=𝜆𝑣,𝑥∈Ω,𝜕𝑢𝜕𝜈+𝑓^′𝜃₁𝑢=0,𝑥∈𝜕Ω,𝜕𝑣𝜕𝜈=0,𝑥∈𝜕Ω. (47) First, observe that all eigenvalues of (47) are real since they are also eigenvalues of the second equation: −𝑑₂Δ𝑣−𝑎₂−𝑏₂𝜃₁𝑣=𝜆𝑣,𝑥∈Ω,𝜕𝑣𝜕𝑛=0,𝑥∈𝜕Ω. (48) Next, note that 𝑏₁/𝑏₂>𝑎₁/𝑎₂>𝑐₁/𝑐₂; by Lemma 1, we have 𝜎₁(𝑑₂,𝑞)<0. Thus (𝜃₁,0) is unstable. In a similar way, we are able to conclude (b).

Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.