#### Abstract

We investigate the global dynamics of solutions of four distinct competitive rational systems of difference equations in the plane. We show that the basins of attractions of different locally asymptotically stable equilibrium points are separated by the global stable manifolds of either saddle points or nonhyperbolic equilibrium points. Our results give complete answer to Open Problem 2 posed recently by Camouzis et al. (2009).

#### 1. Introduction and Preliminaries

We consider the following open problem (see [1, Open Problem ]).

*For**each of the following four distinct systems*

*determine the following:*

*the boundedness character of its solutions,*(ii)

*the local stability of its equilibrium points,*(iii)

*the existence of prime period-two solutions,*(iv)

*the global character of the systems.*

Equation (3.4) is of the form

equation (3.5) is of the form

equation (3.16) is either of the form

or the form

depending on whether it appears as first or second equation in the system; equation (38) is of the form

The typical results are the following theorems. The first theorem is a combination of Theorems 2.3 and 2.5 and the second theorem is Theorem 3.3.

Theorem 1.1. *Consider system (14, 21) and assume that . If , then there exists a set which is invariant and a subset of the basin of attraction of . The set is a graph of a strictly increasing continuous function of the first variable on an interval (and so is a manifold) and separates into two connected and invariant components, namely,
**
which satisfy
** Assume that . Every solution of system (14, 21), with satisfies*

Theorem 1.2. *Consider system (21, 21). There exists a set which is invariant and a subset of the basin of attraction of the unique equilibrium . The set is a graph of a strictly increasing continuous function of the first variable on an interval (and so is a manifold) and separates into two connected and invariant components, namely,
**
which satisfy
*

All considered systems are competitive systems, which we discuss next.

A first-order system of difference equations

where , , , are continuous functions, is *competitive* if is nondecreasing in and nonincreasing in , and is nonincreasing in and nondecreasing in . If both and are nondecreasing in and , the system (1.12) is *cooperative*. A map that corresponds to the system (1.12) is defined as . Competitive and cooperative maps, which are called monotone maps, are defined similarly. *Strongly competitive* systems of difference equations or maps are those for which the functions and are coordinatewise strictly monotone.

If , we denote with , , the four quadrants in relative to , that is, , , and so on. Define the *South-East* partial order on by if and only if and . Similarly, we define the *North-East* partial order on by if and only if and . For and , define the *distance from ** to * as . By we denote the interior a set .

It is easy to show that a map is competitive if it is nondecreasing with respect to the South-East partial order, that is, if the following holds:

Competitive systems were studied by many authors; see [2–17], and others. All known results, with the exception of [2, 3, 18], deal with hyperbolic dynamics. The results presented here are results that hold in both the hyperbolic and the nonhyperbolic case.

We now state three results for competitive maps in the plane. The following definition is from [17].

*Definition 1.3. *Let be a nonempty subset of . A competitive map is said to satisfy condition () if for every , in , implies , and is said to satisfy condition () if for every , in , implies .

The following theorem was proved by DeMottoni-Schiaffino for the Poincaré map of a periodic competitive Lotka-Volterra system of differential equations. Smith generalized the proof to competitive and cooperative maps [14, 15].

Theorem 1.4. *Let be a nonempty subset of . If is a competitive map for which () holds then for all , is eventually componentwise monotone. If the orbit of has compact closure, then it converges to a fixed point of . If instead () holds, then for all , is eventually componentwise monotone. If the orbit of has compact closure in , then its omega limit set is either a period-two orbit or a fixed point.*

The following result is from [17], with the domain of the map specialized to be the cartesian product of intervals of real numbers. It gives a sufficient condition for conditions () and ().

Theorem 1.5. *Let be the cartesian product of two intervals in . Let be a (continuously differentiable) competitive map. If is injective and for all then satisfies (). If is injective and for all then satisfies ().*

The next results are the modifications of [8, Theorem ]. See [18].

Theorem 1.6. *Let be a monotone map on a closed and bounded rectangular region Suppose that has a unique fixed point in Then is a global attractor on *

The following four results were proved by Kulenović and Merino [18] for competitive systems in the plane, when one of the eigenvalues of the linearized system at an equilibrium (hyperbolic or nonhyperbolic) is by absolute value smaller than while the other has an arbitrary value. These results are useful for determining basins of attraction of fixed points of competitive maps.

Our first result gives conditions for the existence of a global invariant curve through a fixed point (hyperbolic or not) of a competitive map that is differentiable in a neighborhood of the fixed point, when at least one of two nonzero eigenvalues of the Jacobian matrix of the map at the fixed point has absolute value less than one. A region is *rectangular* if it is the cartesian product of two intervals in .

Theorem 1.7. *Let be a competitive map on a rectangular region . Let be a fixed point of such that is nonempty (i.e., is not the NW or SE vertex of , and is strongly competitive on . Suppose that the following statements are true. *(a)*The map has a extension to a neighborhood of . *(b)*The Jacobian matrix of at has real eigenvalues , such that , where , and the eigenspace associated with is not a coordinate axis. **Then there exists a curve through that is invariant and a subset of the basin of attraction of , such that is tangential to the eigenspace at , and is the graph of a strictly increasing continuous function of the first coordinate on an interval. Any endpoints of in the interior of are either fixed points or minimal period-two points. In the latter case, the set of endpoints of is a minimal period-two orbit of .*

Corollary 1.8. *If has no fixed point nor periodic points of minimal period-two in , then the endpoints of belong to . *

For maps that are strongly competitive near the fixed point, hypothesis (b) of Theorem 1.7 reduces just to . This follows from a change of variables [17] that allows the Perron-Frobenius Theorem to be applied to give that at any point, the Jacobian matrix of a strongly competitive map has two real and distinct eigenvalues, the larger one in absolute value being positive, and that corresponding eigenvectors may be chosen to point in the direction of the second and first quadrant, respectively. Also, one can show that in such case no associated eigenvector is aligned with a coordinate axis.

The following result gives a description of the global stable and unstable manifolds of a saddle point of a competitive map. The result is a modification of [8, Theorem ].

Theorem 1.9. *In addition to the hypotheses of Theorem 1.7, suppose that and that the eigenspace associated with is not a coordinate axis. If the curve of Theorem 1.7 has endpoints in , then is the global stable manifold of , and the global unstable manifold is a curve in that is tangential to at and such that it is the graph of a strictly decreasing function of the first coordinate on an interval. Any endpoints of in are fixed points of .*

The next result is useful for determining basins of attraction of fixed points of competitive maps.

Theorem 1.10. *Assume the hypotheses of Theorem 1.7, and let be the curve whose existence is guaranteed by Theorem 1.7. If the endpoints of belong to , then separates into two connected components, namely,
**
such that the following statements are true. *(i)* is invariant, and as for every . *(ii)* is invariant, and as for every . **If, in addition, is an interior point of and is and strongly competitive in a neighborhood of , then has no periodic points in the boundary of except for , and the following statements are true. *(i)*For every there exists such that for . *(ii)*For every there exists such that for . *

In this paper we study the global dynamics of four rational systems of difference equations mentioned earlier, where all parameters are positive numbers and initial conditions and are arbitrary nonnegative numbers. Two of these systems have a nonhyperbolic semistable equilibrium point. In general all four systems share the common feature that the global stable manifolds of either saddle points or nonhyperbolic equilibrium points serve as boundaries of basins of attraction of different local attractors or points at infinities. The techniques used here can be applied to treat number of competitive systems which appear in applications, such as Leslie-Gower competition model, see [19], or Leslie-Gower competition model with stocking, see [20], or genetic model, see [13]. An important new feature of our techniques is that they are applicable to nonhyperbolic case as well, which was shown for the first time in [18] where we have completed analysis of basic Leslie-Gower competition model from [19]. Furthermore, system can be considered as a variant of Leslie-Gower competition model, where the first equation has been replaced by another equation, which does not allow extinction of both species. In fact, all four considered competitive systems share common feature that they do not allow the extinction of both species.

#### 2. System (14,21)

Now we consider the following system of difference equations:

where the parameters , and are positive numbers and initial conditions .

System (2.1) was considered in [1, Example ], where it was shown that the associated map is injective and

When , . Therefore, in view of Theorems 1.4 and 1.5 every solution of system (2.1) is eventually componentwise monotonic. If then and four subsequences

of every solution of system (2.1) are eventually monotonic.

Thus, if , the Jacobian matrix of in is invertible.

The Jacobian matrix of the corresponding map is of the form

##### 2.1. Linearized Stability Analysis

The equilibrium points of system (2.1) are solutions of the system of equations

from which we obtain

Lemma 2.1. *(i) If then system (2.1) has a unique equilibrium point:
**
which is a saddle point. **(ii) If then system (2.1) has no equilibrium points.*

*Proof. *By (2.6) and (2.4) the Jacobian matrix evaluated at the equilibrium point has the form
The corresponding characteristic equation evaluated at the equilibrium point is
where

Notice that in view of (2.6) and so

Since and , we need to show(I)(II)

Indeed,

(I)which is satisfied (because and ). Furthermore(II)which is satisfied.

##### 2.2. Global Results

###### 2.2.1. Case

Theorem 2.2. *System (2.1) has no prime period-two solutions. *

*Proof. *System (2.1) can be reduced to the following second-order difference equation:
or to the following second-order difference equation:
Now it is sufficient to prove that both of the difference equations (2.12) and (2.13) have no prime period-two solutions. Assume that this is not true for (2.12), that is, that
is a prime period-two solution of (2.12). Then we have
This implies
By subtraction, we obtain
that is,
and this implies that , which is a contradiction.

Now assume that

is a prime period-two solution of (2.13). Then we have
from which
and this implies that , which is a contradiction.

Theorem 2.3. *Consider system (2.1) and assume that and . Then there exists a set which is invariant and a subset of the basin of attraction of . The set is a graph of a strictly increasing continuous function of the first variable on an interval (and so is a manifold) and separates into two connected and invariant components, namely,
**
which satisfy
*

*Proof. *Clearly, system (2.1) is strongly competitive on . In view of Theorem 2.2 we see that all conditions of Theorems 1.7, 1.9, and 1.10 and Corollary 1.8 are satisfied with and so the conclusion follows.

*Remark 2.4 (see [1]). *If , then system (2.1) can be decoupled as follows:
and every solution of this system (depending of the choice of the initial condition ) is either bounded and converges to an equilibrium point or increases monotonically to infinity.

###### 2.2.2. Case

In this case system (2.1) has no equilibrium points. Now we have the following.

Theorem 2.5. *Assume that and . Every solution of system (2.1), with satisfies
*

*Proof. *If , then
which implies

On the other hand, if , then , and we obtain that the sequence is strictly decreasing. Because for all , we see that is convergent and , since otherwise, that is, the first equation of system (2.1) implies or the second equation of system (2.1) implies , which is a contradiction, since otherwise system (2.1) would have an equilibrium point in the first quadrant.

We see that if then every solution of system (2.1) satisfies

But then the denominator in

is, for all large , strictly less than a constant , which in turn implies
Iterating this inequality we obtain
and this forces to infinity.

The obtained results lead to the following characterization of the boundedness of solutions of system (2.1).

Corollary 2.6. *Consider system (2.1) subject to the condition . If , then all bounded solutions converge to the unique equilibrium with the corresponding initial conditions belonging to the graph of a continuous increasing function in the plane of initial conditions. All solutions that start in the complement of are asymptotic to either or . If , then all solutions are unbounded in the sense that is bounded and approaches .*

#### 3. System (21,21)

Now we consider the following system of difference equations:

where the parameters and are positive numbers and initial conditions .

System (3.1) was considered in [1, Example ], where it was shown that the associated map is injective and

that is, the Jacobian matrix of in is invertible. Therefore, in view of Theorems 1.4 and 1.5, four subsequences

of every solution of system (3.1) are eventually monotonic.

##### 3.1. Linearized Stability Analysis

Equilibrium points of system (3.1) are solutions of the system

Since and we have

where

Since and system (3.1) has a unique positive equilibrium , where

where

Lemma 3.1. *System (3.1) has a unique positive equilibrium point:
**
which is a saddle point.*

*Proof. *The Jacobian matrix of the corresponding map is of the form

By using (3.4) we obtain

The corresponding characteristic equation evaluated at the equilibrium point of system (3.1) is
where

Notice that

Since and , we need to show(I)(II)Now, we get(I)By using (3.4), (3.5), and (3.7) we obtain
Furthermore(II)which is satisfied.

##### 3.2. Global Results

Theorem 3.2. *System (3.1) has no prime period-two solutions.*

*Proof. *System (3.1) can be reduced to the following second-order difference equation:
or to the following second-order difference equation:
Now it is sufficient to prove that both of the difference equations (3.15) and (3.16) have no prime period-two solutions. Assume that this is not true for (3.15), that is, that
is a prime period-two solution of (3.15). Then we have
that is,
from which
and this implies that , which is a contradiction.

Now assume that

is a prime period-two solution of (3.16). Then we have
from which
and this implies that , which is a contradiction.

The global behavior system (3.1) is described by the following result.

Theorem 3.3. *Consider system (3.1). There exists a set which is invariant and a subset of the basin of attraction of . The set is a graph of a strictly increasing continuous function of the first variable on an interval (and so is a manifold) and separates into two connected and invariant components, namely,
**
which satisfy
*

*Proof. *In view of Theorem 3.2 and the injectivity of the map we see that all conditions of Theorems 1.7, 1.9, and 1.10 and Corollary 1.8 are satisfied with and so the conclusion follows.

The obtained result leads to the following characterization of the boundedness of solutions of system (3.1).

Corollary 3.4. *All bounded solutions of system (3.1) converge to the unique equilibrium with the corresponding initial conditions which belong to the graph of a continuous increasing function in the plane of initial conditions. All solutions that start in the complement of are asymptotic to either or .*

#### 4. System (15,21)

Now we consider the following system of difference equations:

where the parameters and are positive numbers and initial conditions . The Jacobian matrix of the corresponding map is of the form

System (4.1) was considered in [1, Example ], where it was shown that the corresponding map is injective and

that is, the Jacobian matrix of in is invertible. Therefore, in view of Theorems 1.4 and 1.5, four subsequences

of every solution of system (4.1) are eventually monotonic.

##### 4.1. Linearized Stability Analysis

Equilibrium points of system (4.1) are solutions of the system

Since , we obtain where

This implies that we have the following three cases for the equilibrium points.

(i)If , then there exist two equilibrium points of system (4.1): (ii)If , then system (4.1) has a unique equilibrium point: (iii)If or , then system (4.1) has no equilibrium points.Next, by using (4.5) we have

The corresponding characteristic equation evaluated at the equilibrium point is

where

Notice that

Lemma 4.1. *If , then the equilibrium point of system (4.1) is locally asymptotically stable and the equilibrium point is a saddle point.**If , then the equilibrium point of system (4.1) is nonhyperbolic.*

*Proof. *First, assume . For the equilibrium point we need to prove that
or equivalently (because ):(I)(II)Indeed,(I)we have
which is true. Furthermore(II)we have
which is true.

For the equilibrium point we need to prove that

that is (because and )(I)(II)Indeed,
Now (I)we have

which is true.

Similarly(II)we have
which is satisfied.

Assume that .

We need to prove that

that is (because and,),

We have

##### 4.2. Global Results

Theorem 4.2. *System (4.1) has no prime period-two solutions.*

*Proof. *The second iterate of map is
Period-two solution satisfies
From this system we have(i)(ii)(iii)(iv)(v)where

In cases (i) and (ii) solutions are equilibrium points and , and in case (v) solution is not in the first quadrant in the plane. It is sufficient to prove that solutions in cases (iii) and (iv) are not in the first quadrant in the plane. Namely, if , and are not real. Supose that . If , then If , then for solution in case (iii). By analogous reasoning we have that the same conclusion for case (iv) holds.

Our linearized stability analysis indicates that there are three cases with different asymptotic behavior, depending on the values of parameters and .

*Case 1. *.

*Case 2. *.

*Case 3. * or .

###### 4.2.1. Global Results—Case 1

Theorem 4.3. *Consider system (4.1) and assume that . Then there exists a set which is invariant and a subset of the basin of attraction of . The set is a graph of a strictly increasing continuous function of the first variable on an interval (and so is a manifold) and separates into two connected and invariant components, namely,
**
which satisfy
*

*Proof. *Clearly, system (4.1) is strongly competitive on . In view of injectivity of , invertibility of , and Theorem 4.2, we see that all conditions of Theorems 1.7, 1.9, and 1.10 and Corollary 1.8 are satisfied and the conclusion of the theorem follows.

###### 4.2.2. Global Results—Case 2

Theorem 4.4. *Consider system (4.1) and assume that . Then there exists a set which is invariant and a subset of the basin of attraction of . The set is a graph of a strictly increasing continuous function of the first variable on an interval (and so is a manifold) and separates into two connected and invariant components, namely,
**
which satisfy
*

*Proof. *In this case system (4.1) has a unique equilibrium point which is nonhyperbolic. For the corresponding characteristic equation is of the form
This implies
and

It is obvious that . We will show that . Indeed

which is satisfied. Thus, .

The eigenvector corresponding to is

It means that all conditions of Theorems 1.7 and 1.10 are satisfied with .

Assume that . Then for all , and sequences , and are monotone and bounded since . Thus these sequences are convergent, which in view of Theorem 4.2 shows that they converge to the equilibrium point. Since is the unique equilibrium point in the statement for follows. The same conclusion is obtained by using Theorem 1.6.

If is in , by Theorem 1.10 the orbit of eventually enters . Assume (without loss of generality) that . An eigenvector associated with the nonhyperbolic eigenvalue is . Choose a value of small enough so that and . Let us show that . Indeed

because
reduces to
where the last equality follows from the condition .

Since , it follows that is a monotonically decreasing sequence in which is bounded above by . Since is coordinatewise monotone and it does not converge (if it did it would have to converge to , which is impossible), we have that has second coordinate which is monotone and unbounded. But