#### Abstract

We use the normal form theory, averaging method, and integral manifold theorem to study the existence of limit cycles in Lotka-Volterra systems and the existence of invariant tori in quadratic systems in .

#### 1. Introduction

It is well known that -dimensional generalized Lotka-Volterra systems are widely used as the first approximation for a community of interacting species, each of which would exhibit logistic growth in the absence of other species in population dynamics. And this system is of wide interest in different branches of science, such as physics, chemistry, biology, evolutionary game theory, and economics. We refer the reader to the book of Hofbauer and Sigmund [1] for its applications. The existence of limit cycles and invariant tori for these models is interesting and significant in both mathematics and applications since the existence of stable limit cycles and invariant tori provided a satisfactory explanation for those species communities in which populations are observed to oscillate in a rather reproducible periodic manner (cf. [2–4] and references therein).

To study the bifurcation of Lotka-Volterra class, we consider three-dimensional generalized Lotka-Volterra systems which describes the interaction of three species in a constant and homogeneous environment, where is the number of individuals in the th population at time and , is the intrinsic growth rate of the th population, the are interaction coefficients measuring the extent to which the th species affects the growth rate of the th, and are parameters, and the values of these parameters are not very small usually.

Over the last several decades, many researchers have devoted their effort to study the existence and number of isolated periodic solutions for system (1). There have been a series of achievements and unprecedented challenges on the theme even if system (1) is a competitive system (cf. [5–12]). In [13], Bobieński and Żołądek gave four components of center variety in the three-dimensional Lotka-Volterra class and studied the existence and number of isolated periodic solutions by certain Poincaré-Melnikov integrals of a new type. In [14], Llibre and Xiao used the averaging method to study the existence of limit cycles of three-dimensional Lotka-Volterra systems. In this paper, we will use the normal form theory to study the same question. And furthermore, we will give the existence of invariant tori in a system of the form (2).

This paper is organized as follows. In Section 2, we obtain some preliminary theorems about a normal form system of degree two in with two small parameters and and other bounded parameters. In Section 3, we first change the system (1) into a system of the form where , and for are functions of the parameters and in system (1), and are bounded parameters, and is perturbation parameter. And then we get the real normal form of the system (2) after a series of transformations. Two examples are provided to illustrate these results in the last section.

#### 2. Preliminary Theorems

In this section, we first consider a normal form system of degree two in . Then, by a series of transformations we introduce some theorems for the normal form. The reader is referred to [15] for more details about the following content.

Consider the 3-dimensional system where is in , and By adding up the 2-parameter linear part we obtain where , with It can be verified that (5) has the following real normal form up to order 3 (see [15]): For convenience, we assume that as in [15]. By the scaling (7) becomes where Then, by introducing polar coordinates (9) further becomes where , , and are periodic in , and , , and . By a further scaling of the form (12) becomes We obtain from (14) where Note that the functions and in (15) are periodic in but may not be well defined at . Thus, we suppose for (15).

The averaging system has a singular point on the half plane if where . By denoting we obtain , and the characteristic polynomial of is . We define

According to Theorem in [15], we can obtain the following theorem.

Theorem 1. *Suppose that (18) holds. Then, (7) has a periodic orbit near the origin for . Further, the periodic orbit is stable (resp. unstable) if one (resp. none) of the following conditions holds:*(a)* and ,*(b)* and ,*(c)*, , and ,**
where is given by (20).*

Then, by letting and and truncating the terms of order , we have from (15) where Thus, in order that (21) has a limit cycle, we necessarily suppose and , , that is, This yields , and hence (21) becomes where For small , (25) has a focus with We define By using the coefficients in (7), we have Then, in 1997, the following result was obtained in [15].

Theorem 2. *Suppose that (24) holds and . Then, for any given there exist an and a function and such that for , (7) has a unique invariant torus near the origin if and has no invariant torus if . Moreover, the torus, if it exists, is stable (resp. unstable) when (resp. ).*

#### 3. Normal Form of System (2)

In this section, we consider system (1) in the first octant , where . We now look for the conditions for the existence of positive equilibria of system (1), which is equivalent to find the positive solutions of the following system:

We suppose that there exists at least one positive solution of (30). Without loss of generality, we assume that the positive equilibrium is . Then, we move it to the origin by doing the change of variables , . Then, system (1) can be written as

Now, we shall investigate a special form of system (31) with a small parameter; we write the perturbed system as Denote , and we suppose is similar to Then, system (32) can be changed into the system (2) by a linear transformation.

In this section, our task is to change system (2) into the normal form of (7). Making the transformation system (2) becomes where Let by changing , where , , and system (35) becomes a complex system of the form where

By the fundamental theory of normal form [16], we know that system (38) can be converted to the normal form by some transformations. So our following task is to find the transformations and work out the normal form of system (38).

We denote (38) as , where , and for simplicity, we write the nonlinear part of (38) as . By doing the following transformation: where , which is to be determined, (38) becomes Then, by noting we can get from (41) where and , . In order to eliminate the quadratic homogeneous polynomial, we need We take , as quadratic homogeneous polynomial, having the form where , , are real undermined coefficients. By inserting (45) into (44) and comparing the coefficients of similar items, we can obtain Note that , . The terms with coefficients , , , and that appeared above cannot be removed. Those terms are called the resonance terms. Then, we have and system (43) becomes Let denote the cubic terms in of (48). Then, from (41) and (42) we have where , , By substituting (42) into the above, we obtain

where

We make a further change , where is homogeneous cubic polynomial, so that (48) becomes where , and has the form as before. In order to eliminate some possibly cubic terms, we consider the equations below Suppose that for , By inserting these representations into (54), we can solve as before Hence, system (53) becomes now where and all of the coefficients are complex. Finally making the change and then taking the real parts of and the coefficients of all terms of the resulting system, we can get a cubic real normal form of the form (7) with