#### Abstract

We study the number and distribution of limit cycles of some planar -equivariant quintic near-Hamiltonian systems. By the theories of Hopf and heteroclinic bifurcation, it is proved that the perturbed system can have 24 limit cycles with some new distributions. The configurations of limit cycles obtained in this paper are new.

#### 1. Introduction

In 1900, Hilbert proposed 23 open mathematical problems [1]; the second part of the 16th problem concerns the maximal number and relative position of limit cycles of the planar polynomial vector fields. Even though there have been many results of obtaining more limit cycles and various configuration patterns of their relative dispositions, it has not been solved completely. To reduce the difficulty one can study the systems with some symmetry. An important symmetry is the -equivariance which was first introduced in [2]. Here we mention some newer results; for more details, see summary work [3–5]. Li et al. [6] proved a cubic -equivariant system having 13 limit cycles; Zhao [7] proved that this system has 13 limit cycles with another distribution. Li and Liu [8] proved another cubic -equivariant system also having 13 limit cycles. Zhang et al. [9] found a quartic system having at least limit cycles. Christopher [10] proved that a -equivariant system has 22 limit cycles. As to the case of quintic polynomial system, there are more results. Xu and Han [11] studied a cubic -equivariant system perturbed by quintic -equivariant polynomials having 13 limit cycles. Li et al. [12] studied a quintic system and obtained at least limit cycles for -equivariant case and 17 limit cycles for -equivariant case. In [13], Wu et al. studied a -equivariant system and found 20 limit cycles. Li et al. [14] found that 24 limit cycles existing in a -equivariant quintic system. Yao and Yu [15] studied a -equivariant quintic planar vector fields by normal form theory and proved that the maximal number of small limit cycles bifurcated from such vector fields is 25. Wu et al. [16] proved that a quintic -equivariant near-Hamiltonian system has 28 limit cycles. In [17], 24 limit cycles are found and two different configurations of them were shown in a -equivariant quintic planar polynomial system.

Our main result is that there can be 24 limit cycles with other distributions for the perturbed quintic -equivariant systems which are different from the known results. Using the methods of Hopf and heteroclinic bifurcation theories, the number and location of limit cycles of the following -equivariant quintic near-Hamiltonian system will be investigated: where is nonnegative and small and the Hamiltonian system is with phase portrait of Figure 1. () is the five-degree polynomial vector invariant under rotation of with respect to the origin . From [2] we know that () is, respectively, the real and imaginary parts of the following complex function: where , , , and . It is direct that Our result is the following.

Theorem 1. *There exist some such that system (1) can have 24 limit cycles with two different distributions, the distributions of these limit cycles are shown in Figure 2.*

**(a) 24 limit cycles**

**(b) 24 limit cycles**

The rest of this paper is organized as follows. Some useful preliminary theorems will be listed in Section 2. In Section 3, some related coefficients of asymptotic expansions are firstly computed; then using this coefficients and preliminary lemmas we prove the main result.

#### 2. Preliminary Lemmas

Let , , and be analytic functions, positive and small, and with compact; then the following system is a planar Hamiltonian system: and the below system is usually called near-Hamiltonian system: Let system (5) have at least one family of periodic orbits defined by which form a periodic annulus ; then the first-order approximation of the Poincaré map of system (6) is which is called the Melnikov function or Abel integral. By the Poincaré-Pontryagin-Andronov theorem, an isolated zero of corresponds a limit cycle of system (6). A popular method to find limit cycles of (6) is to find zeros of and an efficient method to find zeros of is to investigate the asymptotic expansion of near the boundaries of ; see [18].

Let the outer boundary of be a homoclinic loop defined by passing through a hyperbolic saddle at the origin; we have the following.

Lemma 2 (see [19]). *(i) The function has the following expansion:
**
for ; depends on the parameters of , , and .**(ii) Further suppose that, for near ,
**
Then,
**
The values and are, respectively, called the first and second local Melnikov coefficients at the saddle , denoted by and , respectively.*

Now let the outer boundary of be an -polycycle : with hyperbolic saddles, and , and heteroclinic orbits, and , connecting them, defined by . The following lemma was proved in [19].

Lemma 3 (see [19]). *Under the above assumptions, has the form, for ,
**
where
**
where and are, respectively, the first and the second local Melnikov coefficient at the saddle , . In particular,
**
if , .*

When the inner boundary of is a elementary center defined by , the following lemma gives the asymptotic expansion of .

Lemma 4 (see [20]). * has the form, for ,
**
If for near ,
**
the coefficients can be obtained by the formulas in [20].*

*Remark 5. *When the inner boundary is a nilpotent center, a new method of limit cycles bifurcated from the annulus near the center can be found in [21]. Lemma 3 has been developed in [22, 23].

In many cases the Hamiltonian function is not of the form presented in the above lemmas. Then to apply the lemmas we need first to introduce suitable linear change of variables which will cause a change in the first-order Melnikov function. The following lemma gives the relationship between the old and new Melnikov functions.

Lemma 6 (see [24]). *(i) Under the linear change of variables of the form:
**
and time rescaling , where , the system (6) becomes
**
where , , and .**(ii) Let
**
which is the Melnikov function of the system (18); then
*

When systems (5) and (6) are -equivariant, (5) has a compound cycled denoted by , which consists of 8 hyperbolic saddles and 16 heteroclinic orbits , , , , , , , , , , , , , , , and satisfying . contains 8 two-polycycles , where , , , , , , , and . See Figure 3. We suppose that is defined by , . There are 8 centers inside the 2-polycycle , with and . There are 4 families of periodic orbits inside the 2-polycycle , defined by for , , and 4 families of periodic orbits inside the 2-polycycle , defined by for , . Then we have 8 Melnikov functions below: By -equivariance, and , we can only study and . For convenience, the notations are introduced as follows: and , where and . Letting , we introduce The following is directly from Lemma 3.

Lemma 7. *Under the above assumptions, we have the following expansions:
**
for , ,
**
for , , where
**
if . denotes an eigenvalue of for (6) and and are the first and the second Melnikov coefficients at the saddle as defined after Lemma 2.*

By Lemma 4, for the expansions of near the center, we have

To obtain more limit cycles, we have the following.

Theorem 8. *Let (24), (25), and (27) hold.**(1) Suppose that there exists such that
**
Then there exist some near such that (5) has
**
limit cycles in the -polycycles and , where , , , and with being positive and very small, and the location of these limit cycles is as follows: limit cycles near the -polycycle , limit cycles near the center , limit cycles between the center and the polycycle , and limit cycles between the center and the polycycle .**(2) Suppose that there exists such that
**
Then there exist some near such that (5) has
**
limit cycles in the -polycycles and , where , , , and with being positive and very small, and the location of these limit cycles is the following: limit cycles near the -polycycle , limit cycles near the center , limit cycles near the -polycycle , limit cycles between the center , and the polycycle and limit cycles between the center and the polycycle .*

*Proof. *Because of the similarity, we only prove the last conclusion. For , by continuity, there exist zeros of between and and zeros of between and . Thus, for all near or there exist zeros of between and and zeros of between and .

According to the condition, we can take , , , and as free parameters. Hence, we first take satisfying
so that there is a zero of denoted by near satisfying . On the other hand, if , considering , we have and which implies that there is a zero of denoted by near satisfying . If , we are not sure if has a zero. Thus, so far we obtain zero of and zeros of for .

Next, we take , , and satisfying
Then has two new zeros near and near satisfying and , and has a new zero near satisfying . In this step, we get more zeros of and more zero of for , , , , and . Then totally we have zeros for . Therefore, there are limit cycles for some near . This completes the proof.

*Remark 9. *The signs of , , , and can be determined by using the first nonzero coefficients in their expansions.

#### 3. Main Result

In this section, we investigate the distributions of limit cycles of system (1). For , (1) has two level sets, and , defined by and respectively. consists of saddles, , , , , , , , and , as shown in Figure 4. Let , , , , , , , , ,, and so forth are the same as those for Lemma 7. We can write the compound cycles as The equation defines four centers, , , , and , where the equation defines four centers of , , , and . The center is inside the -polycycle for .

We first investigate the -polycycles and . Here, denotes the periodic orbit defined by surrounding the unique center and denotes the periodic orbit defined by surrounding the unique center . Then where . By Lemma 7, we have for , for .

By Lemma 4, for , , and, for , ,

To find the zeros of and , the coefficients in these asymptotic expansions need to be calculated. In order to calculate the coefficients , , , , and so forth, the expressions of heteroclinic orbits are found as follows:

Figure 5 may be helpful to understand the step to calculate the coefficients in the folowing. By (4), and are written as follows: and introduce the following notations: and . By Lemma 7, we have where, for , Thus,

(a) The 2-polycycle |

(b) The 2-polycycle |

By (4), the divergence of (1) at and is as follows: Note that which yields By Lemma 3, we have In the following by letting , then Letting , then Under , we can apply Lemma 7 to calculate the coefficients and . For convenience, the following notations are introduced: . For , , , and By Lemma 7, we have Substituting (52) into (53), with being considered, we have where and Applying Lemma 7, we have The integrals in (44) and (55) and the coefficients in (45) and (56) can be obtained by numeral computation on Maple 13; see Appendix.

In order to find the local coefficient at the saddle we make a change of variables of the form and time rescaling so that the system (1) becomes where Writing functions and as the form then the formula for the second local coefficient at the saddle in Lemma 2 can be applied directly; we have Similarly, we have