#### Abstract

We study the expansions of the first order Melnikov functions for general near-Hamiltonian systems near a compound loop with a cusp and a nilpotent saddle. We also obtain formulas for the first coefficients appearing in the expansions and then establish a bifurcation theorem on the number of limit cycles. As an application example, we give a lower bound of the maximal number of limit cycles for a polynomial system of Liénard type.

#### 1. Introduction

Consider a planar system of the form where is a small parameter and , , and are functions in and with bounded. For , (1) becomes which is a Hamiltonian system. As we know, the system (1) is said to be a near-Hamiltonian system. For (1), the main task is to study the number of limit cycles which are bifurcated from periodic orbits of the unperturbed system (2). On this aspect, the first order Melnikov function of (1) plays an important role. We can use the expansions of it near Hamiltonian values corresponding to a center or an invariant loop to find its zeros and hence the number of limit cycles. See a survey article [1]. There have been many works on this topic. For the study of general near-Hamiltonian systems, see [2–12]; and especially for the system (2) with the elliptic case, one can see [13–17] and references therein. In [2–4], the number of limit cycles of the system (1) near a homoclinic loop with a cusp of order one or two or a nilpotent saddle of order one (for the definition of an order of a cusp or nilpotent saddle, see [5]) was studied. In the heteroclinic case with two hyperbolic saddles, a hyperbolic saddle and a cusp of order one, or two cusps of order one or two, the number of limit cycles of the system (1) was studied in [5, 8, 9], respectively. In this paper, we suppose that the unperturbed system (2) has a compound loop consisting of a cusp of order one, a nilpotent saddle of order one, a homoclinic loop to , and two heteroclinic orbits connecting and , as shown in Figure 1. We aim to study the number of limit cycles of (1) near the loop for small.

#### 2. Main Results with Proof

Now consider the systems (1) and (2). Suppose that (2) has a compound loop denoted by and defined by equation , where is a cusp and is a nilpotent saddle both having order one, are heteroclinic orbits satisfying and , and is a homoclinic loop to . Then, the level curves of define two families of periodic orbits and for on one side of and a family of periodic orbits for on another side of . For the definiteness, let both and exist for and exist for . Thus, we have three Melnikov functions Let denote a closed set with diameter and with center at , . See Figure 2(a). And further introduce Here the Cl. denotes the closure of a set. Then by (3) and (4), for sufficiently small we can write where

**(a)**

**(b)**

By [5], there exist two transformations of the form where is a matrix satisfying such that (1) becomes where for near . Note that for near and for near . Then we have where denote the image of under and , , and denote the image of , , and under , respectively. Then, by using [3, 4] we can obtain the following two lemmas, respectively.

Lemma 1. *Consider system (10) with and suppose (11), (13) hold. Then there are constants satisfying
**
such that
**
for ,
**
for , where at with , and
**
where
*

Lemma 2. *Consider system (10) with and suppose (12), (15) hold. Then we have
**
for ,
**
for , where at with , , and
**
where , and are constants, given by
*

For convenience, let

Theorem 3. *Assume that system (1) has a compound loop as stated before. Then, the functions given in (3) at have the following expansions:
**
for , and
**
for , where
**
where . In particular,
**
if . Here, , are constants and are given in Lemma 1.*

*Proof. *First, by (6), (10) with , (12), (14), (29), and Theorem 2.2 in [4], we directly obtain (27) with given by (30) and (31), respectively. Then we study the expansions of and .

By (5), (7), (29), and Lemmas 1 and 2, we have
for , and