#### Abstract

The bifurcations of heteroclinic loop with one nonhyperbolic equilibrium and one hyperbolic saddle are considered, where the nonhyperbolic equilibrium is supposed to undergo a transcritical bifurcation; moreover, the heteroclinic loop has an orbit flip and an inclination flip. When the nonhyperbolic equilibrium does not undergo a transcritical bifurcation, we establish the coexistence and noncoexistence of the periodic orbits and homoclinic orbits. While the nonhyperbolic equilibrium undergoes the transcritical bifurcation, we obtain the noncoexistence of the periodic orbits and homoclinic orbits and the existence of two or three heteroclinic orbits.

#### 1. Introduction

In recent years, a great deal of mathematical efforts has been devoted to the bifurcation problems of homoclinic and heteroclinic orbits with high codimension, for example, the bifurcations of homoclinic or heteroclinic loop with orbit flip, the bifurcations of homoclinic or heteroclinic loop with inclination flip, and so forth; see  and the references therein. However, most of these papers considered the bifurcation problems of orbits connecting hyperbolic equilibria, and limited work has been done in the corresponding problems with nonhyperbolic equilibria; see . To fill this gap, we investigate the bifurcations of orbit and inclination flip heteroclinic orbits with one nonhyperbolic equilibrium and one hyperbolic saddle. The method is using the fundamental solution matrix of the linear variational system to obtain the Poincarémap, which is easier to get the bifurcation equations.

Consider the following () system and its unperturbed system where , the vector fielddepends on the parameters , , , , , , , , and . Moreover, the parametergoverns bifurcation of the nonhyperbolic equilibrium, whilecontrols bifurcations of the heteroclinic orbits.

Assuming system (2) has a heteroclinic loop connecting its two equilibria , , where , , , , , and . Furthermore, the linearizationhas real eigenvalues, , , andsatisfying; has simple real eigenvalues , , , and fulfilling .

The following conditions hold in the whole paper:where , , , , and mean that is a heteroclinic orbit with orbit flip, is the center unstable manifold of , (resp., ) is the unstable (resp., stable) manifold of , and (resp., is the strong unstable (resp., stable) manifold of , . Moreover, where the first three equations mean that the center unstable manifold of , the stable (resp., unstable) manifold (resp., ) ofare fulfilling the strong inclination property. And the fourth equation implies that the stable manifold is of inclination flip as .

It is worthy of noting that, for any integers and , if we assume and , then all the results achieved in this paper are still valid.

Let be a parameter to control the transcritical bifurcation of system (1), let the -axis be the tangent space of the center manifold at , and let be the vector field defined on the center manifold; then by , we may assume, , , , , where is the component of .

If is true, then system (1) exhibits the transcritical bifurcation, that is, when (or ; in this paper, we only consider the case ; for the case , one may discuss it similarly); there are two hyperbolic saddles and bifurcated from . Denote by and , where and . Moreover, , and .

The present paper is built up as follows. In Section 2, we devote it to deriving the successor functions by constructing a suitable Poincaré Map. The analysis to the bifurcations of system (2) is presented in Section 3, where we establish the existence of the heteroclinic loop, the homoclinic orbits, and the three or two heteroclinic orbits and the coexistence of a periodic orbit and a homoclinic loop, and the difference between the heteroclinic loop with hyperbolic equilibria and nonhyperbolic equilibria is revealed.

#### 2. Normal Form and Poincaré Map

Let the neighborhood of be small enough and straight the local manifolds of , , , and in the neighborhood . And then by virtue of the invariance of these manifolds and a scale transformation and , system (1) has the following expression in: and in it takes the following form: where , , , , ,, ,.

From the normal form (6), (7), and the condition , we may select and such that where is small enough such that and , .

Consider the linear variational system and its adjoint system , where is the transposed matrix of .

Supposing is a fundamental solution matrix of  , then, we arrive at the following lemma.

Lemma 1. If conditions (H1)–(H3) are satisfied, then
(1) there exists a fundamental solution matrix of satisfying such that
(2)   has a fundamental solution matrix fulfilling such that where , , , , .

Now, let be a new local active coordinate system along . Given , then is the fundamental solution matrix of , .

Let , where , . Defining the cross sections of   at and , respectively, .

Now that if and , then Based on the expressions of and , we get their new coordinates of and ; that is,

Next, we divide our establishment of the Poincarémap in the new coordinate system in three steps.

First, consider the map . Put into (1); we have According to the fact and , it then yields to that Integrating the above equation from to , we arrive at Noticing that , then where Together with (17) and , then defines the map , .

Next, to construct the map (where). Let , be the flying time from to ; set and . By virtue of the approximate solution of system (6) and (7), if we neglect the higher terms, then the expression of is and is where , are called Shilnikov coordinates, and Since the nonhyperbolic equilibrium undergoes a transcritical bifurcation based on the structure of orbits in , we may see that the equation holds only for . While for   , the map is well defined only if (see Figure 1). So, we extend the domain of , defining

The final step is to compose the maps and , and then can be expressed as and as

Set , . Combing , (23), (24), (27), and (28), we derive the successor functions : It is easy to see that what we need to do is considering the solutions of with and . This is because the solution of (30) with (resp., , , or , ) means that system (1) has a heteroclinic loop (resp., a periodic orbit; homoclinic loop).

#### 3. Main Results

Based on the expressions of the successor functions and the implicit function theorem, we know that the equation has a unique solution . And putting it into , then we obtain the following bifurcation equations:

Firstly, we consider the case , which means the transcritical bifurcation does not happen. By (23) and (25), (31) turns to Noticing that , which shows , it then follows that

From the above bifurcation equations, we obtain the following results immediately.

Theorem 2. Let the conditions (H1)–(H3) be true and , . Then, for and , one has
(i) for , there exists a codimension 2 surface such that system (1) has a unique heteroclinic loop near if and only if , where the surface has a normal plane at .
(ii) there exists an -dimensional surface such that system (1) has a unique homoclinic loop connecting (resp., connecting ) near if and only if (resp., ).

Proof. The result (i) will be proved by putting into (33).
If we assume and in (33), then which means
It follows that there exists an -dimensional surface given by (35) such that (33) has a unique solution , as and . This implies system (1) has a homoclinic loop connecting . The existence of can be obtained similarly.
This completes the proof.

Remark 3. There is no difficulty to see that has a normal vector at as , while for (resp., ) it has a normal vector (resp., ) at .

Theorem 4. Assume the conditions hold and , . Then for , , and , the periodic orbit and homoclinic loop with of system (1) cannot coexist.

Proof. Theorem 2 shows that if and , then system (1) has a homoclinic loop with. Setting , , and , then (33) is reduced to Notice that and
If , then ; it is obvious that has no sufficiently small positive solutions.
While , then and hold for , which shows that has no sufficiently small positive solution.
Next, we only consider the case and . As , we have , and then, for , we see that In fact, yields that , , and , which shows has no sufficiently small positive solutions. Obviously, the conclusion is correct as.
Similarly, for , , there does not exist a small positive solution for .
The proof is then completed.

Theorem 5. Assume that the conditions (H1)–(H3) hold and , . Let , , and ; then (i)the periodic orbit and the homoclinic loop connecting of system (1) cannot coexist as or ;(ii)at least one periodic orbit and the homoclinic loop connecting of system (1) coexist as , , and ;(iii)a unique periodic orbit and the homoclinic loop connecting of system (1) coexist as , , and .

Proof. By Theorem 2, the condition for implies that system (1) has a homoclinic loop connecting .
(i) Let and eliminatingin (33), we derive Note that as . Moreover,
For and , this means has no sufficiently small positive solutions.
Now we turn to the case , since we are interested in sufficiently small positive solutions of (33), it suffices to consider the sufficiently small positive solutions of satisfying , which implies that (resp., for (resp., . It is easy to see that has no sufficiently small positive solutions as .
(ii) For , we have , which implies that there exists an such that for .
Choosing , then In view of for , so when . As a result, has at least one solution satisfying .
(iii) must fulfill as ; with similar arguments in proof of (ii), we can prove that there exists a such that for . It is easy to compute that for , , and . Combining with the fact , , and , we immediately know that is unique.
This completes the proof.

Now, we turn to discussing the bifurcations of the heteroclinic loop for , when undergoes a transcritical bifurcation. From Figure 1, we know that when , after the creation of the equilibria and , there always exists a straight segment orbit heteroclinic to and , its length is , and we denote this heteroclinic orbit by . Moreover, is a critical position.

Firstly, we take into account the case . In this case, (31) becomes Let ( means and vice versa); by virtue of Taylor’s development for , we have

With similar arguments to , we may easily obtain the following results.

Theorem 6. Suppose the conditions (H1)–(H3) hold, ; then
(i) if , there exists an-dimensional surface such that system (1) has a unique heteroclinic loop if and only if and ;
(ii) there exists an -dimensional surface such that system (1) has one homoclinic loop connecting (resp., connecting ) if and only if and .

Theorem 7. Suppose hypotheses (H1)–(H3) hold, , , , , and . Then, except the homoclinic loop connecting (resp., ), system (1) has no periodic orbits as (resp., ).

Remark 8. It is easy to see that homoclinic loop connecting and heteroclinic loop joining , cannot be bifurcated from , which is exactly determined by the generic condition .

Finally, we consider the case . Due to Figure 1 and (25), it follows from (31) that

Theorem 9. Assume the conditions (H1)–(H3) are true, and . Then, (i)there exists a surface such that system (1) has two orbits heteroclinic to , , as ;(ii)there exists a region in the space such that system (1) has a heteroclinic orbit connecting and for .

Proof. (i) If in (50), then which shows that there exists a surface such that (50) has a solution and for , then system (1) has two heteroclinic orbits, one is heteroclinic toand and the other is heteroclinic to and.
(ii) If in (50), one attains . Eliminating in (50), we achieve which shows that there exists a region such that when , system (1) has one heteroclinic orbit heteroclinic to and .

Remark 10. All the heteroclinic orbits joining will go into in different ways according to different fields of ; see Figure 2.

#### Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.

#### Acknowledgments

This paper is supported by National Natural Science Foundation of China (nos. 11202192, 11226133), the Fundamental Research Funds for the Central Universities (no. 2652012097), and the Beijing Higher Education Young Elite Teacher Project.