#### Abstract

The bifurcation of a nongeneric homoclinic orbit (i.e., the orbit comes from the equilibrium along the unstable manifold instead of the center manifold) connecting a nonhyperbolic equilibrium is investigated, and the nonhyperbolic equilibrium undergoes a pitchfork bifurcation. The existence (resp., nonexistence) of a homoclinic orbit and an 1-periodic orbit are established when the pitchfork bifurcation does not happen, while as the nonhyperbolic equilibrium undergoes a pitchfork bifurcation, we obtain the sufficient conditions for the existence of homoclinic orbit and two or three heteroclinic orbits, and so forth. Moreover, we explore the difference between the bifurcation of the nongeneric homoclinic orbit and the generic one.

#### 1. Introduction

It is well known that the nonhyperbolic equilibrium is unstable and always undergoes a saddle-node (resp., transcritical or pitchfork) bifurcation. So the bifurcation problems of homoclinic or heteroclinic orbits with nonhyperbolic equilibria are more difficult and challenging. And few of the papers take into account the homoclinic or heteroclinic orbits with nonhyperbolic equilibria. Zhu [1] gave the sufficient conditions for the existence of nongeneric heteroclinic orbits accompanied with saddle-node bifurcation by extending exponential trichotomy. Klaus and Knobloch [2] discussed the bifurcation of homoclinic orbit to a saddle-center in reversible system. Liu et al. [3] considered the bifurcations of homoclinic orbit with a nonhyperbolic equilibrium for a high dimensional system; they achieved the persistence of homoclinic orbit and the bifurcation of periodic orbit for the system accompanied by a pitchfork bifurcation. In 2012, we discussed the bifurcations of generic heteroclinic loop accompanied by pitchfork bifurcation [4]. For other works about bifurcations of the homoclinic or heteroclinic orbits with nonhyperbolic equilibria, the readers may see [5–8] and references therein.

Inspired by the above works, we deal with the nongeneric homoclinic bifurcation accompanied by a pitchfork bifurcation in a 4-dimensional system. By extending the method established in [7], we give the sufficient conditions for the existence of a generic (resp., a nongeneric homoclinic) orbit and a periodic orbit when pitchfork bifurcation does not happen, while the nonhyperbolic equilibrium undergoes a pitchfork bifurcation, we achieve the existence of homoclinic orbits connecting the bifurcated equilibrium and three heteroclinic orbits, where we may know the difference of bifurcations between the nongeneric homoclinic orbit and the generic one.

The rest of the paper is organized as follows. In Section 2, we present some hypotheses and give the normal form for the system considered in this paper. The Poincaré map and successor function are achieved in Section 3. Finally, the existence and nonexistence of homoclinic, heteroclinic, and periodic orbits are given in Section 4.

#### 2. Hypotheses and Normal Form

Consider the following () system: and its unperturbed system where , , , and , , , ; namely, the origin is an equilibrium of system (2).

Assume system (2) has a homoclinic orbit connecting the origin with ; denote . Moreover, the linearization has four real eigenvalues , , and satisfying . Obviously, the nonhyperbolic equilibrium has a 2-dimensional stable manifold , an 1- dimensional center manifold , and 1-dimensional unstable manifold .

The following hypotheses will be needed in the whole paper: (H_{1}) = , which means that the homoclinic orbit is nongeneric, and the orbit is generic if it comes from the origin along the center manifold; the bifurcation for generic homoclinic orbit one may see [3]: (H_{2})
where denotes the strong stable manifold of , . (H_{3}) Let -axis be the tangent space of the center manifold at , and let be the vector field defined on the center manifold and satisfies

According to Wiggins [9], under the above assumption, the origin is a pitchfork bifurcation point, and is the parameter controlling the pitchfork bifurcation; that is to say, under small perturbation when the origin is perturbed into three hyperbolic saddles , , (one may see Figure 1). Denote , , and , where , . In the whole paper, the sign “′” denotes the transpose of the vector. It is easy to see that , , .

According to the invariance of the manifolds, we may introduce a scale transformation and straighten the local manifolds of ; then system (2) has the following expression in the small neighborhood of the origin: where , , , for .

Due to the normal form (5) and , we may choose () such that , , where is small enough such that . Obviously, .

Take into account the linear variational system: and its adjoint system where and is the transpose of .

We introduce the following lemma; it is very significant in this paper.

Lemma 1. *Let be a fundamental solution matrix of system (6), we can select , , , such that
**
where , for , and for , for , and for .*

*Proof. *According to the hypotheses () and (), one may easily obtain the existence of the , , and with the given expression at . Based on the condition (), we take , satisfying
Let , where , so we have , satisfying
where , , obviously .

Noticing that the strong inclination property holds, it then follows that , . By diag diag as , one can easily know that for , for , and for .

The proof is then finished.

Let ; from the matrix theory, we know that is the fundamental solution matrix of (7).

Introduce the following local moving frame coordinates: where . Define the cross sections:

Notice that if , , then We may easily obtain the new coordinates for and as follows:

#### 3. Poincaré Map and Successor Function

In this section, we establish the Poincaré map in the new coordinate system and then derive the successor function. (1) Establishment of the map .

Putting into system (1), we have Since and , it then follows that Integrating both sides of the above equation from to , we arrive at Noticing that , then we have the map as follows: where (2) Establishment of the map .

Let be the flying time from to , and set (where ); utilizing the approximate solutions of system (5), it is easily to obtain the expression of : where and the higher order terms are neglected.

*Remark 2. *Figure 1 tells us that holds only when , for , the orbits near will go into , and we may set in this case. While for , the orbits near will keep away from . (3) Establishment of the map .

Composing the maps and , then can be expressed as (4) Establishment of the successor function.

The successor function is given by : where is defined as (20).

#### 4. The Main Results

We will discuss the homoclinic bifurcation accompanied by pitchfork bifurcation using the successor function achieved in Section 3.

It is obvious that system (1) has a homoclinic orbit or heteroclinic orbit (resp., periodic orbit) if and only if the equation has a solution satisfying (resp., ).

According to the implicit function theorem, we know that the equation has a unique solution for , sufficiently small, substituting it into ; then we obtain Equation (25) is called the bifurcation equation.

Firstly, we consider homoclinic bifurcation with ; that is, the pitchfork bifurcation does not happen. Based on (20) and (21), we know that for ; (25) then becomes Note that ; if (resp., ), then we have (resp., ), and (resp., ), omitting the higher order term of ; it then follows that for (for the case , we may discuss similarly), (26) turns to By way of the implicit function theorem, we know that if rank, there exists a function such that for (27) always holds. So we may obtain the following result.

Theorem 3. *Suppose the conditions hold, and rank ; then *(i)*as for , small enough and fixed , system (1) has not any periodic orbit near ;*(ii)*as for , small enough and fixed , system (1) has a unique periodic orbit near ;*(iii)*as for small enough and fixed , system (1) has a unique homoclinic orbit near .*

*Remark 4. *As we know that the homoclinic orbit is nongeneric, so the homoclinic orbit obtained in Theorem 3 (iii) comes from along the unstable manifold, while the orbit may come from the equilibrium along the weak unstable manifold (see Figure 2(b)).

**(a)**

**(b)**

Next, we consider the case ; the origin undergoes a pitchfork bifurcation in this case; namely, there are three equilibria , , and bifurcated from the origin . And there always exist two straight segment orbits, one is heteroclinic to and and the other is heteroclinic to and ; their lengths are , and we denote the heteroclinic orbits by and , respectively. On the other hand, based on the definition of , we will consider the bifurcations with three cases: for , and .

First, we discuss the homoclinic bifurcation for . Based on (20), (25) turns to where we still consider the case , we may discuss the case for similarly. Denote ; then we get

Equation (28) is then becomes

If , , (30) gives a solution by virtue of implicit function theorem. And for fixed and the first equation of (30) we obtain ., differentiating with respect to , we achieve notice that the , according to the relation of and in (20) and (21), we may see that ; then the above equations explore that (resp., ) as (resp., ), which implies that is monotonic with respect to when . Moreover, Therefore, we achieve the following conclusion.

Theorem 5. *Let the hypotheses hold, , and ; then in addition to the heteroclinic orbits and ,*(i)*system (1) has a unique homoclinic orbit with for for fixed ;*(ii)*there exists small enough such that when , , or , system (1) has a unique 1-periodic orbit near .*

Next, we discuss the case , as we know that the orbit will go into (we denote ) in this case. While for the orbit will keep away from . However, the orbit that comes from the equilibrium will be decided by . If , then the orbit comes from ; if , then the orbit comes from ; for , then the orbit comes from (see Figure 1). So we obtain the following result.

Theorem 6. *Let the conditions be true, and , , . Then in addition to the heteroclinic orbits and , *(i)*there exists a region in the space:
* *such that system (1) has a heteroclinic orbit with and as ; see Figures 3(a)–3(d);*(ii)*there exists a region in the space:
* *such that system (1) has a homoclinic orbit connecting as ; see Figures 3(e)-3(f);*(iii)*there exists a region in the space:
* *such that system (1) has a heteroclinic orbit connecting and as ; see Figures 3(g)-3(h);*(iv)*there exists a region in the space:
* *such that system (1) has no other heteroclinic orbit and no homoclinic orbit for .*

**(a)**

**(b)**

**(c)**

**(d)**

**(e)**

**(f)**

**(g)**

**(h)**

*Remark 7. *As we know from [3], the orbits in Figures 3(e)–3(h) cannot be bifurcated from the generic homoclinic orbit, which exactly reveals the difference between the bifurcation of generic homoclinic orbit and the nongeneric one.

#### Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.

#### Acknowledgments

The authors are supported by National Natural Science Foundation of China (no. 11202192), the Fundamental Research Funds for the Central Universities (no. 2652012097), and the Beijing Higher Education Young Elite Teacher Project.