In the present paper, we consider the following Hamiltonian elliptic system : , A new existence result of nontrivial solutions is obtained for the system via variational methods for strongly indefinite problems, which generalizes some known results in the literatures.

1. Introduction and Main Results

The goal of this paper is to study the existence of solutions for the nonperiodic elliptic systems in Hamiltonian form where , , , and . Such a system arises when one is looking for stationary solutions to certain systems of optimal control (Lions [1]) or systems of diffusion equations (Itô [2] and Nagasawa [3]).

In recent years, the systems like or similar to in the whole space were studied by a number of authors. Most of these works focused on the case . An usual way to overcome this difficulty is to consider the corresponding functional in the space of radially symmetric functions. In this way, De Figueiredo and Yang [4] considered the system where , , and They proved that system (1) has a radial solution pair under the assumptions that and are superlinear in and radially symmetric with respect to , and with . This result was later generalized by Sirakov [5] to the system In [6], Zhao et al. considered periodic asymptotically linear elliptic systems where the potential is periodic and has a positive bound from below and and are periodic in , asymptotically linear in as . By using critical point theory of strongly indefinite functionals, they obtained the existence of a positive ground state solution as well as infinitely many geometrically distinct solution for systems (3) under the assumptions that and are odd in . For other results, we refer readers to [717].

Without assumption of periodicity or radially symmetric about nonlinearities, the problem is quite different in nature and there has not been much work done up to now. In a recent paper [18], Wang et al. considered the following nonperiodic elliptic systems in Hamiltonian form: and obtained the following theorem.

Theorem A (see [18, Theorem ]). Suppose that the following conditions are satisfied: .There is such that if .  , where with as uniformly in , and there exist , such that , where .  There is with for , and as uniformly in bounded set of , and for all , where and .Then system (4) has one solution.

In the present paper, we are interested in the existence of solutions for Hamiltonian-elliptic systems involving gradient terms and nonperiodic superquadratic nonlinearities. The class of problems treated here has several difficulties. First, the problem is set on ; a main difficulty when dealing with this problem is the lack of compactness of the Sobolev embedding theorem. Second, the variational functional is strongly indefinite. Therefore, the classic critical point theorem cannot be applied directly. Third, the nonlinearities are nonperiodic in variable and superquadratic at infinity; the method in [6] cannot be applied to obtain the existence of solutions. Finally, the appearance of the gradient terms in the systems also brings us some difficulties; in this case, the variational framework in [18] cannot work any longer. Inspired by recent works of Zhao and Ding [19], we are going to investigate the existence of solutions for the Hamiltonian elliptic systems . By using the critical point theory of strongly indefinite functional which was developed recently by Bartsch and Ding [10, 20] and the reduction methods which was developed in [21, 22], we obtain an existence result of problem , which generalizes Theorem A.

Our fundamental assumptions are as follows: is 1-periodic in for and ., and . is 1-periodic in for .   , where with as uniformly in , and there exist , such that , where . as uniformly in . whenever , where . for some and all with large enough.There is with for , and as uniformly in bounded set of , for all and whenever .

Now we can state our main result.

Theorem 1. Let , , , and be satisfied. Then system has at least one nontrivial solution.

Remark 2. Theorem 1 extends and improves Theorem A. First, we only need to assume that the potential is periodic and has a positive bound from below. Second, the conditions and can be obtained by and . In fact, by (), we know that Consequently, by the conditions and (5), it is easy to see that and hold. Furthermore, similar to the proof of Lemma  2.2(i) in [23], the condition can be obtained by and (5). Indeed, since , we can obtain that . For some , . If , then there exists such that Choose so large that whenever . Then, by , (6), and (7), we obtain It follows that Third, the condition is weaker than the condition and the condition is weaker than the condition . Finally, summing up the above discussion, Theorem A is the special case of Theorem 1 corresponding to .
Throughout this paper, we always assume that denote any positive constant and may be different in different places. For , we define , where .

2. Variational Setting

In this section, we will establish variational framework for the system . For the convenience of notation, let denote the usual -norm and be the usual -inner product. Let and be two Banach spaces with norms and ; we always choose the equivalent norm on the product space . In particular, if and are two Hilbert spaces with inner products and , we choose the inner product on the product space . In order to continue the discussion, we need the following notations. Set and is a matrix operator. Let denote the Schrödinger operator. Denote and Then can be rewritten as

Denote by and the spectrum and the essential spectrum of the operator , respectively. Set ; then we have the following lemmas.

Lemma 3 (see [19, Lemma ]). Suppose that and are satisfied. Then the operator is a self-adjoint operator on with domain .

Lemma 4 (see [19, Lemma ]). Let , , and be satisfied. Then(1) = ; that is, has only essential spectrum;(2) and is symmetric with respect to origin;(3).It follows from Lemmas 3 and 4 that the space possesses the orthogonal decomposition such that is negative definite (resp., positive definite) in (resp. ). Let denote the absolute value of and be the square root of . Let be the Hilbert space with the inner product and norm . possesses an induced decomposition which are orthogonal with respect to the inner products and (the above results can be found in [19]).

Lemma 5 (see [19, Lemma ]). and are equivalent norms. Therefore, embeds continuously into for any and compactly into for , and there exists constant such that On we define the following functional: where . It follows from that, for any , there is such that for all . Thus, Lemma 5 implies that is well defined on . Lemma 4 implies that is strongly indefinite; such type functional appeared extensively when one considers differential equations via critical point theory; see, for example, [2427] and the references therein. Our hypotheses imply that (see Lemma  3.10 in [27]) and a standard argument shows that the critical points of are solutions of .

3. The Abstract Critical Point Theorem

In order to study the critical points of , we now recall a abstract critical point theorem developed recently in [10, 20]. Let be a Banach space with direct sum and corresponding projections onto . We assume that the Banach space is separable and reflexive. Let be a dense subset; for each there is a seminorm on defined by Denote by and the topology induced by seminorm family and the weak-topology on , respectively. denotes the weak-topology on . Now, some notations and definitions are needed.

For a functional , we write , , and .

Suppose() for any , is -closed, and is continuous;()for any , there exists such that for all ;()there exists such that , where .

Theorem 6 (see [10] or [20]). Let be satisfied and suppose there are and with such that where . Then, has a -sequence with . Moreover, if satisfies the -condition for all then has a critical point with .

Lemma 7 (see [10] or [20]). Let Suppose(1) is bounded from below;(2) is sequentially lower semicontinuous; that is, in implies ;(3) is sequentially continuous;(4) is and is sequentially continuous.

Then satisfies .

4. The Limit Equation

In this section, we study the following limit equation related to , where is given in , , and . By virtue of , we have firstly the following lemma.

Lemma 8. and possess the following properties.(i) as and for some .(ii) as .(iii) whenever , where .(iv) whenever large enough.(v) as .

Proof. (i) It is clear by and .
(ii) By , for any , there is such that whenever . Hence for all . Observe that it follows from that as . Letting we get for all .
(iii) Since as , it follows from that (iv) By , for large enough Letting , we obtain (v) It follows from (ii)–(iv) that as .
Now, we set and define the functional By Lemma 8, for any , there is such that It follows from (30) and (31) that is well defined and its critical points are solutions of .

Lemma 9. possesses the following properties:(1) is weakly sequentially lower semicontinuous and is weak sequentially continuous.(2)There is such that , where .

Proof. (1) Suppose in . Going if necessary to a subsequence, we can assume in for and a.e. in . Hence a.e. in . Thus Next it is sufficient to show that is weak sequentially continuous. Indeed, by (30) and in , , it follows from Theorem  A.2 in [27] that Furthermore, for each fixed , one has that, for any , there exists such that Hence, for large , it follows from (30), (33), (34), and Hölder inequality that Therefore,(2) For any , it follows from (31) that The conclusion follows because .
Now, we choose a number such that . From Lemma 8(ii), there is such that whenever . Let be the spectrum family of the operator . It follows from Lemma 4 that is a infinite dimension subspace of and

We have the following result.

Lemma 10. For any finite dimensional subspace of ,

Proof. If not, then there are and with such that for all . Denote , passing to a subsequence if necessary; we can assume that , , and . Then which yields that We claim that . Indeed, if not then it follows from (41) that . Thus , which contradicts with . By (38), we get Hence, there exists such that where . Note that Hence Now the desired conclusion follows from this contradiction.

As a consequence, we have the following.

Lemma 11. Let be given by Lemma 9. Then, letting with , there is such that , where .

Lemma 12. Let be any -sequence for . That is, is such that as . Then it is bounded and . Moreover, there is a subsequence still denoted by satisfying and , as .

Proof. Let be such that Then, for large , one has which implies . If is unbounded in , up to a subsequence if necessary, we can assume that . Set . Then and for each . Note that Hence, one has On the other hand, for and , set Then, by Lemma 8, we have for all and as . For large , one has and Consequently, for large and , whenever one has Since implies For any , we choose . Using the Hölder inequality we have as uniformly in and as . Therefore, as . Let be given. It follows from Lemma 8 that there is such that for all . Consequently, we have for all . By Lemma 8, we define and . By (55), we can take so large that for all . For fixed , i