We consider the multiplicity of nontrivial solutions of the following quasilinear elliptic system , , , , , where , , , , , . The functions , , , , , , and satisfy some suitable conditions. We will prove that the problem has at least two nontrivial solutions by using Mountain Pass Theorem and Ekeland's variational principle.

1. Introduction

In this paper, we consider the multiplicity of solutions to a class of quasilinear elliptic system where , , , , , .

In recent 20 years, existence and multiplicity for a quasilinear elliptic equation and equations have been studied widely. In [1], Yu has studied the existence of a nontrivial weak solution to the quasilinear elliptic problem where is a smooth exterior domain in , , is a positive function, and is a nonnegative function. Problems of this type are motivated by mathematical physics, since certain stationary waves in nonlinear Klein-Gordon or Schrödinger equations can be reduced to this form (see [2, 3]). Problem (2) began to attract more and more attention after the work of Yu [1]; we refer the readers to [48].

In [9], Hsu studied the following elliptic system: where is a bounded domain, , , and , satisfy . The problem (3) is the special case of problem (1), while , . By applying Nehari manifold method, the author proved the problem (3) has at least one positive solution when and has at least two positive solutions when .

This paper is motivated by the results of [912]. Replacing the Nehari manifold methods, we use the Mountain Pass Theory and Ekeland’s variational principle to study the existence of multiple solutions for problem (1). It seems difficult to get the same result by Nehari manifold methods.

Throughout this paper, we make the following hypotheses:, , , ., ., , , , .

In this paper, we let the completion of the space with respect to the norm , and let the completion of with respect to the norm .

We define where , .

The following inequality is called Caffarelli-Kohn-Nirenberg inequality [13]. There is a constant such that where , , , and .

Let denote the usual Sobolev space with the norm Then is a reflexive and separable Banach space endowed with the norm .

Definition 1. We say that is a weak solution of problem (1) if for all , there holds
It is clear that problem (1) has a variational structure. Let be the energy functional corresponding to problem (1) which is defined by where Then, we see that functional and for all , there holds In particular, it follows from (10) that It is clear that the critical points of the energy functional correspond to the weak solutions of problem (1). Thus, to prove the existence of weak solutions for the problem (1), it is sufficient to show that admits a sequence of critical points in . Our main result is the following.

Theorem 2. Assume that hold. There exist such that if satisfy and , then the problem (1) has at least two nontrivial solutions.

2. The Proof of Theorem 2

In this section, we first introduce a compact embedding theory which is an extension of the classical Rellich-Kondrachov compactness theorem; see Xuan [13].

Lemma 3. Assume that is an open bounded domain with boundary and , , . Then the embedding is continuous if and and is compact if and .

The following lemma is called the Mountain Pass Theorem, see [14] (also see [15]).

Lemma 4 (Mountain Pass Theorem). Let be a real Banach space and with . Suppose that satisfies condition and there are , such that when ; there is , such that .
Define Then is a critical value of .

Lemma 5. Assume that hold. There exist such that if the parameters , satisfy , and , then satisfies the assumptions in Lemma 4.

Proof. Let . By (8), By and the Hölder inequality, we have where , .
Similarly, Hence, it follows from (15) and (16) that Since , we obtain from the Hölder inequality and (5) that Similarly, Then, Using the Hölder inequality, we get where , .
Similarly, we have Thus, with .
Let To verify in Lemma 4, it suffices to show that for some .
Note that where or . Then has a minimum at . In order to find , we have So that and Moreover, implies that where .
Thus, it follows from (23) and (27) that there exist such that with and . Then satisfies the assumption in Lemma 4.
We now verify in Lemma 4. Choose      such that Then and (), since . Therefore, there exists large enough, such that . Then we take and . The condition of Lemma 4 is true. This completes the proof of Lemma 5.

Lemma 6. Let - hold.
If and are bounded sequences in , then there exist subsequences, still denoted and , and , such that as

Proof. Let , .
Since is bounded in , is bounded in for for all . By Lemma 3, has a subsequence which converges in . Likewise, the subsequence is bounded in . So that it has a subsequence which converges in . Since is a subsequence of , in . Continuing this process, we obtain a sequence with the following properties: It is clear that a.e in , where By a diagonal process, we take which is a subsequence of . Thus, we have Without loss of generality, we assume that the subsequence is itself. So This implies that for , Now, we prove Indeed, where .
The fact gives that .
Then, (37) implies that This gives (36).
In the following, we show that Since the sequence is bounded in , we can assume (up to a subsequence) that weakly in and for some constant and all .
By (36), we know that for any , there exists so large that Since the embeddding is compact (see Lemma 3), we have Thus, there exists , when , So This shows that in as .
By Brezis-Lieb’s Lemma in [16], we obtain that (39) is true.
Similarly, we can prove

Remark 7. By Brezis-Lieb’s Lemma, it follows from (30) that

Lemma 8. Assume that hold. Then defined by (8) satisfies condition on .

Proof. Let be a sequence of in ; that is, , , in . We first claim that is bounded in . In fact, for large , we obtain Since , we conclude that is bounded. Then, the and are bounded in , respectively. Thus, there exist and a subsequence of , still denoted by , such that Denote Then the fact in implies as .
Since that in , we see that Let From dominated convergence theory and Lemma 6, we can conclude From Hölder inequality, Similarly, we have Then it follows from (48), (49) to (54) that , as .
So Using the standard inequality in given by we have from (55) that , as .
Thus, satisfies condition on .

Proof of Theorem 2. By Lemmas 5 and 8, satisfies all assumptions in Lemma 4. Then there exists such that is a solution of problem (1) by Lemma 4. Furthermore, .
We now seek a solution of problem (1). Choose such that , and then for small and thus for any open ball