#### Abstract

We consider the multiplicity of nontrivial solutions of the following quasilinear elliptic system , , , , , where , , , , , . The functions , , , , , , and satisfy some suitable conditions. We will prove that the problem has at least two nontrivial solutions by using Mountain Pass Theorem and Ekeland's variational principle.

#### 1. Introduction

In this paper, we consider the multiplicity of solutions to a class of quasilinear elliptic system where , , , , , .

In recent 20 years, existence and multiplicity for a quasilinear elliptic equation and equations have been studied widely. In [1], Yu has studied the existence of a nontrivial weak solution to the quasilinear elliptic problem where is a smooth exterior domain in , , is a positive function, and is a nonnegative function. Problems of this type are motivated by mathematical physics, since certain stationary waves in nonlinear Klein-Gordon or SchrÃ¶dinger equations can be reduced to this form (see [2, 3]). Problem (2) began to attract more and more attention after the work of Yu [1]; we refer the readers to [4â€“8].

In [9], Hsu studied the following elliptic system: where is a bounded domain, , , and , satisfy . The problem (3) is the special case of problem (1), while , . By applying Nehari manifold method, the author proved the problem (3) has at least one positive solution when and has at least two positive solutions when .

This paper is motivated by the results of [9â€“12]. Replacing the Nehari manifold methods, we use the Mountain Pass Theory and Ekelandâ€™s variational principle to study the existence of multiple solutions for problem (1). It seems difficult to get the same result by Nehari manifold methods.

Throughout this paper, we make the following hypotheses:, , , ., ., , , , .

In this paper, we let the completion of the space with respect to the norm , and let the completion of with respect to the norm .

We define where , .

The following inequality is called Caffarelli-Kohn-Nirenberg inequality [13]. There is a constant such that where , , , and .

Let denote the usual Sobolev space with the norm Then is a reflexive and separable Banach space endowed with the norm .

*Definition 1. *We say that is a weak solution of problem (1) if for all , there holds

It is clear that problem (1) has a variational structure. Let be the energy functional corresponding to problem (1) which is defined by
where
Then, we see that functional and for all , there holds
In particular, it follows from (10) that
It is clear that the critical points of the energy functional correspond to the weak solutions of problem (1). Thus, to prove the existence of weak solutions for the problem (1), it is sufficient to show that admits a sequence of critical points in . Our main result is the following.

Theorem 2. *Assume that hold. There exist such that if satisfy and , then the problem (1) has at least two nontrivial solutions.*

#### 2. The Proof of Theorem 2

In this section, we first introduce a compact embedding theory which is an extension of the classical Rellich-Kondrachov compactness theorem; see Xuan [13].

Lemma 3. *Assume that is an open bounded domain with boundary and , , . Then the embedding is continuous if and and is compact if and .*

The following lemma is called the Mountain Pass Theorem, see [14] (also see [15]).

Lemma 4 (Mountain Pass Theorem). *Let be a real Banach space and with . Suppose that satisfies condition and** there are , such that when ;** there is , such that .**
Define
**
Then
**
is a critical value of .*

Lemma 5. *Assume that hold. There exist such that if the parameters , satisfy , and , then satisfies the assumptions in Lemma 4.*

*Proof. *Let . By (8),
By and the HÃ¶lder inequality, we have
where , .

Similarly,
Hence, it follows from (15) and (16) that
Since , we obtain from the HÃ¶lder inequality and (5) that
Similarly,
Then,
Using the HÃ¶lder inequality, we get
where , .

Similarly, we have
Thus,
with .

Let
To verify in Lemma 4, it suffices to show that for some .

Note that where or . Then has a minimum at . In order to find , we have
So that and
Moreover, implies that
where .

Thus, it follows from (23) and (27) that there exist such that with and . Then satisfies the assumption in Lemma 4.

We now verify in Lemma 4. Choose â€‰â€‰â€‰â€‰ such that
Then
and (), since . Therefore, there exists large enough, such that . Then we take and . The condition of Lemma 4 is true. This completes the proof of Lemma 5.

Lemma 6. *Let - hold.**If and are bounded sequences in , then there exist subsequences, still denoted and , and , such that as *

*Proof. *Let , .

Since is bounded in , is bounded in for for all . By Lemma 3, has a subsequence which converges in . Likewise, the subsequence is bounded in . So that it has a subsequence which converges in . Since is a subsequence of , in . Continuing this process, we obtain a sequence with the following properties:
It is clear that a.e in , where
By a diagonal process, we take which is a subsequence of . Thus, we have
Without loss of generality, we assume that the subsequence is itself. So
This implies that for ,
Now, we prove
Indeed,
where .

The fact gives that .

Then, (37) implies that
This gives (36).

In the following, we show that
Since the sequence is bounded in , we can assume (up to a subsequence) that weakly in and for some constant and all .

By (36), we know that for any , there exists so large that
Since the embeddding is compact (see Lemma 3), we have
Thus, there exists , when ,
So
This shows that in as .

By Brezis-Liebâ€™s Lemma in [16], we obtain that (39) is true.

Similarly, we can prove

*Remark 7. *By Brezis-Liebâ€™s Lemma, it follows from (30) that

Lemma 8. *Assume that hold. Then defined by (8) satisfies condition on .*

*Proof. *Let be a sequence of in ; that is, , , in . We first claim that is bounded in . In fact, for large , we obtain
Since , we conclude that is bounded. Then, the and are bounded in , respectively. Thus, there exist and a subsequence of , still denoted by , such that
Denote
Then the fact in implies as .

Since that in , we see that
Let
From dominated convergence theory and Lemma 6, we can conclude
From HÃ¶lder inequality,
Similarly, we have
Then it follows from (48), (49) to (54) that , as .

So
Using the standard inequality in given by
we have from (55) that , as .

Thus, satisfies condition on .

*Proof of Theorem 2. *By Lemmas 5 and 8, satisfies all assumptions in Lemma 4. Then there exists such that is a solution of problem (1) by Lemma 4. Furthermore, .

We now seek a solution of problem (1). Choose such that , and then
for small and thus for any open ball