#### Abstract

In this paper, we are concerned with the following coupled Schrödinger equations where ,,, and; is a parameter; and and . Under some suitable conditions that or and with , the above coupled Schrödinger system possesses nontrivial solutions if , where is related to , and .

#### 1. Introduction

We consider the following coupled Schrödinger equations in this paper: where , , , and are the Sobolev critical exponent; is a parameter; and and .

As it is known in [1], this type of systems arises in nonlinear optics. In the past years, under different kinds of assumptions on the potential and the nonlinearity , many authors [2–8] focus on the following kind of Schrödinger equation:

As one knows, single-mode optical fibers are not really “single mode” but actually bimodal because of the presence of birefringence. So recently, the coupled Schrödinger systems are investigated by the authors [9–12]. For more related results and physical background on Schrödinger systems, please see [13–23] and references therein.

In [11], the authors investigated standing waves for the following kind of coupled Schrödinger equations: where , , , , and as . Under the following conditions,

(A0) there exist positive constants and such that , , and ; they obtained the existence of a positive solution for (3) if is sufficiently small. But, if or , then cannot hold. So in the very recent paper [12], Peng et al. investigated the following coupled Schrödinger equations and generalize the result in [11]: where are the same as in (3), . Under the following conditions,

(A1) and , and there exist constants and such that the measure of the sets and are finite

(A2) there exists a constant such that for all ; Peng et al. proved that system (4) has at least one nontrivial solution. An interesting question is what will happen if the nonlinearity is also critical growth in system (4)? Motivated mainly by the above-mentioned results, we will answer this question and prove that system (1), under conditions (A1) and (A2), and

(A3) there exist constants , , , , , , , such that possesses nontrivial solutions if , where is related to , and . As far as we know, similar results for system (1) with a critical exponent have not been investigated by variational methods in the literature. The following condition is similar to condition (A1):

(A1’) and , and there exist constants and such that the measure of the sets and are finite.

Since and , one can choose such that where

and are embedding constants and is the volume of the unit ball in . From (A1’) and (A1), using and , one can let such that

Let and , then system (1) can be rewritten as and the functional of (9) is given by

As is known, the solutions of (1) are the critical points of . The main results are the following.

Theorem 1. *Suppose that (A1)–(A3) or (A1’)–(A3) hold. Then, (9) possesses at least one nontrivial solution such that for .*

Theorem 2. *Suppose that (A1)–(A3) or (A1’)–(A3) hold. Then, (1) possesses at least one nontrivial solution such that for .*

*Remark 3. *Since the presence of the terms , , , and , system (1) is more general than (4), and it is more difficult to deal with the nontrivial solutions. In order to prove that system (1) has nontrivial solutions, we need to find some conditions to restrict , , , and . It seems that there is no literature considering system (1).

#### 2. Preliminaries

Let

From Lemma 1 of [17], by (A1) or (A1’) and the Sobolev inequality, there exists a positive constant independent of such that where . Then, is a Banach space for equipped with the norm given by (12). Moreover, for , one has where is the usual norm in space . From (12), we rewrite as

It is not difficult to see that and

Then, ; moreover,

In the next section, we will prove the main results.

#### 3. Proof of the Main Results

*Proof of Theorem 1. *The proof of Theorem 1 is divided into four steps.

*Step 1. *We first prove that for any , one has
where . From (8), (9), (17), (18), (19), and (A3), we have
Similarly, from (8), (9), (17), (18), (19), and (A3), we have
which together with (21) implies that (20) holds.

*Step 2. *Let , we should prove that there exists a constant and a sequence satisfying
By a standard argument, one can obtain (23) by employing the mountain-pass lemma without the (PS) condition, so we omit the details here.

*Step 3. *We prove that any sequence satisfying (23) is bounded in . From (A2) and Young’s inequality, we have
For , from (15), (16), (23), and (24), we have
For , from (15), (16), (23), and (24), we obtain
It follows from (25) and (26) that is bounded in .

*Step 4. *We show that there exists a nontrivial solution. By Steps 1–3, we know that there exists a bounded sequence satisfying (23) with
Passing to a subsequence, one can suppose that in and , as . Now, we verify that . Arguing by contradiction, assume that , that is, in , so by [24], we have in , , and a.e. on . Since and are sets with finite measure, we have
Similar to [12], from (14), (28), (29), and the Hölder inequality, we obtain
It follows from (15), (16), (23), and (A3) that
From (14), (30), (31), and (32), we have