#### Abstract

We study the fractional Laplacian equation , , here , , is the critical exponent, and is a real potential function. Employing the variational method we prove the existence of nontrivial solutions for small and large.

#### 1. Introduction

We consider the nonlinear Schrödinger equation: whereis the Planck constant. When looking for stationary waves of the formwith, one is led to considering the elliptic equation in; namely, replacingby, one sees thatmust satisfy Settingand, this equation is transformed into Problem (3) has been widely studied in the literature (see, for instance, [1, 2] and references therein), whereis the critical exponentandis a potential well.

The study of existence and concentration of the semiclassical states of Schrödinger equation goes back to the pioneer work [3] by Floer and Weinstein. Ever since then, equations of (3) type with subcritical nonlinearities (for) have been studied by many authors. For critical nonlinearity (for), Clapp and Ding [1, 2] established the existence and multiplicity of positive solutions and minimal nodal solutions which localize near the potential well forsmall andlarge.

The fractional Schrödinger equation is a fundamental equation of fractional quantum mechanics. It was discovered by Nick Laskin as a result of extending the Feynman path integral, from the Brownian-like to Lévy-like quantum mechanical paths. The term fractional Schrödinger equation was coined by Nick Laskin.

Recently, a great attention has been devoted to the fractional and nonlocal operators of elliptic type, both for their interesting theoretical structure and in view of concrete applications in many fields such as combustion and dislocations in mechanical systems. This type of operator seems to have a prevalent role in physical situations and has been studied by many authors [4–9] and references therein. In [5], Di Nezza et al. deal with the fractional Sobolev spaceand analyze their role in the trace theory. They prove continuous and compact embeddings, investigating the problem of the extension domains and other regularity results. In [8], Felmer et al. proved the existence of positive solutions of nonlinear Schrödinger equation involving the fractional Laplacian in. They further analyzed regularity, decay, and symmetry properties of these solutions. Servadei and Valdinoci [9] studied the existence of nontrivial solutions for equations driven by a nonlocal integrodifferential operatorwith homogeneous Dirichlet boundary conditions. They give more general and more precise results about the eigenvalues of a linear operator.

The aim of this paper is to study the fractional Laplacian equation: where,,, andis the usual fractional Sobolev space, andis the corresponding critical exponent. Supposesatisfies the following assumptions. ,, is a nonempty bounded set with smooth boundary, and. There existssuch that wheredenotes the Lebesgue measure in.

The fractional Laplace operatorin (4) can be defined as

We say that a functionsolves (4) in the weak sense if

Define the energy functional by Then we know the critical points ofare exactly the weak solutions of (7). In this sense we will prove the existence of the critical points of the functional. Fréchet derivative ofis

Concerning the Schrödinger equation: Clapp and Ding [1] proved the following.(a)Assumeandhold and. Then, for every, there existssuch that (4) has a least solutionfor each, whereis the first eigenvalue ofonwith boundary condition.(b)Assumeandhold and. Then, there existand for eachthere exist two numbersandsuch that if, then (4) has at least(the number of solutions is bounded from below by a topological invariant) solutions with energy.(c)Every sequence of solutionsof (10) such that,andasconcentrates at a solution of whereis the best Sobolev constant.

Our aim is to show that (a) and (c) can be extended to problem (4). In this paper, we have the following results.

Theorem 1. *Assumeandholdand. Then, for every, there existssuch that (4) has at least a solutionfor each, whereis the first eigenvalue ofonwith boundary condition. There is a great deal of work on; see for example [9]. We have
*

Theorem 2. *Every sequence of solutionsof (4) such that,andasconcentrates at a solution of
**
whereis defined as in.*

Hereis defined as whereis anspace with potential and will be defined in Section 2.

There is a great deal of work on (13); see, for example, [4, 6, 7] and the references therein. Among them Servadei and Valdinoci [4, 6, 7] studied the problem whereis an open bounded set with Lipschitz boundary in, ,,is a real parameter.is defined as follows: Hereis a function such that there existssuch thatandfor any. They proved that problem (15) admits a nontrivial solution for any. They also studied the caseand, respectively.

Clapp and Ding [1] proved the existence of minimizing sequence for energy function of (10) on Nehari manifold and assumed that it is a Palais Smale sequence by Ekeland’s variational principle. Since Palais Smale conditions hold, this finished the proof of (a). For (c), they analyzed the problem directly. We will show that their method can be extended to the case.

This paper is organized as follows. In Section 2, we give some preliminary results. In Section 3, we finish the proof of Theorem 1. In Section 4, we finish the proof of Theorem 2.

#### 2. Preliminary Results

Throughout this paper we writefor thenorm for. We always assume thathold,,,and.is the first eigenvalue ofon.is a nonempty bounded set with smooth boundary.

We consider the fractional Sobolev space: with norm And let be the Hilbert space equipped with norm If, then it is equivalent to the norms Thusis continuously embedded in.

*Remark 3. *We know the embeddingis continuous; see [5] or [8]. So the embeddingis also continuous for any.

Thanks to Remark 3, we can define the constantas in formula (14) and get that.

Lemma 4. *Letbe such thatand. Then, there is asuch that, up to a subsequence,in.*

*Proof. *Ifstrongly in, we prove. Setand. Forlarge enough that, thanks to. So, we get
for every. This implies thatfor a.e.. Hence, sinceis smooth,.

We will show thatstrongly in. Letwithas in, and let. Then
as. Setting, where, and choosingand , we have
as, thanks to. Sincein,
as. By,
as. Thusstrongly in.

We denoteand bythe-inner product and write for. Set, the infimum of the point spectrum of. Observe that and thatis nondecreasing in.

Lemma 5. *For each, there existssuch thatfor. Consequently,
**
for all, , whereis a constant.*

*Proof. *Assume, by contradiction, that there exists a sequencesuch thatfor alland. Letbe such thatand. Then
for alllarge. By Lemma 4 there is asuch that, up to a subsequence,in, and thus. Using Fatou’s theorem, we know
Consequently,
Sinceis the first eigenvalue ofonwith boundary condition, we have. This is a contradiction.

In the following, enlargingif necessary, we assume; thus

#### 3. The Proof of Theorem 1

In this section we will finish the proof of Theorem 1.

The critical points oflie on the Nehari manifold Sinceand, the functionhas a unique maximum pointand. Define and we observe that From Lemma 5, the constantis positive. On the other hand, we define where

Proposition 6. *
Consider.*

*Proof. *Proposition is proved, for instance, in [8, see Section].

is radially diffeomorphic. For, the functionalis So,

We consider the functional on. Its Nehari manifold is radially diffeomorphic. Set

Proposition 7. *Ifandthen
**
whereis defined in formula (14) andis given in Lemma 5.*

*Proof. *By Lemma 5,for all. Taking infima overgives the first inequality. Sinceandfor, it follows that. By [6, see Section] and [10, see Section], we knowandis achieved at some. Thusbecause otherwould be also achieved atwhich vanishes outside, contradicting the maximum principle.

Hence, Proposition 7 is proved.

By definition ofand Proposition 6, there exists a minimizing sequence foronand we note. By Ekeland’s variational principle, we may assume that it is a Palais Smale sequence. So we have as.

Proposition 8. *has at least one critical point with critical valuefor eachand.*

*Proof. *We proceed by steps.*Step* *1*. The sequenceis bounded in. *Proof*. For anyby (46) and (47) it easily follows that there existssuch that
As a consequence of (48) we have
By (49) and the definition ofwe have

Thusis bounded in. *Step **2*. Problem (7) admits a solution. *Proof*. By Stepandis a reflexive space, up to a subsequence, still denoted by, there existssuch thatweakly in; that is,
as. Since Stepand Remark 3, we have thatis bounded in. Sinceis a reflexive space, up to a subsequence
as. While by Lemma 4, up to a subsequence,
as. By (52) and the fact thatis bounded inwe have
as.

Since (47) holds true, for any

Passing to the limit in this expression asand taking into account (51), (53), and (55), we get
for any; that is,is a solution of problem (7).*Step **3.* The following equality holds true:
*Proof.* By Step, takingas a test function in (7), we have
So we get
Hence, Stepis proved.

Now, we conclude the proof of Proposition 8.

We write, and thenweakly in. Moreover, since (54) holds true, by the Brézis-Lieb Lemma, we get
Then,
Byand, we get
As in the proof of Lemma 4 one shows that
as, where. Let. Then, by (34),
Passing to the limit yields. Eitheror. If, the proof is complete. Assuming, we obtain from Step, (45), and (62) that
which is a contradiction. Thus, and
as. This ends the proof of Proposition 8.

We have finished the proof of Theorem 1 by Proposition 8.

#### 4. The proof of Theorem 2

*Proof of Theorem 2. *Letbe a sequence of solutions of (4) such that,and. Then, by Lemma 4, there is asuch that, up to a subsequence,in. Bythat is a solution of (4), we have
for any. Ifthenfor all, so lettingwe obtain
for any. So,is a solution of (13). We write. Then,in.

Sincefor, we get
Byinand the Brézis-Lieb Lemma, we have
So, we can get
We claim that. Assume. Then,
thanks to (73). It follows that
This is a contradiction. Thusand, by (73). Hence, by (71)
Sinceinandfor,
Therefore,and (76) implies thatin.

#### Acknowledgment

This paper is supported by the National Nature Science Foundation of China (11271331).