We study the existence and asymptotic stability in pth moment of a mild solution to a class of nonlinear fractional neutral stochastic differential equations with infinite delays in Hilbert spaces. A set of novel sufficient conditions are derived with the help of semigroup theory and fixed point technique for achieving the required result. The uniqueness of the solution of the considered problem is also studied under suitable conditions. Finally, an example is given to illustrate the obtained theory.

1. Introduction

The stochastic differential equations have been widely applied in science, engineering, biology, mathematical finance and in almost all applied sciences. In the present literature, there are many papers on the existence and uniqueness of solutions to stochastic differential equations (see [14] and references therein). More recently, Chang et al. [5] investigated the existence of square-mean almost automorphic mild solutions to nonautonomous stochastic differential equations in Hilbert spaces by using semigroup theory and fixed point approach. Fu and Liu [2] discussed the existence and uniqueness of square-mean almost automorphic solutions to some linear and nonlinear stochastic differential equations and in which they studied the asymptotic stability of the unique square-mean almost automorphic solution in the square-mean sense. On the other hand, recently fractional differential equations have found numerous applications in various fields of science and engineering [6]. The existence and uniqueness results for abstract stochastic delay differential equation driven by fractional Brownian motions have been studied in [7]. In particular the stability investigation of stochastic differential equations has been investigated by several authors [815].

Let and be two real separable Hilbert spaces with inner products and , respectively. We denote their norms by and . To avoid confusion we just use for the inner product and for the norm. Let be an orthonormal basis of . Throughout the paper, we assume that is a complete filtered probability space satisfying that contains all -null sets of . Suppose is cylindrical -valued Brownian motion with a trace class operator , denote , which satisfies that . So, actually, , where are mutually independent one-dimensional standard Brownian motions. Define as the set of all bounded linear operators with the following norm: It is obvious that is a Hilbert space with an inner product induced by the above norm. Let be called a Hilbert-Schmidt operator. We further assume that the filtration is generated by the cylindrical Brownian motion and augmented, that is, where is the -null sets.

The qualitative properties of stochastic fractional differential equations have been considered only in few publications. El-Borai et al. [16] studied the existence uniqueness, and continuity of the solution of a fractional stochastic integral equation. Ahmed [17] derived a set of sufficient conditions for controllability of fractional stochastic delay equations by using a stochastic version of the well-known Banach fixed point theorem and semigroup theory. Moreover, theory of neutral differential equations is of both theoretical and practical interests. For a large class of electrical networks containing lossless transmission lines, the describing equations can be reduced to neutral differential equations. However, to the author's best knowledge no work has been reported in the present literature regarding the existence, uniqueness, and asymptotic stability of mild solutions for neutral stochastic fractional differential equations with infinite delay in Hilbert spaces. Motivated by this consideration, in this paper we consider the nonlinear fractional neutral stochastic differential equations with infinite delays in the following form: where is the infinitesimal generator of a strongly continuous semigroup of a bounded linear operator in the Hilbert space are two Borel measurable mappings, and is continuous mapping. The fractional derivative is understood in the Caputo sense. In addition, let satisfy , as . Let . Here denote the family of all almost surely bounded, -measurable, continuous random variables with norm . Throughout this paper, we assume that is an integer.

2. Preliminaries and Basic Properties

Let be the infinitesimal generator of an analytic semigroup in . Then, is invertible and generates a bounded analytic semigroup for large enough. Therefore, we can assume that the semigroup is bounded and the generator is invertible. It follows that can be defined as a closed linear invertible operator with its domain being dense in . We denote by the Banach space endowed with the norm , which is equivalent to the graph norm of . For more details about semigroup theory, one can refer [18].

Lemma 1 (see [18]). Suppose that the preceding conditions are satisfied. (a)Let , then is a Banach space.(b)If , then the embedding is compact whenever the resolvent operator of is compact. (c)For every , there exists a positive constant such that .

Definition 2 (see [19]). The fractional integral of order with the lower limit for a function is defined as provided the righthand side is pointwise defined on , where is the gamma function.

Definition 3 (see [19]). The Caputo derivative of order for a function can be written as If is an abstract function with values in , then integrals which appear in the above definitions are taken in Bochner's sense.
According to Definitions 2 and 3, it is suitable to rewrite the stochastic fractional equation (3) in the equivalent integral equation
In view of [18, 3.1] and by using Laplace transform, we present the following definition of mild solution of (3).

Definition 4. A stochastic process is called a mild solution of (3), if (i) is -adapted and is measurable, ; (ii) has càdlàg paths on almost surely and for each , the function is integrable such that the following integral equation is satisfied: (iii), where with a probability density function defined on .

The following properties of and [18] are useful.

Lemma 5. Under previous assumptions on and , (i) and are strongly continuous; (ii)for any and one has

Definition 6. Let be an integer. Equation (8) is said to be stable in th moment if for arbitrarily given there exists a such that

Definition 7. Let be an integer. Equation (8) is said to be asymptotically stable in th moment if it is stable in th moment and, for any , it holds

3. Main Result

In this section, we prove the existence, uniqueness, and stability of the solution to fractional stochastic equation (3) by using the Banach fixed point approach.

In order to obtain the existence and stability of the solution to (3), we impose the following assumptions on (3).(H1) There exist constants and such that . (H2) There exists a positive constant , for every and , such that (H3) There exist such that is -valued, is continuous and there exists a positive constant such that for every and . (H4), where , , and .In addition, in order to derive the stability of the solution, we further assume that(H5) It is obvious that (3) has a trivial solution when under the assumption .

Lemma 8. Let and let be an -valued, predictable process such that . Then,

Theorem 9. Let be an integer. Assume that the conditions hold, then the nonlinear fractional neutral stochastic differential equation (3) is asymptotically stable in the th moment.

Proof. Denote by the space of all -adapted process , which is almost surely continuous in for fixed and satisfies for and as . It is then routine to check that is a Banach space when it is equipped with a norm defined by . Define the nonlinear operator such that and, for , As mentioned in Luo [20], to prove the asymptotic stability it is enough to show that the operator has a fixed point in . To prove this result, we use the contraction mapping principle. To apply the contraction mapping principle, first we verify the mean square continuity of on . Let and let be sufficiently small, and observe that Note that The strong continuity of [18] implies that the right hand of (19) goes to 0 as . In view of Lemma 5 and the Holder's inequality, the third term of (18) becomes where . Since is sufficiently small, the right hand side of the above equation tends to zero as .
Next we consider By the Holder’s inequality, we obtain Therefore, the right hand side of the above equation tends to zero as and sufficiently small. Further, we have As above, the right hand side of the above inequality tends to zero. Similarly, we have as . Thus is continuous in th moment on .
Next we show that . Let . From (18), we have Now, we estimate the terms on the right hand side of (24) by using the assumptions , , and . Now, we have For and for any there exists a such that for .
Therefore, For the fourth term of (24), we have Also, we have By the same discussion as above, we have that (28) tends to zero as . Thus as . We conclude that .
Finally, we prove that has a unique fixed point. Indeed, for any , we have Therefore, is a contradiction mapping and hence there exists a unique fixed point, which is a mild solution of (3) with on and as .
To show the asymptotic stability of the mild solution of (3), as the first step, we have to prove the stability in th moment. Let be given and choose such that satisfies .
If is mild solution of (3), with , then satisfies for every . Notice that on . If there exists such that and for . Then (24) show that which contradicts the definition of . Therefore, the mild solution of (3) is asymptotically stable in th moment.

In particular, when from Theorem 9 we have the following.

Theorem 10. Suppose that the conditions hold. Then, the stochastic fractional differential equations (3) are mean square asymptotically stable if , where .

When , (3) reduces to From Theorems 9 and 10, we can easily get the following result.

Corollary 11. Suppose the assumptions and hold. Then, the stochastic equations (8) are mean square asymptotically stable if .

Example 12. Consider the following stochastic nonlinear fractional partial differential equation with infinite delay in the following form where denotes a standard cylindrical Wiener process and a standard one-dimensional Brownian motion. To write the system (32) into the abstract form of (3), we consider the space and define the operator by with domain where is the orthogonal set of eigenvectors in . It is well known that generates a compact, analytic semigroup in and It is well known that . Take . Since , we can get the inequality . Further, if we impose suitable conditions on , , and to verify assumptions of Theorem 10, then we can conclude that the mild solution of (32) is mean square asymptotically stable.