#### Abstract

The concept of quasivariational inequality problem on proximally smooth sets is studied. Some sufficient conditions for solving the existence of solutions of such a problem are provided; also some interesting cases are discussed. Of course, due to the significance of proximally smooth sets, the results which are presented in this paper improve and extend many important results in the literature.

#### 1. Introduction and Preliminaries

Variational inequality theory is a branch of the mathematics which is important, and it was also the inspiration for researchers to find new works, both in terms of mathematics and applications such as in economics, physical, biological, and engineering science, and other applied sciences. In 1973, Bensoussan et al. [1] introduced and studied the concept of the quasivariational inequality, which is a generalized form of the classical variational inequality that introduced by Stampacchia [2]. Later, many researchers proposed and analyzed the concept of the generalized quasivariational inequality; see, for example [3–18]. It is worth mentioning that the quasivariational inequality problem is of interest to study, since in many important problems the considered set also depends upon the solutions explicitly or implicitly. In fact, the concept of the quasi variational inequality has been applied in many fields such as in economics and transportation equilibrium, control and optimization theory, mathematical programming, and game theory.

In the early period of the research, it should be pointed out that almost all the results regarding the existence and iterative schemes for solving those variational inequalities problems are being considered in the convexity setting. This is because they need the convexity assumption for guaranteeing the well definedness of the proposed iterative algorithm which depends on the projection mapping. However, in fact, the convexity assumption may not be required because it may be well defined even if the considered set is nonconvexs (e.g., when the considered set is a closed subset of a finite dimensional space or a compact subset of a Hilbert space, etc.). However, it may be from the practical point of view one may see that the nonconvex problems are more useful than convex case. Consequently, now many researchers are paying attention to many nonconvex cases.

Let be a mapping and let be a set-valued mapping, where is a family of all nonempty closed subsets of . In this paper, we are interested in the following problem: find such that where is denoted for the proximal normal cone of of . The problem of type (1) was introduced by Bounkhel et al. [19]. In such a paper, they proposed some iterative algorithms for finding a solution of type (1), when the considered mapping is a set-valued mapping, while, in this paper, we will provide sufficient conditions for the existence of a solution of such a problem (1). To do this, let us start by recalling some basic concepts and useful results that will be needed in this work.

Let be a real Hilbert space equipped with norm and inner product . Let be denoted for the class of all nonempty subsets of and denoted for the family of all nonempty closed subsets of .

For each , the usual distance function on to is denoted by , that is: Let and . A point is called the closest point or the projection of onto if The set of all such closest points is denoted by , that is: Further, for each , the proximal normal cone to at is given by

The following is called * the proximal normal inequality*; the proof of this characterization can be found in [20].

Lemma 1 (see [19]). *Let be a closed subset of a Hilbert space . Then
*

We recall also [21] that the Clarke normal cone is given by where means the closure of the convex hull of . It is clear that one always has . The converse is not true in general. Note that is always a closed and convex cone and that is always a convex cone but may be nonclosed (see [20, 21]).

In 1995, Clarke et al. [22] have introduced and studied a new class of nonconvex sets, which are called proximally smooth sets. This class of proximally smooth sets has played an important part in many nonconvex applications such as optimization, dynamic systems, and differential inclusions. Subsequently, the proximally smooth sets have been proposed by many researchers. In recent years, Bounkhel et al. [19], Cho et al. [23], Moudafi [24], Noor [25], Noor et al. [26], Petrot [27], and Pang et al. [28] have considered both variational inequalities and equilibrium problems in the context of proximally smooth sets. They suggested and analyzed some projection type iterative algorithms by using the prox-regular technique and auxiliary principle technique. Here, we will take the following characterization proved in [20] as the definition of proximally smooth sets. Note that the original definition was given in terms of the differentiability of the distance function (see [22, 29]).

*Definition 2. *For a given , a subset of is said to be * uniformly prox-regular with respect to *, say, * uniformly **-prox-regular set*, if for all and for all , one has

For the case of , the uniform -prox-regularity is equivalent to the convexity of (see [22]). Moreover, it is known that the class of uniformly prox-regular sets is sufficiently large to include the class -convex sets, submanifolds (possibly with boundary) of , the images under a diffeomorphism of convex sets, and many other nonconvex sets; see [20, 29].

From now on, we will denote for the class of all uniformly -prox-regular closed subset of , where is fixed positive real number. Also, for each , we write

*Remark 3. *If and are mappings, then the problem of type (1) is equivalent to the following problem: find such that
see [19]. This means, in particular, that the problem (1) contains the well-known Stampacchia's variational inequality, as a special case.

The following lemma which summarizes some important consequences of the uniformly prox-regularity sets is needed in the sequel. The proof of this result can be found in [22, 29].

Lemma 4. *Let and be a nonempty closed subset of . If is uniformly -prox-regular set, then the following holds. *(i)* For all . *(ii)* For all is Lipschitz with constant on . *(iii)* The proximal normal cone is closed as a set-valued mapping.*

*Remark 5. *If is uniformly -prox-regular set, as a direct consequence of Lemma 4 (iii), we have .

The following definition and lemma are also needed, in order to obtain our main results.

*Definition 6. *A set-valued mapping is said to be *-Lipschitz* if there exists such that

Lemma 7 (see [19]). *Let and let be a -Lipschitz set-valued mapping with uniformly -prox-regular values then the following closedness property holds: “For any and with and , one has .”*

#### 2. Main Results

In this paper, we are interested in the following classes of mappings.

*Definition 8. *Let be a mapping. Then is called (a)-strongly monotone if there exists such that
(b)-* Lipschitz * if there exists such that
That is, in other word, we will make the following assumption.

*Assumption *. Let and be mappings. (i) is a -strongly monotone and a -Lipschitz single-valued mapping; (ii) is a -Lipschitz set-valued mapping; (iii) there is such that

*Remark 9. *Let and , for some positive real numbers with . If we define by , where is a mapping defined by for , then we see that is a -Lipschitz mapping, and Assumption (iii) is satisfied with a constant . This means that Assumptions (ii) and (iii) are independent.

The following remark is very useful in order to prove our results. Before seeing that, for the sake of simplicity, let us make a notation: for each and , we will write .

*Remark 10. *Let , and be five positive real numbers such that , and . If is a function defined by
for all , where , then is an increasing continuous function on its domain. Moreover, we can check that is an element of the range of .

Next, for a fixed positive real number , we pick a real number . Here, we notice that . Let us consider now a case when . Then, by the definition of and is an increasing continuous function, for we have that is, Since , this gives This allows us to take a real number such that where .

Now we are in a position to present our main results.

Theorem 11. *Let be a single-valued mapping and let be a set-valued mapping. Assume that Assumption holds and the following control conditions are satisfied:*(i)* and , where ;*(ii)*, where ; *(iii)*, where and are defined as Remark 10. **If there is such that , where and is a real number corresponding to which is chosen as in (19), then the problem (1) has a solution.*

*Proof. *Firstly, we will define a sequence in as follows: consider an element in such that ; we see that
This means . Subsequently, by Lemma 4 (i), we know that . Let . So, by a choice of , we see that
On the other hand, we see that
Using this one together with (21), we obtain
Note that, by , we have . So, since , by (23) we have
Hence , and it follows that . Let . In the same way as obtaining (21) and (23), we see that
By (25), we have
By continuing this process, we can construct a sequence in such that
for all .

Further, let us consider
for each . This implies
Write . From the previous argument, we see that the sequence also has a property that
for all .

Next, we will show that is a convergent sequence, and its limit is nothing but a solution of the problem (1).

Now, by using the Assumption () (iii) and Lemma 4 (ii), we have
Meanwhile, since is a -strongly monotone and -Lipschitz mapping, we see that
Let . Observe that . Moreover, it follows that
By replacing (33) into (31), we get
Let . Then, by a choice of , one can check that . Subsequently, by (34), we have
Hence, for any , we see that
This implies that is a Cauchy sequence in , since . So, by the completeness of , there exists such that as .

We now finish the proof by showing that is a solution of problem (1). To do this, we will start by asserting that . Indeed, since and is a -Lipschitz mapping, we have
Thus, since , we see that . So, by the closedness of , it guarantees that .

Next we show that . Let us observe that
is equivalent to
And this means
Thus, by using the continuity of mapping , from Lemma 7 we see that . This implies , as required. This means that is a solution of the problem (1), and the proof is completed.

*Remark 12. *A condition which has been proposed in the assumptions of Theorem 11 is that “*there is ** such that *,* where *.” Here, in view of Remark 10 together with the following facts, one may see that our choice should be sharpest. (i). (ii) The function is an increasing function on its domain.(iii), where .

*Remark 13. *Assume that all assumptions of Theorem 11 hold. By starting with an element such that , a sequence which is defined by
is well defined. Moreover, it is a convergent sequence and its limit is a solution of the problem (1).

*Remark 14. *Recall that a set-valued mapping is said to be * Hausdorff Lipschitz * if there exists a real number such that
where stands for the Hausdorff distance relative to the norm associated with the Hilbert space , that is:
It is easy to check that the class of Lipschitz mappings, which has been defined in Definition 6, is larger than the class of Hausdorff Lipschitz mappings. Thus, Theorem 11 can also be applied when the Assumption () (ii) is replaced by “ is -Hausdorff Lipschitz set-valued mapping.”

*Remark 15. *Let be a uniformly prox-regular closed subset of . If is defined by
then we see that Assumptions () (ii) and (iii) are satisfied with a constant zero. In this case, Theorem 11 is reduced to a result which was presented by Petrot [27].

It is well known that if is a closed convex set, then it is -prox-regular set for every (see [22]). Using this fact, and by careful consideration of the proof of Theorem 11, one can see that control conditions (i) and (iii) of Theorem 11 can be omitted. So, we have the following results immediately.

Corollary 16. *Let be a single-valued mapping and let be a set-valued mapping, where is a family of nonempty closed convex subset of . If the Assumption holds and , where , then the problem (1) has a solution.*

Finally, in view of Remark 15 and Corollary 16, we can obtain the following classical result immediately.

Corollary 17. *Let be a real Hilbert space, let be a closed convex subset of , and let be a single-valued mapping. If is a strongly monotone and Lipschitz single-valued mapping, then the problem (1) has a solution.*

#### 3. Conclusion

In this paper, we provide some conditions for the existence theorems of the quasivariational inequality problem on a class of nonconvex sets. In fact, there are two constraints on the assumptions of considered mapping in the main Theorem 11, that is, (i) is a strongly monotone and Lipschitz mapping, (ii) the range of mapping is a bounded set. Hence, these lead to some natural questions in the future works for relaxing these constraints. At this point, we desire that the results presented here will be useful for those researchers, because this paper may also be extended and generalized for considering the mutlivalued and set-valued extended general variational inequalities problems.

#### Acknowledgment

This work is supported by the Centre of Excellence in Mathematics, Commission on Higher Education, Thailand.