#### Abstract

We consider a new system of generalized variational inequalities (SGVI). Using the penalty methods, we prove the existence of solution of SGVI in Hilbert spaces. Our results extend and improve some known results.

#### 1. Introduction

Throughout this work, let be real Hilbert space with a norm and inner product . Let be a nonempty closed and convex subset of . Given nonlinear mappings and , , we consider the following problem:

which is called the system of generalized variational inequality problem (SGVI), where and are constants. An element is called a solution of the problem (1.1) if

Special cases of problem (1.1) are as follows.

(1) If , , and , , then problem (1.1) reduces to the variational inequality

Problem (1.3) was introduced by Browder [1, 2] and studied by many authors (e.g., see [3–7]).

(2) If , , , and , then problem (1.1) reduces to the system of variational inequality problem

Problem (1.4) was introduced and studied by Verma [8].

(3) If , , and , then problem (1.1) becomes the following system of nonlinear variational inequalities

which was considered by Chang et al. [9].

*Remark 1.1. *For a suitable choice of and , the problem (1.1) includes many kinds of known systems of variational inequalities as special case (see [4–10] and the references therein). In this work, by using the penalty method, we study the existence of solutions for SGVI.

#### 2. Preliminaries

In the sequel, we give some definitions and lemmas. In what follows, and stand for strong and weak convergence, respectively.

*Definition 2.1 (see [11, pages 96–105]). *Let be a mapping.(i)The mapping is said to be pseudo-monotone if is closed convex set and its restrictions to finite-dimensional subspaces are demicontinuous, and for every sequence , in , the inequality implies that
(ii)The mapping is said to have the generalized pseudo-monotone property if for any sequence with in and in such that
we have and as .(iii)The mapping is said to be monotone if for any , the inequality holds.(iv)The monotone mapping is said to be maximal if the inequality for all (graph of implies .(v)The mapping is said to be coercive if there exists a continuous increasing function with as such that for

*Definition 2.2. *Let be a mapping.(i)The mapping is said to be continuous if there exist constants such that
(ii)The mapping is said to be strongly monotone in the first argument if there exists such that for each fixed , we have
(iii)The mapping is said to be strongly monotone in the second argument if there exists such that for each fixed , we have
(iv)The mapping is said to be coercive in the first (second) argument if there exists a continuous increasing function with as such that for each fixed , we have

*Definition 2.3 (see [11]). *(i) A single-valued bounded demicontinuous monotone operator with the property that is said to be the penalty operator of .

(ii) is demicontinuous if implies that .

(iii) is hemicontinuous if , , , implies that , where stands for weak convergence in .

Lemma 2.4 (see [11, page 267]). *Let be the projection on , then the mapping is a penalty operator of .*

Lemma 2.5 (see [11, page 98]). *(i) Any maximal monotone mapping with is a pseudo-monotone one.**(ii) Any pseudo-monotone mapping has the generalized pseudo-monotone property.**(iii) The monotone mapping is maximal if and only if ,.*

#### 3. Main Results

We consider first the problem of finding such that

which is called the system of nonlinear variational equation, where , and , is the penalty operator of (see Lemma 2.4). We will prove the existence of solutions for problem (3.1).

Lemma 3.1. *Let be real Hilbert space and be a nonempty closed convex subset of . Let mapping be continuous and strongly monotone in the first argument and mapping be continuous and strongly monotone in the second argument. If
**
then for each , problem (3.1) admits a solution .*

*Proof. *Since is continuous monotone operator, is hemicontinuous monotone operator. By Corollary in [11], is maximal monotone. So, problem (3.1) is equivalent to the following problem
where , , and is the identity mapping on .

For any given , we can compute the sequences and by Picard iterative schemes:
Since is nonexpansive [11] and is continuous and strongly monotone in the first argument, then by (3.4), we have
Similarly, we get from (3.5)
By (3.6) and (3.7), we have
where . By (3.2), we know that , and (3.8) implies that and are both Cauchy sequences. Thus, there exist , such that (as . By continuity of , , and and algorithm (3.4) and (3.5), we know that satisfies the following relation:
Therefore, is a solution of problem (3.1). This completes the proof of Lemma 3.1.

*Remark 3.2. *If is *Lipschitz* continuous, then is bounded, that is, map bounded set to bounded set.

Now, we give our main results.

Theorem 3.3. *Let be a real Hilbert space and be a nonempty closed convex subset with . Let be the same as in Lemma 3.1. If the mapping is coercive with respect to the first argument and the mapping is coercive with respect to the second argument, then for any , there exists one solution of the SGVI (1.1).*

*Proof. *Let . Then, by Lemma 3.1, the problem (3.1) has a solution .

By the monotonicity of , we have
since is coercive with respect to the first argument. Hence remains bounded as . Similarly, we have
Hence remains also bounded as . The boundedness of and and the fact that
implies that and for some constant . By reflexivity of , we can choose sequences and , , such that , and as and . Using the fact that and as and taking the limit in the monotonicity relation, we get that
Set with and let be arbitrarily chosen in . Then and by the hemicontinuity of , we get that for all . Hence , that is, . Similarly, we have , that is, .

By (3.1) and monotonicity of , we have
Therefore
that is,
By Lemma 2.5(ii) and Definition 2.1, we deduce that
By (3.1), (3.17), and monotonicity of , we have
Hence
This completes the proof of Theorem 3.3.

If , , , and , then we have the following theorem.

Theorem 3.4. *Let be a real Hilbert space and be a nonempty closed convex subset with . Let be strongly monotone, continuous, and coercive operator. If
**
then for each , there exists at least one solution of problem (1.3).*

#### Acknowledgments

The authors wish to thank the reviewers for many valuable suggestions to improve this paper, and we also would like to thank the editors' kind work. Project supported by National Natural Science Foundations of China (10561007) and The Natural Science Foundations of Jiangxi Province (0411036, 2007GQS2063).