ISRN Geometry

Volume 2012, Article ID 254235, 34 pages

http://dx.doi.org/10.5402/2012/254235

## Poset Pinball, the Dimension Pair Algorithm, and Type Regular Nilpotent Hessenberg Varieties

Department of Mathematics and Statistics, McMaster University, Hamilton, ON, Canada L8S 4K1

Received 28 January 2012; Accepted 15 March 2012

Academic Editors: L. C. Jeffrey, A. Morozov, and E. H. Saidi

Copyright © 2012 Darius Bayegan and Megumi Harada. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

#### Abstract

We develop the theory of *poset pinball*, a combinatorial game introduced by Harada-Tymoczko to study the equivariant cohomology ring of a GKM-compatible subspace of a GKM space; Harada and Tymoczko also prove that, in certain circumstances, a *successful outcome of Betti poset pinball* yields a module basis for the equivariant cohomology ring of . First we define the *dimension pair algorithm*, which yields a successful outcome of Betti poset pinball for any type regular nilpotent Hessenberg and any type nilpotent Springer variety, considered as GKM-compatible subspaces of the flag variety. The algorithm is motivated by a correspondence between Hessenberg affine cells and certain Schubert polynomials which we learned from Insko. Second, in a special case of regular nilpotent Hessenberg varieties, we prove that our pinball outcome is *poset-upper-triangular*, and hence the corresponding classes form a (pt)-module basis for the *S*^{1}-equivariant cohomology ring of the Hessenberg variety.

#### 1. Introduction

The purpose of this paper is to further develop the theory of *poset pinball*, a combinatorial game introduced in [1] for the purpose of computing in equivariant cohomology rings (all cohomology rings in this note are with coefficients), in certain cases of *type ** nilpotent Hessenberg varieties.* One of the main uses of poset pinball in [1] is to construct module bases for the equivariant cohomology rings of *GKM-compatible subspaces *of GKM spaces [1, Definition 4.5]. In the context of this paper, the ambient GKM space is the flag variety equipped with the action of the diagonal subgroup of , and the GKM-compatible subspaces are the nilpotent Hessenberg varieties. It is well recorded in the literature (e.g., [2] and references therein) that GKM spaces often have geometrically and/or combinatorially natural module bases for their equivariant cohomology rings; the basis of equivariant Schubert classes for is a famous example. The results of this paper represent first steps towards the larger goal of using poset pinball to construct a similarly computationally effective and convenient module bases for a GKM-compatible subspace by exploiting the structure of the ambient GKM space.

We briefly recall the setting of our results. Let be a nilpotent operator. Let be a function satisfying for all and for all . The associated Hessenberg variety is then defined as the following subvariety of :
Since we deal exclusively with type in this paper, henceforth we omit this phrase from our terminology. Two special cases of Hessenberg varieties are of particular interest in this paper: when is the principal nilpotent operator (in this case is called a *regular nilpotent Hessenberg variety*) and when is the identity function for all (in this case is called a *nilpotent Springer variety* and is sometimes denoted ). Hessenberg varieties arise in many areas of mathematics, including geometric representation theory [3–5], numerical analysis [6], mathematical physics [7, 8], combinatorics [9], and algebraic geometry [10, 11], so it is of interest to explicitly analyze their topology, for example, the structure of their (equivariant) cohomology rings. We do so through poset pinball and Schubert calculus techniques, as initiated and developed in [1, 12, 13] and briefly recalled below.

The following relationship between two group actions on the nilpotent Hessenberg variety and the flag variety, respectively, allows us to use the theory of GKM-compatible subspaces and poset pinball. There is a natural subgroup of the unitary diagonal matrices which acts on (defined precisely in Section 2). The group , the maximal torus of , acts on in the standard fashion. It turns out that the -fixed points are a subset of the -fixed points . Moreover, the inclusion of into and the inclusion of groups into then induce a natural ring homomorphism
As mentioned above, it is well known in Schubert calculus that the equivariant Schubert classes are a computationally convenient -module basis for . We refer to the images in of the equivariant Schubert classes via the projection (1.2) as *Hessenberg Schubert classes.* Given this setup and following [1], the game of poset pinball uses the data of the fixed points (considered as a partially ordered set with respect to Bruhat order) and the subset
to determine a set of *rolldowns *in . It is shown in [1] that, under certain circumstances (one of which is discussed in more detail below), such a set of rolldowns in turn specifies a subset of the Hessenberg Schubert classes which form a -*module basis* of . Thus poset pinball is an important tool for building computationally effective module bases for the equivariant cohomology of Hessenberg varieties. Indeed, the results of [13] accomplish precisely this goal—that is, of constructing a module basis via poset pinball techniques—in the special case of the Peterson variety, which is the regular nilpotent Hessenberg variety with Hessenberg function defined by for and . Exploiting this explicit module basis, in [13, Theorem 6.12] the second author and Tymoczko give a manifestly positive *Monk formula* for the product of a degree-2 Peterson Schubert class with an arbitrary Peterson Schubert class, expressed as a -linear combination of Peterson Schubert classes. This is an example of equivariant Schubert calculus in the realm of Hessenberg varieties, and it is an open problem to generalize the results of [13] to a wider class of Hessenberg varieties.

We now describe our main results. First, we explain in detail an algorithm which we dub the *dimension pair algorithm* and which associates to each -fixed point a permutation in , which we call the *rolldown of * following terminology in [1] and denoted . In the special cases of regular nilpotent Hessenberg varieties and nilpotent Springer varieties, we show that the set can be interpreted as the result of a *successful game of Betti pinball* (in the sense of [1]). The main motivation for our construction is that a successful outcome of Betti pinball can, under some circumstances, produce a module basis for the associated equivariant cohomology ring (cf. [1, Section 4.3]). In this sense, our algorithm represents a significant step towards the construction of module bases for the equivariant cohomology rings of general nilpotent Hessenberg varieties, thus extending the theory developed in [1, 13]. Although we formulate our algorithm in terms of dimension pairs and permissible fillings following terminology of Mbirika [14], the essential idea comes from a correspondence between Hessenberg affine cells and certain Schubert polynomials which we learned from Erik Insko.

Second, for a specific case of a regular nilpotent Hessenberg variety which we call a 334*-type* Hessenberg variety, we prove that the set of rolldowns obtained from the dimension pair algorithm is in fact *poset-upper-triangular* in the sense of [1]. As shown in [1], this is one of the possible circumstances under which we can conclude that the corresponding set of Hessenberg Schubert classes forms a module basis for the -equivariant cohomology ring of the variety. Thus our result gives rise to a new family of examples of Hessenberg varieties (and GKM-compatible subspaces) for which poset pinball successfully produces explicit module bases. We mention that the dimension pair algorithm also produces module bases in a special case of Springer varieties [15]. Although we do not know whether the dimension pair algorithm always succeeds in producing module bases for the -equivariant cohomology rings for a general nilpotent Hessenberg variety, the evidence thus far is suggestive. We leave further investigation to future work.

We give a brief summary of the contents of this manuscript. In Section 2 we recall some definitions and constructions necessary for later statements. In Section 3.1 we describe the dimension pair algorithm and prove that the result of the algorithm satisfies the conditions to be the outcome of a successful game of Betti poset pinball in the special cases of regular nilpotent Hessenberg varieties and nilpotent Springer varieties. We briefly review in Section 3.2 the theory developed in [1] which show that if the rolldown set obtained from a successful game of Betti poset pinball also satisfies poset-upper-triangularity conditions, then it yields a module basis in equivariant cohomology. In Sections 4 and 5 we prove that the dimension pair algorithm produces a poset-upper-triangular module basis in a special class of regular nilpotent Hessenberg varieties which we call 334-type Hessenberg varieties. We close with some open questions in Section 6.

#### 2. Background

We begin with necessary definitions and terminology for what follows. In Section 2.1 we recall the geometric objects and the group actions under consideration. In Section 2.2 we recall some combinatorial definitions associated to Young diagrams. We recall a bijection between Hessenberg fixed points and certain fillings of Young diagrams in Section 2.3. The discussion closely follows previous work (e.g., [1, 13] and also [16]) so we keep exposition brief.

##### 2.1. Hessenberg Varieties, Highest Forms, and Fixed Points

By the *flag variety *we mean the homogeneous space which is also identified with
A *Hessenberg function* is a function satisfying for all and for all . We frequently denote a Hessenberg function by listing its values in sequence, . Let be a linear operator. The *Hessenberg variety * is defined as the following subvariety of :
If is nilpotent, we say is a *nilpotent Hessenberg variety*, and if is the principal nilpotent operator (i.e., has one Jordan block with eigenvalue 0), then is called a *regular nilpotent Hessenberg variety.* If is nilpotent and is the identity function for all , then is called a *nilpotent Springer variety* and often denoted . In this manuscript we study in some detail the regular nilpotent case, and as such sometimes notate as when is understood to be the standard principal nilpotent operator.

Suppose is a nilpotent matrix in standard Jordan canonical form. It turns out that for many of our statements below we must use a choice of conjugate of which is in *highest form* [16, Definition 4.2]. We recall the following.

*Definition 2.1 (see [16, Definitions 4.1 and 4.2]). *(i) Let be any matrix. We call the entry a *pivot* of if is nonzero and if all entries below and to its left vanish, that is, if and if . Moreover, given , define to be the row of if the entry is a pivot, and 0 otherwise.

(ii) Let be an upper-triangular nilpotent matrix. Then we say is in *highest form* if its pivots form a nondecreasing sequence, namely, .

We do not require the details of the theory of highest forms of linear operators; for the purposes of the present manuscript it suffices to remark firstly that when is the principal nilpotent matrix, then is already in highest form, and secondly that any nilpotent matrix can be conjugated by an appropriate permutation matrix so that is in highest form. However, the following observation will be relevant in Section 2.3.

*Remark 2.2. *In this manuscript we always assume that our highest form has been chosen in accordance to the recipe described by Tymoczko in [16, Section 4]. Since the precise method of this construction is not relevant for the rest of the present manuscript, we omit further explanation here. In the case when is principal nilpotent, we take since is already in highest form and this is the form chosen by Tymoczko in [16]. A more detailed discussion of highest forms as it pertains to poset pinball theory is in [15].

For details on the following facts we refer the reader to, for example, [1, 13, 16] and references therein. Let be an nilpotent matrix in Jordan canonical form and let denote a permutation matrix such that is in highest form. It is known and straightforward to show that the following subgroup of preserves for as above and any Hessenberg function : Here is the standard maximal torus of consisting of diagonal unitary matrices.

This implies that the conjugate circle subgroup preserves . By abuse of notation we will denote both circle subgroups by , since it is clear by context which is meant. The -fixed points of and are isolated and are a subset of the -fixed points of . Since the set of -fixed points may be identified with the Weyl group , and since (resp., ) is a subset of , any Hessenberg fixed point may be thought of as a permutation .

##### 2.2. Permissible Fillings, Dimension Pairs, Lists of Top Parts, and Associated Permutations

Recall that there is a bijective correspondence between the set of conjugacy classes of nilpotent complex matrices and Young diagrams (we use English notation for Young diagrams) with boxes, given by associating to the Young diagram with row lengths the sizes of the Jordan blocks of listed in weakly decreasing order. We will use this bijection to often treat such and as the same data; we sometimes denote by the Young diagram given as above corresponding to a nilpotent .

For more details on the following see [14].

*Definition 2.3. *Let be a Young diagram with boxes. Let be a Hessenberg function. A *filling* of by the alphabet is an injective placing of the integers into the boxes of . A filling of is called a -*permissible filling* if for every horizontal adjacency

*Remark 2.4. *In this manuscript the and will frequently be understood by context. When there is no danger of confusion we simply refer to *permissible fillings. *

*Example 2.5. *Let . Suppose and . Then

We denote a permissible filling of by , in analogy with standard notation for Young tableaux. Next we focus attention on certain pairs of entries in a permissible filling .

*Definition 2.6. *Let be a Hessenberg function and a Young diagram with boxes. A pair is a *dimension pair* of an -permissible filling of if the following conditions hold:(1),
(2) is either(i)below in the same column of , or(ii)anywhere in a column strictly to the left of the column of , and(3)if there exists a box with filling directly adjacent to the right of , then .

For a dimension pair of , we will refer to as the *top part* of the dimension pair.

*Example 2.7. *Let , be as in Example 2.5. The dimension pairs in the permissible filling

Given a permissible filling of , we follow [14] and denote by the set of dimension pairs of . For each integer with , let
so is the number of times occurs as a top part in the set of dimension pairs of . From the definitions it follows that . We call the integral vector the *list of top parts* of .

To each such we associate a permutation in as follows. As a preliminary step, for each with define where denotes the simple transposition in and 1 denotes the identity permutation. Now define the association It is not difficult to see that (2.6) is a bijection between the set of integral vectors satisfying for all and the group . In fact the word given by (2.6) is a reduced word decomposition of and the count the number of inversions in with as the higher integer. The following simple fact will be used later.

*Fact 1. *Suppose *, * are both lists of top parts. Suppose further that for all , we have *. *Then in Bruhat order. This follows immediately from the definition (2.6)*. *

*Example 2.8. *Continuing with Examples 2.5 and 2.7, for the permissible filling

*Example 2.9. *Let be as in Example 2.5. The filling

##### 2.3. Bijection between Fixed Points and Permissible Fillings

For nilpotent Hessenberg varieties, the -fixed points are in bijective correspondence with the set of permissible fillings of the Young diagram , as we now describe. We will use this correspondence in the formulation of our dimension pair algorithm.

Suppose is a Young diagram with boxes. We begin by defining a bijective correspondence between the set of *all* fillings (not necessarily permissible) of with permutations in . Given a filling, read the entries of the filling by reading along each column from the bottom to the top, starting with the leftmost column and proceeding to the rightmost column. The association is then given by interpreting the resulting word as the one-line notation of a permutation. For example, the filling(2.7)
has associated permutation 641523. It is easily seen that this is a bijective correspondence. Given a filling of , we denote its associated permutation by .

*Remark 2.10. *In the case when is the principal nilpotent matrix, the corresponding Young diagram has only one row, so the above correspondence simply reads off the (one row of the) filling from left to right. In this case we abuse notation and denote by just . For instance, the permissible filling of in Example 2.9 has associated permutation 43215.

Now let
denote the set of -permissible fillings of . Recall that elements in are viewed as permutations in via the identification . The next proposition follows from the definitions and some linear algebra. It is proven and discussed in more detail in [15], where the notation used is slightly different.

Proposition 2.11. *Fix a positive integer. Let be a Hessenberg function and a Young diagram with boxes. Suppose is a nilpotent operator in highest form as chosen in [16] (cf. Remark 2.2) with . Let denote the associated nilpotent Hessenberg variety. Then the map from the -fixed points to the set of permissible fillings **
is well defined and is a bijection.*

*Remark 2.12. *In the case when is the principal nilpotent matrix, is the Young diagram with only one row. Thus the map (2.9) above simplifies to where we abuse notation (cf. Remark 2.10) and denote by .

#### 3. The Dimension Pair Algorithm for Betti Poset Pinball for Nilpotent Hessenberg Varieties

In this section we first explain the *dimension pair algorithm* which associates to any Hessenberg fixed point a permutation in . The name is due to the fact that the construction proceeds by computing dimension pairs in appropriate permissible fillings. We then interpret this algorithm as a method for choosing *rolldowns* associated to the Hessenberg fixed points in a game of *Betti poset pinball* in the sense of [1]. The algorithm makes sense for any nilpotent Hessenberg variety, so it is defined in that generality in Section 3.1. However, our proof that the algorithm produces a successful outcome of Betti poset pinball in the sense of [1] is only for the special cases of regular nilpotent Hessenberg varieties and nilpotent Springer varieties. In Section 3.2 we briefly recall the setup and necessary results of poset pinball which allow us to conclude that our poset pinball result yields an explicit module basis for equivariant cohomology.

##### 3.1. The Dimension Pair Algorithm for Nilpotent Hessenberg Varieties

Let be a nilpotent matrix in highest form chosen as in Remark 2.2 and let . Let be a Hessenberg function and the corresponding nilpotent Hessenberg variety.

The definition of the dimension pair algorithm is pure combinatorics. It produces for each Hessenberg fixed point an element in . Following terminology of poset pinball, we denote this function by

*Definition 3.1 (the dimension pair algorithm). *We define as follows.(1)Let and let be its corresponding permissible filling as defined in (2.9).(2)Let be the set of dimension pairs in the permissible filling .(3)For each with , set
as in (2.4) and define .(4)Define where is the permutation associated to the integer vector defined in (2.6).

*Example 3.2. *Let , be as in Example 2.5. The permutation is in , as can be checked. The associated permissible filling is

We next show that the rolldown function defined by the dimension pair algorithm above satisfies the conditions to be a *successful outcome of Betti poset pinball* as in [1] in certain cases of nilpotent Hessenberg varieties. The statement of one of the conditions requires advance knowledge of the Betti numbers of nilpotent Hessenberg varieties, for which we recall the following result (reformulated in our language) from [16].

Theorem 3.3 (see [16, Theorem 1.1]). *Let be a nilpotent matrix in highest form chosen as in Remark 2.2 and let . Let be a Hessenberg function and let denote the corresponding nilpotent Hessenberg variety. There is a paving by (complex) affine cells of such that*(i)*the affine cells are in one-to-one correspondence with , and*(ii)*the (complex) dimension of the affine cell corresponding to a fixed point is*

In particular, Theorem 3.3 implies that the odd Betti numbers of are 0, and the th even Betti number is precisely the number of fixed points in such that . Given the regular nilpotent Hessenberg variety , denote by its th Betti number, that is, We may now formulate the conditions that guarantee that is a successful outcome of Betti pinball. For more details we refer the reader to [1, Section 3]. It suffices to check the following(1) is injective,(2)for every , we have in Bruhat order, and(3)for every , , we have where denotes the Bruhat length of .

We prove each claim in turn. For the first assertion we restrict to two special cases of Hessenberg varieties.

Lemma 3.4. *Suppose that is either a regular nilpotent Hessenberg variety or a nilpotent Springer variety. Then the function is injective.*

*Proof. *Since the association given in (2.6) is a bijection, it suffices to show that the map which sends a Hessenberg fixed point to the list of top parts of its associated permissible filling is injective. Mbirika shows that, in the cases of regular nilpotent Hessenberg varieties and nilpotent Springer varieties, there exists an inverse to this map (Mbirika works with monomials in variables constructed from the list of top parts, but this is equivalent data) [14, Section 3.2]. The result follows.

Lemma 3.5. *For every , one has in Bruhat order.*

*Proof. *Since Bruhat order is preserved under taking inverses, it suffices to prove that is Bruhat-less than . For any permutation , set
and let . Then the association (2.6) applied to the vector recovers the permutation . By definition of and the definition of dimension pairs, the set is always a subset of the set of inversions of the permutation . From Fact 1 it follows that the permutation is Bruhat-less than as desired.

Lemma 3.6. *Let be a nilpotent matrix in highest form chosen as in Remark 2.2 and let . Let be a Hessenberg function and the associated nilpotent Hessenberg variety. For every , , one has
**
where denotes the Bruhat length of .*

*Proof. *By construction, has a reduced word decomposition consisting of precisely simple transpositions. Hence its Bruhat length is . By Theorem 3.3, is precisely the number of fixed points with so the result follows.

The following is immediate from the above lemmas and the definition of Betti pinball given in [1, Section 3].

Proposition 3.7. *Suppose that is either a regular nilpotent Hessenberg variety or a nilpotent Springer variety. Then the association given by the dimension pair algorithm is a possible outcome of a successful game of Betti poset pinball played with ambient partially ordered set equipped with Bruhat order, rank function given by Bruhat length, initial subset , and target Betti numbers .*

*Remark 3.8. *Lemmas 3.5 and 3.6 hold for general nilpotent and Hessenberg functions . Hence to prove that Proposition 3.7 holds for more general cases of nilpotent Hessenberg varieties, it suffices to check that the injectivity assertion (1) above holds. We do not know counterexamples where the injectivity fails. It would be of interest to clarify the situation for more general and .

##### 3.2. Betti Pinball, Poset-Upper-Triangularity, and Module Bases

In the context of a *GKM-compatible subspace of a GKM space* [1, Definition 4.5], it is explained in [1, Section 4] that the outcome of a game of poset pinball may be interpreted as specifying a set of equivariant cohomology classes which, under additional conditions, yields a module basis for the equivariant cohomology of the GKM-compatible subspace. In this paper, the GKM space is the flag variety with the standard -action and the GKM-compatible subspace is with the -action specified above. Consider the -module basis for given by the equivariant Schubert classes . The dimension pair algorithm then specifies the set
where for any the class is defined to be the image of under the natural projection map
induced by the inclusion of groups and the -equivariant inclusion of spaces . We refer to the images as *Hessenberg Schubert classes. *

Following the methods of [1] we view and as subrings of
We denote by , the value of the th coordinate in the direct sums above, for , or , , respectively. If
for all , , then the set in is called *poset-upper-triangular* (with respect to the partial order on induced from Bruhat order) [1, Definition 2.3]. Finally, recall that the cohomology degree of an equivariant Schubert class (and hence also the corresponding Hessenberg Schubert class ) is .

The following is immediate from [1, Proposition 4.14] and the above discussion.

Proposition 3.9. *Let be either a regular nilpotent Hessenberg variety or a nilpotent Springer variety. Let be the dimension pair algorithm defined above. Suppose (3.11) holds for all . Then the Hessenberg Schubert classes form a -module basis for the -equivariant cohomology ring .*

Therefore, in order to prove that the Hessenberg Schubert classes above form a module basis as desired, it suffices to show that they satisfy the upper-triangularity conditions (3.11) for all , . The proof of this assertion, for a special class of regular nilpotent Hessenberg varieties closely related to Peterson varieties, is the content of Sections 4 and 5.

We close the section with a brief discussion of matchings. Following [1, Section 4.3], define
to be the (complex) dimension of the affine cell containing the fixed point in Tymoczko’s paving by affines of in Theorem 3.3. Then from the discussion above we know
and since the cohomology degree of is , we see that the association from is also a* matching *in the sense of [1] with respect to and rank function on given by Bruhat length. Thus the fact that the form a module basis can also be deduced from [1, Theorem 4.18].

#### 4. Poset-Upper-Triangularity of Rolldown Classes for 334-Type Hessenberg Varieties

In this section and in Section 5 we analyze in detail the dimension pair algorithm in the case of a Hessenberg variety which is closely related to the Peterson variety and in particular prove that the algorithm produces a poset-upper-triangular module basis for its -equivariant cohomology ring. Here and below the nilpotent operator under consideration is always the principal nilpotent, so we omit the from the notation and write . Similarly the corresponding Young diagram is always so we omit the from notation and write instead of .

We fix for this discussion the Hessenberg function given by The only difference between this function and the Hessenberg function for the Peterson variety studied in [13] is that the value of is 3 instead of 2. In this sense this is “close’’ to the Peterson case. Thus it is natural that much of our analysis follows that for Peterson varieties in [13], although it is still necessary to introduce new ideas and terminology to handle the Hessenberg fixed points in which do not arise in the Peterson case.

The Hessenberg function in (4.1) is trivial if since in that case which implies that the corresponding Hessenberv variety is equal to the full flag variety . Hence we assume throughout. Under this assumption and following the notation introduced in Section 2, the Hessenberg function is of the form . As such, for the purposes of this manuscript, we refer to this family of regular nilpotent Hessenberg varieties as 334*-type Hessenberg varieties. *

Our main result is the following theorem.

Theorem 4.1. *Let and let be the 334-type Hessenberg variety in . Let be the dimension pair algorithm defined in Section 3. Then
**
for all , . In particular the Hessenberg Schubert classes form a -module basis for the -equivariant cohomology ring .*

For ease of exposition, and because the arguments required are of a somewhat different nature, we prove Theorem 4.1 by proving the two assertions in (4.2) separately, as follows.

Proposition 4.2. *Let , , and be as above. Then,
**
for all .*

Proposition 4.3. *Let , , and be as above. Then,
**
for all , .*

The proof of Proposition 4.2 is the content of Section 5. The main result of the present section is the upper-triangularity property asserted in Proposition 4.3. Its proof requires a number of preliminary results. We first begin by reformulating the problem in terms of Bruhat relations among the fixed points.

Lemma 4.4. *Let , , and be as above. If for all , one has
**
in Bruhat order, then the Hessenberg Schubert classes satisfy (4.4).*

*Proof. *Recall that the equivariant Schubert classes are poset-upper-triangular with respect to Bruhat order on . In particular, for all , we have if . Since the Hessenberg Schubert classes are images of the Schubert classes and the diagram
commutes, it follows that if for all , , we have
in Bruhat order, then (4.4) follows.

The rest of this section is devoted to the proof of (4.5), which by Lemma 4.4 then proves Proposition 4.3.

##### 4.1. Fixed Points and Associated Subsets for the 334-Type Hessenberg Variety

In this section we enumerate the fixed points in the 334-type Hessenberg variety and also associate to each fixed point in a subset of . As we show below, the set of fixed points in the Peterson variety is a subset of the fixed points of the 334-type Hessenberg variety, so the main task is to describe the new fixed points which arise in the 334-type case. We begin with a general observation.

Lemma 4.5. *Let and let be two Hessenberg functions. If for all , , then
**
The inclusion is -equivariant and in particular and .*

*Proof. *Let denote an element in . By definition the regular nilpotent Hessenberg variety associated to is
where is the principal nilpotent operator. Since for all by definition of flags and for all flags, if for all , then automatically implies . We conclude . The -equivariance of the inclusion follows from the definition of the -action of (2.3).

Applying Lemma 4.5 to the Hessenberg function
corresponding to the Peterson variety and the 334-type Hessenberg function (4.1), we conclude that all fixed points in also arise as fixed points in . We refer to the elements of (viewed as elements of ) as *Peterson-type fixed points. *It therefore remains to describe . It turns out to be convenient to do this by first describing .

We first introduce some terminology. Given a permutation in one-line notation and some , , we say that the entries form a *decreasing staircase*, or simply a *staircase*, if for all . For example, for , the segment 432 is a staircase, but 751, though the entries decrease, is not. We will say that a consecutive series of staircases is an *increasing sequence of staircases* (or simply *increasing staircases*) if each entry in a given staircase is smaller than any entry in any following staircase (reading from left to right). For instance, is a sequence of staircases 654, 987, and 321, but is not an increasing sequence of staircases since the entries are not smaller than the entries in the later staircase 321. However, is an increasing sequence of (three) staircases 321, 654, and 987.

It is shown in [13] that the -fixed points of the Peterson variety consist precisely of those permutations such that the one-line notation of is an increasing sequence of staircases. Since such are equal to their own inverses, the permissible fillings corresponding to are precisely those which are increasing sequences of staircases (cf. Remark 2.12). We now describe the permissible fillings which are *not* Peterson-type fillings. We use the language of -tableau trees introduced by Mbirika; see [14, Section 3.1] for definitions. Recall from Remark 2.10 that we identify permissible fillings with permutations in via one-line notation.

Lemma 4.6. *Let and let be the 334-type Hessenberg variety in . Let be a permissible filling for which is not of Peterson type, that is, . Then precisely one of the following hold.*(i)*The one-line notation of is of the form
where is a (possibly empty) staircase such that is also a staircase, and is an increasing sequence of staircases. We refer to these as 312-type permissible fillings. *(ii)*The one-line notation of is of the form
where is a (possibly empty) staircase such that is also a staircase, and is an increasing sequence of staircases. We refer to these as 231-type permissible fillings.**Moreover, any filling satisfying either of the above conditions appears in .*

*Proof of Lemma 4.6. *For any Hessenberg function , Mbirika shows in [14, Section 3.2] that the Level fillings in an -tableau tree are precisely the permissible fillings with respect to . For the Peterson Hessenberg function in (4.10) Mbirika’s corresponding -tableau tree has the property that for every with and every vertex at Level , there are precisely 2 edges going down from that vertex to a Level vertex (this is because the corresponding *degree tuple * [14, Definition 3.1.1] has for all ). In the case of the 334-type Hessenberg function, by definition the -tableau tree also has precisely 2 edges going down from every vertex at Level for all , . However, at Level 2, each vertex has not 2 but 3 edges pointing down to a vertex at Level 3.

From [14, Section 3] (cf. in particular [14, Definition ]) it can be seen that for the case of the Peterson Hessenberg function, the corresponding -tableau tree at Level 2 has vertices and , whereas for the 334-type Hessenberg function, the Level 2 vertices have the form and . Here the bullets indicate the locations of the -permissible positions available for the placement of the next index 3, in the sense of [14, Section 3] (cf. in particular [14, Lemma ]). In particular, since we saw above that the edges going down from Level 3 onwards are identical in both the Peterson and 334-type Hessenberg case, it follows that the branches of the tree emanating downwards from the two Level 3 vertices , (coming from ) and the two vertices , (coming from ) are identical to the corresponding branches in the -tableau tree for the Peterson Hessenberg function. Hence all permissible fillings at the final Level of these branches are of Peterson type. In contrast, the branches emanating from and do not appear in the Peterson -tableau tree, and none of the fillings appearing at Level in these branches can be Peterson permissible fillings since a 3 appears directly before a 1. Hence it is precisely these branches which account for the permissible fillings which are not of Peterson type. As noted above, the rest of the branch only has 2 edges going down from each vertex with -permissible positions determined exactly as in the Peterson case. In particular, except for the exceptional 3 appearing directly to the left of a 1, the fillings must consist of decreasing staircases and all possible arrangements of decreasing staircases do appear. The result follows.

*Example 4.7. *Suppose . Then is an example of a 312-type permissible filling where and . An example of a 231-type permissible filling is where and . Neither of these are permissible with respect to the Peterson Hessenberg function since a 3 appears directly to the left of a 1. Nevertheless, both of these fillings are closely related to the Peterson-type permissible filling ; this relationship is closely analyzed and used below.

We now give explicit descriptions of the corresponding non-Peterson-type elements in , obtained by taking inverses of the permissible fillings described in Lemma 4.6.

*Definition 4.8. *Let . We say is a 312*-type (resp., *231*-type) fixed point* if its inverse is a permissible filling of 312-type (resp., 231-type).

As observed above, since Peterson-type permissible fillings are equal to their own inverses, in that case there is no distinction between the fillings and their associated fixed points. For the 312 and 231 types, however, this is not the case. We record the following. The proof is a straightforward computation and is left to the reader.

Lemma 4.9. *Let be a 312-type (resp., 231-type) permissible filling. Let be the integer such that is the first entry (resp., second entry) in the one-line notation of . Let be the corresponding 312-type (resp., 231 type) fixed point. Then,*(i)*the one-line notation of is the same as that of for all th entries with ,*(ii)*if is 312-type, then the first entries of the one-line notation of are
*(iii)*if is 231-type, then the first entries of the one-line notation of are
*

In the case of the Peterson variety, there is a convenient bijective correspondence between the set of -fixed points of the Peterson variety and subsets of given as follows [13, Section 2.3]. Let be a Peterson-type fixed point. Then the corresponding subset is In the case of the 334-type Hessenber variety, it is also useful to assign a subset of to each fixed point as follows.

*Definition 4.10. *Let . The *associated subset of **corresponding to *, notated , is defined as follows.(i)Suppose is of Peterson type. Then is defined to be the set in (4.15).(ii)Suppose is 312-type. Consider the permutation (i.e., swap the and the in the one-line notation (4.13)). This is a fixed point of Peterson type. Define .(iii)Suppose is 231-type. Consider the permutation
(i.e., move the 1 to the right of the 2 in the one-line notation (4.14)). This is a fixed point of Peterson type. Define .

*Example 4.11. *Suppose .(i)Suppose is the Peterson-type fixed point . Then . This agrees with the association used in [13].(ii)Suppose is the 312-type fixed point (corresponding to the 312-type permissible filling 43127658). Then and .(iii)Suppose is the 231-type fixed point (corresponding to the 231-type permissible filling 25431768). Then and .

*Remark 4.12. *The three fixed points , , and , which are, respectively, of Peterson type, 312 type, and 231-type, all have the same associated subset .

It is useful to observe that the 312-type and 231-type fixed points have associated subsets that always contain 1 and 2.

Lemma 4.13. *Let be a 334-type Hessenberg fixed point. Suppose further that is not of Peterson type. Then .*

*Proof. *From the explicit descriptions of the one-line notation of the 312 type (resp., 231-type) fixed points given above, we know that the initial segment (resp, ) in the one-line notation is such that . From Definition 4.10 it follows that the first decreasing staircase of the associated Peterson-type fixed point (resp., ) is of length at least 3. In particular, the first staircase starts with an integer which is ≥ 3. The result follows.

As noted in Remark 4.12, the association given in Definition 4.10 is *not* one-to-one and hence in particular not a bijective correspondence. This makes our analysis more complicated than in [13], but the notion is still useful for our arguments below.

##### 4.2. Reduced Word Decompositions for 334-Type Fixed Points and Rolldowns

In this section we fix particular choices of reduced word decompositions for the fixed points in which we use in our arguments below. We also compute, and fix choices of reduced words for, the rolldowns of the fixed points.

The association of the previous section allows us to describe these reduced word decompositions in relation to that of the Peterson-type fixed points. Let be a positive integer and a nonnegative integer. Recall that a reduced word decomposition of the maximal element (the full inversion) in the subgroup is given by
For the purposes of this manuscript, we call this the *standard reduced word (decomposition)* for the maximal element. (This is different from the choice of reduced word decomposition used in [13, Section 2.3].) We denote a consecutive set of integers for positive and a nonnegative integer by . We say that is a *maximal consecutive substring* of if and neither nor are in . It is straightforward that any subset of uniquely decomposes into a disjoint union of maximal consecutive substrings
For instance, for , the decomposition is . For any , denote by the full inversion in the subgroup . Then it follows from Definition 4.10 (see also [13, Section 2.3]) that the Peterson-type fixed point associated to , which we denote by , is the product
We fix a choice of reduced word decomposition of given by taking the product of the standard reduced words (4.17) for each of the full inversions appearing in (4.19). For the purposes of this manuscript we call this the *standard reduced word decomposition of a Peterson-type fixed point *.

*Example 4.14. *Let and let be a Peterson-type fixed point. Then the two decreasing staircases are 4321 and 765, the associated subset is with maximal consecutive strings and . The standard reduced word decomposition of is

We now fix a reduced word decomposition of the non-Peterson-type fixed points.

Lemma 4.15. *Let be a fixed point which is not of Peterson type and let be the associated subset with its decomposition into maximal consecutive substrings.*(i)*If is 312-type, then a reduced word decomposition for is given by
*(ii)*If is 231-type, then a reduced word decomposition for is given by
where the in the above expressions are assumed to be given the reduced word decomposition described in (4.17).*

*Proof. *For the first assertion, observe that the explicit description of the one-line notation 312-type fixed points in (4.13) implies that has precisely 1 fewer inversion than . An explicit computation shows that the given word (4.21) is equal to , so it is a word decomposition of with exactly as many simple transpositions as the Bruhat length of . In particular it must be reduced. A similar argument proves the second assertion.

*Example 4.16. *Suppose . Suppose is a 312-type fixed point. Then the reduced word decomposition of given in Lemma 4.15 is
Similarly suppose is a 231-type fixed point. Then the reduced word decomposition of given in Lemma 4.15 is

Henceforth, we always use the reduced words given above.

Next we explicitly describe the rolldowns associated to each in by the dimension pair algorithm. We begin with the Peterson-type fixed points. It turns out there are two important subcases of Peterson-type fixed points.

*Definition 4.17. *We say that a Peterson-type fixed point *contains the string *321 (or simply *contains *321) if, in the one-line notation of , the string 321 appears (equivalently, if ). We say *does not contain the string *321 (or simply *does not contain *321) otherwise.

*Remark 4.18. *Note that Definition 4.17 is different from the standard notion of pattern-containing or pattern-avoiding permutations since here we require the one-line notation of to contain the string 321 exactly.

Given a subset and corresponding Peterson-type fixed point , we call the permutation
the *Peterson case rolldown of *. Note that the word (4.25) is in fact a reduced word decomposition of this permutation; we always use this choice of reduced word. The terminology is motivated by the fact that (4.25) is the (inverse of the) permutation given in [13, Definition 4.1]. (The fact that it is the inverse of the permutation used in [13] does not affect the theory very much, as is explained in [13, Proposition 5.16]).

Lemma 4.19. *Let and let be the 334-type Hessenberg variety in . Let be a Peterson-type fixed point and let be its associated subset.*(i)*Suppose does not contain 321. Then is the Peterson case rolldown of .*(ii)*Suppose does contain 321, that is, for and and . Then is**
In particular, if a Peterson-type fixed point contains 321, then its rolldown is Bruhat-greater and has Bruhat length 1 greater than the Peterson case rolldown of .*

*Proof. *If contains a 321, then by Definition 2.6, the pairs , and are all dimension pairs in . Hence 3 appears precisely twice as a top part of a dimension pair and 2 appears precisely once. Thus by constructing the dimension pair algorithm the permutation begins with the word . With respect to all other indices , the 334-type Hessenberg function is identical to the Peterson Hessenberg function and hence, for each such , the index appears precisely once as a top part of a dimension pair of and thus contributes precisely one to . Taking the inverse yields (4.26) as desired.

If does not contain 321, then 3 appears at most once as the top part of a dimension pair in , and again for all other indices the computations are identical to the Peterson case as above. Hence is identical to the Peterson case rolldown. This completes the proof.

Next, we give an explicit description, along with a choice of reduced word decomposition, of the rolldowns corresponding to the non-Peterson-type fixed points.

Lemma 4.20. *Let and suppose that is not of Peterson type. Let for some .*(1)*If is of 312-type, then the dimension pair algorithm associates to the permutation
*(2)*If is 231-type, then the dimension pair algorithm associates to the permutation*

*Proof. *Suppose is a 312-type fixed point so is a 312-type permissible filling. By definition of dimension pairs, 2 does not appear as the top part of any dimension pair (since it appears to the right of a 1). Also by definition, 3 appears as a top part of the two dimension pairs and . The form of the 312-type permissible fillings described in Lemma 4.6 and the definition of imply that the other dimension pairs are precisely the pairs for (for ), from which it follows that . Taking inverses yields (4.27). The proof of the second assertion is similar.

*Example 4.21. * (i) Suppose . This is of Peterson type. Then .(ii)Suppose . This is 312-type. Then .(iii)Suppose . This is 231-type. Then .

We conclude the section with a computation of the one-line notation of the rolldowns for different types; we leave proofs to the reader.

Lemma 4.22. *Let be a 334-type Hessenberg fixed point and let be its associated subset with its decomposition into maximal consecutive substrings. Suppose is of Peterson type that contains 321, 312-type, or of 231 type. Then , and the first entires of the one-line notation of are
**
for of Peterson type that contains 321,
**
for -type, and
**
for -type.*

##### 4.3. Bruhat Order Relations

In this section we analyze the properties of the association with respect to comparisons in Bruhat order.

The first two lemmas are straightforward and proofs are left to the reader.

Lemma 4.23. *Let and let be the Peterson-type filling associated to . Then is maximal in the subgroup of generated by the simple transpositions . In particular, is Bruhat-bigger than any permutation .*

Lemma 4.24. *Let . Suppose is not of Peterson type. Then is Bruhat-less than the Peterson type fixed point corresponding to .*

We also observe that a Bruhat relation implies a containment relation of the associated subsets.

Lemma 4.25. *Let , and let , be the respective associated subsets. Let be a simple transposition. Then,*(1)* if and only if ,*(2)* if and only if ,*(3)*if or , then .*

*Proof. *Bruhat order is independent of choice of reduced word decomposition for . Therefore, a simple transposition is less than in Bruhat order if and only if appears in a (and hence any) reduced word decomposition of . In particular, to prove the first claim it suffices to observe that by the definitions of , the index appears in precisely when appears in the choice of reduced word for given above. A similar argument using the explicit reduced words given for in Lemmas 4.19 and 4.20 proves the second claim. The last claim follows from the first two.

We have just seen that implies . In the case of the Peterson variety these Bruhat relations are precisely encoded by the partial ordering given by containment of the ; specifically, by Lemma 4.23, if and only if . In our 334-type Hessenberg case this is no longer true although the sets do still encode the Bruhat data. The precise statements occupy the next several lemmas.

We take a moment to recall the *tableau criterion* for determining Bruhat order in the Weyl group (see, e.g., [17]) which will be useful in the discussion below. For , denote by the descent set of , namely,
For example, for the descent set is .

Theorem 4.26 (the tableau criterion [17, Theorem 2.6.3]). *For , , let be the th element in the increasing rearrangement of , and similarly for . Then in Bruhat order if and only if
*

For example, suppose and . Since , we examine the three increasing rearrangements of initial segments of and of lengths 3, 5, and 7, respectively, which we may organize into Young tableaux: (4.34)

Comparing corresponding entries, there are two violations of the tableau condition of the proposition in the upper-left corner, so we conclude that .

Now we observe that some Bruhat relations never arise.

Lemma 4.27. *Let , . Let be the associated subset of with its decomposition into maximal consecutive substrings. Suppose one of the following conditions hold:*(1)* is of Peterson type that does not contain 321 while is not,*(2)* is 231-type while is either of Peterson type that contains 321 or is 312-type.** Then and .*

*Proof. *If is of Peterson type that does not contain 321, then by definition of the associated subsets. All other types (Peterson type that contains 321, or 312-type, or 231-type) have associated subsets containing by Lemma 4.13 and by definition of . The claim (1) now follows from Lemma 4.25.

Next suppose is 231-type and is of Peterson type that contains 321. Then the first two entries of the one-line notation of must be both strictly greater than 1, and . Similarly if is a 312-type fixed point, then . From (4.13) it follows that the first two entries in the one-line notation of are also strictly greater than 1, and . On the other hand, the one-line notation for a 231-type fixed point, in (4.14) has a 1 in the second entry. By the tableau criterion, if , then, since in both cases under consideration, we must have that one of the first two entries of is equal to 1, but we have just seen that is impossible. Hence . The assertion that follows by a similar argument using (4.29), (4.30), and (4.37).

For the next lemma and below, we say two fixed points are *of the same type *if both are Peterson-type, or both are 312-type, or both are 231-type.

Lemma 4.28. *Let , . Suppose one of the following conditions hold:*(i)* and are of the same type, or*(ii)* is of Peterson type and does not contain 321, and is either 312-type or 231-type, or*(iii)* is either 312-type or 231-type, and is of Peterson type.**Then,
*

*Proof. *Since the lemma above shows that implies , for all cases it suffices to show the reverse implication. First suppose and are of the same type and . An examination of the reduced word decompositions of the 334-type fillings given in the above discussion and an argument similar to that in [13] implies . Now suppose is of Peterson type and does not contain 321 and is either of 312 type or 231-type. Then since , either or . From the explicit reduced word decompositions of 312 or 231-type fixed points chosen above it can be seen that is Bruhat-greater than both and . The claim now follows from Lemma 4.23. Finally suppose is either 312-type or 231-type and is of Peterson type. Since we know from Lemma 4.23 that . Lemma 4.24 shows that so the result follows.

The next step is to show that Bruhat relations between certain Hessenberg fixed points are connected to lengths of initial maximal consecutive substrings in the associated subsets. We need some notation. Let . Recall we denote by the Peterson-type fixed point associated to . For the purposes of this discussion we let (resp., ) denote the 312-type (resp., 231-type) fixed point with associated subset . Thus for for some with , we have in one-line notation. For general subsets with and , the definitions 312-type and 231-type fixed points imply that