Fixed Point Theory and Applications
Volume 2010 (2010), Article ID 323069, 31 pages
doi:10.1155/2010/323069
Research Article

A Poincaré Formula for the Fixed Point Indices of the Iterates of Arbitrary Planar Homeomorphisms

Francisco R. Ruiz del Portal1 and José M. Salazar2

1Departamento de Geometría y Topología, Facultad de CC.Matemáticas, Universidad Complutense de Madrid, Madrid 28040, Spain
2Departamento de Matemáticas, Universidad de Alcalá, Alcalá de Henares, Madrid 28871, Spain

Received 11 November 2009; Accepted 1 March 2010

Academic Editor: Marlène Frigon

Copyright © 2010 Francisco R. Ruiz del Portal and José M. Salazar. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Let 𝑈 ⊂ ℝ 2 be an open subset and 𝑓 ∶ 𝑈 → ℝ 2 be an arbitrary local homeomorphism with F i x ( 𝑓 ) = { 𝑝 } . We compute the fixed point indices of the iterates of 𝑓 at 𝑝 , 𝑖 ℝ 2 ( 𝑓 𝑘 , 𝑝 ) , and we identify these indices in dynamical terms. Therefore, we obtain a sort of Poincaré index formula without differentiability assumptions. Our techniques apply equally to both orientation preserving and orientation reversing homeomorphisms. We present some new results, especially in the orientation reversing case.

1. Introduction

There is abundant literature about the fixed point index of a homeomorphism 𝑓 , in a neighborhood of an isolated fixed point and the local dynamical behavior of 𝑓 . There are results in both directions, that is, bounds (or explicit computation) for the fixed point index from dynamical properties of 𝑓 and conversely how the knowledge of the fixed point index is used to describe the dynamics locally.

One can notice that due to the systematic use of Brouwer's translation arcs theorem (see [1] or [2]), most of the known results are limited to orientation preserving homeomorphisms.

It is well known that the classical Poincaré index formula relates the index of a planar vector field with the elliptic and hyperbolic regions in a neighborhood of a critical point. Such a formula, for the iterates of an arbitrary homeomorphism, will give a geometric interpretation of the fixed point indices of the iterates, it could help to attack some open problems and it will provide simple proofs of many of the strongest theorems in the subject. This is the main goal of this article.

The Ulam's problem about the existence of minimal homeomorphisms in the multipunctured plane was solved completely in the negative by Le Calvez and Yoccoz in [3]. The main technique in the proof of their theorem is the computation of the fixed point index of all iterates of an orientation preserving homeomorphism in a neighborhood of a fixed point 𝑝 which is an isolated invariant set, neither an attractor nor a repeller. Given an orientation preserving local homeomorphism 𝑓 ∶ 𝑈 ⊂ ℝ 2 → ℝ 2 , they carry out a detailed local study, near the fixed point 𝑝 . Then they prove the existence of integers 𝑟 , 𝑞 ≥ 1 such that 𝑖 ℝ 2 𝑓 𝑘 = , 𝑝 1 − 𝑟 𝑞 i f 𝑘 ∈ 𝑟 ℕ , 1 i f 𝑘 ∉ 𝑟 ℕ . ( 1 . 1 )

The authors, in [4], using Conley index ideas, gave, in a quite simple way, a general theorem extending the above result to arbitrary local homeomorphisms. In particular, if 𝑓 reverses the orientation, there are integers 𝛿 ∈ { 0 , 1 , 2 } and 𝑞 such that 𝑖 ℝ 2 𝑓 𝑘 = , 𝑝 1 − 𝛿 i f 𝑘 o d d , 1 − 𝛿 − 2 𝑞 i f 𝑘 e v e n . ( 1 . 2 )

Later, Le Calvez extended his theorem with Yoccoz to arbitrary isolated fixed points of orientation preserving planar homeomorphisms. Again the fixed point indices of the iterations of the homeomorphism have periodical behavior. Le Calvez, in [5], uses in a very clever way the nice Carathéodory's prime ends theory (see [6, 7]). The idea of applying the compactification of Carathéodory to study planar dynamical problems is not new. It was introduced by Pérez-Marco in [8] and it was used more recently by the first author, in [9], to prove that the index of arbitrary stable planar fixed points is equal to 1 .

On the other hand, Baldwin and Slaminka, in [10], dealt with the problem of relating the fixed point index of an orientation and area preserving homeomorphism around an isolated fixed point 𝑝 and the number of branches in which the stable/unstable “manifold” of 𝑝 decomposes. The results of Baldwin and Slaminka were improved by Le Roux, in [11], where the fixed point index is used not only to detect stable/unstable branches but also Leau-Fatou petals around 𝑝 . The authors, in [12], gave a stable/unstable “manifold” theorem for arbitrary planar homeomorphisms near a fixed point admitting nice filtration pairs.

There are some papers dedicated to the study of the analogous problem in dimension 3. See [13–16] and its references.

The computation of the fixed point index of any iteration of any planar homeomorphism at an isolated fixed point laying in an isolated invariant compactum was done by the authors in [4, 12]. As we said above, when 𝑝 does not belong to any isolated invariant compactum and the homeomorphism is orientation preserving, Le Calvez improved a result of Brown, see [17], showing that the sequence of indices is periodic. We will find with our methods the same formula for orientation preserving homeomorphisms and we shall solve the problem also for orientation reversing homeomorphisms. The main fact to obtain our results is the existence of special classes of filtration pairs in the Carathéodory's prime ends compactification that will allow us to by-pass the technical problem that occurs if the fixed point does not lay in an isolated invariant compactum.

Roughly speaking, if a fixed point 𝑝 does not lay in arbitrary small isolated compacta, we can consider any disc 𝐽 containing 𝑝 in its interior and take 𝐾 𝑝 , the component containing 𝑝 of the maximal invariant set contained in 𝐽 . By using the Carathéodory's compactification of 𝑆 2 ⧵ 𝐾 𝑝 , we work in a disc and we can compute the index at 𝑝 from the local indices (in semidiscs) of the fixed prime ends that now will admit isolating blocks. The existence of such isolating blocks around the fixed prime ends not only provides a simple technique to compute the index of the iterations of arbitrary homeomorphisms but also allows to identify such indices in a geometrical way. Given a disc 𝐽 the existence of isolating blocks, around the fixed points that appear in the compactification, allows to find dynamical objects (generalized stable/unstable branches and generalized attracting/repelling petals whose definitions we will precise later) which are the keys for the computations of the indices.

Essentially, the index of the homeomorphism at 𝑝 only provides “optimal” dynamical information if 𝑝 admits isolating blocks. Otherwise, the set of indices of the induced homeomorphism in the Carathéodory's compactification of 𝑆 2 ⧵ 𝐾 𝑝 at the new fixed points provides much more information than the index at 𝑝 .

The main goals of this paper are the following:(a)The first goal is to provide a general geometrical method to compute the fixed point index of the iterations of an arbitrary local homeomorphism at an isolated fixed point;(b) Given any Jordan domain 𝐽 , I n v ( c l ( 𝐽 ) , 𝑓 ) ∩ 𝜕 ( 𝐽 ) ≠ ∅ and an isolating block, 𝑁 , is a neighborhood that isolates the fixed (or periodical) prime ends of the component of I n v ( c l ( 𝐽 ) , 𝑓 ) containing 𝑝 , to prove that 𝐽 and 𝑁 determine canonically a number of generalized unstable (stable) branches and generalized repelling (attracting) petals around the fixed point (see Definition 2.6). Their number depends on 𝐽 and 𝑁 but their difference depends just on the germ of 𝑓 ;(c)The third goal is to provide some dynamical consequences. We shall give new and short proofs of some known results and new theorems in the orientation reversing framework.

The paper is organized as follows: in Section 2 we start with some preliminary definitions. We will dedicate subsections to recall the results we will need in the special case where the fixed point is an isolated invariant set and to give a brief presentation of the Carathéodory's prime ends theory. At the end of the section, we give the statement of the main results. Section 3 is devoted to the computation of the fixed point indices of the iterations of arbitrary planar homeomorphisms at an isolated fixed point. In Section 4, we will give the proof of the main theorems and the dynamical meaning of the indices. First we shall study the case where the homeomorphism has a finite number of periodic prime ends. The general case follows easily from this previous simpler case (see Remark 2.12). Finally Section 5 contains the proofs of a number of corollaries of our techniques.

2. Preliminary Definitions and Results. The Main Construction and the Statement of the Principal Results

2.1. Preliminary Definitions

Given 𝐴 ⊂ 𝐵 ⊂ 𝑁 , c l ( 𝐴 ) , c l 𝐵 ( 𝐴 ) , i n t ( 𝐴 ) , i n t 𝐵 ( 𝐴 ) , 𝜕 ( 𝐴 ) and 𝜕 𝐵 ( 𝐴 ) will denote the closure of 𝐴 , the closure of 𝐴 in 𝐵 , the interior of 𝐴 , the interior of 𝐴 in 𝐵 , the boundary of 𝐴 , and the boundary of 𝐴 in 𝐵 , respectively.

Let 𝑈 ⊂ 𝑋 be an open set. By a (local) semidynamical system, we mean a local homeomorphism 𝑓 ∶ 𝑈 → 𝑋 . The invariant part of 𝑁 , I n v ( 𝑁 , 𝑓 ) , is defined as the set of all 𝑥 ∈ 𝑁 such that there is a full orbit 𝛾 with 𝑥 ∈ 𝛾 ⊂ 𝑁 .

I n v + ( 𝑁 , 𝑓 ) (resp., I n v − ( 𝑁 , 𝑓 ) ) will denote the set of all 𝑥 ∈ 𝑁 such that 𝑓 𝑗 ( 𝑥 ) ∈ 𝑁 for every 𝑗 ∈ ℕ (resp., 𝑓 − 𝑗 ( 𝑥 ) is well defined and belongs to 𝑁 for every 𝑗 ∈ ℕ ).

A compact set 𝑆 ⊂ 𝑋 is invariant if 𝑓 ( 𝑆 ) = 𝑆 . A compact invariant set 𝑆 is isolated with respect to 𝑓 if there exists a compact neighborhood 𝑁 of 𝑆 such that I n v ( 𝑁 , 𝑓 ) = 𝑆 . The neighborhood 𝑁 is called an isolating neighborhood of 𝑆 .

An isolating block 𝑁 is a compactum such that c l ( i n t ( 𝑁 ) ) = 𝑁 and 𝑓 − 1 ( 𝑁 ) ∩ 𝑁 ∩ 𝑓 ( 𝑁 ) ⊂ i n t ( 𝑁 ) . Isolating blocks are a special class of isolating neighborhoods.

We consider the exit set of 𝑁 to be defined as 𝑁 − = { 𝑥 ∈ 𝑁 ∶ 𝑓 ( 𝑥 ) ∉ i n t ( 𝑁 ) } . ( 2 . 1 )

If 𝑋 is a locally compact ANR (absolute neighborhood retract for metric spaces), 𝑖 𝑋 ( 𝑓 , 𝑆 ) will denote the fixed point index of 𝑓 in a small enough neighborhood of 𝑆 . The reader is referred to the text of [18–22] for information about the fixed point index theory.

An isolated fixed point 𝑝 is said to be indifferent if for every small enough disc 𝐷 such that 𝑝 ∈ i n t ( 𝐷 ) , I n v ( 𝐷 , 𝑓 ) ∩ 𝜕 ( 𝐷 ) ≠ ∅ .

An isolated fixed point 𝑝 is accumulated if 𝑝 ∈ c l ( P e r ( 𝑓 | 𝑉 ) ⧵ { 𝑝 } ) for every neighborhood 𝑉 of 𝑝 .

2.2. Strong Filtration Pairs

The next definition is based on the notion of filtration introduced by Franks and Richeson, in [23]. It is the key for the direct computation of the fixed point index of any iteration of any homeomorphism of the plane.

Definition 2.1. Let 𝑓 ∶ 𝑈 ⊂ ℝ 2 → ℝ 2 be a local homeomorphism. Suppose that 𝐿 ⊂ 𝑁 is a compact pair contained in the interior of 𝑈 . The pair ( 𝑁 , 𝐿 ) is said to be a strong filtration pair for 𝑓 provided 𝑁 and 𝐿 are each the closure of their interiors and(1) 𝑁 and 𝜕 ( 𝑁 ⧵ 𝐿 ) are homeomorphic to a disc and 𝑆 1 , respectively.(2) c l ( 𝑁 ⧵ 𝐿 ) is an isolating neighborhood.(3) 𝑓 ( c l ( 𝑁 ⧵ 𝐿 ) ) ⊂ i n t ( 𝑁 ) (i.e., 𝐿 is a neighborhood of 𝑁 − in 𝑁 ).(4) For any component 𝐿 𝑖 of 𝐿 , 𝜕 𝑁 ( 𝐿 𝑖 ) is an arc and there exists a topological disc 𝐵 𝑖 such that 𝜕 𝑁 ( 𝐿 𝑖 ) ⊂ 𝐵 𝑖 ⊂ 𝐿 𝑖 , 𝐵 𝑖 ∩ 𝑁 − ≠ ∅ , and 𝑓 ( 𝐵 𝑖 ) ∩ c l ( 𝑁 ⧵ 𝐿 ) = ∅ .

Theorem 2.2 (see [4, 12]). Let 𝑓 ∶ 𝑈 ⊂ ℝ 2 → 𝑓 ( 𝑈 ) ⊂ ℝ 2 be a homeomorphism. Suppose that there exists a strong filtration pair, ( 𝑁 , 𝐿 ) , for 𝑓 and let 𝐾 = I n v ( c l ( 𝑁 ⧵ 𝐿 ) , 𝑓 ) . Then, there are an absolute retract for metric spaces, 𝐷 0 , containing a neighborhood 𝑉 ⊂ ℝ 2 of 𝐾 , a finite subset { 𝑞 1 , … , 𝑞 𝑚 } ⊂ 𝐷 0 , and a map 𝑓 ∶ 𝐷 0 → 𝐷 0 such that 𝑓 | 𝑉 = 𝑓 | 𝑉 and for every 𝑘 ∈ ℕ , F i x ( ( 𝑓 ) 𝑘 ) ⊂ 𝐾 ∪ { 𝑞 1 , … , 𝑞 𝑚 } .
Moreover, (a) if 𝑓 preserves the orientation, then 𝑖 ℝ 2 𝑓 𝑘 = , 𝐾 1 − 𝑟 𝑞 i f 𝑘 ∈ 𝑟 ℕ , 1 i f 𝑘 ∉ 𝑟 ℕ , ( 2 . 2 ) where 𝑘 ∈ ℕ , 𝑞 is the number of periodic orbits of 𝑓 in { 𝑞 1 , … , 𝑞 𝑚 } , and 𝑟 is their period;(b) if 𝑓 reverses the orientation, then 𝑖 ℝ 2 𝑓 𝑘 = , 𝐾 1 − 𝛿 i f 𝑘 o d d , 1 − 𝛿 − 2 𝑞 i f 𝑘 e v e n , ( 2 . 3 ) where 𝛿 ∈ { 0 , 1 , 2 } and 𝑞 are the number of fixed points and period two orbits of 𝑓 in { 𝑞 1 , … , 𝑞 𝑚 } , respectively.

Definition 2.3. Under the setting of the above theorem, the integer 𝑟 ( 𝑟 = 2 if 𝑓 is orientation reversing) is called the period of the strong filtration pair ( 𝑁 , 𝐿 ) .

We conclude this subsection with the next theorem that resumes the main results of [12]. We will construct a family of branches of the stable and unstable “manifolds” associated to a fixed point 𝑝 which admits a strong filtration pair ( 𝑁 , 𝐿 ) . The minimum number of elements of these families depends on the fixed point index 𝑖 ℝ 2 ( 𝑓 𝑟 , 𝑝 ) with 𝑟 being the period of the strong filtration pair ( 𝑁 , 𝐿 ) . In order to make the paper as self-contained as possible, we will sketch the proof which contains some ingredients we will need in the future.

Theorem 2.4. Let 𝑓 ∶ 𝑈 ⊂ ℝ 2 → 𝑓 ( 𝑈 ) ⊂ ℝ 2 be a homeomorphism with 𝑝 being an isolated fixed point of 𝑓 , and let us assume that there is a strong filtration pair of period 𝑟 , ( 𝑁 , 𝐿 ) , such that 𝑝 ∈ i n t ( 𝑁 ⧵ 𝐿 ) , 𝐿 ≠ ∅ , 𝑓 𝑗 ( c l ( 𝑁 ⧵ 𝐿 ) ) ⊂ 𝑈 for 𝑗 ∈ { 1 , … , 𝑟 } and F i x ( 𝑓 𝑟 ) ∩ c l ( 𝑁 ⧵ 𝐿 ) = { 𝑝 } . Let us suppose that the connected component of 𝐾 = I n v ( c l ( 𝑁 ⧵ 𝐿 ) , 𝑓 ) which contains 𝑝 is 𝐾 𝑝 = { 𝑝 } . Then there exist trivial shape continua 𝑆 1 , … , 𝑆 𝑠 , 𝑈 1 , … , 𝑈 𝑠 in c l ( 𝑁 ⧵ 𝐿 ) , with 𝑠 = 1 − 𝑖 ℝ 2 ( 𝑓 𝑟 , 𝑝 ) , such that: (1) ⋃ 𝑠 𝑖 = 1 𝑆 𝑖 ⊂ 𝐾 + 𝑝 and ⋃ 𝑠 𝑖 = 1 𝑈 𝑖 ⊂ 𝐾 − 𝑝 , with 𝐾 + 𝑝 and 𝐾 − 𝑝 being the connected components of 𝐾 + = I n v + ( c l ( 𝑁 ⧵ 𝐿 ) , 𝑓 ) and 𝐾 − = I n v − ( c l ( 𝑁 ⧵ 𝐿 ) , 𝑓 ) which contain 𝑝 ;(2) 𝑆 𝑖 ∩ 𝑆 𝑗 = 𝑈 𝑖 ∩ 𝑈 𝑗 = { 𝑝 } for all 𝑖 ≠ 𝑗 and 𝑆 𝑖 ∩ 𝑈 𝑗 = { 𝑝 } for all 𝑖 , 𝑗 ∈ { 1 , … , 𝑠 } ;(3) 𝑓 𝑟 ( 𝑆 𝑖 ) ⊂ 𝑆 𝑖 , 𝑓 − 𝑟 ( 𝑈 𝑖 ) ⊂ 𝑈 𝑖 , and ⋂ 𝑛 ∈ ℕ 𝑓 − 𝑛 𝑟 ( 𝑈 𝑖 ⋂ ) = 𝑛 ∈ ℕ 𝑓 𝑛 𝑟 ( 𝑆 𝑖 ) = { 𝑝 } for every 𝑖 ∈ { 1 , … , 𝑠 } ;(4) the sets 𝑆 𝑖 ∩ 𝜕 ( c l ( 𝑁 ⧵ 𝐿 ) ) and 𝑈 𝑖 ∩ 𝜕 ( c l ( 𝑁 ⧵ 𝐿 ) ) alternate in 𝜕 ( c l ( 𝑁 ⧵ 𝐿 ) ) .

Proof. If 𝐿 = 𝐿 1 ∪ ⋯ ∪ 𝐿 𝑚 , let us consider the quotient space 𝑁 𝐿 obtained from c l ( 𝑁 ⧵ 𝐿 ) by identifying each 𝜕 𝑁 ( 𝐿 𝑖 ) to a point 𝑞 𝑖 for 𝑖 = 1 , … , 𝑚 . Take the projection map 𝜋 ∶ c l ( 𝑁 ⧵ 𝐿 ) → 𝑁 𝐿 and the retraction 𝑟 ∶ 𝑁 → c l ( 𝑁 ⧵ 𝐿 ) . The map 𝑓 = 𝜋 ∘ 𝑟 ∘ 𝑓 ∘ 𝜋 − 1 ∶ 𝑁 𝐿 ⧵ 𝑞 1 , … , 𝑞 𝑚 ⟶ 𝑁 𝐿 ( 2 . 4 ) induces in a natural way a continuous map 𝑓 ∶ 𝑁 𝐿 → 𝑁 𝐿 . It is easy to see that 𝑓 ( { 𝑞 1 , … , 𝑞 𝑚 } ) ⊂ { 𝑞 1 , … , 𝑞 𝑚 } . Let 𝜃 = { 𝑝 1 , … , 𝑝 𝑠 } be the biggest subset of { 𝑞 1 , … , 𝑞 𝑚 } on which 𝑓 acts as a permutation. It is clear that 𝜃 is an attractor for 𝑓 (is locally constant for every 𝑝 𝑖 ∈ 𝜃 ). Let 𝐴 be the region of attraction of 𝜃 , 𝐴 = 𝑥 ∈ 𝑁 𝐿 ∶ t h e r e i s 𝑛 0 s u c h t h a t 𝑓 𝑛 0 ( 𝑥 ) ∈ 𝜃 ( 2 . 5 ) and let 𝐴 ( 𝑝 𝑗 ) be the component of 𝐴 containing 𝑝 𝑗 ∈ 𝜃 . Let us observe that 𝐾 − 𝑝 and 𝐾 + 𝑝 are trivial shape continua such that l i m 𝑘 → ∞ 𝑓 − 𝑘 ( 𝑥 ) = 𝑝 for every 𝑥 ∈ 𝐾 − 𝑝 and l i m 𝑘 → ∞ 𝑓 𝑘 ( 𝑥 ) = 𝑝 for every 𝑥 ∈ 𝐾 + 𝑝 (see the Main Theorem in [12] for a proof). Then it is not difficult to see that 𝑝 ∈ c l ( 𝐴 ( 𝑝 𝑗 ) ) for all 𝑗 = 1 , … , 𝑠 .
Let 𝐾 𝑖 = ⋂ 𝑛 ∈ ℕ ( 𝑓 ) 𝑛 𝑟 ( c l ( 𝐴 ( 𝑝 𝑖 ) ) for 𝑖 ∈ { 1 , … , 𝑠 } . Since ( 𝑓 ) 𝑟 ( c l ( 𝐴 ( 𝑝 𝑖 ) ) ⊂ c l ( 𝐴 ( 𝑝 𝑖 ) ) , it is clear that 𝐾 𝑖 is a continuum with { 𝑝 , 𝑝 𝑖 } ⊂ 𝐾 𝑖 = ( 𝑓 ) 𝑟 ( 𝐾 𝑖 ) ⊂ c l ( 𝐴 ( 𝑝 𝑖 ) ) . Then we have that ⋃ 𝑖 ∈ { 1 , … , 𝑠 } ( 𝐾 𝑖 ⧵ { 𝑝 𝑖 } ) ⊂ 𝐾 − 𝑝 , then 𝜕 𝑁 ( 𝐿 𝑖 ) ∩ 𝐾 − 𝑝 ≠ ∅ for all 𝑖 = 1 , … , 𝑠 .
Let us define the continuum 𝑈 𝑖 = 𝜋 − 1 ( 𝐾 𝑖 ) ∩ 𝐾 − 𝑝 . We have that 𝑈 𝑖 is negatively invariant for 𝑓 𝑟 and contains 𝑝 .
On the other hand, 𝑈 𝑖 ∩ 𝐾 = { 𝑝 } . In fact, since ⋂ 𝑛 ∈ ℕ 𝑓 − 𝑛 𝑟 ( 𝑈 𝑖 ) is an invariant continuum for 𝑓 𝑟 which contains 𝑝 , then ⋂ 𝑛 ∈ ℕ 𝑓 − 𝑛 𝑟 ( 𝑈 𝑖 ) ⊂ 𝐾 𝑝 = { 𝑝 } . If 𝑥 ∈ 𝑈 𝑖 ∩ 𝐾 , then ⋂ 𝑥 ∈ 𝑛 ∈ ℕ 𝑓 − 𝑛 𝑟 ( 𝑈 𝑖 ) ⊂ 𝐾 𝑝 = { 𝑝 } .
Let us see that 𝑈 𝑖 has a trivial shape. In fact, if 𝑈 𝑖 has a hole 𝑉 , then there are 𝑎 ∈ 𝑉 and 𝑛 0 ∈ ℕ such that 𝑓 𝑟 𝑛 0 ( 𝑎 ) ∈ i n t ( 𝐿 𝑖 ) and 𝑓 𝑟 𝑛 ( 𝑎 ) ∈ c l ( 𝑁 ⧵ 𝐿 ) for all 𝑛 ∈ ℤ , 𝑛 < 𝑛 0 . Then it is immediate that 𝑎 ∈ 𝑈 𝑖 which is a contradiction.
Let us prove that 𝑈 𝑖 ⊂ 𝜋 − 1 ( 𝐴 ( 𝑝 𝑖 ) ) ∪ { 𝑝 } . If 𝑥 ∈ 𝑈 𝑖 ⧵ { 𝑝 } , then there exists 𝑛 0 ∈ ℕ such that 𝑓 𝑛 𝑟 ( 𝑥 ) ∈ c l ( 𝑁 ⧵ 𝐿 ) for all integer 𝑛 < 𝑛 0 and 𝑓 𝑛 0 r ( 𝑥 ) ∈ i n t ( 𝐿 𝑖 ) (if this is not true, 𝑥 ∈ 𝐾 and we have 𝑥 = 𝑝 ). Then it follows that 𝑥 ∈ 𝜋 − 1 ( 𝐴 ( 𝑝 𝑖 ) ) . As a corollary, we obtain that 𝑈 𝑖 = ( 𝜋 − 1 ( 𝐴 ( 𝑝 𝑖 ) ) ∪ { 𝑝 } ) ∩ 𝐾 − 𝑝 .
It is obvious that 𝑈 𝑖 ∩ 𝜕 ( c l ( 𝑁 ⧵ 𝐿 ) ) ⊂ 𝜕 𝑁 ( 𝐿 𝑖 ) .
If we repeat this construction for 𝑖 ∈ { 1 , … , 𝑠 } , we obtain 𝑈 1 , … , 𝑈 𝑠 with 𝑈 𝑖 ∩ 𝑈 𝑗 = { 𝑝 } for every 𝑖 ≠ 𝑗 .
Let us construct the sets 𝑆 1 , … , 𝑆 𝑠 . Let us consider the set 𝜃 = { 𝑝 1 , … , 𝑝 𝑠 } with 𝑝 𝑖 − 1 adjacent to 𝑝 𝑖 (there is an arc 𝛾 ⊂ 𝜋 ( 𝜕 ( 𝑁 ⧵ 𝐿 ) ) joining 𝑝 𝑖 − 1 with 𝑝 𝑖 such that 𝛾 ∩ 𝜃 = { 𝑝 𝑖 − 1 , 𝑝 𝑖 } ). If 𝑝 𝑖 − 1 𝑝 𝑖 is the arc in 𝜋 ( 𝜕 ( c l ( 𝑁 ⧵ 𝐿 ) ) which makes adjacent 𝑝 𝑖 − 1 and 𝑝 𝑖 , we have that there is a component 𝐾 𝑝 𝑖 ⊂ 𝐾 + 𝑝 of 𝜕 ( 𝐴 ( 𝑝 𝑖 ) ) separating 𝑝 𝑖 from 𝜃 ⧵ 𝑝 𝑖 (see the Main Theorem in [12]) with 𝐾 𝑝 𝑖 ∩ 𝑝 𝑖 − 1 𝑝 𝑖 ≠ ∅ .
Let 𝐵 𝑖 be the connected component of c l ( 𝑁 ⧵ 𝐿 ) ⧵ ( 𝑈 𝑖 − 1 ∪ 𝑈 𝑖 ) which contains 𝜋 − 1 ( 𝐾 𝑝 𝑖 ∩ 𝑝 𝑖 − 1 𝑝 𝑖 ) . Then we define 𝑆 𝑖 = ( 𝐵 𝑖 ∪ { 𝑝 } ) ∩ 𝐾 + 𝑝 . Following the steps given with the family { 𝑈 𝑖 } , it is easy to prove the analogous properties for the family { 𝑆 𝑖 } .

2.3. Carathéodory's Prime Ends

Let 𝐵 ⊂ ℂ be the unit open disc and let 𝑓 ∶ 𝐵 → 𝐺 ⊂ ℂ ∪ { ∞ } be an onto and conformal mapping. The problem whether 𝑓 admits an extension to c l ( 𝐵 ) = 𝐵 ∪ 𝑆 1 , by defining 𝑓 ( 𝑧 ) = l i m 𝑥 → 𝑧 𝑓 ( 𝑥 ) for 𝑧 ∈ 𝑆 1 , has a topological answer. Indeed, 𝑓 admits that an extension if and only if 𝜕 ( 𝐺 ) is locally connected. The problem whether 𝑓 has an injective extension has also an answer of topological nature: 𝑓 has an injective extension if and only if 𝜕 ( 𝐺 ) is a Jordan curve (Carathéodory's Theorem, see [24]). If 𝜕 ( 𝐺 ) is locally connected but not a Jordan curve, there are points of 𝜕 ( 𝐺 ) that have several preimages. The situation becomes much more complicated if 𝜕 ( 𝐺 ) is not locally connected. Carathéodory introduced the notion of prime end to describe this setting. The points 𝑧 ∈ 𝑆 1 correspond one-to-one to the prime ends of 𝐺 and the limit 𝑓 ( 𝑧 ) exists if and only if the prime end has only one point (Prime End Theorem, see [24]).

Let 𝐷 ⊂ ℝ 2 be a simply connected open domain containing the point at infinity such that 𝜕 ( 𝐷 ) contains more than one point. Then 𝜕 ( 𝐷 ) is bounded. A cross-cut is a simple arc, 𝐶 , lying in 𝐷 , except in the end points, with different extremities. If 𝐶 is a cross-cut of 𝐷 then 𝐷 ⧵ 𝐶 has exactly two components 𝐴 1 and 𝐴 2 such that 𝐷 ∩ 𝜕 ( 𝐴 1 ) = 𝐷 ∩ 𝜕 ( 𝐴 2 ) = 𝐶 ⧵ { e n d p o i n t s } .

A sequence { 𝐶 𝑛 } of mutually disjoint cross-cuts and such that each 𝐶 𝑛 separates 𝐶 𝑛 − 1 and 𝐶 𝑛 + 1 is called a chain. A chain of cross-cuts induces a nested chain of domains (bounded by each 𝐶 𝑛 ) ⋯ 𝐷 𝑛 + 1 ⊂ 𝐷 𝑛 ⋯ . Each chain of cross-cuts defines an end. Two chains of cross-cuts, { 𝐶 𝑛 } and { 𝐶 𝑛 } , are equivalent if for any 𝑛 ∈ ℕ there is 𝑚 ( 𝑛 ) such that 𝐷 𝑚 ⊂ 𝐷 𝑛 and 𝐷 𝑚 ⊂ 𝐷 𝑛 for every 𝑚 > 𝑚 ( 𝑛 ) . Equivalent chains of cross-cuts are said to induce the same end. If 𝑃 and 𝑄 are ends represented by chains of cross-cuts { 𝐶 ( 𝑃 ) 𝑛 } and { 𝐶 ( 𝑄 ) 𝑛 } such that for every 𝑛 , 𝐷 ( 𝑃 ) 𝑚 ⊂ 𝐷 ( 𝑄 ) 𝑛 for 𝑚 > 𝑚 ( 𝑛 ) , we say that 𝑃 divides 𝑄 . A prime end 𝑃 is an end which cannot be divided by any other.

Let 𝑃 be a prime end. The set of points of 𝑃 is the intersection ⋂ 𝐸 = 𝑛 ∈ ℕ c l ( 𝐷 ( 𝑃 ) 𝑛 ) where { 𝐷 ( 𝑃 ) 𝑛 } is the sequence of domains bounded by any sequence of cross-cuts representing 𝑃 . A principal point of 𝑃 is a limit point of a chain of cross-cuts representing 𝑃 tending to a point. The set 𝐻 𝑃 ⊂ 𝐸 of principal points of a prime end 𝑃 is a continuum (compact connected set) (see [6] or [7] for details).

Each chain of cross-cuts inducing a prime end 𝑃 determines a basis of neighborhoods of 𝑃 . We obtain in this way a topology in the set of prime ends of 𝐷 . More precisely, if ℙ is the set of prime ends of 𝐷 and 𝐷 ∗ is the disjoint union of 𝐷 and ℙ , we can introduce a topology in 𝐷 ∗ in such a way that it becomes homeomorphic to the closed disk and the boundary being composed by the prime ends. It is enough to define a basis of neighborhoods of a prime end 𝑃 ∈ ℙ . Given the sequence of domains { 𝐷 ( 𝑃 ) 𝑛 } , we produce a basis of neighborhoods { 𝑈 𝑛 } of 𝑃 in 𝐷 ∗ . Each 𝑈 𝑛 is composed by the points in 𝐷 ( 𝑃 ) 𝑛 and by the prime ends 𝑄 such that 𝐷 ( 𝑄 ) 𝑚 ⊂ 𝐷 ( 𝑃 ) 𝑛 for 𝑚 large enough.

If 𝑆 2 is the 2-sphere ℝ 2 ∪ { ∞ } and ∞ ∈ 𝐷 ⊂ 𝑆 2 is a simply connected open domain, the natural compactification, due to Carathéodory, see [6], of 𝐷 obtained by attaching to 𝐷 a set homeomorphic to the one-dimensional sphere 𝑆 1 is called the prime ends compactification of 𝐷 . We identify ℝ 2 = ℂ and we consider a conformal homeomorphism 𝑔 ∶ 𝐷 → 𝑆 2 ⧵ 𝐵 (where 𝐵 is the disc 𝐵 = { 𝑧 ∈ ℂ ∶ | 𝑧 | ≤ 1 } ). Now a one-dimensional sphere 𝑆 1 is attached to 𝐷 using 𝑔 . Each point of 𝑆 1 corresponds to a prime end of 𝐷 .

2.4. The Main Construction

Let 𝑓 ∶ 𝑈 → 𝑊 be a local homeomorphism with 𝑈 , 𝑊 ⊂ ℝ 2 open subsets and let 𝑝 be a nonaccumulated and indifferent fixed point in a small enough Jordan domain 𝐽 with { 𝑝 } being the unique periodic orbit contained in c l ( 𝐽 ) and such that 𝐾 𝑝 ∩ 𝜕 ( 𝐽 ) ≠ ∅ for 𝐾 𝑝 being the connected component of 𝐾 = I n v ( c l ( 𝐽 ) , 𝑓 ) which contains 𝑝 . We will suppose that 𝑝 ∈ 𝜕 ( 𝐾 𝑝 ) (e.g., if 𝑝 is not stable and 𝐽 is small enough, then 𝑝 ∈ 𝜕 ( 𝐾 𝑝 ) ).

Remark 2.5. Let us observe that, given 𝑝 being a non-accumulated and indifferent fixed point, if 𝑖 ℝ 2 ( 𝑓 𝑘 , 𝑝 ) ≠ 1 for some 𝑘 ∈ ℕ , then we can select a Jordan domain 𝐽 , as above, with 𝑝 ∈ 𝜕 ( 𝐾 𝑝 ) . In fact, if 𝑝 ∈ i n t ( 𝐾 𝑝 ) for every small enough Jordan domain 𝐽 , then 𝑝 is stable for 𝑓 𝑘 and 𝑖 ℝ 2 ( 𝑓 𝑘 , 𝑝 ) = 1 (see [9, 25]).

It is easy to see that the set 𝐾 𝑝 ⊂ c l ( 𝐽 ) has a trivial shape, that is, 𝐾 𝑝 and ℝ 2 ⧵ 𝐾 𝑝 are connected.

We follow with some of the most important notions of the paper: the generalized stable/unstable branches and generalized attracting/repelling petals. The first ones are essentially branches, in a classical sense, for the map that our homeomorphism 𝑓 induces in the compactification of ℝ 2 ⧵ 𝐾 𝑝 at a fixed prime end.

Let 𝑝 ∈ 𝐽 be an indifferent and non-accumulated fixed point for 𝑓 in the above conditions. Given the open domain 𝑆 2 ⧵ 𝐾 𝑝 , for each open arc 𝛾 ⊂ 𝐽 with end-points 𝑎 , 𝑏 ∈ 𝐾 𝑝 (we do not exclude the possibility 𝑎 = 𝑏 ) such that 𝛾 ∩ 𝐾 𝑝 = ∅ , we call 𝐷 𝛾 the bounded connected component of ℝ 2 ⧵ ( 𝛾 ∪ 𝐾 𝑝 ) . The set 𝐷 𝛾 is an open ball contained in 𝐽 .

Definition 2.6. A continuum 𝑈 𝑝 ⊂ c l ( 𝐽 ) is a generalized unstable branch for 𝑓 at 𝑝 if:(i) 𝑈 𝑝 ∩ 𝐾 𝑝 is an invariant continuum contained in 𝜕 ( 𝐾 𝑝 ) such that 𝑝 ∈ 𝑈 𝑝 ∩ 𝐾 𝑝 and 𝑈 𝑝 ⧵ 𝐾 𝑝 ⊂ 𝐽 is nonempty and has trivial shape components;(ii) 𝑓 − 1 ( 𝑈 𝑝 ) ⊂ 𝑈 𝑝 and ⋂ 𝑛 ∈ ℕ 𝑓 − 𝑛 ( 𝑈 𝑝 ) = 𝑈 𝑝 ∩ 𝐾 𝑝 ;(iii) there exists an open ball 𝐷 𝛾 associated to an open arc 𝛾 , as above, with 𝑈 𝑝 ⊂ c l ( 𝐷 𝛾 ) , 𝑈 𝑝 ∩ 𝛾 a compact set, and such that:(a) the set 𝑈 𝑝 is locally maximal, that is, if 𝑈 𝑝 ⊂ c l ( 𝐷 𝛾 ) satisfies conditions (i) and (ii), then 𝑈 𝑝 ⊂ 𝑈 𝑝 ;(b) for every open neighborhood 𝑉 of 𝑈 𝑝 , there exists 𝑥 ∈ 𝐷 𝛾 ∩ 𝑉 with 𝑓 − 𝑛 𝑥 ( 𝑥 ) ∉ c l ( 𝐷 𝛾 ) for some 𝑛 𝑥 ∈ ℕ .

In an analogous way, we define generalized stable branches 𝑆 𝑝 for 𝑓 at 𝑝 . We only have to replace 𝑓 by 𝑓 − 1 in (ii) and (iii).

A set 𝑅 𝑝 is a generalized repelling petal for 𝑓 at 𝑝 if:(i) 𝑅 𝑝 = c l ( 𝐷 𝛾 ) ⊂ c l ( 𝐽 ) with 𝐷 𝛾 being an open ball associated to an open arc 𝛾 , as above, such that 𝑐 𝑙 ( 𝛾 ) = 𝛾 ∪ { 𝑞 1 , 𝑞 2 } with 𝑝 ∉ { 𝑞 1 , 𝑞 2 } ;(ii) 𝑓 − 1 ( 𝑅 𝑝 ) ⊂ 𝑅 𝑝 and ⋂ 𝑛 ∈ ℕ 𝑓 − 𝑛 ( 𝑅 𝑝 ) ⊂ 𝜕 ( 𝐾 𝑝 ) is an invariant continuum for 𝑓 which contains 𝑝 .

In an analogous way, we define generalized attracting petals for 𝑓 at 𝑝 . We only have to replace 𝑓 by 𝑓 − 1 in (ii).

Remark 2.7. The stable and unstable branches in the classical sense associated to 𝑓 at 𝑝 and constructed in the proof of Theorem 2.4, are, of course, particular cases of generalized unstable and stable branches if we consider the map 𝑓 = 𝑓 𝑟 and 𝐾 𝑝 = { 𝑝 } . It is easy to obtain an adequate arc 𝛾 𝑗 ⊂ c l ( 𝑁 ⧵ 𝐿 ) for each unstable (stable) branch 𝑈 𝑗 .

Let 𝑈 be a Jordan domain such that c l ( 𝐽 ) ⊂ 𝑈 ⊂ 𝑈 ⊂ 𝑆 2 and let 𝑓 ∶ 𝑆 2 → 𝑆 2 be a homeomorphism such that 𝑓 | 𝑈 ′ = 𝑓 . The Carathéodory's compactification of 𝑆 2 ⧵ 𝐾 𝑝 is a disc (obtained by gluing 𝑆 1 to 𝑆 2 ⧵ 𝐾 𝑝 ) which we call 𝐷 . The homeomorphism 𝑓 | 𝑆 2 ⧵ 𝐾 𝑝 ∶ 𝑆 2 ⧵ 𝐾 𝑝 → 𝑆 2 ⧵ 𝐾 𝑝 can be extended to a homeomorphism 𝑓 ∶ 𝐷 → 𝐷 . Let us denote 𝐷 ⧵ ( 𝑆 2 ⧵ 𝐾 𝑝 ) = 𝜕 ( 𝐷 ) and let us consider the set of prime ends obtained from the accessible points 𝐾 𝑝 ∩ 𝜕 ( 𝐽 ) (by arcs on 𝑈 ⧵ c l ( 𝐽 ) ) and which we call 𝒫 ( 𝐾 𝑝 ∩ 𝜕 ( 𝐽 ) ) ⊂ 𝜕 ( 𝐷 ) .

If 𝑓 is orientation preserving and there exist periodic orbits for 𝑓 | 𝜕 ( 𝐷 ) , then all of them have the same period 𝑟 . If 𝑓 is orientation reversing, then 𝑓 | 𝜕 ( 𝐷 ) has exactly two fixed points and period two periodic orbits.

Let us see that the compact sets P e r ( 𝑓 | 𝜕 ( 𝐷 ) ) and 𝒫 ( 𝐾 𝑝 ∩ 𝜕 ( 𝐽 ) ) are disjoint. Let 𝒫 0 be a prime end in 𝒫 ( 𝐾 𝑝 ∩ 𝜕 ( 𝐽 ) ) associated with a point 𝑝 0 ∈ 𝐾 𝑝 ∩ 𝜕 ( 𝐽 ) . Then 𝒫 0 ∉ P e r ( 𝑓 | 𝜕 ( 𝐷 ) ) . In fact, if this is not true, 𝒫 0 is a fixed prime end for 𝑓 𝑟 ( 𝑟 = 2 if 𝑓 is orientation reversing) and, since 𝑝 0 is accessible by an arc 𝛾 𝑝 0 ⊂ 𝑈 ⧵ c l ( 𝐽 ) such that c l ( 𝛾 𝑝 0 ) ⧵ 𝛾 𝑝 0 = { 𝑝 0 } , then the principal points of the fixed prime end 𝒫 0 are the continuum, invariant for 𝑓 𝑟 , 𝐻 𝒫 0 = { 𝑝 0 } ( 𝐻 𝒫 0 ⊂ c l ( 𝛾 𝑝 0 ) ⧵ 𝛾 𝑝 0 = { 𝑝 0 } ). Then, 𝑝 0 must be a fixed point for 𝑓 𝑟 . But this is a contradiction.

Remark 2.8. Note that both 𝑓 and the set of fixed prime ends of 𝑓 depend on the Jordan domain 𝐽 such that I n v ( c l ( 𝐽 ) , 𝑓 ) ∩ 𝜕 ( 𝐽 ) ≠ ∅ . See Example 2.9.

Example 2.9. Let us consider the dynamical system of Figure 1, which gives us a homeomorphism 𝑓 of ℝ 2 with 𝑝 being a non-accumulated and indifferent fixed point.
The Jordan domains 𝐽 1 and 𝐽 2 of Figure 1 are such that I n v ( c l ( 𝐽 1 ) , 𝑓 ) = 𝐾 1 𝑝 is a “petal” which contains 𝑝 and such that 𝐾 1 𝑝 ∩ 𝜕 ( 𝐽 1 ) ≠ ∅ . On the other hand, I n v ( c l ( 𝐽 2 ) , 𝑓 ) = 𝐾 2 𝑝 are two “petals” which contain 𝑝 and such that 𝐾 2 𝑝 ∩ 𝜕 ( 𝐽 2 ) ≠ ∅ .
The maps 𝑓 ∶ 𝐷 → 𝐷 have the dynamical behavior in Figure 2.
The map 𝑓 for 𝐽 1 has, in 𝜕 ( 𝐷 ) , a fixed prime end 𝑝 1 and the map 𝑓 for 𝐽 2 has, in 𝜕 ( 𝐷 ) , two fixed prime ends { 𝑝 1 , 𝑝 2 } .
Following with the main construction, there are two possible situations:(a) P e r ( 𝑓 | 𝜕 ( 𝐷 ) ) is a finite set of 𝑛 points;(b) P e r ( 𝑓 | 𝜕 ( 𝐷 ) ) is an infinite set of points.
Let us suppose that we are in case (a). Remark 2.12 permit us to reduce case (b) to case (a) by identifications to points of adequate intervals in 𝜕 ( 𝐷 ) . If 𝑓 is an orientation preserving homeomorphism, we have that 𝑛 = 𝑞 𝑟 for certain 𝑞 , 𝑟 ∈ ℕ with 𝑟 being the period of the periodic orbits of 𝑓 | 𝜕 ( 𝐷 ) and 𝑞 the number of periodic orbits. On the other hand, if 𝑓 is orientation reversing, we obtain 𝑞 periodic orbits of period 2 and two fixed points in 𝜕 ( 𝐷 ) . It is obvious that 𝑛 = 2 𝑞 + 2 .
Let us suppose that 𝐷 ⊂ 𝑆 2 and let us denote by 𝑓 𝑠 ∶ 𝑆 2 → 𝑆 2 the homeomorphism obtained by pasting along 𝜕 ( 𝐷 ) a symmetric copy of 𝑓 ∶ 𝐷 → 𝐷 .

323069.fig.001
Figure 1
fig2
Figure 2

The next lemma is needed for the computation of the fixed point index 𝑖 ℝ 2 ( 𝑓 𝑘 , 𝑝 ) by using strong filtration pairs.

Lemma 2.10. Given a fixed poin