#### Abstract

A population has two types of individuals, with each occupying an island. One of those, where individuals of type 1 live, offers a variable environment. Type 2 individuals dwell on the other island, in a constant environment. Only one-way migration () is possible. We study then asymptotics of the survival probability in critical and subcritical cases.

#### 1. Introduction

Multitype branching process in random environment is a challenging topic with many motivations from population dynamics (see, e.g., [1–3]). Very little is known in the general case and in this paper we consider a particular two-type branching process with two key restrictions: the process is decomposable and the final type individuals live in a constant environment.

The subject can be viewed as a stochastic model for the sizes of a geographically structured population occupying two islands. Time is assumed discrete, so that one unit of time represents a generation of individuals, some living on island 1 and others on island 2. Those on island 1 give birth under influence of a randomly changing environment. They may migrate to island 2 immediately after birth, with a probability again depending upon the current environmental state. Individuals on island 2 do not migrate and their reproduction law is not influenced by any changing environment. Our main concern is the survival probability of the whole population.

An alternative interpretation of the model under study might be a population (type 1) subject to a changing environment, say in the form of a predator population of stationary but variable size. Its individuals may mutate into a second type, no longer exposed to the environmental variation (the predators do not regard the mutants as prey). Our framework may be also suitable for modeling early carcinogenesis, a process in which mutant clones repeatedly arise and disappear before one of them becomes established [4, 5]. See [6] for yet another possible application.

The model framework is that furnished by Bienaymé-Galton-Watson (BGW) processes with individuals living one unit of time and replaced by random numbers of offspring which are conditionally independent given the current state of the environment. We refer to such individuals as particles in order to emphasize the simplicity of their lives. Particles of type 1 and 2 are distinguished according to the island number they are occupying at the moment of observation. Our main assumptions are(i)particles of type 1 form a critical or subcritical branching process in a random environment,(ii)particles of type 2 form a critical branching process which is independent of the environment. Let and be the numbers of particles of type 1 and of type 2, respectively, present at time . Throughout this paper it is assumed (unless otherwise specified) that and . We investigate asymptotics of the survival probability as . In all cases addressed here we have Therefore, in view of we focus on the asymptotic behavior of .

In Section 2 we recall known facts for constant environments. They will then be compared to the results of this paper on random environments. In Section 3.1 we describe IID environments (Independent and Identically Distributed environmental states) and then in Section 3.2 Markovian environments. The main results of the paper are (i)Theorem 4.1 in Section 4 on the critical case with an IID environment, (ii)Theorem 5.1 in Section 5 on the subcritical case with an IID environment, (iii)Theorem 6.1 in Section 6 on the critical case with a Markovian environment, (iv)Theorem 7.1 in Section 7 on subcritical case with a Markovian environment. Theorems 6.1 and 7.1 treating the case of Markovian environment are extensions of Theorems 4.1 and 5.1 obtained under rather restrictive conditions and yielding qualitatively the same asymptotic behavior as in the case of IID environment.

*Notation 1. *In asymptotic formulae constants denoted by the same letter are always assumed to be fixed and independent of the parameter that tends to infinity (or zero).

#### 2. Two-Type Decomposable Branching Processes

Consider a two-type BGW process initiated at time zero by a single individual of type 1. We focus on the decomposable case where type 1 particles may produce particles of types 1 and 2, while type 2 particles can give birth only to type 2 particles. Put(i) the number of type 2 daughters produced by the particles of type 1 present at time , in particular, ,(ii) the first time when , so that ,(iii), so that gives the total number ever of type 1 particles,(iv), so that gives the total number of type 2 daughters produced by all particles of type 1.

The aim of this section is to summarize what is already known about such branching processes in the case of a constant environment. This will pave our way in terms of notation and basic manipulation with generating functions towards branching processes in IID random and then Markovian environments.

If the environment is constant from generation to generation, two-type decomposable BGW processes are fully described by a pair of probability generating functions where and represent the numbers of daughters of type 1 and 2 of a mother of type 1, while stands for the number of daughters (necessarily of type 2) of a mother of type 2. Let be the first two moments of the reproduction laws. Concerning the second type of particles we assume that implying that the probability of extinction (of a single-type BGW process evolving in constant environment with the probability generating function ) satisfies [7, ] It follows that We will be interested in two kinds of reproduction regimes for particles of type 1, critical and subcritical. In the constant environment setting with , the critical case corresponds to , and the subcritical case is given by . In the critical case with a constant environment we have and according to [8, ]

Next we outline a proof of (2.8) based on the representation preparing for the proof in the random environment case, to be given in Section 4. Thanks to (2.7) and (1.2), it is enough to verify that in order to prove (2.8). However, by the branching property the total progeny of a single-type branching process is 1 plus independent daughter copies of . In terms of the Laplace transform where . As , a Taylor expansion of as a function of yields For , after removing the negligible terms, we get a quadratic equation whose solution shows that Replacing by and using (2.6), we obtain It remains to verify, see (2.9), that This holds, indeed, since by (2.7) and for any fixed the probability is much smaller than the target value of order . (In [9, 10] infinite second moments in decomposable two-type critical processes were allowed.)

On the other hand, in the subcritical case (2.12) implies that so that by (2.6) In view of we conclude that in the subcritical case See [11] for a comprehensive study of subcritical decomposable branching processes in a constant environment.

#### 3. Branching Processes in a Random Environment

A randomly changing environment for BGW processes is modeled by a random sequence of probability generating functions for the offspring distributions of consecutive generations. Throughout this paper we assume that the offspring distribution for type 2 particles is the same across the different states of the environment and characterized by the same generating function . This restriction that greatly simplifies analysis still allows new interesting asymptotic regimes.

We consider two types of stationarily changing environments: IID and Markovian.

##### 3.1. IID Environment

Our description of the IID environment case starts with a simple illustration based on just two alternative bivariate generating functions and with mean offspring numbers and , respectively. We assume that at each time the environment is say “good” with probability , so that type 1 particles reproduce independently according to the , and with probability the environment is “bad” and particles of type 1 reproduce according to the law. In other words, the generating function should be treated as a random function having distribution In particular, the vector of the mean offspring numbers takes values and with probabilities and .

More generally, our two-type branching process in an IID random environment is characterized (besides the fixed reproduction law for the type 2 particles) by a sequence of generating functions independently drawn from a certain distribution over probability generating functions so that In this setting the respective conditional moments , , , and should be treated as random variables. An important role is played by the random variable representing the step size of the so-called associated random walk [12] formed by the partial sums with .

*Notation 2. *Characteristics of the reproduction law in generation are denoted by adding an extra lower index to the generic notation, like in (3.2).

##### 3.2. Markovian Environment

One way to relax the IID assumption on the environment is to allow for Markovian dependence among its consecutive states. We implement this by modelling changes in terms of an irreducible aperiodic positive recurrent Markov chain with countably many states . Assuming a stationary initial distribution , we associate with each state of this chain a probability generating function , so that the changing environment for the branching process is governed by the sequence of identically reproduction laws with Markovian dependence. Due to the stationarity we can again write (3.2) and use the same notation for the marginal moments of the reproduction laws as in the IID case.

To build a bridge to the IID environment case we use an embedding through a sequence of regeneration moments defined as The times between consecutive regenerations are independent and all distributed as . The embedded process defined as is a decomposable branching process in an IID environment with two types of particles and and conditional reproduction generating functions where stands for the -fold iteration of .

*Notation 3. *For all characteristics of the embedded process and related constants appearing in the asymptotic formulae we use the same notation as for the process in the IID case just adding the hat sign.

The key difference from the IID case is that the reproduction law for the -type particles is dependent on the random environment. However, this dependence is of specific nature which we are able to manage using the law of large numbers for renewal processes. Notice that on its own the -type particles form a so-called degenerate critical branching process in an IID random environment [12]: its conditional offspring mean is deterministic . Meanwhile, the conditional variance is random .

Taking the first- and second-order derivatives of (3.6), we can express the moments of the reproduction law of the embedded process in terms of the moments of the consecutive reproduction laws with Markovian dependence. In what follows we use Notation 2 again, while keeping in mind that the sequence now consists of dependent random vectors. It can be shown that where, as usual, the product of the elements of an empty set is one. Furthermore, setting we can write

Lemma 3.1. *Let
**
For the following sum of a random number of random variables
**
a version of the Wald identity holds: . *

*Proof. *For any state consider the function
According to [13, ] this defines a stationary measure which is necessarily of the form . The constants are such that
It follows that

Developing the example of two environmental states from Section 3.1, let us consider a Markov chain with transition probabilities and a stationary distribution . (Notice that corresponds to the IID case.) Under stationarity the regeneration time satisfies implying that If are the two possible values for , we can write and in full agreement with Lemma 3.1.

#### 4. Critical Processes in IID Environment

The single-type critical branching process with an IID environment displays an asymptotic behavior that is in stark contrast with the constant environment formula (2.7). According to [14, ], if then for some positive constant (A much more general limit theorem is obtained in [12].) The following theorem shows that in the decomposable case the difference between constant and random environments is even more striking. For constant environments the survival probability decays as , see (2.8), but in random environments the decay is like .

Theorem 4.1. *Consider a critical decomposable branching process in an IID environment satisfying (2.3) (4.1), (4.2), and
**
If for some positive **
then there exists a constant such that
*

Before turning to the proof, we make some comments on the conditions and statement of this theorem.

*Notation 4. *We will often use the abbreviations and .

Condition (4.5) is needed for the following properties to hold for any fixed ; recall notation agreements Notations 2 and 4,
Each of them is proven via an intermediate step like
relying on the IID assumption for consecutive environmental states. The constant in the statement of Theorem 4.1 is the same as in the asymptotic formula from [15] concerning the total number of particles of type 1 ever appeared in the process:
This constant has a complicated nature and is not further explained here. It is necessary to mention that the representation (4.11) has been proved in [15] under conditions (4.1), (4.2), and (4.4) only for the case when the probability generating functions are linear fractional with probability 1. However, the latter restriction is easily removed using the results established later on for the general case in [12, 14].

Our proof of Theorem 4.1 uses the next lemma.

Lemma 4.2. *Consider conditional moments of the entity defined at the beginning of Section 2:
**
Under conditions (4.1), (4.5),
**
For any fixed , in Notation 4,*

*Proof. *For any fixed ,
Notice that according to (4.3)
Thus, using (4.7) and (4.11) we get
To obtain a similar estimate from the previously mentioned we write, recalling Notation 4,
which together with (4.8), (4.11), and (4.16) yields
Finally, according to (4.9)

*Proof of Theorem 4.1. *We will show that
using a counterpart of (2.9)
and Lemma 4.2.

First we prove the second inequality in (4.21). It follows from (4.22) and the monotonicity of that for any fixed
Recall that and
with the latter inequality being valid thanks to the monotonicity of the second derivative of the generating function. Therefore,
where the last inequality is due to (2.5). It follows that given ,
for sufficiently large . As a result, we see that for large
Now, to finish the proof of the second inequality in (4.21) it remains to use (4.16), Lemma 4.2, and
which again due to Lemma 4.2 follows from the Tauberian theorem [16, , ] applied to the right hand side of

Next, we verify the first inequality in (4.21). From the estimates , valid for , and , we conclude that for all sufficiently large
Thus,
and (4.21) follows due to (4.8) and (4.16).

#### 5. The Subcritical Case with an IID Environment

We continue studying BGW processes in IID environment, but now assume that instead of (4.1). Results rely upon a theorem from [17] giving the asymptotics for as . It requires an important technical assumption, namely, the existence of a constant such that If, in addition, for some and either or then, according to [17], there exists a constant such that It is also known [18–20] that under (5.1) and (5.2)

Theorem 5.1. *If conditions (5.1), (5.2), (5.3), and either (5.4) or (5.5) hold, then
**
for some positive constant , given by (5.15) latter, where
*

*Proof. *Referring to (5.7), put and notice that
From the first equality in (4.22) and the evident inequality we obtain for
On the other hand, we have a similar upper bound

It remains to observe that due to (2.5) and (5.6), the same Tauberian theorem [16, , ] applied to the right-hand side of
yields
with

#### 6. The Critical Case with a Markovian Environment

As compared to the IID case, Markovian environments require extra conditions on the underlying Markov chain. First we assume that the two-type critical process evolves in a stationary Markovian random environment as defined in Section 3.2. Besides, we suppose the validity of (3.11) and that for some This implies that and due to Lemma 3.1 conditions and become equivalent. Moreover, under condition (6.1) the sequence of regeneration times (3.4) satisfies for an arbitrarily small , cf. [21].

Theorem 6.1. *Assume (2.3), (3.11), (6.1), and
**
Further, for some positive let
**
Then , and there exists a constant such that
*

*Proof. *The statement is divided into two steps: first
and then
together with

Fix , and write
Here the last two terms are treated with the help of and (6.2), while the main term is analyzed by means of ideas from the proof of Theorem 4.1. Letting be the number of type daughters produced by particles of type and putting , we deduce from
a lower bound
Hence, applying arguments used to derive (4.21) in Theorem 4.1 one can show that for some
proving (6.6).

To demonstrate that (6.7) follows from (6.6) observe first that due to
we have
Setting we obtain
and for any we get
It follows that
On the other hand, again by (6.2) as
Thus,
A similar estimate shown latter follows from
Finally, relation (6.8) is derived from (6.13) by the law of large numbers argument.

#### 7. Subcritical Processes with a Markovian Environment

Assume now that the two-type subcritical process evolves in a stationary Markovian random environment as defined in Section 3.2. Here, similarly to Section 6 the auxiliary branching process in IID environment with probability generating functions (3.6) and (3.7) plays an important role.

Single-type subcritical processes with a Markovian environment were recently studied in [22]. According to [22] under the conditions of our next theorem one has, similarly to (5.7), that

Theorem 7.1. *Assume that assumption (2.3) holds,
**
and conditions (5.2), (5.3) and either (5.4) or (5.5) are valid for the corresponding random variables related to the embedded process with the key constant replaced by . Suppose, in addition, that
**
Then, there exists a constant , given by (7.16) and (7.20) latter, such that, see (5.9),
*

*Proof. *Our main arguments here are similar to that used in the proof of Theorem 6.1. Fix and a sufficiently large and with write
where
Clearly,
Further, if then according to [21] under condition (7.3) we have
Thus,
and therefore, denoting by the event complementary to , we get
Due to (1.2) and (7.2)
It follows that in view of (1.2), (7.7), and (7.8) one can find such that
On the other hand, using (2.5) and
one can show, arguing as in Theorem 5.1, that for
giving
with
A similar upper bound in view of (1.2) and (7.11) yields

If then condition (7.3) entails
and, as before, this implies
where

We proceed by recalling (6.15). For any
and as
It follows from [21] and our conditions that
Thus,
so that
The corresponding lower bound is obtained similarly.

#### Acknowledgments

E. Dyakonova was supported in part by the Program of RAS “Theoretical Problems of Contemporary Mathematics” and by INTAS Grant no. 03-51-5018. S. Sagitov was supported by Swedish Research Council Grant no. 621-2010-5623. V. Vatutin was supported in part by the Program of RAS “Theoretical Problems of Contemporary Mathematics” and by a Wenner-Gren Foundation Visiting Researcher Scholarship.