Computational Intelligence and Neuroscience

Volume 2015, Article ID 236285, 19 pages

http://dx.doi.org/10.1155/2015/236285

## Stabilization Methods for a Multiagent System with Complex Behaviours

Department of Computer Science and Engineering, “Gheorghe Asachi” Technical University of Iaşi, D. Mangeron 27 Street, 700050 Iaşi, Romania

Received 29 November 2014; Accepted 27 April 2015

Academic Editor: Reinoud Maex

Copyright © 2015 Florin Leon. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

#### Abstract

The main focus of the paper is the stability analysis of a class of multiagent systems based on an interaction protocol which can generate different types of overall behaviours, from asymptotically stable to chaotic. We present several interpretations of stability and suggest two methods to assess the stability of the system, based on the internal models of the agents and on the external, observed behaviour. Since it is very difficult to predict a priori whether a system will be stable or unstable, we propose three heuristic methods that can be used to stabilize such a system during its execution, with minimal changes to its state.

#### 1. Introduction

Multiagent systems have enjoyed increasing popularity by both researchers and practitioners, due to the continuation of trends in computing such as ubiquity, interconnection, intelligence, delegation, and human-orientation [1]. Agent-based systems can be used as a tool to understand social and economic systems, since the aggregation of simple rules of behaviour for individual agents often leads to an extremely rich range of emergent behaviours, which can help to understand complex phenomena in human societies [2–12].

The actions of individual agents are often based on a certain process of decision making. Therefore, stability, intuitively understood as the property of a system to exhibit bounded behaviour [13], is one of the most desired features in multiagent systems, because of the importance of predicting their response to various conditions in the environment.

The initial motivation of developing the interaction protocol which will be analysed in the present paper was to design a set of simple interaction rules which in turn can generate, through a cascade effect, different types of overall behaviours, that is, stable, periodical, quasi-periodical, chaotic, and nondeterministically unstable. They can be considered metaphors for the different kinds of everyday social or economic interactions, whose effects are sometimes entirely predictable and can lead to an equilibrium while, in some other times, fluctuations can widely affect the system state, and even if the system appears to be stable for long periods of time, sudden changes can occur unpredictably because of subtle changes in the internal state of the system.

We organize our paper as follows. In Section 2, we review some of the related work in the areas of dynamical and evolutionary behaviours, stabilization, and phase transitions in multiagent systems. In Section 3, we describe the interaction protocol employed by the multiagent system under study. In Section 4, we present different interpretations of stability and suggest two methods to assess the stability of the system, based on the internal model and the external, observed behaviour. In Section 5, we give some empirical evidence about the difficulty of predicting a priori whether the system will be stable or unstable and present experiments that can help to identify the nature of phase transitions. Section 6 describes three heuristic methods for the stabilization of the system with minimal changes, along with case studies and discussion. Section 7 contains the conclusions of the paper and addresses some future lines of investigation.

#### 2. Related Work

An important aspect of dynamical systems is the issue of stability. In the multiagent systems literature, the study of stability is mainly applied for three broad types of problems. The first is the question of stability in evolutionary systems. The second is formation control, including the stability of simulated swarms of particles or groups of mobile robots. The third is the general issue of convergence of agent interactions, consensus in networks of agents modelled using graphs, and so forth.

Nonlinear effects are characteristic of evolutionary game theory [14], which aims to enhance the concepts of classical game theory [15] with evolutionary issues, such as the possibility to adapt and learn. In general, the fitness of a certain phenotype is, in some way, proportional to its diffusion in the population. The strategies of classical game theory are replaced by genetic or cultural traits, which are inherited, possibly with mutations. The payoff of a game is interpreted as the fitness of the agents involved [16].

Many such models have been proposed, based on the different ways in which agents change their behaviours over time. Among them we can mention replicator dynamics [17, 18], its replicator-mutator generalization [19], and the quasi-species model [20], which have been used to model social and multiagent network dynamics [21–23].

The emergence of cooperation within groups of selfish individuals, where cooperators compete with defectors, is an interesting research direction because it may seem to contradict natural selection. Studying the evolutionary stability in games involving cooperative and noncooperative behaviours such as the repeated prisoners’ dilemma, a set of conditions were found which enable the emergence of cooperation and its evolutionary stability [24]. Recent results reveal that the evolution of strategies alone may be insufficient to fully exploit the benefits of cooperative behaviour and that coevolutionary rules can lead to a better understanding of the occurrence of cooperation [25].

Cyclic interactions, which occur in simple games such as rock-paper-scissors, emerge spontaneously in evolutionary games entailing volunteering, reward, and punishment and are common when the competing strategies are three or more regardless of the particularities of the game. Cyclic dominance is also a key to understanding predator-prey interactions or mating and growth strategies of biological organisms. A review of these issues [26] also presents an analysis of stability of spatial patterns that typically emerge in cyclic interactions.

Another study of the group interactions on structured populations including lattices, complex networks, and coevolutionary models highlights the synergic contribution of statistical physics, network science, and evolutionary game theory to the analysis of their dynamics [27]. In general, it is considered that group interactions cannot be reduced to the corresponding sum of pairwise interactions.

The evolution of public cooperation on complex networks is particularly important and has been studied, for example, in the context of public goods games [28] or the emergent behaviour of agent social networks [29].

In the context of diffusion, which allows players to move within the population, the analysis of the spatiotemporal patterns reveals the presence of chaos, which fits the complexity of solutions one is likely to encounter when studying group interactions on structured populations [30].

The applicability of the concept of evolutionary games can be found in social and natural sciences, with examples such as an RNA virus [31], ATP-producing pathways [32], and traffic congestion [33].

By using ideas from evolutionary computing, a multiagent system can be seen as a discrete Markov chain and its evolution as a Markov process, possibly with unknown transition probabilities [13]. In a model using this approach, in which the number of agents varies according to the fitness of the individuals, a definition for the degree of instability is proposed based on the entropy of the limit probabilities [34].

In the second category, some authors analysed the stability of small groups of fully interconnected particles [35–37]. A centralized algorithm for a system of particles that leads to irregular collapse and a distributed algorithm that leads to irregular fragmentation were proposed [38]. A universal definition of flocking for particle systems with similarities to Lyapunov stability was also suggested [39].

The stability of formation control was analysed in terms of the limitations of the number of communication channels, showing that stability is maintained by appropriately constructing the switching sequence of network structure [40]. Stable flocking motion can be obtained using a coordination scheme that generates smooth control laws for particles or agents, based on attractive and repulsive forces, and stability is investigated with double integrator dynamics [41]. The coordinated control of mobile robots where the agents are periodically interconnected leads to the formulation of a theoretical framework in which the stability of many distributed systems can be considered [42]. For the problem of formation control of a group of homogenous agents, necessary and sufficient conditions for stability in case of arbitrary time-invariant communication topologies were proposed, which reduced the analysis of multiagent systems with any number of agents to the analysis of a single agent [43].

Conditions for swarm stability of nonlinear high-order multiagent systems were also described based on the idea of space transformation, showing that swarm stability can be ensured by sufficient connectivity of graph topology and dissipative property regulated by relative Lyapunov function, with two independent variables, for time-varying or heterogeneous models [44].

Distributed control strategies for motion coordination were proposed and demonstrated to work for the rendezvous problem of planar roving agents using contraction theory. It was found that even if noise were present in the network, rendezvous would still be achieved with some final bounded error because of the contractivity of all the agents guaranteed by the protocol [45]. The rendezvous problem is related to the classical consensus problem in networks and therefore can be solved using a similar approach.

For systems that belong to the third category, stability was analysed for the problem of iterative contracting of tasks. The interactions were resource reallocations through a memoryless bidding protocol where every agent calculated its next bidding price using a discount drawn from a Gaussian distribution. The results showed that the network of agents reached an equilibrium distribution, even in the presence of noise [46].

Another model of a network of agents interacting via time-dependent communication links was proposed which can be applied in domains such as synchronization, swarming, and distributed decision making. Each agent updates its current state based on the current information received from its neighbours. It was shown that, in case of bidirectional communication, convergence of the individual agents’ states to a common value is guaranteed if, during each (possible infinite) time interval, each agent sends information to every other agent, either through direct communication or indirectly via intermediate agents. In case of unidirectional communication, convergence is proven if a uniform bound is imposed on the time it takes for the information to spread over the network [47].

In a scenario in which leaders are required to recruit teams of followers, satisfying some team size constraints, and where agents have only local information of the network topology, an algorithm was proposed and shown to converge to an approximate stable solution in polynomial time. For general graphs, it was found that there can be an exponential time gap between convergence to an approximate solution and convergence to a stable solution [48].

The problem of coordination in multiagent systems becomes more difficult when agents have asynchronous, uncertain clocks. For this situation, necessary and sufficient conditions for stability were suggested based on linear matrix inequalities. The analysis was applied to networked control with random sampling times, as well as an asynchronous consensus protocol. The stability of linear multiagent systems with noisy clocks was also studied [49].

In case of a supervisory control scheme that achieves either asymptotic stability or consensus for a group of homogenous agents described by a positive state-space model, necessary and sufficient conditions for the asymptotic stability, or the consensus of all agents, were derived under the positivity constraint [50].

Some approaches are hybrid, for example, trying to obtain stabilizing control laws for problems such as state agreement with quantized communication and distance-based formation control. Stabilizing control laws are provided when the communication graph is a tree [51].

For heterogeneous, interconnected multiagent systems where the interaction topology is an arbitrary directed graph, a general method was proposed to derive the transfer functions between any pair of agents with different dynamics. A class of multiagent systems is presented for which a separation principle is possible, in order to relate formation stability to interaction topology [52].

Another important issue when analysing the behaviour of a dynamical system is the presence of phase transitions. A phase transition in a system refers to the sudden change of a system property when some parameter of the system crosses a certain threshold value. This kind of changes has been observed in many fields, such as physics, for example, Ising magnetization model [53, 54], graph theory, for example, random graphs [55], cellular automata [56, 57], or biology, for example, the evolution of the numbers of two species [58]. Phase transitions have been observed in multiagent systems displaying simple communication behaviour, that is, when agents update their states based on the information about the states of their neighbours received under the presence of noise [59]. It was shown that, at a noise level higher than some threshold, the system generates symmetric behaviour or disagreement, whereas, at a noise level lower than the threshold, the system exhibits spontaneous symmetry breaking or consensus.

Our proposed multiagent system described in Section 3 also belongs to the third category but exhibits certain differences from other existing models.

#### 3. Description of the Multiagent Interaction Protocol

The main goal in designing the structure and the interactions of the multiagent system was to find a simple setting that can generate complex behaviours. A delicate balance was needed in this respect. On the one hand, if the system is too simple, its behaviour will be completely deterministic. On the other hand, if the system is overly complex, it would be very difficult to assess the contribution of the individual internal components to its observed evolution. The multiagent system presented as follows is the result of many attempts of finding this balance. The wide range of observed behaviours from stable and periodic to chaotic and nondeterministically unstable was described in two previous papers [60, 61].

The proposed system is comprised of agents; let be the set of agents. Each agent has needs and resources, whose values lie in their predefined domains , . This is a simplified conceptualization of any social or economic model, where the interactions of the individuals are based on some resource exchanges, of any nature, and where individuals have different valuations of the types of resources involved.

It is assumed that the needs of an agent are fixed (although it is possible to consider an adaptive mechanism [62, 63] as well), that its resources are variable, and that they change following the continuous interactions with other agents.

Also, the agents are situated in their execution environment: each agent has a position and can interact only with the other agents in its neighbourhood . For simplicity, the environment is considered to be a bidimensional square lattice, but this imposes no limitation on the general interaction model; it can be applied without changes to any environment topology.

##### 3.1. Social Model

Throughout the execution of the system, each agent, in turn, chooses another agent in its local neighbourhood to interact with. Each agent stores the number of previous interactions with any other agent , , and the cumulative outcome of these interactions, , which is based on the profits resulting from resource exchanges, as described in the following section.

When an agent must choose another agent to interact with, it chooses the agent in its neighbourhood with the highest estimated outcome: .

The parallelism of agent execution is usually simulated by running the agents sequentially and in random order. Since one 3 of the goals of the system is to be deterministic, we define the execution order from the start. Thus, at any time, it can be known which agent will be active and with which other agent the active agent will choose to interact. In our case the random order is not necessary to generate complex behaviours. Even if the agents are always executed in lexicographic order (first A1, then A2, then A3, etc.), sudden changes in utilities still occur, although the overall aspect of the system evolution is much smoother.

##### 3.2. Bilateral Interaction Protocol

In any interaction, each agent tries to satisfy the needs of the other agent as well as possible, that is, in decreasing order of its needs. The interaction actually represents the transfer of a resource quantum from an agent to the other. Ideally, each agent would satisfy the greatest need of the other.

For example, let us consider 3 needs () and 3 resources () for 2 agents and : , , , , and . Since need 2 is the maximum of agent , agent will give 1 unit of resource 2. Conversely, will give 1 unit of resource 3.

In order to add a layer of nonlinearity, we consider that an exchange is possible only if the amount of a resource exceeds a threshold level and if the giving agent has a greater amount of the corresponding selected resource than the receiving agent : and .

In the previous situation, if we impose a threshold level , agent will still give 1 unit of resource 2, but will only satisfy need 1 for agent .

Based on these exchanges, the resources are updated and the profit is computed for an agent as follows:

A bilateral interaction can bring an agent a profit greater than or equal to 0. However, its utility should be able to both increase and decrease. For this purpose, we compute a statistical average of the profit, , increase the utility of an agent if the actual profit is above , and decrease the utility if the profit is below .

Thus, the equation for updating the utility level of an agent iswhere the adjusted number of interactions is , is the maximum number of overall interactions that the agent can “remember” (i.e., take into account), and is the rate of utility change. At the beginning, the utility of the agent can fluctuate more, as the agent explores the interactions with its neighbours. Afterwards, the change in utility decreases but never becomes too small.

Similarly, the social outcome of an agent concerning agent is updated as follows:

In this case, the social model concerns only 1 agent and thus the use of the actual number of interactions can help the convergence of the estimation an agent has about another.

Regarding the computation of the average profit, a global parameter of the system, we used a statistical approach where we took into account 1000 continuous interactions between two randomly initialized agents, which exchange resources for 100,000 time steps. The average profit depends on the number of resources, their domain, and the interaction threshold.

Since the social outcome depends on the average profit, the latter must be a fixed point of the corresponding system. If the initial value , the result of the simulation will provide a value and vice versa. Therefore the correct value can be iteratively approximated by increasing or decreasing until .

In the following sections, we will present some heuristic methods to stabilize the utility functions of the agents, such that they reach an equilibrium, in a minimally invasive way, that is, with the least amount of change to the system. We will explore the effect of small endogenous (internal) perturbations, in terms of alternative decisions made by the agents, to the stability of the multiagent system.

#### 4. Interpretations of Stability

##### 4.1. Statistical and Game Theoretical Interpretations

From a statistical point of view, the evolution of the multiagent system can be viewed as a Markov process with an initial distribution and transition matrix. The state of the system at time step is represented by a random variable , which is a vector that contains the probabilities of certain parameters that define the actual state of the system (e.g., agent locations, utilities, and characteristics). In this case, the system can be considered to be stable if the distribution of states converges to an equilibrium distribution; that is, , when . In other words, the system is stable if the probability distribution of system states becomes independent of the time step , for large values of [13].

However, this interpretation cannot be easily applied for the proposed multiagent interaction protocol. First, the visible output of the system is continuous, but it could be discretized into a convenient number of states. Secondly, all the interactions in the system are deterministic; while it is possible to make a statistical interpretation of a larger number of behaviours, the probabilistic framework would generate only approximations of the real evolution of a particular configuration. Thirdly, the most important aspect is that the accumulation of the nonlinear effects makes it very difficult to estimate the transitions between different states, because these transitions do not depend on the decisions of one agent at a time but on the aggregation of the decisions of all agents involved.

Stability is a relatively well-understood concept in physics, where it is regarded as a property of a stable equilibrium, that is, a state where the system returns on its own after a small perturbation. For multiagent systems, stability can also be defined with respect to perturbations of some sort, such as noise, variations in parameter values, or addition or removal of agents. If a small initial disturbance later becomes significant, the system is considered unstable. A multiagent system can be considered in equilibrium when the statistical properties of its performance indicators remain stationary when the external conditions that can impact the system vary [46].

In our case, it has been demonstrated that small perturbations can sometimes change the final state entirely, while in other cases the perturbations fade away [60, 61]. In the present paper, we use small perturbations to correct instability, such that the behaviour of the system should become easier to predict.

Multiagent systems can also be seen as a set of agents engaged in a multiplayer game. In game theory, stability is also a property of equilibrium, and the problem of finding an equilibrium is equivalent to the choice of an optimal strategy. Stability can then be used to describe the characteristics of the set of strategies in equilibrium, where the strategy of each player is the best response to the strategy of the others and where no player has any incentive to deviate, for example, Nash equilibrium [64].

Another related definition of stability considers the set of all actions performed by all agents in the system. An equilibrium point is one where no agent needs to further operate within the system and accepts the currently reached state. A set of actions, by the different agents, is stable if, assuming that an oracle could inform each agent about all the actions in the set performed by the other agents (and all the external events in the environment), each agent would do exactly what is in the set; that is, its observable behaviour would be exactly what the set predicts [65].

Considering again our multiagent system, the main difference in a game theoretic approach is that we take into account the observed behaviour and not the decisions that agents make. We need methods to find an equilibrium in an implicit not explicit way by considering the dynamic succession of interactions between agents.

Therefore, in the following sections, we try to identify some definitions of stability which are more appropriate for our multiagent system.

##### 4.2. External Behaviour Interpretation

From the practical point of view, stabilization can be defined in a straightforward way as the lack of further change to the agent utilities, starting from a certain time step of the simulation (Figure 1).