Structural Learning about Directed Acyclic Graphs from Multiple Databases

Abstract

We propose an approach for structural learning of directed acyclic graphs from multiple databases. We first learn a local structure from each database separately, and then we combine these local structures together to construct a global graph over all variables. In our approach, we do not require conditional independence, which is a basic assumption in most methods.

1. Introduction

Graphical models including independence graphs, directed acyclic graphs (DAGs), and Bayesian networks have been applied widely to many fields, such as data mining, pattern recognition, artificial intelligence, complex systems, and causal discovery [1–4]. Graphical models can be used to cope with uncertainty for a large system with a great number of variables. Structural learning of graphical models from data is an important and difficult problem and has been discussed by many authors [1–5]. There are two main kinds of structural learning methods. One is constraint-based learning and the other is score-based learning. Most of the structural learning approaches deal with only one database with completely observed data. With the development and popularity of computers, various databases have been built, which may contain different sets of variables and overlap with each other. For example, in medical research, a researcher collects data of these variables, another researcher may collect data of other variables, and they have some common variables.

In this paper, we discuss how to learn the structures of DAGs from multiple databases with different and overlapped variables. In our approach, we first learn a local structure from each database separately, and then we combine these structures together to construct a global graph over all variables. Several theoretical results are shown for the validity of our algorithm. Our approach can validly discover DAGs from multiple databases. In our approach, we only need a weaker condition than conditional independence, which is a basic assumption in most methods [1–5]. This approach can also utilize the prior knowledge of conditional independencies to reduce the number of variables in each conditional set.

Section 2 gives notation and definitions. In Section 3, we show how to construct the DAG with multiple databases. We give an example in Section 4 to illustrate our approach for recovering a DAG. Finally a discussion is given in Section 5.

2. Notation and Definitions

Let 𝐺^→_𝑉=(𝑉,^→𝐸_𝑉) denote a DAG where 𝑉 is a set of 𝑛 vertices {𝑎,𝑏,…} and ^→𝐸_𝑉⊆𝑉×𝑉 is a set of directed edges. A directed edge from a vertex 𝑎 to another vertex 𝑏 is denoted by ⟨𝑎,𝑏⟩, and we say that 𝑎 is a parent of 𝑏 and 𝑏 is a child of 𝑎. We denote the set of all parents of a vertex 𝑏 by 𝑝𝑎(𝑏). A path 𝑙 between the two distinct vertices 𝑎 and 𝑏 is a sequence of distinct vertices that starts with 𝑎, ends with 𝑏, and two consecutive vertices are connected by an edge; that is, 𝑙=(𝑐₀=𝑎,𝑐₁,…,𝑐_𝑛−1,𝑐_𝑛=𝑏), where ⟨𝑐_𝑖−1,𝑐_𝑖⟩ or ⟨𝑐_𝑖,𝑐_𝑖−1⟩ is contained in ^→𝐸_𝑉, for 𝑖=1,…,𝑛 (𝑛≥1), and 𝑐_𝑖≠𝑐_𝑗, for all 𝑖≠𝑗. We say that 𝑎 is an ancestor of 𝑏 and 𝑏 is a descendant of 𝑎 if there is a path from 𝑎 to 𝑏 in 𝐺^→_𝑉 and all edges on this path point at the direction toward 𝑏. Denote the set of ancestors of 𝑏 as 𝑎𝑛(𝑏). A path 𝑙 is said to be separated by a set of vertices 𝑍 if and only if there exists at least one 𝑖∈{1,2,…,𝑘−1}, such that 𝑐_𝑖∈𝑍. And 𝑙 is said to be 𝑑-separated by 𝑍 if and only if (1)𝑙contains a “chain” 𝑖→𝑚→𝑗 or a “fork” 𝑖←𝑚→𝑗 such that the middle vertex 𝑚 is in 𝑍, or (2)𝑙 contains an “inverted fork” 𝑖→𝑚←𝑗 such that the collision vertex 𝑚 is not in 𝑍 and no descendant of 𝑚 is in 𝑍.

Two sets 𝐴 and 𝐵 of vertices are separated (𝑑-separated) by a set 𝐶 if 𝐶 separates (𝑑-separates) every path from any vertex in 𝐴 to any vertex in 𝐵, and we say 𝐶 is a separator (𝑑-separator) of 𝐴 and 𝐵. If 𝑎→𝑐←𝑏, ⟨𝑎,𝑏⟩∉^→𝐸_𝑉 and ⟨𝑏,𝑎⟩∉^→𝐸_𝑉 in a DAG 𝐺^→_𝑉, then the triple (𝑎,𝑐,𝑏) is called an immorality and 𝑎 and 𝑏 are called two parents of the immorality.

Note that the two vertexes 𝑎 and 𝑏 of the DAG are nonadjacent if and only if they are 𝑑-separated by some subset 𝑆⊂𝑉{𝑎,𝑏}. Though this is obvious the case by taking 𝑆 to be either 𝑝𝑎(𝑎) or 𝑝𝑎(𝑏) for DAGs, there are certain types of graphs in which nonadjacency is not sufficient for separability, for example, the ancestral graph in [6].

Example 2.1. Consider a DAG 𝐺^→_𝑉=(𝑉,^→𝐸_𝑉) in Figure 1, where 𝑉={𝑎,…,𝑗} and ^→𝐸_𝑉={⟨𝑎,𝑏⟩,⟨𝑎,𝑐⟩,⟨𝑎,𝑑⟩,⟨𝑏,𝑒⟩,⟨𝑐,𝑒⟩,⟨𝑐,𝑓⟩,⟨𝑑,𝑓⟩,⟨𝑓,ℎ⟩,⟨𝑔,𝑒⟩,⟨𝑔,ℎ⟩,⟨𝑔,𝑖⟩,⟨ℎ,𝑗⟩}. In this DAG, we have 𝑝𝑎(𝑒)={𝑏,𝑐,𝑔} and that (𝑐,𝑒,𝑏), (𝑏,𝑒,𝑔), (𝑐,𝑒,𝑔), (𝑐,𝑓,𝑑), and (𝑓,ℎ,𝑔) are immoralities. The path 𝑙=(𝑎,𝑐,𝑓,ℎ) between 𝑎 and ℎ is 𝑑-separated by {𝑐} or {𝑓}, while the path 𝑙^′=(𝑎,𝑏,𝑒,𝑔) between 𝑎 and 𝑔 is 𝑑-separated by the empty set. Vertex 𝑎 is an ancestor of ℎ and ℎ is a descendant of 𝑎. The sets 𝐴={𝑎} and 𝐵={𝑒,𝑓,𝑔,ℎ,𝑖,𝑗} are 𝑑-separated by the set 𝐶={𝑏,𝑐,𝑑}.

Figure 1 

A directed graph 𝐺→𝑉.

Let 𝑛 vertices {𝑎,𝑏,…} in a DAG 𝐺^→_𝑉 denote 𝑛 variables {𝑋₁,…,𝑋_𝑛}. If a joint distribution or density of variables 𝑋₁,…,𝑋_𝑛 satisfies 𝑃𝑥₁,…,𝑥_𝑛=^𝑛_𝑖=1𝑃𝑥_𝑖∣𝑝𝑎_𝑖, (2.1) where 𝑃(𝑥_𝑖∣𝑝𝑎_𝑖) is the conditional probability or density of 𝑋_𝑖 given 𝑝𝑎(𝑋_𝑖)=𝑝𝑎_𝑖, then DAG 𝐺^→_𝑉 and the distribution 𝑃 are said to be compatible [3] and 𝑃 obeys the global directed Markov property of 𝐺^→_𝑉 [2]. We use the notation in [7] to denote independence. Let 𝑋⟂⟂𝑌 denote the independence of 𝑋 and 𝑌 and 𝑋⟂⟂𝑌|𝑍 the conditional independence of 𝑋 and 𝑌 given 𝑍 for any variables or sets of variables 𝑋, 𝑌, and 𝑍. Since in our discussion we always think the joint distribution corresponds to an underlying DAG, we do not differentiate the usage of letters as 𝑋,  𝑌,  𝑍,  𝐴,  𝐵,  𝐶 to denote variables or vertexes; however, we will make an obvious reference if the context is not clear.

As pointed out by [3], if sets 𝑋 and 𝑌 are 𝑑-separated by 𝑍, then 𝑋 is independent of 𝑌 conditionally on 𝑍 in every distribution that is compatible with 𝐺^→_𝑉. In this paper, we assume that all the distributions are compatible with 𝐺^→_𝑉. We also assume that all independencies of a probability distribution of variables in 𝑉 can be checked by 𝑑-separation of 𝐺^→_𝑉, called the faithfulness assumption in [4]. The faithfulness assumption means that all independencies and conditional independencies among variables can be represented by 𝐺^→_𝑉. For a distribution which obeys the faithfulness assumption, we can learn the underlying DAG by checking the pairwise conditional independence 𝑋⟂⟂𝑌∣𝑍, where 𝑋 and 𝑌 are two random variables and 𝑍 is a subset of variables.

A hypergraph is a collection of vertex sets [8, 9]. Multiple databases 𝒞={𝐶₁,…,𝐶_𝐻} are depicted as a hypergraph, where a hyperedge 𝐶_ℎ is an observed variable set in a database and ∪^𝐻_ℎ=1𝐶_ℎ=𝑉 [5, 10]. A database with an observed variable set 𝐶_ℎ is treated as a sample from a marginal distribution of the variable set 𝐶_ℎ. Let 𝐷_ℎ=𝐶_ℎ∩(∪_𝑘≠ℎ𝐶_𝑘), which is the intersection of 𝐶_ℎ and the other sets. Given a collection of databases 𝒞, there is no information on higher interactions over different databases.

Example 2.2. Let 𝒞={𝐶₁={1,2,3,4},𝐶₂={1,3,5,6},𝐶₃={4,6,7}} be a hypergraph, as shown in Figure 2. We can get that 𝐷₁={1,3,4}, 𝐷₂={1,3,6}, and 𝐷₃={4,6}.

Figure 2 

A hypergraph.

3. Structural Learning of DAGs

In this section, we propose an approach for structural learning of DAGs. In our approach, we first learn a local structure from each database, and then we combine these local structures together to construct a global graph over all variables.

Note that for a distribution obeying the faithfulness assumption with respect to a certain DAG 𝐺^→_𝑉, there may not exist a DAG which can fully represent all the conditional independencies in the marginal distribution; see [6] for more discussion on this issue. Though this fact implies that we may not expect to learn a DAG for each database, it will be shown that under a certain condition a marginal structure could be learned which partially reflects the original true structure.

We consider a joint distribution on a set of variables 𝑉 which satisfies the faithfulness assumption and denote by 𝐺^→_𝑉=(𝑉,^→𝐸_𝑉) the DAG which can fully represent the conditional independencies in this joint distribution. We consider the problem of structure recovery from multiple databases. To facilitate our discussion, we first give the definition of the local structure.

Definition 3.1. For a DAG 𝐺^→_𝑉=(𝑉,^→𝐸_𝑉) and a subset 𝑉^′⊂𝑉, we define the local structure 𝐺_𝑉′ to be an undirected graph 𝐺_𝑉′=(𝑉^′,𝐸^′) where the edge set 𝐸^′={(𝑢,𝑣)∶𝑢,𝑣∈𝐸^′𝑎𝑛𝑑𝑢⟂⟂_¨^¨𝑣∣𝑆for all 𝑆⊂𝑉^′⧵{𝑢,𝑣}}.

From the definition, it is known that to judge whether 𝑢 and 𝑣 are adjacent in the local structure 𝐺_𝑉′, we need only to search for a 𝑑-separator 𝑆 from all possible variable subsets 𝑆⊂𝑉^′ such that two variables 𝑢 and 𝑣 are independent conditionally on 𝑆. With the faithfulness assumption, this is equivalent to test whether 𝑢 and 𝑣 are independent conditionally on 𝑆 and this can be done by using data observed on 𝑉^′ only. Note that the edges in the local structure may be spurious in the sense that its two vertexes are not adjacent in the original DAG; we call these edges spurious edges. However, in the section below, we show that such learned local structure could be used to identify part of the true structure of the original DAG under one additional condition. We give a lemma to be used in proofs of theorems.

Lemma 3.2. If 𝑢 is not an ancestor of 𝑣, then 𝑢 and 𝑣 are 𝑑-separated by a subset of 𝑉 if and only if they are 𝑑-separated by 𝑝𝑎(𝑢).

Proof. See [2].

The following two theorems show the relationship between the local structure and the true structure of the original DAG.

Theorem 3.3. Let 𝐴, 𝐵, and 𝐶 be a partition of 𝑉. If 𝐴 and 𝐵 are separated by 𝐶, then two vertices in 𝐴 are 𝑑-separated by a subset of 𝑉 if and only if they are 𝑑-separated by a subset of 𝐴∪𝐶.

Proof. The necessity is obvious since 𝑉⊇𝐴∪𝐶. For the sufficiency, let 𝑢 and 𝑣 be two vertices in 𝐴 that are 𝑑-separated by 𝑆  (⊆𝑉). Thus there is no edge connecting 𝑢 and 𝑣 in 𝐺^→_𝑉. Since 𝑢 and 𝑣 are contained in 𝐴 and 𝐴 and 𝐶 are separated by 𝐵, 𝑝𝑎(𝑢) and 𝑝𝑎(𝑣) are contained in 𝐴∪𝐶. Without loss of generality, suppose that 𝑢 is not an ancestor of 𝑣. From Lemma 3.2 we have that 𝑝𝑎(𝑢)  (⊆𝐴∪𝐶)  𝑑-separates 𝑢 and 𝑣.

From Theorem 3.3, we can see that two vertexes in 𝐴 are adjacent in the DAG 𝐺^→_𝑉 if and only if they are adjacent in the local structure 𝐺_𝐴∪𝐶. This means that with the faithfulness assumption, the existence of edges falling into 𝐴 can be determined validly from the marginal distribution of 𝐴∪𝐶.

Theorem 3.4. Suppose 𝐴, 𝐵, and 𝐶 is a partition of 𝑉 and 𝐴 and 𝐵 are separated by 𝐶. Let 𝑢∈𝐶 and 𝑣∈𝐴∪𝐶. If 𝑢 and 𝑣 are 𝑑-separated by a subset of 𝐴∪𝐶, then they are 𝑑-separated by a subset of 𝑉.

Proof. The result is obvious since 𝐴∪𝐶⊆𝑉.

According to Theorem 3.4, we can see that 𝑢  (∈𝐶) and 𝑣  (∈𝐴∪𝐶) are not adjacent in the DAG 𝐺^→_𝑉 if they are not adjacent in the local structure 𝐺_𝐴∪𝐶. This means that we may get spurious edges in the local structure 𝐺_𝐴∪𝐶.

According to the two Theorems above, we can get that an edge whose two vertices are contained only by one database 𝐶_ℎ can be determined by using the marginal distribution of 𝐶_ℎ without requirement of the other databases. Those edges crossing 𝐶_ℎ⧵𝐷_ℎ and 𝐷_ℎ or falling into 𝐷_ℎ may be spurious. Their existence must be determined according to multiple databases.

Now we give the algorithm for structural learning of directed acyclic graphical models.

Algorithm 3.5. Construct a DAG from Multiple Databases(1) Input: Multiple databases 𝒞={𝐶₁,…,𝐶_𝐻}.(2) Construct a local structure 𝐺_ℎ from database 𝐶_ℎ separately for each ℎ:(a) Initialize 𝐺_ℎ as a complete undirected graph;(b) Delete edge (𝑥,𝑦) from 𝐺_ℎ if there exists a subset 𝑍 of 𝐶_ℎ such that 𝑥 and 𝑦 are conditional independent given 𝑍. (3) Construct the global undirected graph 𝐺_𝑉:(a) Initialize the edge set 𝐸 of 𝐺_𝑉 as the union of all edge sets of 𝐺_ℎ for ℎ=1,…,𝐻;(b) For an edge (𝑥,𝑦), which falls into some 𝐷_ℎ, delete it from 𝐺_𝑉 if it is absent in some other 𝐺_ℎ′.(4) Delete spurious edges to construct the global skeleton:(a) For an edge (𝑥,𝑦) with 𝑥 in some 𝐶_ℎ⧵𝐷_ℎ and 𝑦 in 𝐷_ℎ, delete it from 𝐺_𝑉 if there exists a subset 𝑍 of 𝑛𝑒(𝑦) such that 𝑥 and 𝑦 are conditional independent given 𝑍;(b) For an edge (𝑥,𝑦), which fall in some 𝐷_ℎ, delete it from 𝐺_𝑉 if there exists a subset 𝑍 of 𝑛𝑒(𝑥)⧵{𝑦} or 𝑛𝑒(𝑦)⧵{𝑥} such that 𝑥 and 𝑦 are conditional independent given 𝑍.(5) Construct the equivalence class:(a) Orient the local skeleton 𝑥−𝑧−𝑦 as 𝑥→𝑧←𝑦 if 𝑥 and 𝑦 are not adjacent in 𝐺_𝑉 and 𝑧∉𝑍;(b) Orient other edges if each opposite of them creates either a directed cycle or a new immorality.(6) Output: The equivalence class of DAGs.

Note that 𝑛𝑒(𝑦) at step 4 denotes all the vertices that are adjacent with vertex 𝑦. According to Theorems 3.3 and 3.4, the equivalence class constructed by the above algorithm is valid.

4. Illustration of Structural Learning

In this section, we illustrate our algorithm using the ALARM network in Figure 3 that is often used to evaluate structural learning algorithms [4, 11, 12]. The ALARM network in Figure 3 describes causal relations among 37 variables in a medical diagnostic system for patient monitoring. Using the network, some researchers generate continuous data from normal distributions and others generate discrete data from multinomial distributions [4, 12]. Our approach is applicable for both continuous and discrete data. Since the validity of our algorithm can be ensured by Theorems 3.3 and 3.4, the algorithm is illustrated by using conditional independencies from the underlying directed acyclic graph in Figure 3 rather than conditional independence tests from simulated data.

Figure 3 

The ALARM network and multiple databases.

Suppose that we have three databases as depicted by the hypergraph in Figure 3. Database 𝐶₁ contains variables {1,2,3,4,5,6,7,8,11,27,28}, database 𝐶₂ contains variables {8,9,28,29,30,31,32}, and database 𝐶₃ contains variables {9,10,11,12,13,14,15,16,18,19,20,22,23,24,25,26,33,34,35,36,37}. Thus 𝐷₁={8,11,28}, 𝐷₂={8,9,28}, and 𝐷₃={9,11}. Note that 𝐶₁⧵𝐷₁ and 𝑉⧵𝐶₁ are not conditional independent given𝐷₁. At Step  2, the local structures are obtained separately from the three databases, as shown in Figures 4(a), 4(b), and 4(c), respectively, for example, the undirected edge (6,27) because 6 is independent of 27 conditional on {5}.

(a) The local structure 𝐺1 for database 𝐶1

(b) The local structure 𝐺2 for database 𝐶2

(c) The local structure 𝐺3 for database 𝐶3

Figure 4 

Local structures for all databases.

At step 3, we combine three local structures together to construct the global undirected graph, as shown in Figure 5. At step 4, we delete the spurious edges to get the global skeleton, which is the undirected version of Figure 6. For example, the spurious edge (8,11) can be deleted since variables 8 and 11 are conditional independent given {7,10}.

Figure 5 

The global undirected graph.

Figure 6 

The partially directed acyclic graph.

At step 5, we determine immoralities and orient edges as much as possible. For example, the direction of the undirected edge (1,2) is determined as ⟨2,1⟩ by ⟨4,2⟩ so as not to create a new 𝑣-structure, and the direction of the undirected edge (16,23) is determined as ⟨16,23⟩ by ⟨16,20⟩ and ⟨20,23⟩ so as not to create a cycle. At last we obtain the equivalence class in Figure 6, in which all directed edges are oriented correctly, except that four undirected edges (5,27), (10,33), (21,34), and (35,37) cannot be oriented because any of their orientation leads to a Markov equivalent DAG.

5. Discussion

In this paper, we presented an approach for structural learning of directed acyclic graphs from multiple databases. In our approach, we require that 𝐶_ℎ⧵𝐷_ℎ and 𝑉⧵𝐶_ℎ are separated by 𝐷_ℎ, which is a weaker condition than the condition that 𝐶_ℎ⧵𝐷_ℎ and 𝑉⧵𝐶_ℎ are 𝑑-separated by 𝐷_ℎ. This condition can be judged with experts’ prior knowledge of associations among variables, such as Markov chains, chain graphical models, and time series.

There are several obvious advantages of our approach for structural learning. First we do not require conditional independence, which is a basic assumption in most methods. Second we search 𝑑-separators in 𝐶_ℎ or 𝑛𝑒(𝑥), which is much smaller than 𝑉. At last, the theoretical results proposed in this paper can be applied to scheme design of multiple databases. Without loss of information on structural learning of DAGs, a joint data set can be replaced by a group of incomplete data sets based on the prior knowledge.

Acknowledgments

I would like to thank the referees for their valuable comments and suggestions. This research was supported by NSFC (11001155), Doctoral Fund of Shandong Province (BS2010SW030), and A Project of Shandong Province Higher Educational Science and Technology Program (J11LA08).