Abstract
The task of discovering equivalent entities in knowledge graphs (KGs), so-called KG entity alignment, has drawn much attention to overcome the incompleteness problem of KGs. The majority of existing techniques learns the pointwise representations of entities in the Euclidean space with translation assumption and graph neural network approaches. However, real vectors inherently neglect the complex relation structures and lack the expressiveness of embeddings; hence, they may guide the embeddings to be falsely generated which results in alignment performance degradation. To overcome these problems, we propose a novel KG alignment framework, ComplexGCN, which learns the embeddings of both entities and relations in complex spaces while capturing both semantic and neighborhood information simultaneously. The proposed model ensures richer expressiveness and more accurate embeddings by successfully capturing various relation structures in complex spaces with high-level computation. The model further incorporates relation label and direction information with a low degree of freedom. To compare our proposal against the state-of-the-art baseline techniques, we conducted extensive experiments on real-world datasets. The empirical results show the efficiency and effectiveness of the proposed method.
1. Introduction
The knowledge graph (KG) is directed graph-structured data that captures intricate relationships of real-world objects. KGs cover a wide range of real-world facts in the form of triples to make them machine-readable. Following the resource description format, a triple is a single unit in KG and it can be represented in the form of which mirrors human language structure. More specifically, it can be categorized into relation triple and attribute triple . Taking an example from DBpedia, is one of the relation triples and is an attribute triple. Due to the recent growth of the Linked Open Data, KGs such as DBpedia [1], ConceptNet [2], and WordNet have rapidly proliferated in a variety of domains and languages by enabling AI upstream and downstream applications (e.g., question-answering [3], recommendation system [4], and semantic search [5]) to harness abundant machine-readable knowledge-powered data.
Despite its advancement, there still remain many challenges such as incompleteness due to the ever-evolving nature of KGs under the open-world assumptions. Besides, KGs are often developed for different purposes by individual experts and even in heterogeneous human languages, which widens the knowledge gap between them. As a solution for bridging the knowledge gap, KG alignment techniques aim to find the corresponding entities between multiple KGs, also known as anchor links, to complement each other. However, only a limited number of anchor links exist in real-world cases, and finding latent links manually is costly and can be easily misguided.
To handle such tasks automatically in an efficient way, a line of study has been conducted to develop effective KG alignment techniques by adopting machine learning methods. The most renowned techniques are based on the representation learning approach which represents resources of KGs as points in multidimensional real coordinate spaces. The main goal of such techniques is to retain the expressiveness of each point, so-called embeddings, as much as possible by extracting salient alignment-oriented features. Given partially aligned KGs, the alignment model learns meaningful embeddings to automatically discover the links between the equivalent entities (or relations) based on their pairwise distances in the vector space. The representation learning-based KG alignment techniques can be divided into two main categories: semantic-based and neighborhood structure-based techniques. Semantic-based [6–8] techniques concentrate on capturing the semantic information contained in the triples. One representative method used for learning the semantic pattern of triples is TransE [9] which considers the relation in triples as translation from head entity to tail entity. In recent years, a number of state-of-the-art graph neural networks (GNNs) [10–12] have approaches such as GCN which have been largely utilized for capturing the structural information in the KGs. The fundamental intuition of GNNs is to incorporate local neighborhood information for representing entity embeddings. They have demonstrated their robustness in learning neighborhood-aware vector representations with various message-passing schemes.
However, there is still much room for improvement regarding the following challenges:(i)Relation Patterns and Structure. There are some significant challenges to be addressed for semantic-based techniques which use the translation assumption. First, the translation assumption cannot cover triangular structure (e.g., cyclic) since all the relation embeddings involved have to converge to 0 which results in a contradiction. Second, the same contradiction occurs for capturing symmetry patterns of relations which frequently appear in practice [13] and has shown its robustness at capturing triangular relation structures; however, it requires extra graphs which represent the structure of the relations, hence resulting in a scalability problem when it comes to large-scale KGs. These listed limitations can lead the embeddings to learn incorrect semantic information which leads the encoded vectors to be obsolete. Thus, to learn better representations for KGs, it is stressed to carefully model such intricate relation structures with a more efficient and scalable approach.(ii)Relation-Aware Neighborhood Information. GNN-based models have shown their superiority in learning local neighborhood structure; however, they often overlook to consider several important factors such as the multiple perspectives of relational information. For example, the adjacency matrix that propagated through GCN layers only preserves simple connection information while losing informative indicators by ignoring other additional relational information (e.g., direction, pattern, and relation type). Existing works [14, 15] attempt to incorporate type and direction information of relations; however, such techniques either rely on abstract concepts or suffer from a high degree of freedom. Moreover, it exclusively learns representations of entities while embeddings of relations play a significant role in many tasks.(iii)Expressiveness of Embeddings. Existing GCNs are limited to being trained in Euclidean space where the vectors are represented with real numbers. However, it has been manifested by recent studies that the embeddings in complex space have more potential to retain rich feature information compared to those in real vector space [16–18] by using complex computations. For example, unlike the dot product in real vector space, the Hermitian product in complex space can capture higher-order features of relation based on the antisymmetric nature of it [17]. Hence, the GCN that operates in the complex space is worth to be explored to go beyond the existing works.
To tackle the aforementioned issues, we propose a complex embedding-based model combined with a novel complex GCN framework, namely, ComplexGCN. ComplexGCN takes advantage from two modules: rotation-based embedding module (RE) and deep integration module (DI) to complement two respective perspectives of knowledge semantic and neighborhood information simultaneously by learning embeddings in complex space. RE learns alignment-oriented complex vector representations in complex space of both entities and relations with rotation constraints. By leveraging the rotation concept, it can inherently model all of the relation patterns (e.g., hierarchy, symmetry, and composition) resulting in better semantic-preserving embeddings compared to the translation assumption. We further discover that cyclic structures such as a set of three triples (e.g., , ) can be captured properly with the rotation assumption. With the rotation score function, we fuse two loss functions to encourage the equivalent entities to be generated close to each other. We further combine DI with RE to consider both semantic and neighborhood information. Different from vanilla GCN, DI has the capability to generate both entity and relation embeddings while keeping track of multiple perspectives of relation information (e.g., direction and relation type). It is worth noting that our proposal prevents overparameterization with the weight-sharing mechanism. Also, it effectively passes the complex embeddings through GCN layers with the Hadamard product. Therefore, our model successfully preserves the expressiveness of the embeddings with more meaningful and intense computation. Note that, we not only train the embeddings based on alignment likelihood but also the plausibility of the triples via the end-to-end learning framework of the DI module. After generating two separate embedding spaces with SE and DI, the alignment result is then estimated by combining the alignment scores generated by the respective module.
1.1. Contributions
The main contributions of this paper can be summarized as follows:(i)ComplexGCN learns the representation of both entities and relations in complex spaces from the two viewpoints: semantic-based (RE module) and neighborhood information (DI).(ii)RE module generates knowledge semantic-preserving embeddings in complex space to better capture both triangular structure and relation patterns [16] than conventional translation assumption. We further prove that Hadamard products can capture not only relation patterns but also cyclic structure in KGs.(iii)DI module successfully incorporates additional relation information (e.g., direction and type) while propagating KGs through GCN layers. To surpass the expressiveness of real vector representations, it learns embeddings with complex numbers with an efficient calculation method inspired by QGNN [19].(iv)Different from existing methods, our DI module requires simpler equations and fewer parameters. Moreover, it is a pioneering approach to model the representation learning of knowledge graphs in complex spaces. Different from complex-based GNNs [19, 20], our DI module incorporates directional, self-loop, and relational information that are specific for knowledge graphs.(v)The fusion of RE and DI modules guarantees to retain the semantic and topological information at the same time by complementing each other. The vanishing problem of semantic features of DI is supplemented by the RE, while the lack of structural information of the RE module is filled up with DI.(vi)We conduct extensive experiments on popular publicly available real-world datasets to evaluate ComplexGCN with state-of-the-art baselines. The empirical results demonstrate that our model produces the best performance against the baselines.
The remainder of this paper is organized as follows. The motivations of our work are discussed in Section 2. In Section 3, the problem is formulated and a succinct overview of our framework is presented. In Section 4 and Section 5, the details of our proposal are demonstrated. Our empirical results are organized in Section 6. Finally, the related works are reviewed in Section 7, and Section 8 concludes the paper.
2. Preliminaries
2.1. Motivations
In Figure 1, we offer an example of a KG entity alignment problem containing four main relation patterns: symmetry, inversion, composition, and cyclic patterns. Furthermore, the KG in the upper container is English-version KG, and the one in the lower container is German-version KG, respectively. The black solid arrow connecting entities in different KGs depicts seed alignments (e.g., owl:sameAs), a solid arrow connecting entities in a single KG is an example of typical relation, and all the dotted arrows or entities in a single KG are newly discovered resources. As we can observe from the illustration, there are four types of relation patterns depicted in different colors of arrows: a set of green arrows depicts a composition pattern, a set of pink arrows describes an inversion pattern, blue arrows represent a symmetry pattern, and a set of orange arrows shows the cyclic structure.
It is crucial to carefully encode these complex relation patterns into the relation embeddings since the former works have proven that the local semantic information greatly impacts on the meaningful alignment-oriented embeddings [6, 7, 9]. However, the score function that is constrained by translation assumption [9] has its limitations at coping with these relation patterns. For example, when we attempt to encode the semantics of three triples , , and based on the translation assumption in a 2-dimensional Euclidean space, it faces discrepancy in relation embedding of relation by producing multiple embeddings for it. Such a problem can cause the inconsistency of embeddings and can seriously harm the aligners by hindering to capture of semantic information.
2.2. A Solution Based on Complex Vector Space
In recent years, the majority of representation learning methods have focused on generating embeddings in Euclidean space where each element of a vector is a real number. However, the computations in Euclidean space have shown their limitations in capturing diverse facets of KGs. One remedy for this problem is that recent studies have been started to utilize complex space where each entry of a vector is composed of a complex number with the form, . By involving additional imaginary parts, complex vector representations enable more expressive rendering with intense computations.
In particular, instead of using a real-valued space, entities and relations are represented in a complex space, in which for example the head entity has a real part and an imaginary part [21]. ComplEx [17] introduced a complex vector space that can capture both symmetric and antisymmetric relations using the Hermitian dot product for the composition of the relation, head, and the conjugate of the tail. Inspired by Euler’s identity, RotatE [16] proposed a rotational model that considers the relation as a rotation from head entity to tail entity in the complex space, where the elementwise Hadamard product is used for composition. QuatE [22] extends the complex-valued space to a hypercomplex space by using a quaternion that has three imaginary components, and the Hamilton product is used as the compositional operator for the head entity and relation. Both RotatE and QuatE can capture inversion and composition patterns, as well as symmetry and antisymmetry. However, QuatE incurs more computational overhead than RotatE, as it uses the Hamilton product to capture latent interdependencies within the 4-D space of entities and relations. Especially for the KG representation learning tasks, complex vector representations successfully reflect the complicated relationships of entities compared to real embeddings. Starting from ComplEx [17], RotatE [16] to MRotatE [18], encodings of complex embedding for KG have been explored extensively by applying various calculus and measurements. We further develop our model by generating complex embeddings which capture not only the semantic information but also the topological information of KGs.
One remedy for this issue can be the representation learning method in complex spaces. Based on the expressive computation using the Hadamard product in complex spaces [16], we can successfully capture all four relation patterns based on the rotation approach resulting in semantic well-preserved embeddings. For instance, in Figure 2, the rotation assumption-based RotatE [16] in complex space can capture cyclic relation patterns which have been neglected by the translation assumption in Euclidean space. Despite the significance of the rotation-based score function, only resorting to the local semantic information can lead the model to neglect the global scale of information. To consider both local and global information, the GNN mechanism that works in complex spaces is highly needed. We believe that a reliable and promising alignment can be performed by fusing the semantic-preserving and neighborhood-aware methods that operate in complex spaces.
3. Problem Statement
3.1. Knowledge Graph Structure
Before getting to the formulation, some necessary notations are provided as follows: Resources. Resources in KGs denote every component in KG such as a unique identifier or literal value (e.g., string and integer) indicating entities and predicates (e.g., relation type) placed between subject and object in triple. KG consists of the set of each resource , , and , where denotes the set of entities, denotes the set of relation types, and indicates the set of relation triples. Triples. Triples in the set of relation triples indicate the relationships of real-world objects or events following the format of . Subjects and objects are from the set of entities , and the predicates are selected for the set of relations . For example, is a relation triple where it denotes the relationship that Barack Obama Sr. is the child of the former president Barack Obama with a given triple format. Triangular Structure. One of the most representative triangular structures is a cycle structure. Specifically, such as a set of relation triples , , and can form a cyclic structure. It is crucial for KG embedding models to be able to express such intricate relation structures explicitly to generate meaningful embeddings. Relation Patterns. There are three main relation patterns frequently observed in KGs: symmetry, inversion, and composition. More precisely, symmetry can be a pattern such as and , inversion can be a pattern between and , and lastly, an example of composition can be a pattern where the exists when and hold.
3.2. Knowledge Graph Alignment
Formally, the problem of embedding-based KG entity alignment can be defined as follows.
Problem 1. With two different input KGs, and where each component denotes entities, relations, and relation triples in order, KG alignment techniques focus on computing distance between embeddings of two entities and whereand.
Given the distances between every entity pair from to , it identifies the corresponding entities under the premise that the entity with the shortest distance from the current entity embedding is the most likely to be equivalent to it. Seed Alignment.Most existing KG entity alignment techniques train their model in a supervised or semisupervised manner which requires a sufficient amount of prealigned entities as their reference. These prealigned entities are often called as seed alignment and can be obtained with various approaches such as leveraging semantic owl:sameAs in DBpedia. However, seed alignments demand lots of human effort to acquire which makes the process costly resulting in a lack of seed alignment in real-world datasets. Entity Similarity. The similarity between entities is decided by the distance of the embeddings in a pointwise space. The shorter distance denotes that the two entities are more likely to be the same entities. The final alignments are determined by the top-1 closest entity among other candidates. The distance can be calculated in the embedding space with various types of distance metrics. The most popular measures are Manhattan distance, L2 norm, and cosine similarity.Importation notations are summarized in Table 1.
3.3. Requirements
In the scope of this work, there are three main requirements that need to be satisfied which are as follows: (R1) Relation Pattern-Aware Embedding. To guarantee accurate embeddings, it is crucial to integrate relation pattern information. In order to capture the aforementioned four major relation patterns, it has been validated that the rotation constraint [16] can be applied in complex spaces. While making the embeddings to capture the semantic structure with relation patterns, we also need to consider alignment-oriented features to enforce the equivalent embeddings to be close as much as possible. The details of the implementation of this process are provided in Section 4. (R2) Neighborhood-Aware Complex Embedding. Along with the semantic features of triples, we need to consider the similarity of neighborhood structure information of entities. Since a number of research studies have shown that the sophisticated calculus can be performed with embeddings in complex spaces, it is highly on demand to explore the GCNs that operate with such expressive computations. In the scope of this paper, we propose a novel GCN framework that generates complex embeddings by using the Hadamard product-inspired method. We discuss it in more detail in Section 5. (R3) Auxiliary Relation Information Integration. The expressiveness and the accuracy of the embeddings can be further improved by integrating auxiliary relation information such as relation type and direction. However, the majority of the existing GCNs ignores the supplementary relation information and simply propagates the connectivity information. Besides, integrating such additional information often requires excessive parameters and increases the complexity of the computation. To overcome such problems, we design simple but effective approaches to integrate relation type and direction information. A more in-depth explanation about this approach is provided in Section 5.
3.4. Framework Overview
To realize the existing challenges, we designed a novel framework, namely, ComplexGCN, consisting of two main modules: rotation-based embedding module (RE) and deep integration (DI) module. As demonstrated in Figure 3, there are two main components in our framework: rotation-based embedding module in Section 4 and deep integration module in Section 5. The overview of our framework is presented as follows: Rotation-Based Embedding Module. First, to be able to capture the four prevalent relation patterns, we exploit the rotation score function [16] which rotates the head entity with the constrained relation embedding and minimizes the distance between the rotated head entity and tail entity in each triple in complex spaces. By rotating the entities by given angles instead of adding them [9], the relation patterns can be aptly captured by solving contradictions that the translation assumption often neglects (R1). Note that, this module is different from the original RotatE in the sense that we enforce the embeddings to secure both semantic and alignment information by optimizing the semantic-preserving objective function and alignment loss function at the same time. Deep Integration Module. Despite the promising direction of the RE module, there still remains space for improvement in terms of neighborhood information. Neighboring entities and the relations that connect the entity with its neighbors play an essential role in the alignment task since there is a tendency that the entities on par to have a similar neighborhood. In our work, we propose a deep integration module (DE) which is a generalized GCN framework that is inspired by reference [19] to generate complex embeddings while incorporating neighboring information (R2) with extra relational information. Specifically, the DE module captures the direction and type information of relations with specific weight matrices and rotation composition operations (R3). Besides, our DE module enables the Hadamard product-inspired computation between direction-specific weight matrices and hidden node embeddings to give more expressiveness through a more sophisticated calculus (R2).
4. Rotation-Based Embedding Module
This section first discusses the learning approach of the semantic-preserving embeddings based on the rotation assumption which was devised by the authors in reference [16]. We then train them to retain both semantic and alignment features by constraining with two individual loss functions.
4.1. Rotation-Based Embedding
The ultimate goal of the RE module is to learn alignment-oriented complex vector representation of entities and relations by preserving relation triple information. Hence, we leverage the rotation function [16] to generate complex embeddings that better reflect complicated relational structures and relation patterns in complex space . To estimate the legitimacy of each relation triple , we calculate the rotation-based score function aswhere denotes complex vector representation, indicates entrywise product, is the Manhattan distance metric, and , , and denote head entity, relation, and tail entity, respectively. In essence, the score function is based on Euler’s formula which is another form of the complex number . By constraining the modulus of every relation representation () to be equal to 1, entrywise multiplication between head entity embedding and relation embedding can be regarded as rotation from head entity to tail entity by radians. For example, if we assume is , then the multiplication of and equals to . However, the rotation-based score function is not tailored for the alignment problems since the purpose of such a score function is to recognise the semantics in triples. Hence, we further combine two different loss functions to ensure that the RE module is alignment-oriented.
It is worth noting that such rotation calculus in complex spaces guarantees that the model is able to capture three popular relation patterns along with the cycle structure. The following definitions demonstrate the effectiveness of the rotation function in preserving the relation patterns and structure.
Definition 2. A relation is symmetric, if and only if the phase of relation embedding equals to 0 or .
Definition 3. A relation is an inverse of and vice-versa, if and only if is a conjugate of (or in reverse).
Definition 4. A relation and compose , if and only if .Along with the abovementioned three relation patterns [16], we define additional definitions for the cyclic structure which is prevalent in real-world KGs.
Definition 5. A relation can reflect a cycle structure, if its phase equals to .The examples of the definitions are illustrated in Figure 2. The listed definitions ensure that the intricate relation patterns can be expressed in embedding spaces using the rotation concept.
4.2. Semantic and Alignment-Preserving Loss Functions
Based on this assumption, the RE module attempts to ensure that the rotated head entity is as close as possible to the corresponding tail entity from the positive relation triple set , while keeping the rotated head entity and tail entity to be far away from each other where those are from the negative triple set . Formally, the knowledge semantic-preserving objective function is defined as follows:where is the margin hyperparameter, denotes the rotation-based score function, is max(, 0), and and indicate the positive and negative relation triples, respectively. Note that, we further guarantee the distance between positively aligned entity pairs to be close as much as possible with the following loss function:where is the cosine distance, is the set of entity seed alignments, and returns the argument if it is larger than 0, otherwise 0. The final loss function for the RE module is the sum of , and the is multiplied by the importance coefficient as
The is chosen from the list of candidates , and we select the one that provides the best result.
5. Deep Integration Module
In this section, the details of the Hadamard product-inspired GCN framework that works in complex spaces are discussed. To incorporate additional relation information, we use composition operator and specific weight matrices during the propagation.
5.1. Complex Graph Convolutional Networks
To harness the advantage of hypercomplex space, QGNN [19] has first introduced a GNN model that can produce embeddings within the quaternion space. Borrowing the computation method of the Hamilton product, the product between the weight matrix and the hidden node vector representation has been defined as follows:where is the activation function, is the set of neighbors of , denotes the Hamilton product, is the quaternion embedding of node in th layer, denotes the edge constant, and is the weight matrix. Since QGNN leverages the weight-sharing method across the real and imaginary component vectors of a node representation, it enables capturing the subtle differences between different nodes. To the best of our knowledge, such a GCN framework is compatible with complex spaces that have not been handled in our research field so far. Despite the achievement of QGNN, it is still worth developing a generalized GCN framework that works in complex spaces since embeddings in complex spaces still show the runner-up performance in practice while having less degree of freedom.
5.1.1. Direction-Aware Hadamard Product
To reflect direction information, we adopt two types of direction-aware weight matrices and for incoming and outgoing neighbors, respectively. We use individual weight matrix which is specified for self-loop to integrate the embedding of the source entity from the previous layer with a relation that connects it to itself. Moreover, we aim to replace the Hamilton product with the Hadamard product for the following reasons. First, the former is applicable to square matrices, while the latter can be applied to matrices of any size. This is suitable for imbalanced graph alignment, where the two input graphs do not have the same size. Second, the Hadamard product is straightforward to compute, as it requires only elementwise multiplications.
More precisely, going beyond QGNN [19], we reconstruct the weight matrix as a nodewise concatenation of the direction-aware matrices and the individual matrix, that is, . In that, each inside matrix is defined as follows:where denotes the real part of the corresponding weight matrix and is the imaginary part of the weight matrix. Each , , and denotes outgoing directions, incoming directions, and self-loop of relations. Unlike R-GCNs [15] where a specific weight matrix is assigned to each relation type, we leverage the weight-sharing mechanism for the three aforementioned relation directions so that the model can avoid the overparameterization problem. While keeping the simplicity of the model, it still can capture meaningful features by incorporating relation-type information with the rotation-based composition operator.
The new version of the weight matrix allows the Hadamard product between the weight matrix and the node representation to successfully generate complex embeddings with richer information. More precisely, the Hamilton product in equation (9) is replaced by the Hadamard product between the abovementioned weight matrices and the hidden complex representation of entities that can be described as follows:
For the sake of simplicity, the equation can be expressed as matrix multiplication represented as where is the real part of the weight matrix, denotes the imaginary part of the weight matrix, is the real part, and is the imaginary part of the hidden node embedding. By transforming the hidden representations in complex space instead of the Euclidean space through the Hadamard product, we can more rigorously tune the embeddings which can result in capturing the features more precisely.
After propagating through the GCN framework, the final vector representations are generated by concatenating the hidden node representations at all GCN layers and then performing the linear transformation on them. Hence, we can incorporate local and global aspects of the neighborhood information to make the embeddings more consolidated.
5.1.2. Entity-Relation Composition
The key idea to incorporating relational-type information is to feed forward the synthesis of entity and relation through the GCN layers. To keep its expressiveness and simplicity in complex spaces, we leverage the rotation concept using the Hadamard product as follows:where denotes the Hadamard product between and , represents the entity embedding, and denotes the relation embedding that connects entity and . Note that, to follow the track of relation information, the hidden embedding in equation (12) is replaced by equation (13).
5.1.3. Put It Altogether
In the end, we design a novel GCN mechanism that learns complex embeddings in a more careful manner while considering both relation type and direction information at the same time via end-to-end learning. As the embeddings in the complex plane (i.e., the Argand plane) can imply meaningful semantic information through the entrywise product (i.e., the Hadamard product), we also leverage this property into our GCN model to move beyond the traditional multiplication between weight matrix and embeddings. More precisely, the embeddings of entities are updated via the following propagation rule per node:where denotes the activation function, denotes the set of pair of neighboring nodes and corresponding relations, and are trainable weight matrices, is the hidden representation of node in layer , is the Hadamard product-based composition operator inspired by rotate [16], and denotes the Hadamard product between the weight matrix and node representation. Note that, the reasons and are separated instead of combined into one weight matrix as the former represents neighbor information, while the latter preserves the node identity across layers.
In summary, we use a different approach compared to QGNN to incorporate the information of relation types by updating embeddings not only with its neighbors but also with the neighbors and their corresponding relations that connect them to the source entity.
5.2. Loss Function
To enforce aligned entities to be closer than those that are not, we utilize prealigned entities set as a reference when training the DI module. The model aims to minimize the below margin-based scoring function [13].where represents the hyperparameter, is a set of corrupted entity pairs, and denotes the cosine distance. is built by replacing either or in truly alignment entity pairs with an entity in -nearest neighbor candidates [13]. By sampling the negative alignments with close neighbors instead of randomly selecting from a uniform distribution, it can enforce the model to be able to distinguish subtle differences between a truly corresponding counterpart and the falsely corresponding one which has a high alignment possibility.
It is noteworthy that the real and the imaginary parts of the hidden node embeddings are obtained via the backward propagation when training the model with the loss function in equation (15). In particular, and are treated as two variables in equation (11); thus, they can be obtained using an optimizer such as stochastic gradient descent. The partition of hidden embeddings into real and imaginary parts enables the representation learning of high-order features, thus capturing the subtle difference between correct and incorrect alignments.
5.3. Alignment Inference
The final alignment score is determined by considering both similarity results generated from RE and DI modules. To effectively integrate two individual similarity results, it is efficient to utilize the late-fusion method instead of the early-fusion approach to maintain the characteristics of separated embedding spaces. Before we calculate the final result, we concatenate the embeddings generated from each GCN layer in order to consider from local range to the global scope of neighborhood information. Then, we fuse and by following the late-fusion method as follows:where denotes the balance coefficient that defines the importance of each module. Note that, in our model, similarity scores are calculated by measuring the cosine distance between entities in every entity pair. Finally, the likely-to-be-aligned counterparts are selected by the popular heuristic approach such as the greedy search algorithm. To retrieve the one-to-one alignments, we first sort the possible candidates of each entity in a descending order of similarity score and choose the one with the highest cosine similarity as the final alignment. Then, the entries in the similarity matrix which represent the similarity between or and other nodes are deleted. With the updated similarity matrix, we repeat the same procedure and end the iterations when all the entities of one of the source and target KGs have been matched.
6. Experiments
6.1. Experiment Setup
6.1.1. Datasets
We conduct experiments on the eight benchmark datasets constructed by the authors in reference [23] using the IDS (iterative degree-based sampling) algorithm. Based on the renowned real-world KGs: DBpedia [1], Wikidata [24], and YAGO [25], we adopt two monolingual and three multilingual KG datasets with tens of thousands of entities to evaluate the performance in various cases. The chosen curated datasets are divided twofold depending on the density of the average node degree: (1) V1: sparse version and (2) V2: denser version. V2 has a more similar structure to the real-world dataset by keeping the nodes with a high degree while pruning those that are connected to fewer neighbors. The detailed statistics of the V1 and V2 datasets are organized in tables Tables 2 and 3.
6.1.2. Metrics
We utilize three representative metrics that are widely used for assessing the performance of KG alignment techniques.(i)Hits@m (the higher the better): Hits@m is one of the widely adopted measures that aims to estimate the alignment performance based on whether the true positive alignment appears within the top-m ranked candidates.(ii)MR (mean rank, the lower the better): to examine the alignment in terms of how high the anchor node is ranked among other anchor candidates, we use MR (mean rank) formulated as follows: where indicates the rank of a truly aligned target node among the other candidates.(iii)MRR (mean reciprocal rank, the higher the better): another popular metric is MRR, represented as follows: where alignment performance is measured based on the average of reciprocal rank of truly aligned entities.
6.1.3. Comparative Models
To investigate the ability of our model, we carefully choose 7 competitive state-of-the-art KG alignment approaches organized as follows:(i)MTransE. It is based on the translation assumption to train the alignment-oriented embeddings of entities and relations and then learns the transformation matrix to assist truly matching pairs to have similar embeddings [8](ii)JAPE. It utilizes both relation and attributes triples to generate embeddings and uses the parameter-sharing method to unify the embedding spaces [6](iii)GCN-Align. It employs GCN (graph convolutional networks) to produce entity embeddings while considering both relation and attribute triples simultaneously [14](iv)BootEA. It augments the training data during iterations by labeling close embeddings as newly generated seed alignments by using the bootstrapping method [7](v)RDGCN. It generates relation-aware embeddings by encouraging attentive interaction between input KGs and their dual relation graphs. Then, it feed-forwards the embeddings through two-layer GCNs with highway gates [13](vi)AliNet. It extends the scale of the neighborhood to incorporate informative distant neighbors and then aggregates neighboring nodes by giving high importance to helpful neighbors based on attention GNNs and gating mechanism [26](vii)MultiKE. It generates three perspectives of entity features (entity name, relation, and attribute) and then incorporates them with three combination strategies to consider multiple views of alignment information [27](viii)Dual-AMN. It captures both intragraph and cross-graph information while minimizing computational complexity with a method for selecting challenging negative samples with reduced loss impact [28]
Note that, there are recent graph embedding methods such as QGNN [19] and magnet [20]. However, they mostly focus on single-graph tasks such as node classification and link prediction, and thus, they are not applicable to our setting. We include their evaluations in subsection 6.3 for reference.
6.1.4. Training Details
Following reference [23], we set the ratio of the seed alignments for training as 20 %, validation as 10 %, and test as 70 % for all datasets. To search for the suitable hyperparameters, we take the listed values as candidates: learning rate in , margin and in , matrix balancing weight in , loss balancing weight in , number of GCN layers in , and embedding dimension size in . The final selected hyperparameters are learning rate , , , , , number of GCN layers , and embedding dimension size for RE module and for DI module . In addition, we set the maximum epoch number as 2000 and check the result at every 10 epochs on the validation set and terminate the training before it hits the maximum epoch when starts to decrease in succession. Table 4 summarizes the hyperparameters.
6.1.5. Implementation
We leveraged the PyTorch library along with Python 3.6 to implement and run our model. To execute the experiments in a fair manner, we ran the experiment over 100 times for each technique. Experiments were conducted on an AMD Ryzen Threadripper 3.8 GHz system with 64 GB RAM and four GTX Titan X graphic cards. We used random initializations.
6.2. End-to-End Comparison
We conduct an end-to-end comparison between ComplexGCN and selected state-of-the-art baseline techniques on real-world monolingual and multilingual KG datasets.
6.2.1. Monolingual Entity Alignment
The objective of the monolingual entity alignment task is to find out matching entities across two different input KGs constructed in the same language (i.e., English). For the evaluation metrics, we take four measures into consideration: Hits@1, Hits@10, MR, and MRR. The alignment technique is regarded as efficient with a higher Hits@k score and MRR, while a lower MR indicates high performance. Table 5 provides convincing results for our experiment. As can be seen from Table 5, our model consistently outperforms all other baseline techniques on four monolingual datasets. Specifically, ComplexGCN achieves up to on Hits@1, on Hits@10, and 0.983 on MRR.
From the experiment result, we can rationalize the effectiveness of two main properties of our model: (1) the Hits@1 and MRR of the result of the Dual-AMN, RDGCN, and GCN-Align demonstrate that incorporating multiperspective relation information assists the performance gain. From this, it can be found that our rotation-based composition operator in the DI module makes our model efficient. (2) The large gap in accuracy between MTransE and ComplexGCN is observed in all scenarios, which implies that our DI module takes an important role in alignment performance improvement.
Following ComplexGCN, both BootEA and RDGCN and Dual-AMN have the highest results among the baseline techniques, while MTransE obtained the lowest output among all the other baseline techniques. This indicates that more complicated architecture often achieves better performance than solely relying on the translation constraint method. Moreover, GNN-based techniques such as GCN-Align, RDGCN, AliNet, and Dual-AMN consistently exhibited promising results which indicate the significance of GNNs on the KG alignment task.
6.2.2. Multilingual Entity Alignment
In practice, there often exists a huge knowledge gap between different language-version KGs. It is crucial to discover equivalent entities between KGs in different languages; however, such a task requires a more sophisticated approach to address. The same alignment metrics are adopted to estimate the performance of the techniques on multilingual datasets for the monolingual test. From Table 6, the results that are similar to the experiment on monolingual datasets can be observed.
Different from the monolingual test, Dual-AMN, RDGCN, and MultiKE were the runner-up techniques among the baselines in most of the scenarios. It can be assumed that the algorithms that involve various pieces of information about KG are more suitable for multilingual settings. In addition, the margin between the Hits@1 results of our model and the runner-up technique was larger than those in a monolingual environment. Such a large margin of up to on Hits@1 demonstrates that our model performs better on multilingual datasets, which validates the robustness of the relation-aware method as well as the power of the architecture of our model on multilingual datasets.
6.2.3. Running Time
Time complexity is one of the important metrics that evaluate the performance of the models since the shorter the running time, the more practical they are at the application level. As depicted in Figure 4, the running time of our model is quite competent in the D-Y-V2, D-Y-V1, D-W-V2, and D-W-V1 datasets (BootEA is the slowest method). However, our model is much slower in EN-FR-V2, EN-FR-V1, EN-DE-V2, EN-DE-V1, DBP-V2, and DBP-V1 datasets. This could be explained by the fact that two different languages have inherently different semantics that could not be aligned perfectly, whereas our rotation-based embedding tries to find the perfect entity calibration, which leads to slow convergence.
There are possible improvements to reduce run time such as simplifying tensor products with circular correlation operation [29] and NeuralLP [30]. However, these methods are often applied to more than one million entities, which is out of the scope of our datasets. Thus, they will be considered as our orthogonal future work.
6.3. Ablation Test
To analyse the effectiveness of each module in our model, we conduct the ablation study on the V1 and V2 D-W-15K datasets. We compare the results of our model and three variants as follows:(i)Var-1. In this variant, we discard the DI module and only keep the SE module, in order to evaluate the impact of the DI module(ii)Var-2. In this variant, we discard the SE module and only keep the DI module to see the impact of the SE module(iii)Var-3. In this variant, we replace the DI module with vanilla GCN to validate the effectiveness of our relation-aware DI module(iv)Var-4. In this variant, we use a single weight matrix instead of three as in equation (10). This is actually a better version of QGNN [19] that leverages the design of our DI module(v)Var-5. In this variant, we replace the DI module with a magnet [20] to validate the effectiveness of our relation-aware DI module.
As it is demonstrated in Table 7, the original version of our model outperforms other variants. More precisely, the margin of at least on Hits@1 is observed between Var-1 and the original ComplexGCN which indicates that our DI module significantly contributes to performance improvement. Second, another variant Var-2 also shows lower accuracy which justifies that it is an effective approach to take both SE and DI modules into consideration. This shows that both semantic and neighborhood information is worth incorporating to enhance the quality of the alignment-powered embeddings. Third, the comparison between Var-3 and our model proves the comparative superiority of our newly proposed GCN architecture compared to vanilla GCN. According to the results, our GCN framework ranks the truly aligned entities higher than vanilla GCN overall which can evidence that our proposal works better on the entire entities evenly. We assume that this is because of the capability to capture various types of relation patterns. In other words, our model can generate more accurate embeddings for the entities that are connected with specific relation patterns along with general relations. Fourth, the variant Var-4 is close to our performance by being the runner-up in some cases. This is reasonable as it is almost equivalent to QGNN. However, it does not separate the weight matrix to capture the direction information and self-loop information, thus losing some contexts for the graph alignment setting. Finally, the variant Var-5 is worse than our original ComplexGCN, even though it leverages a recent graph embedding process for complex spaces. It could be explained by the fact that it was originally designed for node classification and link prediction only.
6.4. Supervision Level Study
The ability to achieve high performance with a small amount of prealigned entities is a crucial challenge for KG alignment techniques. We trained our model and the other baseline techniques with different ratios of training data to see their degree of performance degradation depending on the amount of the seed alignment. From Figure 5, our model maintains the highest result or the same level of accuracy with all the baselines even under the condition where at least of seed alignments are given. Other datasets are omitted for brevity’s sake because they show similar trends. This result demonstrates that our model can even preserve its performance in real-world practice where labeled data are hard to be accumulated.
7. Related Works
7.1. Embedding-Based KG Completion
In the past few years, extensive embeddings have been performed to handle the incompleteness problem of KGs. The major idea in this field is to effectively encode embeddings of resources of a single KG by capturing the plausibility of the relation triples in KGs so that the knowledge semantics can be expressed in Euclidean space. The generated embeddings are then leveraged to fill out the missing links (link prediction) or to find out which class of unlabeled entities should belong to (node classification). The most renowned approaches [31, 32] are based on the translation assumption which was first proposed by TransE [9]: . The parameters are trained by inducing the score of positive triples to have lower values than those of the negative ones. Despite its simplicity and effectiveness, there remain several problems: (1) it fails to address the cyclic structure and (2) different embeddings are encoded for the same relation when it is symmetry. To mitigate these problems, vector representations where each entry is a complex number have emerged in the field [16, 17]. Unlike vectors of real numbers, complex vectors can harness the intuition behind Euler’s formula where is a value at which the length of 1 bar is rotated from the real axis by radians. Taking advantage of this property, RotatE [16] constrains every relation embedding to have modulus 1 so that the relation embeddings can be deemed as rotation from head entity to tail entity. By bringing the rotation concept, all three relation patterns as well as cyclic structures can be easily captured.
7.2. Conventional KG Entity Alignment Methods
As aforementioned, before the era of representation learning of KGs started, text-based EA approaches were frequently adopted in many domains to enrich KGs with newly aligned entities in different KGs. Text-based EA approaches provide straightforward intuition to align entities in distinctive KGs under the assumption that equivalent entities have the same literal information such as entity name. The primary EA method appeared as the problem suited for relational data, mainly relying on the rich textual information (e.g., entity description) of entities in data. Most representative traditional methods such as PARIS [33], RiMOM-IM [34], and LogMap [35] are tailored for finding matching between entities (instances) or even for relations (also known as schema) and classes as well. To detect alignments, text-based approaches for KGs exclusively employ attribute triples as their input where the source entity is connected to a textual value by an attribute type. In the case of cross-lingual KG matching, the use of machine translation (e.g., Google translate) is imperative to convert non-English-version KG into English for making them easily comparable. It is noteworthy that such conventional approaches are subject to a lack of a sufficient amount of textual information on entities in real-world KGs leading to requiring novel methods to take account into more conducive information for alignment tasks.
7.3. Embedding-Based KG Entity Alignment
Unlike conventional KG alignment methods, embedding-based KG alignment approaches have shown to be beneficial at leveraging multifaceted information and simplifying the inference process to compute matchable probability between every entity in source and target KGs. The core idea is to embed entities or relations into continuous vector space and measure the similarity between each vector point, so-called embedding, by measuring the distance between them with a given particular distance metric. Embedding-based KG alignment approaches can be categorized into two main groups: translation-based models and neighborhood-integration models. Translation-based models [6, 7] follow the concept of TransE [9] to conserve the knowledge logic of the input triples. Over the last few years, the interest in graph neural networks (GNN) has exponentially grown due to their capability of integrating neighborhood information. An entity can be expressed with the consolidations of its neighboring nodes since entities do not have any meaning without connection to other nodes while it can be defined with its connections to its neighbors. Neighborhood-integration models [13, 14, 26, 36–41] utilize GNN methods (e.g., recurrent GNN [42], graph convolutional network (GCN) [11], and graph attention network [12]) to incorporate neighborhood structure to infer alignment likelihood following the assumption that the equivalent entities in KGs tend to have similar neighborhood structure.
7.4. Embeddings in Complex Space
In recent literature, researchers have discovered that the entities encoded in complex or even hypercomplex spaces can have much expressiveness in comparison with the embeddings represented with only real values. Formally, the pointwise representation of entities in complex space consists of a complex number which holds the form of where and are real values and denotes the imaginary unit. Each element in a complex vector follows the aforementioned form which allows the embeddings to have denser representation, hence enabling a variety of computations to operate such as rotation and conjugate [16–18]. Due to such properties, complex embedding learning methods have shown their promising performance which opens much potential to be further developed. ComplEx [17] is the first model to successfully capture symmetric and antisymmetric relation patterns. RotatE [16] further proposes a rotation-based score function to model additional composition patterns. There has been an effort to build a generalized GNN model in quaternion space, the so-called QGNN [19], by employing the Hamilton product in the GNN model. Similar to our effort is magnet [20], which introduces complex spaces for GNNs. However, magnet only focuses on node classification and link prediction; thus, it lacks important information for knowledge graph alignment. To the best of our knowledge, this is a pioneering paper in handling complex embeddings for graph alignment, especially knowledge graphs. Our novelty is a careful combination of a rotation-based embedding augmentation, a new loss function, and a new parametrization for weight matrices.
8. Conclusion and Future Works
This paper proposed a novel framework, namely, ComplexGCN that generates complex embeddings via rotation constraint and the advanced GNN module that performs in complex spaces. The model combines RE and DI modules to complement each other and guarantees the semantic and relation information-preserving embeddings which lead to more accurate alignment results. To this end, ComplexGCN first learns the semantic-aware complex node representations by constraining the triples with rotation assumption. Then, the model incorporates the multiperspective of relation information through the rotation composition operator and gains the expressiveness of the embeddings with Hadamard product-inspired GNN architecture. Extensive experiments conducted on various datasets demonstrate the superiority of our model against the state-of-the-art baseline techniques and the comparison between the variants of our model justifies the design of our model. For future work, we plan to incorporate multimodal data and attribute triple information to further enhance the expressiveness of alignment-oriented embeddings.
Data Availability
The experiments were performed on public datasets (see Section 6.1.1).
Conflicts of Interest
The authors declare that they have no conflicts of interest.