Abstract
Discovering the influences between paintings and artists is very important for automatic art analysis. Lately, this problem has gained more importance since research studies are looking into explanations about the origin and evolution of artistic styles, which is a related problem. This paper proposes to build a multiplex artwork representation based on artistic formal concepts to gain more understanding about the aforementioned problem. We complement and built our approach on the previous notion of Creativity Implication Network. We used the recently proposed MultiRank algorithm to suggest possible explanations of the dynamic of some artistic styles. Our results corroborate some well-known facts about the artists analyzed and give qualitative and quantitative information that show the possibilities and strengths of the proposed framework. We plan to expand our analysis to include more abstract artworks. Ideally, we are going to be able to validate more our results and test how our methodology could be used to generate visual artifacts too.
1. Introduction
Discovering the influences between paintings and painters is still an arduous task. As suggested by Saleh et al. [1], a big part of the problem depends on the fact that determining influence is a subjective matter. When an observer looks at a painting, he can see objects, colors, lines, and shapes, capture design principles, and possibly even recognize the genre or style. Having multiple ways of describing an artwork adds to the complexity when tasked with influence identification and analysis. Saleh et al. [2] affirm that measuring similarity between paintings is fundamental when shedding light on possible influences. The authors have investigated specific artist distance measures to quantify similarity and have suggested influences with some success. Since paintings from a given artist can cover a wide period of time and influences might come from several categories such as formal aspects, historical events, or social context, we know there is still a large space to cover and make improvements in previous ideas [1, 2].
As stated by the authors, some of the descriptions used by observers to speak about a painting might be translated into a computational domain. In particular, characterizations based on object presence have been the path followed by some authors in the past as they solved the aforementioned or similar problems [1–7]. Saleh et al. quoted [1] “Although the meaning of a painting is unique to each artist and is completely subjective, it can somewhat be measured by the symbols and objects in the painting” (pg. 2). Within the related problems to influence analysis, art style and genre classification are two that stand apart. These two problems have occupied an important part of automatic art analysis, both achieving interesting results [7–9]. Although we know that finding the elements that characterize styles is one of the most important tasks of art historians [10] and is still an important research objective, things are starting to move elsewhere. Elgammal et al. quoted [11] “… classifying style by the machine is not what interests art historians. Instead, the important issues are what machine learning may tell us about how the characteristics of style are identified and the patterns or sequence of style changes” (pg. 1). Automatic art analysis research is trying to find quantitative and qualitative explanations as stated by the authors and is one of the goals of our work.
In addition to recognizing influences between artists, trying to identify creativity associated with particular paintings has also been studied. Elgammal et al. [12] proposed a computational framework based on constructing a network between paintings to give some understanding in terms of originality and influence. For the authors, a creative product must be original compared to previous samples and valuable or influential when taking into account future artifacts. The authors suggest that, based on the visual aspect of interest, a similarity function or measure should be used during the process of building the proposed network. The calculated scores, measure creativity along the visual aspect, or dimension associated with the similarity function are used. The authors also comment on the fact that this implies that several networks can be constructed: one for every similarity function. The proposed procedure applies a series of transformations to the network and reduces the analysis of creativity to a variant of a network centrality problem [12].
Network and complex systems theory are making their way into automatic art analysis [13, 14]. Some authors recognize that most complex systems are not simply formed by one or two simple networks [15, 16]. Instead, these systems are better explained with several layers or networks. Multiplex networks, as they are called, are a special type of multilayer networks [17–19]. They are formed by a fixed set of nodes connected by different types of relations or interactions. In general, every one of these interactions gives birth to a distinct layer. Based on the increasing interest in multiplex networks, several algorithms have been suggested to measure the centrality of nodes in multiplexes [20]. Rahmede et al. [20] briefly review some of the most relevant algorithms proposed during the past several years. The authors pay close attention to how previous research solves the problem of understating how centrality in one layer might affect the same measurement in another layer. Finally, the authors used previous ideas that suggest measuring simultaneously the importance of nodes and layers based on random walks [21].
In this paper, we are interested in discovering and understating possible influences based on formal artistic concepts between some abstract art paintings and the implications they might have giving plausible explanations of the origin and evolution of the artistic styles to which these artworks belong. We claim that recognizing novel, interesting, and valuable paintings is the first step towards our objective. As suggested by Kim et al. [10], we firmly believe that identifying important factors of each style is the key ingredient to comprehend the uniqueness of every piece of art and the relationships with other similar pieces while trying to understand their creativity. We propose a multiplex artwork representation in order to achieve our research goal. We believe that, in abstract art, in which most of the samples were found to lack the presence of common day objects, formal aspects of the artwork become more relevant for any analysis. As a byproduct of our proposal, we are going to identify potentially creative, original, and visually valuable pieces that could be further analyzed and also try to identify possible microstyles in particular artists.
To work towards our objective, we are going to execute the following steps which are described in the next paragraphs. First, we suggest expanding some previous works related to the six-step methodology (6SM) to analyze art [22] (see Section 2.1). In contrast to what has been done by some authors using features of convolutional neural networks to gain some understanding and insights about style [3, 5, 7, 10, 11, 23], we propose to use an art-based representation that focuses more on human perceptual features and expand on it [24, 25]. This representation is based on some formal artistic concepts, design principles, and binary relations. The main reason to choose a different direction is related to the fact that, in terms of possible explanations, deep neural features are not very understandable by humans right now [11].
We then propose to apply some clustering to our representations (see Section 2.2). From the results obtained by this procedure, we are going to build several distance matrixes from our categorical and numerical features that are going to be the base for our similarity function. Following the claims of previous authors [1], we hope that our similarity function is better suited to capture additional stylistic characteristics and as a byproduct, identify better influences between paintings. We believe that having the option to include categorical information into the representation gives alternatives to encode information related to time relations between the artworks that is relevant to capture stylistic changes. During this research, we have only scratched the surface of this aspect with the inclusion of shape hierarchies or some simple categorical features. In the future, we plan to include more artistic and contextual material that we believe is key for our main objective.
Based on our similarity function, we suggest building some networks or layers from our multiplex (see Section 2.3). We hope to explore how information in a multiplex can benefit from what has been suggested. We propose to expand and complement previous ideas by Elgammal et al. [12]. We take into account more visual and formal aspects into the similarity measurement and suggest separating or aggregating this information as we see fit. We hypothesize that doing this could give us more plausible explanations as to the evolution of paintings in our dataset. This perspective can be more realistic in terms of explaining the creation of artistic styles or microstyles in which there are several aspects to be considered. For every internal representation, we are going to build an artwork network over which we can gain some insights into the connections between paintings, their influences, and originality.
Using the ideas suggested by the authors, we propose to build several Creativity Implication Networks. We deviate from the authors’ proposed parameters and suggest new ways to fix those values. In particular, we used part of the hierarchical clustering procedure to fix some parameters that in the original paper [12] were global. Some of these Creativity Implication Networks are going to be the building blocks of our multiplex artwork representation. We think that our proposal gives a better approximation to capture a larger quantity of stylistic attributes because it allows us to measure creativity, originality, and influence in a comprehensive way. Elgammal et al. [12] suggest having several similarity functions focused on specific artistic aspects, but do not consider the impact of utilizing all aspects at the same time. We are going to do some experiments and try to support that our hypothesis is plausible based on our results. As far as we know, nobody has tried to make a similar analysis using a multiplex artwork representation before.
Next, we propose to use the MultiRank algorithm [20] (see Section 2.4) to give some possible explanations of the evolution of artworks and styles. Based on the layer ranking suggested by the algorithm, we analyze the results and find correlations with arts’ previously known theoretical facts. Finally, we present the results of our experiments (see Section 3) by providing insights into possible interpretations and future work (see Section 4).
2. Materials and Methods
2.1. Artwork Representation and Knowledge Base Construction
To represent the abstract works of art used in this research, we followed the steps described in Gutiérrez et al. [22]. The procedure basically consists of 6 steps: (a) image segmentation, (b) straight skeleton and centrality measurement, (c) color information, (d) shape classification, (e) design principles and binary relations extraction, and (f) artwork representation. The dataset used for this work consists of art by Mark Rothko, Clyfford Still, and Barnett Newman. In total, we downloaded 288 images from WikiArt [26]. From this total, 158 samples belong to Rothko, 86 to Newman, and 45 to Still. The time span covered by our dataset is between 1934 and 1976.
2.1.1. Image Segmentation
The image segmentation step uses the algorithm proposed by Syu et al. [27] which builds hierarchical image segmentation. The authors suggest a two-phase procedure based on an iterative and contraction process. The first phase uses raw pixels from the original image and groups the pixels into regions. The second phase employs color, size, texture, and intertwining of the regions of the previous phase to build the final segmentation. For this research, we used a maximum of 60 segments along with the other parameters described in [22]. Figures 1 and 2 give examples of the result of the segmentation process we described.
(a)
(b)
(c)
(d)
(a)
(b)
(c)
(d)
2.1.2. Straight Skeleton and Centrality Measurement
The second step builds a planar graph called straight skeleton over every region. The purpose of this graph is to induce a terrain model as described in [22] over which a centrality measure can be calculated. This measurement can define a generalized notion of center of mass that is going to be used to represent every region during the remaining steps. Figure 3 shows the terrain model induced by the straight skeleton of a region and visualizes the center of the Euclidean graph.
2.1.3. Color Information
The third step extracts color information based on Itten’s model [28]. As described in [22], we follow Sartori et al. [29] proposal to build a 180-color palette. We replace every original color with a representative of the color swatch. As a byproduct of this procedure, we obtain 11 groups of colors that are going to be used elsewhere in our procedures. Figure 4 shows the color palette built over the 288 artworks in our dataset. Figure 5 shows the result of switching the color value of every pixel of an artwork with the nearest centroid that makes up the color swatch.
(a)
(b)
2.1.4. Shape Classification
The fourth step is a shape classification process. The goal of this step is to do a clustering procedure to find a reduced number of general shapes that can represent the regions extracted in previous steps. As described elsewhere [22], most of the details of this phase are based on techniques of statistical shape analysis. Every region undergoes a simplification process that selects points from the border of the straight skeleton generated in the second step. Using the Ramer–Douglas–Peucker algorithm, we reduce the original region one point at a time until only three points are left. Figure 6 shows some steps of this simplification. All these approximated representations of the original region are stored for further use.
(a)
(b)
The clustering procedure starts with the 13-point representation of all the regions extracted. Out of the original 288 samples in the dataset used for this research, a total of 2831 regions were extracted. After some empirical experimentation, we found that setting the cluster number parameter to 200 gave good results for our purpose. The two conditions used to establish this number were the following: and , where is the arc-length of the approximated curve, is the arc-length of the original curve, is the area of the approximated curve, and is the area of the original curve. The clustering procedure uses the ideas presented by Vinué et al. [30]. The authors suggest an extension to the original k-means algorithm with the objective of applying the clustering directly over configuration matrixes using Procrustes analysis and the Riemannian distance. The result of the shape classification can have several uses as discussed in Gutiérrez et al. [22]. In particular and for the purposes of this research, we use it as a means of simplifying all the regions to be able to extract the binary relations that will be described in the following paragraphs. Once the 200 clusters were identified and the respective mean shape of each cluster was calculated, the centroids were used to replace the original regions preserving as much information as possible.
2.1.5. Binary Relations and Design Principles
The fifth step, binary relations and design principles extraction, calculates relations on pairs of regions based on measurements over the mean shape bounding box, direction, and aspect ratio as well as some contrast relations based on size and color.
(1) Normalized Area Contrast. The first binary relation between regions that we calculate is that related to the area. Every area region is normalized using the width and height of the original image. We define the normalized area contrast (NAC) for a pair of regions as follows:where is the normalized area over the total area of the original image. This measurement gives us an idea of the difference in size between the pair of regions. This number tends to zero when both regions have approximately the same area and tends to one when one of the regions is large compared to the other.
(2) Aspect Ratio Contrast. The next binary relation that we define is related to the aspect ratio. Using the straight skeleton and the generalized mass center of our regions, we can calculate the shortest path between the two furthest points of the straight skeleton’s outer border as it passes through the center. To calculate this path, we compute the distance between every pair of points of the border and take the maximum. Then, using the interior points of the straight skeleton, we find the path that connects both end points and passes through the center of the graph. Figure 7 shows an example of a region that exhibits the generalized center, the full straight skeleton, and the shortest path between the end points that generate the maximum distance. These two end points are going to be used as the reference points of the Bookstein shape coordinates transformation [31]. After applying this linear transformation that takes the end points that achieve the maximum distance to the coordinates and , respectively, we calculate the shape bounding box. This procedure can be applied to the centroids or every original region or simplified region. We define the aspect ratio contrast (ARC) for a pair of regions as follows:where of the bounding box after applying the Bookstein shape coordinates transformation as just described. This measurement allows us to get an idea of how different in terms of proportion both regions are. If one of the regions is very flat or horizontal and the other one is thin and vertical, this measurement is going to be large.
(3) Orientation Weighted Histogram Contrast. The third binary relation we are going to define is related to orientation contrast. Using the shortest path just described, we implement a way to calculate an orientation based on the ideas presented in [22]. Instead of using 6 bins, we suggest having 13 bins in the histogram. Every bin is 15°, and the range starts at −7.5°. We use every segment of the shortest path and extract the angle of inclination. We calculate the histogram counting the number of elements in every bin, weighted by the distance of the segment normalized by the path’s full distance. Once the histogram is calculated, we can have an approximation to the orientation of the longest path and a general notion of the orientation of the region. We define the orientation weighted histogram contrast (OWHC) for a pair of regions as follows:
If this measurement tends to 1, then both regions are parallel. If on the contrary, the value tends to 0, then both regions are somehow perpendicular.
(4) HUE Color Wheel Contrast. The next binary relation is related to hue contrast. Based on the color obtained after the image segmentation, we convert from the RGB color space to the HSV color space. The maximum possible contrast for the hue of the analyzed color is located at exactly 180 degrees. Based on this fact, we define the hue color wheel contrast (HCWC) aswhere corresponds to the angle in the HSV color space. We do the same procedure for the color palette sample used to switch the region’s original color.
(5) Lightness Color Contrast. The fifth binary relation we are going to define is related to lightness color contrast. Using once again the color conversion from RGB to HSL color space, we take the lightness coordinate and define the lightness color contrast (LCC) as follows:
(6) Color Temperature Contrast. The following binary relation we are going to use is related to the contrast between cold and warm colors. To calculate this measurement, we follow the procedure described by Wang et al. [32]. We define the color temperature contrast (CTC) for a pair of regions as follows:where is calculated as the authors suggest.
(7) Color Weight Contrast. The next binary relation we are going to define is associated with the perceptual contrast between heavy and light colors. To define this measurement, we use the ideas suggested by Wang et al. [32]. The equation to calculate the color weight contrast (CWC) is as follows:
(8) Color Saturation Contrast. The last binary relation we are going to define is related to saturation. Using the conversion from RGB color space to HSL color space, we capture the saturation in this last space. We define the color saturation contrast (CSC) as follows:
2.1.6. Artwork Representation
The last step in our art representation is building the internal vector representation. Out of the final results from the image segmentation, we find those regions that contain other regions inside. We know this is always possible because of the way this hierarchical segmentation was defined. We refer to these segments as simple regions. After analyzing all the segmentations of our dataset, we found that on average there were less than 12 simple regions and the maximum number was 17. Our procedure is flexible enough, and in case more regions are required, everything that is going to be described can be modified.
We construct a complete graph of the first 20 simple regions or less. Figure 8 shows an example of our procedure. The order in which the graph is built depends on the size of the regions. We start with the largest region and end with the smallest. We propose to build several vector representations of every artwork. These vector representations are going to be categorical-numerical, numerical, or only categorical. The first part of our representation is going to capture the global aspects of the sample.
(a)
(b)
We suggest the following three different global subvectors:where “artist,” “year,” and “style” are categorical attributes. “Year” and “style” correspond to the reported creation time and style in the WikiArt site. From 288 artworks in the dataset, all the samples have an attribute “artist” filled in. Five records do not have an established year of creation, so we filled in the values with the “unknown” value. All of the samples have a “style” attribute. “Width” and “height” are numeric and correspond to the original image parameters. After the global aspects, we propose 11 different region-specific information representations. The details of how we construct every representation are in the supplementary materials (see S.2.1.1.). Aspects of area, aspect ratio, the identifier of the cluster to which every region belongs, the coordinates of the straight skeleton center, the major and minor axis of the original region and simplified shape approximation, eccentricity, solidity, perimeter, orientation information, and original color and palette color information are taken into account. We name these region representations from now on as . Finally, we come to the binary relations representation part. For this purpose, we construct 17 different subvectors that take into account all the binary relations previously defined. See details in the Supplementary Material for a description of every representation (S.2.1.2.). We refer to these representations as .
2.2. Representation Clustering
After all the previous steps, our final dataset ended with 166 artworks. Most of the original samples could not be processed because the algorithm to calculate the straight skeleton was not able to finish. Five artworks without year information were taken out to be able to build the final multiplex. Since our internal representations can contain categorical-numerical vectors, we decided to use a variant of the Growing Hierarchical Self-Organizing Map (GHSOM) [33] implemented by Malondkar et al. [34]. We decided to use this algorithm for two reasons. The first one is that besides GHSOM being a dynamic variant of the SOM algorithm that generates a multilevel hierarchy of SOM maps, there is no need to specify any parameter besides the input data. The second reason is that the authors made an extension of the original GHSOM algorithm to work with mixed attributes. The authors make use of the distance hierarchy [35] approach to modify the optimization function of the original GHSOM.
2.2.1. Distance Hierarchy Construction
To be able to use the algorithm suggested by Malondkar et al. [34] we had to build a distance hierarchy for every categorical attribute. For most of the categorical attributes present in our representations, we use a simple distance hierarchy tree consisting of only two levels. In the first level, we only have the root element. In the second element, we place all the different categorical values for the specific attribute and set the weight of every edge as 0.5. This simple distance hierarchy tree has two properties. The first one is that the distance between any pair of leaves of the tree is 1. The second characteristic is that all the categorical values have the same importance. In our experiments, attributes such as “artist,” “year,” or “style” are basically all equal. We intend to build a more complex distance hierarchy tree in the future that considers the relationship between “years” and “style” so that years that are historically related to a specific style are more similar than far away “years”.
For the cluster identifier categorical value, we propose to construct a distance hierarchy based on the result of the Riemannian distance of the clustering procedure. Our first task to achieve this goal is to map the distance matrix of the 200 clusters to a tree. We accomplished that using the neighbor-joining tree estimation method developed by Saitou et al. [36]. After normalizing the distance matrix and applying the algorithm, we validated the error of the approximation running a linear regression. We obtained an . Figure 9 shows the result of the validation. The final distance hierarchy consists of 398 nodes of which 200 leaves correspond to the centroids of the clustering procedure. Figure 10 shows a simplified version of the cluster distance hierarchy.
After having all the distance hierarchies for the categorical attributes defined for our vector representations, we have to establish our measurement of variance. Following Malondkar et al.’s [34] ideas, for the simple categorical attributes, those who have a distance hierarchy tree of two levels, we use the coefficient of unlikability. This coefficient is defined as follows:where , in which is the ith value of the attribute in its domain and is the absolute frequency of for the attribute in and is the set of all n input instances. For the shape cluster distance hierarchy attribute, we propose to use the within-data variance [37] measurement that is defined as follows:wherel is any categorical attribute, and are any input vectors, and corresponds to the distance hierarchy mapping for the specific categorical attribute l. Remembering that this mapping locates a point in the tree, we can give the precise definition for the ith input vector as in [34]:where is the anchor or the leaf symbol or categorical value and is the offset of the point from the root. Finally, equation (12) can be defined aswhere is the offset of the least common point of X and Y. is defined as(1)Either X or Y, if X or Y refers to the same point(2)X if X is an ancestor of Y; this means the X lies on the path from the root to Y(3)The least common ancestor of X and Y
The general distance function between any pair of input vectors is defined aswhere M corresponds to the total number of attributes in the specific representation. More details can be found in Malondkar et al. [34].
2.2.2. GHSOM Results
For the research in this paper, we used the following parameters for applying the GHSOM. The number of epochs used was 10, for the criterion associated with or the criteria to expand a map in any level; we suggest the value , for the criterion associated with or the decision to expand a neuron in any map, and we propose the value . For the decision of , we followed the guidelines proposed by Malondkar et al. [34]. The authors suggest that this value is going to let the algorithm find general clusters within the first levels and does not create maps with a large quantity of neurons. For the decision of , we did some preliminary experiments. Initially using a value of 0.8, on average only 25% of the neurons of the first level were candidates to new iterations. This amount was too small, and we achieved large clusters of neurons in the first level, while for 25% of the neurons we ended up with hierarchies with 2 or 3 levels deep and clusters with between 2 and 20 elements. When we set , we achieved on average that 50% of the neurons in the first level were candidates for new iterations, while the levels of each independent neuron hierarchy were no more than three levels deep.
With additional experiments, we were able to corroborate that with the combination of and we had fewer clusters of 4 or fewer elements. This fact made the run time for the algorithm shorter. Table 1 lists all the internal representations used in this work. All the representations are built out of global, regions, and/or binary relation characteristics. Table 2 shows the results of GHSOM using the parameters described previously and some of the initial and more important results that we are going to use next. Every internal representation has its , the size of the map in level 1, the total number of maps, and the number of levels of the final hierarchy. All of these results are going to be important in the multiplex construction process. Figures 11 and 12 show examples of the internal representation and of the contrast binary relationship between color palette elements present in .
2.3. Network Construction
To build all the networks or layers of multiplex in this research, we propose to start from the distance function defined in equation (15) for the internal representations discussed previously. Even do we have several aggregated representations such as , , and , we are going to concentrate initially in the last one. The reason behind this decision is based on empirical visual evaluations of the clusters in the first level of all these three networks. Since we have the style classification of all the artworks analyzed, we observed a better separation in the style classes in representation . From this point on, we are going to refer to these networks as , , and . We also built two additional networks that take into account only the region’s aspects or the relational characteristics, respectively. These two networks are associated with the internal representations and and are denoted by and . We plan to analyze the relationships between all these five networks in some future works. Finally, we suggest a multiplex composed of 23 layers that correspond to the internal representations for . We are going to name our multiplex and are going to reference each layer using the same indexes of the respective internal representation.
Following Elgammal et al.’s [12] ideas and using the time labels from the artworks in our dataset, we construct our networks as directed graphs where each vertex corresponds to a painting. A relationship between painting and takes place if was created before . This relationship has a positive weight named that corresponds to the similarity between both paintings. To define the similarity between any two paintings in our dataset, we start with the distance matrixes based on our internal representations and generated in the previous GHSOM procedure. We first normalize these distance matrixes. Then we define a similarity matrix for every normalized distance as one minus the normalized distance. We denote by the adjacency matrix for where an entry or 0. Since our time labels describe only the year the painting was created and an artist can produce more than one painting per year, we assume that if a case like that happens, all the paintings by the same artist in that year are related to each other. If an artist painting spans more than one year of production, we assume that all the previous paintings of the same artists and other artists are related. We also assume that a painting is not related to itself; then, for every i and k, .
Elgammal et al. [12] suggest that, to use the adjacency matrixes to build a creativity implication network, we have to interpret the weights in a specific way. Starting with an assignment of a creativity value equal for every painting in the network, the authors propose to interpret an incoming edge from painting to painting with a high weight, showing that painting is very similar to , and therefore, meaning that is not novel. They also suggest that, based on the last fact, the creativity value of should be lowered and, in opposition, the value of should be increased. If the contrary situation happens, the authors suggest that the creativity value of should be increased and that of lowered. Considering the outgoing edges of , a different situation comes into play.
According to the creativity definition proposed by Elgammal et al. [12], for to be creative, it is not enough to be novel, and it also has to be influential. This last fact in our context means that future paintings must imitate . If we have a high weight between and , the creativity value of has to increase and that of should be reduced. On the contrary, a lower weight between and shows that is not influential over . In this last situation, the score of has to go down and the creativity value of has to increase. To formalize the notions of low and high, the authors introduce what they call a balancing function on the graph. They define a value m(i) for every node that basically determines for every edge weight connected to node ; if the weight is over m(i), then it is considered high and the other way around. The linear balancing function used in [12] is defined as follows:
The authors state that this balancing function basically converts weights lower than m(i) to negative values. This fact implies that the more negative the weight of an edge, the more creative the subsequent nodes or paintings and the less influential the node . By contrast, if we have a more positive weight over an edge, then the following node is considered less creative and the origin node must be considered more influential. As suggested by the authors, negative weights provide a solution for low weights, but are problematic when considering the propagation of creativity scores. To solve this last issue, the authors propose to formalize the intuition that a negative edge between a node and should really mean that a positive and equivalent weight between and should exist. Since by construction the original adjacency graph was antisymmetric, this last reversal of some links is not problematic. This last transformation is what Elgammal et al. [12] consider a “Creativity Implication Network” and is denoted in our context by for .
To finally implement the construction of a “Creativity Implication Network,” Elgammal et al. [12] suggest limiting the incoming connections into a painting since more recent paintings will have a higher number of incoming edges, making them highly biased. They propose to use the top K most similar prior paintings. In the experiments conducted by the authors, this parameter is global and fixed. If we limit our similarity function to a specific aspect, this global parameter is going to have a lot of impact in the results of the analysis. If the analysis takes into account paintings from a very wide time span, the top K similar paintings are not going to take into account information about style that in some sense should be crucial to defining the connections of the “Creativity Implication Network”. The authors introduce a “temporal prior” to overcome this situation. They suggest taking into consideration the prior K painting and combining the similarity function with this temporal constraint to reduce the number of incoming edges. The authors also propose to use a temporal neighborhood to locally calculate the parameter m(i) for every node of the graph.
We propose to use the information of the unsupervised GHSOM clustering procedure to give an alternative solution to the global parameter K of the most similar paintings, the temporal prior, and the local balancing function parameter m(i). We know that having a fixed global K parameter treats all the inputs in a uniform manner irrespective of being located in dense or sparse regions. Using the cluster of painting , we could replace the K topmost similar nodes for the cluster’s real members. In this case, we would not be fixing globally this parameter and the GHSOM algorithm will let us achieve a similar result without possibly inducing any bias in our construction procedure. Following Elgammal et al. [12] discussion on the temporal prior, we believe our proposal of using the cluster can give a better approximation to capture more stylistic aspects than rather constraining the incoming connections based on a fixed and global number over the entire graph.
Our approach borrows some ideas from the work described in [38, 39] that use clustering heuristics for network construction methods. Finally and trying to give a more local perspective on the local balancing function, we suggest ranking the distances between all the members of the cluster in the first level and consider including an additional 20 percent of the size of the cluster of similar elements that do not belong to the cluster to define the value m(i). There are two general cases. The first one is that the maximum distance between any pair of elements from the cluster applied to the respective similarity matrix already contains the additional 20 percent of new elements. In this case, the final m(i) value is the last similarity measurement of the member that was added to the cluster. The rationale behind this decision is to include similar elements that were not assigned to the specific cluster after the GHSOM procedure or to take into account more diversity and measure the impact over the “Creativity Implication Network” of similar paintings.
2.4. Creativity Scores and Multiplex Measurements
To complement the ideas presented in the previous section to our multiplex, we propose to use the MultiRank algorithm developed by Rahmede et al. [20]. The aim of this algorithm is to rank nodes and layers in large multiplex networks. The authors suggest that the MultiRank takes into account the full multiplex network structure and uses the information present in nodes and layers. A random walk hopping through links of different layers with different probabilities determined by the centrality of the layers or influences is the main idea of the algorithm proposed by Rahmede et al. [20]. Solving a coupled set of equations that determine the centrality of nodes and the importance of layers is the way the ranking works. The authors state that this algorithm can be applied to weighted and directed multiplexes as is our case.
According to Rahmede et al. [20], the equation that governs the node centrality scores is as follows:where is taken as 0.85, which is usual in the context of the PageRank algorithms, and , , , and are defined as follows:
In equation (18), corresponds to the entries of the adjacency matrix of layer that in our case are the weights and are the influence of the layers for . The second equation coupled with equation (17) gives the ranking of the layers as follows:where indicates a normalization constant, , , and is a parameter. Based on the comments of the authors, when the parameter , nodes with low centrality contribute more than when the parameter value is 1. On the contrary, if , the contribution of central nodes is less than that of the linear case, i.e., when .
3. Results and Discussion
3.1. Creativity Implication Network Scores
Table 3 contains the color key conventions used in all the networks in the figures presented in this research. Figure 13 shows network (based on ). The network layout is ordered by year of creation (starting from 1934 and ending in 1976) and colored by style. The weight of the links is represented in the thickness of the arcs. The illustration has the following seven artistic styles: (1) Color Field Painting (68.07%), (2) Abstract Expressionism (13.86%), (3) Expressionism (7.23%), (4) Surrealism (6.63%), (5) Abstract Art (2.41%), (6) Minimalism (1.2%), and (7) Abstract Expressionism-Color Field Painting (0.6%). Figure 13(a) corresponds to the creativity scores calculated using the method discussed in Section 2.3. We can see the effect of the balancing function especially in some of the first nodes from left to right. Since the color of the edges corresponds to the color of the originating nodes, we see an important number of purple edges pointing towards blue nodes. These paintings correspond to the Expressionism style and were created between 1936 and 1938. These nodes receive a lot of reversed connections from paintings belonging to the Color Field Painting style (purple nodes) because the similarity between them and examples of future Color Field Paintings is not that much.The first interesting fact about Figure 13(a) is that almost all the styles have a node that has a high creative score compared to the rest of the elements in the network belonging to the same style class. These more creative paintings are located at the beginning of the development of the specific styles. We could also see that as time passes, the creative nodes are more creative than previous ones. Following Elgammal et al.’s [12] ideas, we also calculate the creativity score giving different weights to originality and influence. Figure 13(b) uses , to give 90% of weight to influence during the calculation of the creativity scores. We can see that the older paintings have more importance under these circumstances. On the contrary, the more recent nodes appear smaller. Figure 13(c) uses , which gives a lot of importance to originality. As suggested by the authors, the recent paintings increase their creative score. To try to visualize the impact of exterior influences and see the correlation with originality and influence, we did the same visualization as in Figure 13, but to the paintings of Mark Rothko exclusively (based on but taking into account only 91 elements).
(a)
(b)
(c)
Figure 14 has the following five artistic styles: (1) Color Field Painting (65.93%), (2) Abstract Expressionism (13.19%), (3) Surrealism (12.09%), (4) Expressionism (7.69%), and (5) Minimalism (1.1%). Figure 14(a) corresponds to the creativity scores calculated using the same techniques already discussed but applied to the new dataset. Figure 14(b) shows once again the impact of the linear combination between originality and influence. We used the same values for as before. We can visually see the differences between the original creativity scores of the full dataset and those of the dataset based on Rothko’s artworks. Without considering the external influences of Newman and Still’s works, Rothko’s Surrealism artworks (orange nodes) gain a lot of creative value. This can be attributed in part to the small time period of this style in Rothko’s artistic production that lasted only five years. Based on the construction of our networks, having less influence in future paintings gives the previous nodes more creativity.
(a)
(b)
(c)
Additionally, and as is mentioned by Breslin [40], Newman’s influence was clear for Rothko’s surrealism period, because since 1942, Gottlieb, Newman, and Rothko discussed surrealism art exhibitions that took place in New York and incorporated several of the artistic movement’s manifestos into their work. A similar effect happens when we analyze the Abstract Expressionism paintings. As commented by Hess et al. [41], Newman is recognized as one of the major figures in Abstract Expressionism and the influence in Rothko’s artworks since the 1940s is clear. In particular, the authors discuss the impact of Newman’s Onement series for the birth of Color Field Painting style. Figure 15 shows more clearly what has been mentioned. In the illustration, the creativity scores of Rothko’s paintings are compared. The green points correspond to the score calculated over the first dataset and the red points correspond to the score without external influences. We see a lot of changes in the values up to 1950 where both series were stabilized for the period between 1951 and 1956 (red box in Figure 15).
As described previously, the first part of the chart corresponds to the following three artistic styles: Expressionism, Surrealism, and Abstract Expressionism. Even though the creativity scores change considerably, the most creative paintings remain the same with different values. An interesting exception to the fact just stated is Primeval Landscape of 1945 (purple box in Figure 15). Figure 16 shows this painting. The change in creativity score can be explained using the clusters of the GHSOM process. Figure 16(b) illustrates the cluster of this painting calculated using the full dataset and only Rothko’s artworks. In the first case, we can see that there are 13 elements of the cluster that belong to Newman or Still paintings, and roughly 38% of the elements are external samples. We can also see that 7 paintings of Rothko that appeared in the first cluster over the full dataset are not present in the second cluster. From that number, more than half correspond to artworks that belong to Expressionism or Surrealism styles. The samples of Abstract Expressionism remain almost constant independent of the dataset.
(a)
(b)
(c)
Similar explanations can be given to some other paintings that have an important change in scores. We believe we are correct in claiming that the external paintings that are not present in the second clusters are those specifically important and influential in Rothko’s future works and positively impact his creative scores. Examples of these external paintings are the 13 elements that force the reduction of creativity of Primeval Landscape. Another interesting fact is illustrated in Figure 15 (red box). This period corresponds to the development of Color Field Painting. Since the creativity score values are very similar in both datasets, we hypothesize that during this period, Rothko developed his own Color Field Painting style, and it was not affected in important ways by external influences. During this time, we can see the appearance of his signature large blocks of color horizontally placed in the canvas. Lastly, the clusters generated by GHSOM for both datasets are basically the same.
The final remark for Figure 15 is related to the black square paintings. This last period starts around 1964 and goes until the last created painting of Rothko in 1970. The general trend of both series during this time is very similar. If we only focus on the second dataset, we see there were not any variations on Rothko’s Color Field Painting. We only see two paintings that change their creative scores in a relevant way. When we analyze Rothko’s last paintings in the context of the three artists, what we can conclude is that Rothko’s paintings increase their values because Newman and Still start exploring different variations on their own Color Field Painting styles. Rothko’s last paintings are definitely more creative within that context because the three artists stop developing new paintings and were not similar to previous examples. The visualization of the comparison of creativity scores when originality and influence change weights is in the supplementary material section. In particular, Figure S.3.1 shows as expected that based on influence, some of the first artworks developed by Rothko turn out to be more influential if we only take into account the works of the artist. In terms of originality, the last works of Rothko gain more importance but keep a similar trend with relation to the complete dataset. This fact is explained based on the network construction process that quantifies originality in terms of not having similar future paintings.
3.2. Creativity Scores in Multiplex
Figure 17 shows some of the layers of our multiplex (based on for ). Once again, every network layout is ordered by year of creation and colored by style (see Table 3) to be able to compare visually with network . This sample of 9 layers shows the diversity of creativity scores considering particular visual aspects or design principles. We can also see that the weight links have a different importance in every layer. The impact of different balancing functions used to build every layer’s final network can also be characterized. In particular, we can see that Figure 17(a) has more balanced links pointing to the first paintings of the dataset, while Figure 17(g) shows that on that layer almost all the edges of paintings belonging to Color Field Painting point towards the last paintings. The rest of the creativity scores are in the supplementary section (see Figure S.3.2).
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
The next experiment we conducted with the dataset was to visualize the evolution of creativity scores. To have a general insight into the evolution of the full dataset, we split the complete time period between 1934 and 1976 into smaller slices of 6 years each. The last period has more years since only Clyfford Still produced artworks after 1970. The goal behind our analysis is to see the correlations between the creativity scores and the birth and evolution of artistic styles. Even do we have 7 different artistic styles in our dataset, we decided to focus on Color Field Painting. The first reason behind this decision has to do with the large number of samples of art from that respective style. The second reason has to do with having all three artists develop diverse works of art belonging to this style. The last reason has to do with the large time period of development of Color Field Painting. Compared to Abstract Art, Minimalism, or Surrealism, Color Field Painting lasted three or four times more than those artistic styles. Part of the hypothesis we would like to validate is that to gain insights on the birth of a new artistic style, we would have to analyze the creativity scores, especially the originality and influential variants, because to be defined as an artistic style there has to be a novelty and quite some influence. In particular, Color Field Painting has all the ingredients that make it a very good candidate to validate our intuitions and see how the global creativity scores correlate with specific artistic concepts or design principles that can help us understand better and possibly complement the art expert knowledge used to classify and analyze art.
Figure 18 shows the 6 general time subperiod between 1934 and 1976 in which we initially split the dataset. Every consecutive period aggregates the paintings of the following six years. From the network visualization in this figure, we see the variation in creativity scores along the different time periods. Figure 18(b) shows the first time an artwork was categorized as belonging to Color Filed Painting style (purple node—Figure 19(A) shows Barnett Newman’s Moment 1946 painting). In the next time slice, we see the appearance of more paintings belonging to Color Field Painting style. We see a shift in the creativity scores of some Surrealism paintings. This is normal, since we already know that previous samples lose creativity scores as soon as new styles appear. The evolution of the behavior of the scores is better appreciated in Figure 19. In this illustration, we can see the nine most creative Color Field Painting images. The first painting was created in 1946, but it was not until 1949 that its creativity score gained value. This painting appeared once again with a higher score in 1964. This is the only painting that has that behavior. The other creative works gained a higher creativity score but lose it gradually. In general, that is the trend we see in Figure 19. There is also a period of time between the creation of the work and the high score that is not less than three or four years.
(a)
(b)
(c)
(d)
(e)
(f)
Figure 19 also shows us the very stationary period between 1950 and 1956 that was a moment in which Rothko developed his own style. We can see that, during that short period, Newman kept on exploring his vertical and long compositions that is one of the main characteristics of his Color Field Painting style. On the other hand, the most creative work of 1956 let us identify what is going to be Rothko’s distinctive sign. Once again, we have more facts that let us support what theoreticians claim about the importance of Newman for the development of this particular style [40, 41]. Six out of nine of the works were developed by him, Rothko helped with two, and Still only has one painting.
In terms of our multiplex and to try to capture some insights in the origin of Color Field Painting, we generated two variants. The first one was generated only using the paintings of our dataset up to the year 1952 (Figure 18(c)). The second multiplex was built using all the paintings available. Using the definition of creativity that is suggested by Elgammal et al. [12], we are going to focus initially in the influential part. By construction of our networks and as described previously, influence is a crucial factor. In the supplementary material, Figure S.3.2.1 shows that the time series of all the paintings belonging to the Color Field Painting style is quite similar to that of Figure 19. We use this fact to compare our previous measurements to the ones obtained using our two variants of multiplex and different datasets.
Figure 20 shows the results of calculating the MultiRank measures using several parameters. Following what Rahmede et al. [20] comment about the parameter, we know that, for , nodes with low centrality contribute more and on the contrary, for , nodes with low centrality contribute less to the measurements. In terms of the evolution of an artistic style, we can think that might give us some possible explanations of the origin of the artistic style. This hypothesis is based on the fact that, by construction, nodes that do not have a high creativity score are basically those nodes that have more similarity with previous creative nodes or on the contrary are isolated paintings that might be creative but do not have enough or any influence when calculating the scores. To analyze the left part of the graphs presented in Figure 20, we assume that these regions correspond to an exploration using the design principles or main characteristics of the particular style. We believe that influence is quite important for the final establishment of an artistic style. Based on these two hypotheses, we are going to interpret the results of Figure 20.
(a)
(b)
(c)
(d)
The first row of Figure 20 shows the rankings of the paintings from 1934 up to 1952. This time period has 6 years of Color Field Painting artwork productions that is equivalent to 35.9% of the samples. Abstract Expressionism accounts for 29.49%, Expressionism for 15.38%, Surrealism for 14.1%, and finally, Abstract Art with the remaining 5.13%. We see that, across all the parameter variations, the artworks 4, 41, 35, and 14 are always ranked in the top 4 (Figure 21). Artwork number 4 (Figure 21(a)) belongs to Expressionism and is one of the earliest paintings in the dataset. We have to remember that in general, when we give influence more weight, older nodes in the network become more important. Our findings corroborate this fact once again. Painting number 41 belongs to Abstract Expressionism and is also one of the first in Barnett Newman’s production. The last two paintings (35 and 14) belong to Mark Rothko and are classified as Color Field Painting. These artworks are, respectively, from 1948 and 1951. We recall that painting number 14 was also classified as very creative (Figure 19(B)). In terms of the layer importance and following the hypotheses stated previously, we are interested in elite layers and use , to try to gain some insights into the origin of the artistic Color Field Painting style.
(a)
(b)
(c)
(d)
The second row of Figure 20, columns (a) and (b), shows that the most important layers are those based on , , , and . These representations correspond to the following relations or aspects of our representation, respectively: (1) binary lightness color palette contrast characteristics, (2) binary normalized area contrast characteristics, (3) binary saturation contrast characteristics, and (4) binary lightness contrast characteristics. At first, we can see that 3 out of 4 characteristics are related to color. The other characteristic is a binary relationship that compares areas of the regions that make up the main composition of the artwork. We know these characteristics are very general and that almost all artworks irrespective of artistic style are going to have color as one of the most important aspects and composition as the other one. Even do and in particular, for Color Field Painting, these two characteristics are more important since these are the main formal aspects that theoreticians use to classify these paintings [40, 41]. It is also interesting that, in Figure 20(a), three of our characteristics remain constant during the layer ranking. Even do our network layers are highly connected (see supplementary material Figure S.3.2.2 (a)), the diverse density coefficient (see supplementary material Figure S.3.2.2 (b)) of every layer helps the multiplex capture different aspects in terms of influence.
To give some insights in relation to the stabilization of the Color Field Painting style, we are going to use the information in Figure 20 in columns (c and d). In this case and based on the parameters used, we are giving more influence to popular layers. In our context, this means that layers with more active nodes are more important. Focusing in the region associated with , we see quite some differences in both graphs. When we take into account the normalization of the ranking using the total weight of every layer (Figure 20(c)), we see that there is not a lot of variation in the graph. On the contrary, if we do not normalize the centrality scores, we see a lot of layer importance variations. The top-ranked layers do not change that much. Once again, we see that the most important aspects in terms of the stabilization of the styles in the time period analyzed are related to , , , , , , and . We see the appearance of the binary relation of color temperature based on the color palette and some other new design principles that were not present before.
Besides the facts just mentioned, it can be more interesting to see the behavior of Figure 20(d) in the lower layers. In particular, the layers associated with the internal representations , , , , , , and are the ones that frequently shift in importance. We see from the graph that and loses importance very fast. These two aspects are related to color characteristics of regions and shape cluster identifier of regions. It is interesting that when is near 0, these two aspects are quite important. We know from experts’ descriptions that shape and color are important for Color Field Painting and is the main characteristic of this style [40, 41]. We see that these two layers are probably important because there are quite an important number of paintings that share common shapes and colors. As soon as we move towards 1 in the parameter range, binary lightness contrast characteristics gain more importance. We believe the measurements are capturing one of the most important aspects of Rothko’s works of this period that is the color relationship between its large color blocks.
To see the evolution of the characteristics of Color Field Painting, we used our second multiplex with all the paintings in our dataset. Figures 22(a) and 22(b)) give us the most important paintings based on creativity and influence (Figure 23). We see a different result compared with the previous dataset. Now we observe that the four top-ranked nodes are very early paintings of Rothko and Still. These paintings shift in importance from graph to graph, but they remain important along the full parameter range. The painting numbered 24 is once again the most important painting. It seems that influence is the prevalent characteristic of our ranking and that is probably why older paintings rank higher. In terms of the layer importance and focusing on , Figures 22(a) and 22(b) only have layer 15 and 17 in common in the top 5. As was already commented, and are related to binary lightness contrast and saturation contrast characteristics, respectively. These two aspects are fundamental to Color Field Painting artistic style. The other relevant layers are , , , , and . These representations correspond to the following relations or aspects: (1) normalized area-related characteristics, (2) binary normalized area contrast characteristics, (3) binary normalized weighted orientation contrast histogram characteristics, (4) binary color weight contrast characteristics, and (5) binary color temperature color palette contrast characteristics.
(a)
(b)
(c)
(d)
(a)
(b)
(c)
(d)
In contrast to what we found for the previous dataset, this time most of our characteristics are related to compositional aspects like area or orientation. In terms of additional color characteristics, we see that temperature and weight are new important characteristics. We believe this fact can be attributed to Mark Rothko’s late Color Field Painting artworks that are darker and in perceptual terms heavier. Examples of this are given in Figure 24. This fact is also supported by Figures 22(c) and 22(d), in particular looking at , in which we can see that the two most important layers in both cases are and .
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
4. Conclusions
In this paper, we used a hierarchical representation [22] to extract information related to shapes, color, size, orientation, and some binary contrast relationships. All the features that make up this representation try to capture some design principles grounded on a perceptual basis. We presented several internal representations built over 166 artworks from Mark Rothko, Barnett Newman, and Clyfford Still. The paintings in our dataset span a time period between 1934 and 1976 and are classified under 7 artistic styles. Our internal representations were built to try to give information or insights of specific art theory formal aspects present in almost all artworks and in particular in abstract paintings.
Based on Elgammal et al.’s [12] ideas and our internal representations, we built several creativity implications networks. We experimented over some of the main parameters and procedures used by the authors to build these networks. In particular, we used the results of an extension to the traditional GHSOM clustering procedure that works over categorical and numerical attributes, to replace the global parameter K used by Elgammal et al. [12]. In our opinion, this decision frees us from having to decide subjectively over the temporal prior value described by the authors and also let us build a more robust balancing function. We argued that giving a local approach based on the cluster size and elements of every painting let us treat the input space in a nonuniform manner and capture dense regions or sparse in different ways. It is our belief that these facts generate a better approximation to some stylistic aspects that we tried to argue about along our paper.
Additionally, we analyzed the influences of certain artistic external information in the creative process of Mark Rothko’s work. We showed visually and through numerical comparison that some time periods of the artist were more prone to external influences. We gave some possible explanations as to the fluctuations in creativity scores and the impact of these measurements in previous artistic styles in which Mark Rothko had worked. Furthermore, we also showed the support of our findings with some known art theorists claims about relations and influences between Mark Rothko, Newman, and Still. Expanding on these ideas and in the context of creativity in general, our methodology might be used to validate and do further research in physiological creativity (P-creativity) and historical creativity (H-creativity) based on Boden’s ideas [42]. The relationship between these two types of creativities can give us new perspectives to try to understand the evolution of artistic styles in general or to simply grasp ideas as how external influences act against artistic medium and in particular artists. Since the full artistic production is so large, the rankings presented here give an alternative to focus on interesting and particular examples to understand general trends in the art world. We believe it can also help with what Elgammal et al. [12] suggest as being an important characteristic of a creative agent that is to have: “… ability to assess its creativity as well as judge other agents’ creativity” (pg. 2).
We also presented a multiplex artwork representation using our internal representations. We showed the application of the MultiRank algorithm [20], whose aim is to rank nodes and layers in a multiplex. We compared the results obtained of an aggregated network to that of the multiplex and the measurements from both models. In particular, we used the linear combination proposed by Elgammal et al. [12] over originality and influence to contrast our findings. We discussed some of the most creative paintings identified and showed visually the different interpretations if originality guides our measurements or if we use influence instead. We tried to argue the advantage of our representation in identifying relevant nodes belonging to particular artistic styles and the relation that exists between these paintings and the birth, evolution, and decay of these styles. We gave quantitative and qualitative arguments that support how our methodology can provide insights into the general evolutionary trends of Color Field Painting style. We analyzed the relations between the MultiRank results and parameter range based on Rahmede et al.’s [20] comments. We tried to show the relationship between the layer rankings and impact for creativity scores and explanations of formal aspects that can be related to understating particular features from Color Field Painting. Our analyses permit us to further hypothesis what goes on when a specific artist can explore the creative medium he works on.
In future work, we propose to do more experiments with other artistic styles to see if our results can generalize or in what forms they differ if they do. We believe that our hypothesis that microstyles or styles should have specific time periods of high creativity scores followed by more influence in future artwork production can be explored further. We think the evidence we presented allows us to support our hypothesis. Doing more experiments in broader Abstract Art styles should give us the chance to correlate the duration of these periods with some artistic concepts or design principles that can help us understand better and possibly complement the art expert knowledge used to classify and analyze artworks. Achieving this helps us in the broader art understating or analysis that are important for creativity and other areas of knowledge. In conquering our goal of trying to identify new promising microstyles in an artistic domain, we hope our ideas can help us gain progress in that direction. We firmly believe that, in terms of explanation, our multiplex artwork representation can be improved to give even more interesting insights that help us achieve our objective. We also proposed to validate the power of our multiplex representation to separate artwork paintings based on previous stylistic information to give more support to our results. In that process, we propose to include more design principles related to balance, rhythm, and possible aspects associated with composition.
Data Availability
The data used in this research are available from the corresponding principal author upon request.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Supplementary Materials
The supplementary material contains the regions and relations information vector description based on the internal representations discussed in the paper. Additionally, comparison of creativity scores when originality and influence change weights are presented and also the associated time series are presented. These materials extend some of the analyses discussed in the paper. Finally, some important measurements of the multiplex used are presented [43]. (Supplementary Materials)