Abstract
Since the 1980s, nonrealistic drawing techniques have become one of the main focuses of research in computer graphics. In recent years, with the rapid development of computer graphics, nonrealistic drawing techniques have become capable of rendering any digital image into an image with an alternative artistic style. As a result, many scholars have focused on the pencil sketch style. Against this backdrop, research into how to create more realistic art drawings has become a hot topic. Although the rapid development of computer technology has led to advances in art drawing techniques, traditional fine art painting methods are not yet able to simulate realistic pencil sketches in a quite realistic way. In other words, the following issues have been noted. For example, they fail to simulate the gradation of shades and densities of real sketches, the direction of the shadows does not conform to the general rules of sketching, and the results produced are not artistic and realistic. After all, unlike realistic graphics, which seeks a photo-like realism, nonrealistic graphics wants to highlight the message of the image with the help of computer simulations. As a result, nonrealistic graphics often ignores unimportant details in the image. On the other hand, nonrealistic graphics can also simulate an artist’s painting to a certain extent, thus producing an artistically effective image. Many nonrealistic drawing methods have been devised to simulate the styles of oil, watercolour, and ink, but not many algorithms have been developed to generate colour sketches. In order to address this topic, this study improves a fine art drawing generation approach based on the line integral convolution techniques. To be specific, this investigation begins by segmenting the colour images through cluster analysis. Two basic colours are then assigned to each region by calculating the colour variability, and the density of each colour is calculated using a two-tone mapping technique. After that, the line integral convolution technique is applied to generate sketch textures for each of the two basic colour layers, and the two layers are fused to produce the coloured textures. At the same time, a neon transform is used to generate the sketch contour lines. Eventually, the contours are blended with the coloured textures to obtain the coloured sketch effect. The experimental results show that the art painting generation method designed in this paper is able to automatically transform from colour images to coloured pencil drawings, thus improving the efficiency of the process as well as meeting the real-time requirements.
1. Introduction
Fine art painting is a formal artistic creation that allows the use of different coloured lines to express the shapes, tones, and contrasting grey effects of things in the real world [1]. In addition to this, fine art drawing can also express ideas, concepts, attitudes, feelings, and even abstract forms of things. As a result, as the basis of all plastic arts, painting can accurately express the inherent properties of an object [2]. It focuses not only on totality and colour but also on the structure and form of things. Fine art painting is often about still life, animals, landscapes, and people [3]. Also, the drawing of the human figure is often of greater artistic interest. Nowadays, with the continuous development of networking, sharing photos through multimedia social networks has become an everyday life [4]. Therefore, more and more people are uploading their digital photos in artistic styles to different social media platforms to share their life stories with their friends. In this context, with the development of computer graphics, the technique of rendering digital images into various artistic styles by computer has become a reality. Also, the advanced development of computer technologies, such as machine learning [5, 6], total life cycle assessment [7, 8], deep learning [9, 10], and numerical modelling [11], has caused significant impacts on people’s daily life.
In the 1970s, computer graphics was born as an important part of the field of computer science. The application of computers for realistic graphics gradually became a major theme in the development of computer graphics [12]. As the development of realistic graphics continued, realistic drawing tended to become more realistic and more relevant to real-world scenarios. In the 1990s, a large number of scholars began to realise that computer graphics could be used to convey information beyond realism [13, 14]. The pursuit of objective realism has therefore been a constant goal of computer graphics since its inception. Classical realism lies in the ability to produce images with a simulated effect and to reproduce the real world to the level of a photograph [15]. Until now, realistic drawing techniques have generally been based on physical modelling. This means that computers are used to simulate the creation of physical phenomena in the natural world, starting with a physical model of the various properties of the scene [16]. This can be achieved by fading, simulating light, and object interaction to produce a realistic result. However, despite the growing sophistication and near-perfection of realistic drawing techniques, they still have their shortcomings. One of the main problems is that the resulting images are too precise and inflexible. After all, realistic images are not conducive to presenting complex information to the user in a way that is easy to understand [17]. In other words, in order to better communicate complicated information to the outside world, some kind of visual abstraction of that information is required. Hand-drawn graphics, however, meet similar requirements very well. In medicine, for instance, people sometimes choose to use hand-drawn illustrations rather than photographs for textual descriptions. This is because small or hidden parts can be more clearly represented by hand. In addition, in the field of construction, there is also a need to express the progress of a design by drawing it with the bare hands [18]. From the point of view of fine art painting, it is the abstraction and reworking of natural scenes rather than their mere reproduction. The purpose of art is to express the author’s intentions, thus forming symbolic motifs. Fine art painting therefore requires the exposure of the spiritual world rather than just a straightforward representation [19]. Realistic painting, on the other hand, seeks to reproduce the work without any reservations and does not reflect this artistic appeal. Before the introduction of the camera, people used to express the richness of the real world by means of drawings and paintings.
Nonrealistic drawing techniques began to emerge in the 1980s. Nonrealistic drawing is the use of techniques from the field of computer vision to simulate and produce artistic effects in a hand-drawn style. As a more abstract image generation technique, nonrealistic drawing techniques are becoming a focus of research in the field of computer graphics. Nonrealistic drawing is a technique that adapts to the content of an image, preserving certain key details in the image [20]. The aim is to use the computer to produce stylised images with artistic effects that look like hand-drawn artwork. In fact, nonrealistic drawing is an abstraction of the model, omitting some details and highlighting others that need to be enhanced. As a result, the designer’s intentions are more clearly reflected in the nonrealistic drawing technique than in the realistic drawing [21]. Hence, nonrealistic drawings are more effective than realistic drawings in the fields of engineering and technical drawings, medicine, and architecture. The main art forms simulated by nonrealistic drawing are drawing, oil painting, watercolour, pen and pencil drawing, ink and wash drawing, and cartoon animation [22, 23]. Drawing simulation is a flourishing and independent branch of nonrealistic drawing, which has received a great deal of attention and research in recent years, both nationally and internationally. Because of its emphasis on artistic expression and subjective sensory awareness, nonrealistic drawing techniques are of great value for theoretical research and application in various fields.
The results that have been produced on fine art painting can be divided into two categories of research ideas. The first category is the generation of pencil sketching effects from the point of view of simulating the constructional principles of matter itself [24]. Specifically, the artist can gain insight by looking at the pencil marks on the drawing paper under the electron microscope. The graphite content of the pencil, the thickness and hardness of the paper, etc., are counted for each pencil stroke, and the interaction between pencil and paper is modelled and considered in order to obtain a realistic pencil sketch [25]. The advantage of this approach is that it is able to simulate realistic pencil strokes to the greatest extent possible. Nevertheless, the disadvantage is that the user has to set detailed and complex parameters for each stroke and blend them together to achieve a realistic fine art drawing. In the field of real art painting, there is a great deal of variation in the brushstrokes. As a result, it is clearly impractical to use this model to simulate a realistic painting.
A large body of research has exploited the ability of line integral convolution to visualise vector fields, combined with black and white noisy images, to implement an image-based algorithm for automatic black and white sketching [26]. Nevertheless, the transition from black and white to colour cannot be achieved by simply overlaying channels. The main reason for this is that the range of colours available in coloured pencil drawings is quite limited and it is often necessary to combine several colours to obtain a particular colour [27]. There are also significant differences in the way that colour sketches are mixed compared to oil or watercolour. To be specific, in oil painting, the artist mixes the colours on a palette before applying them to the canvas [28]. In the case of coloured pencil drawings, one draws directly onto the sketching paper, layer by layer, using different coloured pencils. This means that they can use the overlapping and interspersing of different colour layers to achieve a tonal effect. There are two main forms of colour mixing in pencil sketching, as shown in Figure 1. The first one is superimposition between layers of colour, where the light reflected from the underlying colour can penetrate the upper layer of paint to render the colour [29]. The other one is a juxtaposition of colours, where two colours are placed very closely next to each other [30]. As a result, the reflected colours can be mixed between the different colours.
Nonrealistic drawing means that the images are no longer realistic or photographic, but have a more artistic quality. At the same time, it gives the imagination room to create a wide range of creative possibilities. Nonrealistic drawing is more abstract than realistic drawing. The lack of a hard indicator makes computer simulation possible, and the development of computers, which is bound to simplify life and make art easier, also provides a realistic meaning for nonrealistic drawing. It is well known that there are various requirements for the environment in which art is created. For example, in sketching, the material of the pencil, the greyscale, the hardness, the effect of the paper, etc., are all factors to be considered. This means that it is not easy for professionals to create a fine art painting. It is even more difficult for a nonexpert to create a fine art drawing. The idea of adopting advanced computer technology to facilitate artistic expression for professionals has therefore become a hot topic in the field of image processing and is the starting point for this research.
2. Generation of Fine Art Painting Effects
2.1. Image Segmentation
In the image preprocessing stage, it is necessary to use a number of image segmentation algorithms in order to obtain a significant image segmentation result. Actually, the main purpose of image segmentation, which is the basis of image analysis, is to separate the image objects. In other words, image segmentation divides an image into two parts, the object and the background. The image algorithms used in this study are the Mean Shift algorithm, which is applied for automatic image clustering and segmentation, and the Graph Cut algorithm, which adopts the clustering results of the Mean Shift algorithm as a basis for semiautomatic manual segmentation with the help of user interaction. The Graph Cut algorithm is based on the clustering results of the Mean Shift algorithm and is used for semiautomated manual segmentation with the help of user interaction. Furthermore, this study applies an image processing algorithm to separate the background, skin, and clothing areas of the input image in order to prepare for the next application.
2.1.1. Mean Shift Algorithm
In the field of image processing, the Mean Shift algorithm is a cyclic process in which a single or multiple pixel points are shifted according to a certain offset rule until a target is finally reached and the iteration ends. To be specific, the detailed meaning of Mean Shift algorithm is shown by the following equation:where refers to the location of the mean value in the image, indicates the i-th pixel point in the image, and means that the offset vectors of the i-th sample points in the image are summed with respect to point k and then averaged.
The image feature space is a prerequisite for the application of the Mean Shift algorithm for image segmentation. An image feature space is a way of mapping the input domain of an image to a subset of data by projection. In other words, an image feature space can be applied to describe the characteristics of a neighbourhood of an image by mapping the parameters of the multidimensional space of that neighbourhood to a single pixel point and then describing the characteristics of that neighbourhood based on that pixel point. The feature space is an effective way of extracting the underlying data and can provide the basis for obtaining further high-level information about the image. The detailed process of Mean Shift algorithm can be seen in Figure 2.
In fact, the Mean Shift algorithm is a continuous probability density function that extends the Mean Shift algorithm from two dimensions to a multidimensional feature space. After obtaining a multidimensional feature space for all local regions of the image, the feature space can be discriminatively categorised according to the change in the size of the adjacent neighbourhoods, thus enabling the segmentation and clustering of the image.
2.1.2. Graph Cut Algorithm
The detailed process of Graph Cut algorithm can be seen in Figure 3. First, some of the pixels are interactively specified as objects and backgrounds, resulting in hard constraints that reflect the user’s segmentation intentions. The soft constraints are then combined with information from the image itself, and the image to be segmented is abstracted into a graph-type data structure. Finally, the maximum flow can be obtained by solving for the minimum cut of the graph to perform the segmentation.
In the process of determining the hard constraints for image segmentation, the user can specify certain points in the image as background points. In other words, in the segmented image, the background area must contain the specified background points and the foreground area must contain the specified foreground points. The region and boundary properties of the image can be used as soft constraints on the segmentation. As a result, the combination of hard and soft constraints can form an energy function. By finding the minimum value of the energy function, the globally optimal solution can be guaranteed for the segmentation. The energy function of Graph Cut algorithm is shown by the following equation:where refers to the energy function of Graph Cut algorithm, refers to the coefficient controlling the weight of the area term and the boundary term, indicates the region term, and means the boundary term. Furthermore, the boundary term can be expressed by the following equation:
Therefore, the entire Graph Cut algorithm is illustrated in Figure 4.
2.2. Basic Colour Density Calculation
For a given colour block, the painter will first lay the groundwork in one of the main shades. This is then supplemented by one or two secondary shades to achieve a tonal effect. In accordance with this characteristic, two basic colours are assigned to each segmented area after the image has been divided. Figure 5 shows the six basic colours used in this study.
The basic colours can be assigned interactively by the user or automatically by calculation. In this research, two base colours are automatically assigned to each region, and the process is shown in Figure 6.
2.3. Generation of Noise Images
In a real art drawing, the carbon particles of the refill are scattered randomly with the bumpiness of the paper. To simulate this feature, the pixel values in the image are randomly assigned to black or white with a certain probability. As a result, the resulting image is highly grainy and is also known as a black and white noise map. Based on this principle, noise images can be generated as primary and secondary colours, respectively. It was found that a black and white noise map described by only two values of 0 and 255 was too monotonous. Therefore, the number of random grey values in the noise image can be increased by dividing 0 to 255 into three different ranges using a hierarchical approach. This gives the final pencil sketch a stronger sense of light and dark.
Adaptive noises, sometimes referred to as signal-dependent noises, vary in their noise grey value with the grey value of the input image. Their variance is also related to the grey value of the input image, making adaptive noise inherently more suitable for shadow line simulation in the art painting process. Poisson noise is a common type of adaptive noise, and an image degraded by Poisson noise can be described as follows:where refers to the original image and refers to the Poisson random function.
Therefore, the variance of the Poisson noise can be expressed by the following equation:
The above equation shows that when the parameter is fixed, the grey value of each pixel in the Poisson noise degraded image changes according to the grey value of the original input image. This means that the Poisson noise is related to the original image and is adaptive.
In addition, the multiplicative noise is also a type of adaptive noise and the image degraded by multiplicative noise can be represented by the following equation:where can be different random distribution models.
Then, the variance of the multiplicative noise can be expressed by the following equation:where refers to the mean of and is the variance of .
As a result, the grey value of each pixel in the image degraded by the multiplicative noise also changes according to the change in the grey value of the original input image. This means that the multiplicative noise is related to the original image and is adaptive noise.
In order to determine which of Poisson noise or multiplicative noise is more suitable for application in fine art painting simulations, this study tests both noise models. Specifically, the effect of grey scale values on the variance of these two types of adaptive noise was investigated. Using the same three-colour grey-scale striped map as input, the noise degradation maps were obtained using Poisson noise and multiplicative noise, respectively. The variance of Poisson noise is linearly correlated with the grey scale value, whereas the variance of multiplicative noise is quadratically correlated with the grey scale value, as shown in Figure 7. As a result, the variance of multiplicative noise grows more rapidly, and the greater the variance of the input image degraded by multiplicative noise, the more prominent the noise becomes, which is more conducive to generating clear shadow lines.
2.4. Line Integral Convolution
Line integral convolution can overcome the disadvantages of the point noise method in accurately reflecting the direction of the vector when the direction of the vector changes drastically, while solving the problem of loss of characteristic information in the convolution algorithm. The algorithm also has far-reaching implications for the visualisation of vector fields when dealing with two-dimensional vector fields. It has also been successfully applied to the fields of graphic image processing, computer art, computer vision, and other related fields.
Line integral convolution selects a noisy texture as the input texture, and the output texture value for each pixel is obtained by line integral convolution. The streamline is first generated by integrating the pixel symmetrically along the vector in both positive and negative directions. The input noise values corresponding to all the pixel points on the streamline are then convolved with the convolution kernel of a one-dimensional low-pass filter, and the result is used as the pixel value for the output texture. The points of the input white noise image are not correlated with each other, while the texture is correlated in the direction of the streamlines, thus showing the directional information of the vector field. In a random white noise degraded image, the probability of white noise appearing is related to the grey scale value of the input image. The higher the grey value of the input image, i.e., the brighter it is, the higher the probability of white noise appearing. Conversely, the probability of white noise is lower. As a result, if the brightness of an area of the input image is extremely dark, the probability of white noise will become quite low. The detailed process can be expressed by the following equation:where refers to the normalization parameter, indicates the white noise, and refers to the filter.
Next, the discrete sampling of equation (8) can generate the following equation:where refers to the length of the integration.
3. Conclusion
With the rapid development of the global animation, film, and television entertainment and multimedia data transmission industries, the research on nonrealistic drawing techniques has been driven by the rapid development of these industries. In this context, nonrealistic drawing has gradually become a hot topic of research in computer graphics, computer art, and computer vision. Compared to existing methods, the fine art painting method proposed in this research is different from them in terms of drawing ideas, implementation methods, algorithmic complexity, and interactivity and has the following advantages. First, the method proposed in this study is based on an image transformation that enables automatic conversion from the original image to a colour painting and also allows the user to adjust the painting effect by interactively specifying the base colour. Second, the method simulates both contours and textures, allowing for a comprehensive simulation of the real painting process. In addition, the method allows different colour layers to be overlaid and blended to produce different coloured textures, in keeping with the real painting process. This method of blending different colour layers up and down can be extended to simulate similar styles of painting such as crayons and pastels, as they are blended in the same way as coloured pencils.
The approach proposed in this paper is capable of modelling the colour mixing process for fine art painting. However, the existing method is still at the experimental stage and needs to be further optimised in the following areas. In the future, better results will be achieved by using deep learning algorithms to differentiate between content and context, using different textures, from the perspective of image understanding. Furthermore, although line integral convolution can produce textures that approximate pencil sketches, they are still somewhat more rigid than real hand-drawn effects, so new methods of texture generation need to be investigated.
Data Availability
The labeled dataset used to support the findings of this study is available from the corresponding author upon request.
Conflicts of Interest
The author declares that there are no conflicts of interest.
Acknowledgments
This study was supported by Henan Vocational University of Science and Technology.