Abstract
A novel simulation method of the brush stroke is proposed by applying force feedback technology to the virtual painting process. The relationship between force and the brush deformation is analyzed, and the spring-mass model is applied to construct the brush model, which can realistically simulate the brush morphological changes according to the force exerted on it. According to the deformation of the brush model at a sampling point, the brush footprint between the brush and the paper is calculated in real time. Then, the brush stroke is obtained by superimposing brush footprints along sampling points, and the dynamic painting of the brush stroke is implemented. The proposed method has been successfully applied to the virtual painting system based on the force feedback technology. In this system, users can implement the painting in real time with a Phantom Desktop haptic device, which can effectively enhance reality to users.
1. Introduction
In virtual painting, many researchers have focused on Japanese calligraphy and painting [1–4] and western painting [5–9]. These methods yield good results. However, these methods are not suited to Chinese calligraphy and painting, which usually consist of typical brush strokes created with various painting materials and techniques to convey the artist’s deep feelings of a painted object. The simulation of brush stroke in Chinese calligraphy and painting has attracted much attention of researchers [10–22].
The brush stroke is defined by the users as a list of position and pressure samples [1]. But the physical model of brush has not been considered, which has impact on the effects of the brush stroke.
Wong and Ip [12] use a parameterized model to simulate the physical process of the brush stroke creation. Yet in the process of stroke generation, the brush tip is in the middle of the painting stroke and fails to produce the biased-tip strokes.
An empirical one-dimensional brush model is proposed in the simulation of the brush stroke [13]. Nonetheless, the effects of the first brush footprint and end brush footprint cannot be simulated.
Mi et al. [15, 16] apply the droplet model in their simulation of the brush stroke. Sun et al. [17] propose a practical 3D brush model in the simulation of the brush stroke. Zhang et al. [18] propose a virtual hairy brush model based on triangular mesh to simulate the brush stroke. Zhang et al. [20] propose a statistic-based method to model the brush footprints between brush bundles and the paper. However, these methods have not referred to the force feedback technology, which can effectively enhance reality to users during the virtual painting process.
The force feedback technology is introduced in [6–8, 21], and their methods are described as follows. The brush in [6, 7] is modeled with a spring-mass particle system skeleton and a subdivision surface. In their feedback force model, the stretch spring is used as brush skeletons to deform subdivision surface, and the force computation is separated from the brush deformation computation, so this approach makes limited use of the haptic feedback device’s ability to recreate an accurate haptic sensation of the virtual brush. Baxter and Lin [8] focus on generating a wider variety of brushes that are found in western painting with a versatile multispine modeling approach. They also employ the bending spring to evaluate the force for each brush spine and use the average as the feedback force. They model the brush dynamics based on energy minimization techniques which are similar to [10], and this method of modeling can simulate small-scale deformation of the brush instead of large-scale bending due to the restriction of constrained energy minimization. Besides, differential and derivative operations are needed for the optimizer, which introduce high overhead. The brush bristle model in [21] is similar to the algorithm in [6, 7]. But they use bending springs instead of stretch springs as the skeletons of the brush in order to simulate the dynamics of the brush. The haptic simulation in [21] is modified from the method in [6, 7], and the calculations of feedback force and the brush deformation are splitted, so this method has the same problem as [6, 7]. For the characteristics of Chinese brushes, certain feature in Chinese calligraphy and painting such as brush flattening and bristle spreading due to the exerted force on the brush is difficult to simulate by those methods, and the automatic on-the-fly generation of the brush stroke has not been studied either.
In this paper, the relationship between force and the brush deformation is analyzed, and the spring-mass model is applied to construct the Chinese brush model based on the force feedback technology. In this model, a virtual spring which is perpendicular to the paper plane and deforms along the normal of the paper plane is used for calculating the exerted force on the brush, and then the deformation of the brush such as the brush flattening and bristle spreading is simulated. According to the deformation of the brush model at a sampling point, the brush footprint between the brush and paper is calculated in real time. Then, along the sampling points, the brush stroke is obtained by superimposing footprints of different sizes and shapes. In the meantime, force information is sent back to the force feedback device to simulate the feeling that user touches a paper with a Chinese brush. Users also can adjust the painting to the desired effects according to the feedback force. The proposed method has been applied to the virtual painting system based on the force feedback technology. In this system, users can paint the brush stroke in real time with a Phantom Desktop haptic device.
2. The Brush Model Based on Force Feedback Technology
In the virtual painting process, an expressive brush model is beneficial to the simulation of the brush stroke. According to the characteristics of the real Chinese brush, we construct the brush model, which includes two components: brush geometry and brush dynamics.
2.1. The Brush Geometry
The geometry is closely related to the dynamics. A well-structured geometry can not only reduce the computational complexity and improve the real time performance, but also simulate the deformations of various brushes. Like some previous models [10, 11], we represent the geometry (Figure 1) in two layers: the skeleton and the surface.
The skeleton handles the general bending of the brush. We represent the skeleton as a connected sequence of line segments (spine segments) that become progressively shorter toward the tip. Since the brush tip is usually much softer than the brush root, it bends much more. In fact, usually only the tip and the belly are used to paint. Therefore, for modeling efficiency, progressively shorter segments are used towards the brush tip so as to dedicate higher resolution to tip. Suppose the skeleton has nodes, , with as the tip node and as the root node. The length of the line segment is denoted by . form arithmetic progression, and the general formula is shown as follows:where is a positive integer and ; is related to the soft and hard degree of the brush: when the same force is exerted on the brush, the softer the brush is, the easier the bristles near the brush tip deform; thus the value of is smaller but, on the contrary, the value of is larger. The value of is given according to experiments. The common difference is determined according to the initial length of the skeleton () and the value of .
When the brush bends, all the spine segments are in the same plane (Figure 2). The inclination angle of brush holder (the angle between the brush holder and the paper plane) is denoted by . The angle between the spine segment and the paper plane is denoted by (). When the brush is unbent, . In the virtual painting, in order to control the outline of the brush stroke when it is painted with the brush, define the cross section which passes node () as the outline controlling plane of the brush (the plane bisects the angle between the two adjacent spine segments and is perpendicular to the plane of spine segments). The deformation of the brush can be simulated by controlling the positions and sizes of the outline controlling planes.
The brush surface is represented as a triangular mesh surface defined by the skeleton and the varying outline controlling planes of the brush. When the brushes are moistened and unbent, the shape of the Chinese brush is similar to a cone. Therefore, the outline controlling planes of the brush are circles along the entire skeleton. We predefine these initial diameters of circles for various types of brushes. In the virtual painting, the brush root connects with the brush holder. Therefore, the diameter of the circle which passes remains unchanged, and the outline controlling plane of the brush transforms itself into ellipses when the pressure and friction are exerted on the brush (Figure 3). This representation is computationally efficient and does not differ much from the real brush deformation.
In Figure 3, is the diameter of the circle; is the major diameter of the ellipse; and is the minor diameter of the ellipse. The mathematical expression of is shown as follows:where is the pressure factor, which is defined as the ratio of the pressure to the maximum output force of the force feedback device. In our system, the maximum output force provided by the Phantom Desktop haptic device is 7.9 N. Therefore, . The value range is ; in order to adjust the sizes of ellipses in different painting conditions (e.g., the painting is implemented using brushes with different soft and hard degree), we set the adjustment factors and which are determined by painting experiments to simulate the most realistic brush deformations; the mathematical expression of the frication influence factor is , and refers to the friction factor.
The minor diameter () of the ellipse is computed according to the conservation of area:
2.2. The Brush Dynamics
The aim of the brush dynamic is to simulate brush flattening and bristle spreading due to the force exerted on the brush during the painting process.
A spring-mass model (Figure 4) is adopted to represent the brush dynamics in order to better describe the relationship between the force and brush deformation. Set a virtual spring between the root node and its projection point on the paper plane (). The spring is perpendicular to the paper plane and deforms along the normal of the paper plane. When the brush just contacts the paper plane and it is unbent, point coincides with the brush tip node , and the initial length of the skeleton is denoted by . The spring moves downward when the pressure is exerted on the brush, and the positions of nodes change, while the length of each spine segment remains unchanged. The feedback pressure () is proportional to the downward displacement of the brush, and the mathematical expression of is shown as follows:
In the expression, is the force feedback factor which is used for controlling the magnitude of . The value of is related to hardware and is determined according to experiments. The unit of is N/mm. is the hardness factor of the brush, and . The larger the value of is, the harder the brush is; thus, the exerted force is larger when the brush moves down the unit displacement. is the downward displacement of the brush at a sampling time and is also the deformation amount of the spring. The unit of is mm.
In the virtual painting, the friction () between the brush and paper is proportional to the pressure, and the mathematical expression is shown as follows:where is the pressure which is determined by (4), and the unit of is N.
3. The Simulation of the Brush Stroke
In the painting process, when the force is exerted on the brush, the brush footprint is formed between the brush and paper. Then, the brush stroke is obtained by superimposing brush footprints along the painting direction.
3.1. The Control of Force to the Brush Stroke
Different effects of the brush stroke are simulated by controlling the magnitude and direction of the force which is exerted on the brush. When the painting direction remains unchanged, the brush footprint varies with different magnitude of force exerted on the brush, which will result in different effects of the brush stroke. In the real painting process of the Chinese calligraphy and painting, the most common painting techniques include Zhongfeng, Pianfeng, and Cefeng. In the virtual painting, define the direction of the brush tip ( in Figure 4) as the bending direction of the brush. When the bending direction is opposite to the painting direction, the effects of the brush stroke with Zhongfeng are simulated. When the bending direction is perpendicular to the painting direction, the effects of the brush stroke with Pianfeng are simulated. When the brush tip is on one side of the painting stroke while the brush holder is on the other side, the effects of the brush stroke with Cefeng are simulated.
3.2. The Generation of the Brush Stroke
In the real painting process, the brush is in contact with the paper surface, which forms the brush footprint of the “raindrop” shape (Figure 5(a)). Similar to [9–11], we suppose that the brush model intersects with the paper plane and consider the orthogonal projection of the penetrating portion onto the paper plane as the brush footprint (Figure 5(b)). Then, the complete brush stroke is obtained by superimposing the footprints along the painting direction.
(a)
(b)
The angle between the bending direction of the brush () and -axis is denoted by , of which the value range is . The coordinate values of the brush tip node are determined by
The coordinate values of node are determined by
The points and () are the endpoints of the minor diameter () of the ellipse which passes node , and the coordinate values of are determined by where the value range of is .
The coordinate values of brush root node are determined by (9), where is the diameter of circle which passes the brush root node ; is the downward displacement of the brush at a sampling time. Hence,
In the plane , the outline controlling curve of the brush is obtained by B-spline fitting according to points and . The line segment which is the projection of the brush skeleton intersects with the outline controlling curve of the brush at the point . is the intersection between the line segments and , and the coordinate values of are determined by the following equation:
The outline controlling plane of the brush which passes node intersects with the paper plane at the line segment . In the outline controlling plane (Figure 6), the length () of the line segment is determined by
The outline of the brush footprint is symmetric, and the axis of symmetry is . Then, the coordinate values of points and are determined by
In the paper plane, the outline of the brush footprint is obtained by B-spline fitting according to points , , , and .
The brush stroke is obtained by superimposing brush footprints along sampling points. In Figure 7, two lines which are parallel to (the painting direction) are tangent to the outline of the brush footprint, and the tangent points are A and B. Define A and B as the effective points of the footprint. Along the painting direction, the brush footprint is divided into the front zone (f) and the back zone (b). Since the footprints cover each other in the real painting process, in order to reduce the computational complexity and improve the real time performance during the virtual painting, the brush stroke zone is composed of three parts: the b zone of the footprint at the first sampling point, the zone formed by sequentially connecting effective points at every sampling point, and the f zone of the footprint at the last sampling point.
4. Simulation Experiment and Analysis
Our method is applied to the virtual painting system based on the force feedback technology. In the system, MS VC 2005 is adopted as our integrated development environment (IDE). Qt framework is used for graphical user interface (GUI). Open Inventor is adopted as the graphical kernel library. The haptics effects are designed with the combination of widely used standard modules provided by the OpenHaptics library from SensAble Technologies Inc. The hardware components include HP xw 8600 workstation for graphic and haptics rendering and a Phantom Desktop device for haptics interaction.
We have developed a novel virtual 3D brush model based on the force feedback technology. With the force feedback users can experience the interaction between Chinese brush and paper more realistically. The position, motion, and force information of the virtual brush can be obtained from the Phantom Desktop haptic device to accomplish the brush stroke simulation, and the painting stroke also can be adjusted by users to the desired effects through haptic device. The schematic diagram of our system is shown in Figure 8 to illustrate how various system components are integrated.
In our system, users can implement the painting with different Chinese brush and paper models in real time, rather than defining sets of parameters to describe the brush footprint at any given instant, which makes the painting process more natural and spontaneous. It means that if a Chinese brush and paper model are selected, the brush footprint information will be taken from the Chinese brush model directly with the given current state of the dynamic painting simulation. This process is similar to that of the real Chinese calligraphy and painting.
The parameters in our paper can be classified into the parameters for the geometry of the brush, the parameters for the dynamic of the brush, and the parameters between the brush and paper. The parameters for the geometry include the length () of the line segment , the numbers of nodes in brush skeleton (), the initial length of the skeleton (), and initial diameters of circles. The parameters for the dynamic of the brush include the hardness factor of the brush () and the adjustment factors ( and in (2)). The friction factor () is the parameter between the brush and paper. The initial length of the skeleton () and initial diameters of circles are predefined for simulating various types of Chinese brush. When other parameters remain unchanged, the deformation of the brush is more realistic with the increase of the numbers of nodes in brush skeleton (), which will increase the computation amount; on the contrary, the computation amount is small. Considering the computational complexity and the real time performance during the virtual painting, the reasonable value range of is 8–13. The value range of reasonable of Chinese brushes is estimated by considering the magnitudes of exerted force on brushes when brushes bend nearly 90°, and the reasonable hardness factors are about 0.3–0.7. With these hardness factors, reasonable , , and are about 0.8–1.4, 1.3–1.8, and 0.6–0.8, respectively. The friction factor () is estimated according to different brushes and papers, and the value range is about 0.2–0.3. Using these values, the forces generated for haptic feedback are also reasonable.
In a simulation experiment, main parameters are shown in Table 1. The brush deformations and footprints under the actions of different pressure are shown in Figure 9. The area of brush footprint increases with higher values of the pressure from Figure 9.
With the same parameters in Table 1, another simulation experiment is implemented, and the effects of the brush strokes with the pressure of different magnitude and painting techniques are shown in Figure 10.
In Figure 10, the bending direction of the brush is denoted by red arrow, and the painting direction is denoted by green arrow.
Using the Phantom Desktop device, users can paint the desired strokes in real time with the preferred painting technique, and some common strokes which are painted in Chinese regular script are shown in Figure 11. Some Chinese calligraphic characters (Chinese regular script) which are created with different painting techniques are shown in Figure 12(b). Compared with the Chinese copyright of related characters (Figure 12(a)), we conclude that some real characteristics in Chinese characters can be simulated by our system.
(a)
(b)
There are two main characteristics that we should consider to select an input device for controlling the virtual 3D brush: the number of input degrees of freedom (DOF) should be as close to the six degrees of freedom of the real Chinese brush as possible; the magnitude and direction of force felt by painters give a useful indication of the brush’s state at each moment during the painting process of Chinese calligraphy and painting; thus the input device should provide the haptic sensation which is similar to the real painting process. Some input devices which are commonly used in virtual painting are listed in Table 2. The Phantom haptic device is able to deliver an arbitrary force to the user under programmatic control compared with the Wacom Intuos Tablet from Table 2, and our system supports both Phantom Desktop device and mouse as the input device. When the brush stroke is painted with mouse, users should input the magnitude of the force through keyboard and control the bending direction and painting direction of the virtual brush through mouse in order to implement the painting process. If the brush stroke is not the desired one from observation, they should change the exerted force on the virtual brush and repeat the above steps. In the whole process, the fun of painting would be lost without real time interaction and force feedback. On the other hand, when the brush stroke is painted with the Phantom Desktop device, the direction and magnitude of the exerted force on brush can be input from the Phantom Desktop device. If the stroke is not the desired one, users can change the exerted force through haptic device, and then the deformation and stroke of the brush are displayed in real time. This force feedback process is similar to the real painting process of Chinese calligraphy and painting.
In order to verify the effectiveness of the haptic feedback, a simulation experiment is implemented by six users that include two art students and four ordinary volunteers. In our system, all the users are able to pick up the haptic stylus and start painting immediately, with little training or detailed instruction. The Chinese character “qie” is painted by these users under the same painting condition with two devices, (I) mouse and keyboard and (II) Phantom Desktop device, and the two devices are randomly used in sequence. The used time of the painting process with the two devices is shown in Figure 13. The used time with the mouse and keyboard is more than double that with Phantom Desktop device from Figure 13.
After the experiment, six users are also asked to use a pen to check which painting manner is more attractive in accomplishing this task, whether it is painted with I or II. The survey results show that five out of six users express their preference for painting with II (Phantom Desktop device) compared to I (mouse and keyboard). From Figure 13 and the survey results, we can conclude that the painting process of Chinese calligraphy and painting with haptic feedback is better than that without haptic sensation. It would be interesting to conduct a more thorough study over a substantially larger group of users to confirm our conclusion as well as to evaluate the effectiveness of various parameters in our system.
Some of the sample paintings created with our system are shown in Figure 14.
5. Conclusion
In this paper, a simulation method of the brush stroke is proposed by applying force feedback technology to the virtual painting process. Firstly, a new brush model is adopted to simulate the brush deformation according to the force exerted on it. Then, different effects of the brush stroke are simulated by controlling the magnitude and direction of the force exerted on the brush. Finally, based on the hardware components, HP xw 8600 workstation and Phantom Desktop device, we establish the virtual painting system based on the force feedback technology, and then different effects of the brush strokes with the pressure of different magnitude and painting techniques are simulated in real time, which can effectively enhance reality to users.
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.
Acknowledgment
This work was supported by the National Natural Science Foundation of China (no. 51175058).