Abstract
A novel haptic decorating method on the surface of virtual clay model is proposed. The relationship between brush deformation and endured force is researched for the first time by applying the spring-mass model to construct the 3D brush model. Then, the collision detection between virtual hairy brush and virtual clay model is researched based on collision algorithm of weighted average distance. When the hairy brush initially collides with 3D exterior, the tactility is simulated and the interactive virtual painting on the 3D exterior is carried out practically. The 3D brush stroke is formed by superimposing 3D brush footprints along the painting direction and controlling the stress of the brush. The ink quantity in the brush footprint is determined according to the proposed positive correlation between the exerted pressure on brush and outflow ink quantity of the brush. A painting storage method is also presented for storing and displaying 3D stroke painting results. The proposed method has been successfully applied in the 3D virtual painting system based on real-time force feedback technology. With this system, the 3D brush strokes with 3D half-dry and ink diffusion results can be painted with a Phantom Desktop haptic device, which effectively enhance reality to users.
1. Introduction
Product exterior design, of which main research fields include product form design and exterior decoration, plays a more and more important role in the modern product development. The exterior decoration is usually carried out by using computer aided industrial design (CAID). Traditionally, in this process, the texture mapping is applied for transforming 2D graphics onto the 3D object exterior. However, the process of texture mapping is complex and occupies more computer resources, so it is difficult for designers to catch design inspiration. With the development of virtual painting technology, direct painting and decoration on 3D product exterior are introduced. In this paper, the novel haptic decorating method on the surface of virtual clay model is proposed. This method has been realized in the 3D virtual painting system based on the real-time force feedback technology with higher performance.
2. Related Works
Igarashi and Cosgrove [1] proposed an adaptive unwrapping mechanism where their system can dynamically create a tailored UV-mapping for painting polygons during the interactive painting process. Nevertheless, the repeated reprojection of the existing texture gradually degrades the quality of the bitmap, so their system is only suitable for simple paintings with a limited number of strokes.
ArtNova system [2, 3] used a user-centric viewing technique that integrated haptic and visual presentation by taking into account the user’s haptic manipulation in the dynamic determination of the new viewpoint locations. However, a 3D brush model was not constructed in their method that has a negative impact on the reality of the interactive painting.
Fu and Chen [4] presented a haptic painting system which provided an easy way for users to accurately add color on the mesh models. Their painting system provided virtual brushes based on a haptic device with an input of 3 degrees of freedom. Since a real brush has 6 DOFs, their system cannot provide various brush effects.
The artist and designer can use the system of Kim et al. [5–7] to create and refine a 3D multiresolution polygonal mesh. However, a 6 DOFs haptic device was not integrated into their system.
The painting system in [8] for 3D objects provided intuitive user interaction. However, their method failed to simulate real hairy brush features that include brush flattening and bristle spreading produced by exerting real-time force on a hairy brush.
The system of Chen et al. [9] can not only simulate the motion of brush bristle and painting liquid, but also imitate the liquid transfer processes among different presentations. However, their method does not mention force feedback technology, so the sense of reality was reduced during the painting process of virtual interaction.
The brush in [10] is modeled with a spring-mass particle system skeleton and a subdivision surface. In their feedback force model, the stretch springs are used as brush skeletons to deform subdivision surface, and the force computation is separated from the brush deformation computation, so this approach makes limited use of the haptic feedback device’s ability to recreate an accurate haptic sensation of the virtual brush.
In this paper, a new decorating method on the exterior of virtual clay model is proposed based on force feedback technology with six degrees of freedom input device. The relationship between brush deformation and endured force is researched for the first time by applying the spring-mass model to construct the 3D brush model. Then, the collision detection between virtual hairy brush and virtual clay model is researched based on collision algorithm of weighted average distance. When the hairy brush initially collides with 3D exterior, the tactility is simulated and the interactive virtual painting on the 3D exterior is carried out practically. In the virtual painting process, the collision related points can be found out based on the theory of minimum bounding ball, and then it can calculate the average normal vector of the collision point to determine the projection plane. In our 3D brush model, a virtual spring which is perpendicular to the projection plane and deforms along the normal of the projection plane is used for calculating the exerted force on the brush, and then the deformation of the brush such as the brush flattening and bristle spreading on the projection plane can be simulated. According to the deformation of the brush model at a sampling point, the 2D brush footprint between the brush and the projection plane is calculated in real-time. Then, the 3D brush footprint can be obtained by projecting the 2D brush footprint onto the 3D object surface in real-time. At the same time, along the sampling points, the 3D brush stroke is formed by superimposing 3D brush footprints of different sizes and shapes along the painting direction and controlling the stress of the brush. In the meantime, force information is sent back to the force feedback device to simulate the feeling that user touches the surface of 3D object. Users can also adjust the painting to the desired effects according to the feedback force. Therefore, the dynamic painting of the 3D brush with deformation is implemented. In our simulation system, the ink quantity in the brush footprint is determined according to the proposed positive correlation between the exerted pressure on brush and outflow ink quantity of the brush. A painting storage method is also presented for storing and displaying 3D stroke painting results.
3. Force Feedback Model of Hairy Brush
It is necessary to establish a virtual 3D brush model with rich representation in order to realize the haptic decorating of 3D model surface [11, 12]. According to the actual characteristics of the brush, the 3D brush models which include geometric model and mechanical model are constructed. This model can reflect the physical characteristics of the actual brush and meet the requirements of decorating of the virtual 3D model surface as well.
3.1. Geometric Model
The geometry is closely related to the mechanical dynamics. A well-structured geometry can not only reduce the computational complexity and improve the real-time performance, but also simulate the deformations of various brushes. We represent the geometry (Figure 1 [11]) in two layers: the skeleton and the surface.
The skeleton handles the general bending of the brush. We represent the skeleton as a connected sequence of line segments (spine segments) that become progressively shorter toward the tip. Since the brush tip is usually much softer than the brush root, it bends much more. In fact, usually only the tip and the belly are used to paint. Therefore, for modeling efficiency, progressively shorter segments are used towards the brush tip so as to dedicate higher resolution to tip.
Suppose the skeleton has nodes, , with as the tip node and as the root node. The length of the line segment is denoted by . form an arithmetic progression, and the general formula is shown as follows:where stands for the length of the spine between the control node and , refers to the number of segments of the spine, and is related to the soft and hard degree of the brush: when the same force is exerted on the brush, the softer the brush is, the easier the bristles near the brush tip are deformed; thus, the value of is smaller; on the contrary, the value of is larger. The value of is given according to the experiments. The common difference is determined according to the initial length of the skeleton () and the value of .
When the brush bends, all the spine segments are in the same plane (Figure 2 [11]). The inclination angle of brush holder (the angle between the brush holder and the paper plane) is denoted by . The angle between the spine segment and the paper plane (virtual projection plane) is denoted by . When the brush is unbent, . In the virtual painting, in order to control the outline of the brush stroke when it is painted with the brush, we define the cross-section which passes the node as the outline controlling plane of the brush (the plane bisects the angle between the two adjacent spine segments and is perpendicular to the plane of spine segments). The deformation of the brush can be simulated by controlling the positions and sizes of the outline controlling planes.
The brush surface is represented as a triangular mesh surface defined by the skeleton and the varying outline controlling planes of the brush. When the brushes are moistened and unbent, the shape of the brush is similar to a cone. Therefore, the outline controlling planes of the brush are circles along the entire skeleton. When the brush is moved down and deformed under pressure, the spine of the nib begins to be bent and deformed. The outline controlling is deformed from circle to ellipse by pressure and surface friction, as shown in Figure 3 [11]. With the increasing in pressure, the short axis of the elliptical surface is getting shorter and shorter, and the long axis is getting longer and longer. The elliptical surface will become more flat, and the entire brush head is changed from a cone to an elliptical cone. This representation is computationally efficient and does not differ much from the real brush deformation.
When the brush head gets the force of , can be deduced. As the pressure increases, the formula for calculating is as follows:where and are given by empirical values; is the pressure factor, which is defined as the ratio of the pressure to the maxim output force of the force feedback device. In our system, the maximum output force provided by the Phantom Desktop haptic device is 7.9 N. Therefore, , with a range of ; is frictional factor which can be obtained by the formula with a range of . In order to adjust the sizes of ellipses in different painting conditions, we set the adjustment factors and , which are determined by painting experiments to simulate the most realistic brush deformations. The algorithm of this paper assumes that the area of the outline controlling plane remains unchanged in the deformation before and after, so the short axis can be calculated as follows:
3.2. Mechanical Model
The aim of the brush mechanical model is to simulate brush flattening and bristle spreading due to the force exerted on the brush during the painting process.
We use a mass-spring model (Figure 4) to simulate the brush dynamic deformation when real-time force is exerted on the brush. In the process of virtual interactive painting, a compression spring which deforms in the normal plane is placed between and its projection . That is to say, the spring is perpendicular to the paper plane (virtual projection plane) and deforms along the normal of the paper plane (virtual projection plane). When the brush just contacts the paper plane, and it is unbent, the point coincides with the brush tip node , and the initial length of the skeleton is denoted by LE. The spring moves downward when the pressure is exerted on the brush, and the positions of nodes change, while the length of each spine segment remains unchanged.
In the initial condition of painting, the feedback pressure () is related to the downward displacement of the brush. is given in
A proper output force will be transferred to the user for the purpose of simulating; a coefficient () of feedback force is adopted to control the magnitude of force. The (unit N/mm) coefficient is obtained from many painting experiments using the Phantom Desktop haptic device. The value of the brush softness coefficient () is given inwhere is the Elastic Modulus of the brush spring, is the diameter of the brush spring wire, is the middle diameter of the brush spring, and is the effective number of turns of the brush spring. The brush is much softer with the decrease in , so the downward displacement is much larger when the same force is exerted on the brush. (unit mm) refers to the amount of deformation in the spring, which is also the downward displacement of the brush.
Friction can help users to perceive the surface roughness of 3D objects and control the three-dimensional brush. When is the angle between the normal vector of the surface of the 3D model and the normal vector of the projective plane, and its range is , we can compute the friction () which is proportional to (unit N) according to
The friction coefficient () of the brush is given in (7), where is the humidity factor of the brush which is obtained experimentally.
4. The Collision Detection between Virtual Hairy Brush and Virtual Clay Model
To determine the relative location between the hairy brush and 3D product, a contact force in a small tolerance range is introduced in this paper. When the distance between the hairy brush and product exterior of virtual clay model is in the tolerance range, a contact force is transferred to users by Phantom Desktop haptic device and the collision between virtual hairy brush and virtual clay model can be detected.
The distance between the hairy brush and product exterior of virtual clay model is set as . In this paper, the exteriors of hairy brush and virtual clay model are described as triangle mesh models, and then is represented as the weighted average distance between brush tip and triangle mesh vertexes which are closest to . where , , , and are weighted coefficient, the normal vector of product exterior vertexes, coordinate vector of product exterior vertexes, and the vector of brush tip , respectively. . is the maximum of .
When is in the tolerance range, the hairy brush collides with the product exterior of virtual clay model. A contact force is transferred to users by Phantom Desktop haptic device, and the tactility in the real painting is simulated. Then, the button in the stylus of Phantom Desktop haptic device is pressed and the contact force is invalid and the interactive virtual painting on the 3D exterior is begun practically. The coordinate vector of brush tip is denoted as where the weighted average vector is denoted as
5. Virtual Painting on 3D Object Exterior
5.1. Modeling a Footprint Profile of the Virtual Brush
A local mapping method is presented for mapping the 2D brush footprint into the 3D brush footprint on 3D object exterior. The 3D brush stroke is obtained by superimposing the 3D brush footprints along the painting direction.
The brush interacts with the plane when the brush is under pressure. The nib of the brush begins to deform when subjected to force, and it contacts with the plane to form a “raindrops” type of strokes [13]. This paper chooses the orthographic projection of the contact area between the brush and the plane as the brush stroke. In the process of virtual 3D painting, the actual brush strokes are simplified and formed by two symmetrical B-spline curves in order to simulate the footprint profile, as shown in Figure 5. When the brush is subjected to different pressures, the size and shape of the stroke can be changed by controlling the number and position of the control points of the brush strokes.
According to the geometric relation, the line is the intersection of the contour circle and the plane whose center of circle is . The coordinates of are obtained by intersecting the left outline of the brush with the plane in which the stroke is located. Then, the coordinates of the control node of brush tip can be obtained, and the coordinates of the control node of contour central line of the brush can be obtained by the coordinates of , and then the coordinates of the control points of the brush stroke contour can be calculated. The specific algorithms are shown as follows.
The coordinates of point are as follows:where the range of is . The coordinates of the contour control points on the spine of the brush are as follows:
The contour control point of the brush strokes , is the intersection point of the control circle of brush outline and the drawing plane at the control node ; then the coordinates of the point can be calculated bywhere
Since and are symmetrical about the line , the coordinate expressions about are as follows:
Then a B-spline curve is used to form a closed two-dimensional brush footprint by , , , and ; Figure 5(b) is a two-dimensional brush footprint generated by the system.
5.2. The Establishment of Projective Plane
According to the smallest enclosing ball algorithm [14], we compute the bounding sphere of the bent brush according to an effective ball expanding operation until all the vertices of the exterior of the bent brush are in the bounding sphere.
In Figure 6, vertex is an arbitrary vertex of the bent brush. On the exterior of the bent brush, the farthest vertex from vertex is obtained by traversing all the vertices of the exterior of the bent brush [15]. In the same way, the farthest vertex from vertex is obtained. The midpoint and half-length of line segment are denoted by and , respectively. The initial bounding sphere is defined as .
The length of the line segment , which is obtained by connecting an outlier vertex and , is denoted by . is positioned along the line segment at distance from vertex , and the updated sphere is defined as . In the initial sphere, the diameter is perpendicular to the line segment . Vertex and diameter form an isosceles triangle with base , and the radius of the updated sphere is obtained by
Next, the length of line segment () is computed below:
The coordinate values of the updated center of the sphere are calculated according to and the coordinate values of .
Repeat the above steps until the smallest bounding sphere is obtained, which encloses all the vertices of the bent brush. When the distance between the center of the smallest bounding sphere and an object exterior vertex is less than or equal to the radius of the smallest bounding sphere, these object exterior vertices are defined as effective vertices. The normal of the 2D brush footprint plane is defined by the average normal of these effective vertices (Figure 7), and this plane went through the projection point of the brush root node on the object exterior. Then, the 3D brush footprint profile is obtained by mapping the 2D brush footprint onto the 3D object exterior.
5.3. The Formation of 3D Brush Stroke
In general, the ink quantity of the 3D brush footprint at a sampling point is proportional to the pressure exerted on the brush and the current ink quantity of the brush. Suppose that is the initial brush ink quantity, and is the pressure factor of the th sampling point. The 3D brush footprint ink quantity at the first sampling () is determined bywhere is the ink quantity factor, which is used for controlling the ink quantity of the 3D brush footprint under different painting conditions, and given by painting experiments.
The ink quantity of the 3D brush footprint at the th sampling () is determined by
When the brush moves along a certain path, the ink contained in the nib is transferred to the surface of the three-dimensional model from the brush. The ink quantity is converted into color intensity using theory of KM [16], and the 3D stroke is formed on the surface of 3D model. The specific steps are shown in Figure 8.
During the virtual 3D painting, the 3D brush stroke is simulated by superimposing 3D brush footprints along the sampling points (Figure 9).
The methods of a two-dimensional semidry painting and ink diffusion based on the force feedback are also presented in our previous papers [17, 18]. In the painting process of virtual 3D, the preset value and corresponding parameters of rice paper fiber model are determined according to the 3D product exterior property, and then the half-dry stroke and the ink diffusion on 3D product exterior (useful out of shape) are simulated by the mapping of 2D half-dry stroke and ink diffusion, respectively, which support designers to catch design inspiration.
6. Painting Storage Method for Virtual Clay Model
In virtual clay modeling, the compressed voxel model is defined by three orthogonal dexel ray groups, perpendicular to , , and plane, respectively [19–21]. Each dexel ray is located on the intersection between square grids (Figure 10). A list of intersections between dexel rays and 3D object is denoted by nodes in linked list. Each node in linked list stores position (), material (), and normal vector () (Figure 11).
Example data structure of the compressed voxel model is shown in Algorithm 1, where dimension denotes the division space of voxel grid. plane describes the information of the dexels parallel to the axis. This paper extends the structure of the compressed voxel model, and the painting information (color intensities RGB) on virtual 3D product exterior is saved directly in this model (Algorithm 2). For example, RGB (209, 207, and 210) at the intersection () between 3D object and the dexel (, ) describes the color intensities of this point.
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||
7. Experimental Verification
In this paper, the 3D virtual painting system based on the real-time force feedback technology has been developed with Visual Studio 2005, Qt, and Open Inventor. The 3D painting is implemented in this system running on the HP Xw8600 workstation with a 6DOF Phantom Desktop device.
In our simulation system, the spring-mass model is applied to simulate the brush deformation as long as real-time force is exerted on the brush. A two-dimensional brush footprint between the brush and the brush plane is simulated according to the brush’s deformation. Next, the 3D brush footprint is obtained by projecting the 2D brush footprint onto the outer surface of the 3D object in real-time. Finally, the 3D brush stroke is simulated by superimposing 3D brush footprints along the direction of painting. The schematic diagram of the system for haptic decorating is shown in Figure 12; the haptic device provides the brush motion and location information that can compute the force exerted on the brush. Some of the input devices commonly used in haptic decorating are listed in Table 1. The Phantom Desktop (Figure 13) uses the automatic calibration mechanism that the internal virtual model movements are controlled by the motor. The phantom haptic device is able to deliver an arbitrary force to the user under programmatic control compared with the Wacom Intuos Tablet from Table 1. The interactive haptic decorating systems are shown in Figure 14.
In a simulation experiment, using the hairy brush model in Table 2, we apply the corresponding parameters listed in the table to simulate the Chinese brush. When we used different pressures () on the brush, then the brush deformation is shown in Figure 15.
As a simple example, the Chinese character “七上八下” is drawn on the exterior of a bowl (Figure 16).
Some concrete examples of the interactive haptic decorating system created on the surface of the 3D objects are shown in Figure 17.
In another simulation experiment, using the hairy brush model in Table 1, Chinese character “漫步人生” is drawn on the exterior of a bowl (Figure 18(a)). Compared with the painting results on the 2D painting plane in “Moxi” system (Figure 18(b)), the 3D brush stroke with ink diffusion effects is displayed with our method.
(a)
(b)
In order to verify the effectiveness of the haptic feedback, a simulation experiment is implemented by six users which include two art students and four ordinary volunteers. In our system, all the users are able to pick up the haptic stylus and start painting immediately, with little training or detailed instruction. The same Chinese character “漫步人生” is painted by these users under the same painting condition with two devices: (I) mouse and keyboard; (II) Phantom Desktop device, and the two devices are randomly used in sequence. The used time of the painting process with the two devices is shown in Figure 19. The used time with the mouse and keyboard is more than double that with Phantom Desktop device. After the experiment, six users are also asked to use a pen to check which painting manner is more attractive in accomplishing this task, whether it is painted with (I) or (II). The survey results show that five out of six users express their preference for painting with (II) (Phantom Desktop device) compared to (I) (mouse and keyboard). We can conclude that painting with haptic feedback is better than that without haptic sensation.
8. Conclusion
In this paper, a new haptic decorating method on the exterior of virtual clay model is proposed. Firstly, the force feedback model of hairy brush is constructed to simulate the deformation of hairy brush. The 2D brush footprint is simulated according to the brush deformation. Then, a local mapping method is presented for mapping the 2D brush footprint into the 3D brush footprint on 3D object exterior, and the 3D brush stroke is created by superimposing 3D brush footprints. A painting storage method is also presented for storing and displaying 3D stroke painting results. The proposed methods have been applied in the 3D virtual painting system based on the real-time force feedback technology. With this system, 3D brush strokes with 3D half-dry and ink diffusion results can be painted with a Phantom Desktop haptic device, which effectively enhance reality to users.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.
Acknowledgments
This work was supported by the National Natural Science Foundation of China (no. 51175058).