Abstract
A novel variable stiffness 3D virtual brush model and haptic decoration technique suitable for the surface of the three-dimensional objects for the automobile industry are introduced based on real-time haptic feedback mechanism using a 6 DOFs input device, and the haptic behavior of an expressive virtual 3D brush with variable stiffness is studied in detail for the first time. First, the intrinsic mechanism between the deformation of real hair brush and the applied external forces (such as the bending moment) is analyzed and studied in detail by introducing a bending spring to express the basic mechanical behavior for the 3D hair brush. Based on this brush model, many important painting features can be simulated, such as the softer brush tip, brush flattening, and bristle spreading. And a useful algorithm (named the weighted-average distance) for dealing with collision checking among the two objects (3D clay and the 3D brush) is presented. As long as the brush head is close to the 3D object, within a tolerance range, the computational tactile sensation force will be emerged, and the interactive painting process is implemented actually on the outer surface of the virtual object. We then calculate the related bounding ball for deformed 3D brush using a fast ball-expanding search algorithm to determine the virtual projection plane. Based on the real-time deformation about the virtual brush head at a sampling point, the 2D painting footprints, which is produced between the brush head and virtual projection painting plane, is calculated and rendered. Next, the 3D painting footprint could be easily produced via mapping the 2D painting footprints onto the surface of the 3D model in real time. Finally, the 3D painting strokes are formed via controlling the exerted force and overlapping the virtual 3D painting footprints with different shape, size, and following the moving direction of the 3D brush. Experiment result shows that the adopted method can effectively enhance reality to users, with high performance.
1. Introduction
Product exterior design occupies an important position at the conceptual design stage of modern industrial products whether from the perspective of enterprises or consumers. Product exterior which includes the shape and surface decoration of the product is generally completed by Computer Aided Industrial Design (CAID). The CAID technology is based on computer graphics (CG) and computer aided design (CAD), which has become an effective way for designers to quickly realize product exterior design and display product function. The CAID can optimize the process of modern industrial design and improves the efficiency of product development. Traditionally, the CAID system uses texture mapping technology [1] to complete the conversion process of 2D graphics onto 3D object exterior, so as to realize the exterior decoration of 3D products. However, the transformation from 2D graphics to 3D object exterior is prone to distortion and aliasing. The entire texture mapping process is complex and occupies more computer resources. Designer’s inspiration plays an important role in product innovation design, but texture mapping technology needs to map the 2D graphic onto the surface of the 3D object, which is not conducive for designers to capture design inspiration. With the development of interactive 3D painting technique, directly painting on the 3D virtual object surfaces is introduced, which can not only overcome the shortcomings of texture mapping technology but also enable designers to fully catch design inspiration. The brush is the tool of virtual painting, and the artist completes the creation process by controlling the brush deformation. Therefore, constructing a reasonable and expressive virtual brush model is very important at the stage of the interactive haptic painting.
At present, many researchers at home and abroad have carried out a lot of related work and made many important progresses about the problem of virtual brush modeling and virtual painting.
ArtNova painting system [2] employed a user-centric viewing technology which integrates visual and haptic perception by considering the haptic manipulation of the user for dynamically determining the new viewpoint location. Nevertheless, the 3D virtual hair brush was not created and introduced in the painting system which could not enhance the authenticity of the interactive virtual painting.
Gregory et al. [3] provided an intuitive 3D interface for interactively editing and painting on a polygonal mesh based on a haptic device. Their system enables users to naturally create complex patterns aided not only by visual feedback but also through their tactile sensation. However, the 6 DOFs force feedback device was not used in their paint system and there is no two-handed interaction in the painting process.
Fu et al [4] proposed a digital painting system that provided a simple way for adding colors onto the 3D object mesh accurately. And their virtual painting system integrated the virtual brush with a haptic feedback device, equipping an input of the 3 degrees of freedom. Since real hair brushes have six degrees of freedom, their painting system could not provide various expressive stroke effects.
In terms of virtual brush modeling, Chu and Tai [5] proposed a brush model based on the outline of brush strokes, which simulated the formation of brush strokes by using Bezier curves connected back and forth.
Laehyun et al. [6] obtained a new brush model on the basis of the original brush model by adding the geometric model of the bristle bundle and its deformation algorithm, and it can simulate the deformation of the brush better.
Saito et al. [7] constructed a brush model with the Bezier curve as the skeleton of the brush head. And it used a quasi-static energy optimization method to drive the Bezier curve to simulate the morphological changes of the brush head.
Baxter [8] can simulate the deformation of real brushes such as bifurcation and plastic deformation of brushes by improving the original brush model and used the energy minimization method to enable a novel geometric representation of the brush head to change. It can generate brushes with different shapes of brush head and realize the simulation of different strokes.
Yin et al [9] proposed a brush model based on the pressure sensitive model in 2005. This model consisted of the pressure-sensing model and contact model, which were used to describe the relationship between pressure sensitivity and stroke width, and the interaction between the brush tip and the paper surface, respectively. It improved the authenticity of the user’s writing with the brush and the fluency of painting complex strokes.
Baxter and Govindaraju [10] proposed a data-driven three-dimensional virtual brush model that stored the measured data of the actual brush deformation in tables in 2010, and the deformation of the virtual brush was simulated dynamically by using the data in the table. This model can effectively simulate complex behavior of the brush and had higher computational efficiency and good numerical stability.
The painting system for 3D objects in [11] presents an intuitive user interaction interface. However, their system cannot simulate real hairy brush properties such as brush flattening and bristle spreading caused via real-time force exerted on the virtual brush.
In 2017, Guo et al. [12] proposed a novel 3D brush model for Chinese painting and calligraphy via real-time force feedback. The relationship between the force exerted on the brush head and the resulting brush deformation has been analyzed, and a virtual spring-mass model is applied to construct a model of the 3D brush. The proposed methods have been applied in a virtual 3D interactive painting system via haptic feedback technology. However, their brush model does not take into account the mechanical properties of the variable stiffness that exists in a real brush.
In this paper, a novel variable stiffness brush model and haptic painting technique at the surface of the 3D objects are proposed for the first time, via real-time haptic feedback technique using a 6 DOFs input device. And the haptic behavior of virtual 3D brush based on variable stiffness is studied in detail. First, the related mechanism between the applied bending moment and the real-time deformation for the 3D virtual brush is studied by employing a bending spring mass to represent the virtual 3D hair brush mechanical model. Many important brush features can be simulated, such as softer brush tip, brush flattening, and bristle spreading, using our variable stiffness brush model. Given the bending moment exerted at the brush root and the spatial coordinates of the node of the brush root provided by the haptic input device, the spatial coordinates of the brush tip node, the rotation angle at any node of the brush skeleton, and the posture of brush-holder can be calculated by constructing the elastic-plastic bending equation of the brush head. Then, collision contact checking among the two objects (here, it refers to the brush and 3D object) is studied by employing a contact checking method named average-weighted distances. As the 3D brush head is close to a virtual three-dimensional object within a tolerance range, the tactile sensation is immediately emulated, and then the 3D haptic painting or drawing on the surface of the virtual three-dimensional object is actually implemented. To achieve accurate 3D tactile rendering, we calculate a related enclosure ball to the deformed hair brush via a novel ball search algorithm, until all the points on the surface of the bent brush are in the bounding sphere. Then, it can calculate the average normal vector of the object surface points in minimum bounding ball to determine the projection plane. The 2D painting footprint of the brush, which is produced among the contact region between the brush head and virtual projective plane is computed according to the deformation of the hair brush model at a sampling point. Then, through projecting the painting footprint of the two-dimensional space onto the three-dimensional model surface in real time, we could obtain real-time 3D painting footprint quickly. Then, 3D brush stroke can be obtained by controlling the exerted force and superimposing different shapes or sizes of three-dimensional footprint of the hair brush following the real-time moved direction of the brush. Meanwhile, virtual force feedback information will be sent back to Phantom Desktop (a haptic feedback device), to mimic the tactile sense. So, by using this mechanism, our users could control the painting result to the satisfied effect via obtaining the information of the force feedback, which can effectively enhance the reality of the virtual 3D painting process.
2. Haptic Model of Virtual 3D Hairy Brush
It is important to construct an expressive virtual 3D brush model to implement intuitive haptic decorating on the surface of the 3D object exterior [13, 14]. We built a 3D brush model with a two-layered structure incorporating geometry and dynamics based on real characteristics of the brush. This brush model can mimic some characteristics of the real physical 3D hair brush and also satisfy the decorating requirements on the surface of the virtual 3D model (such as the clay model).
2.1. Geometric Model of the Brush
The brush geometry model is very closely related with the brush mechanical model. A well-structured geometry model of the could not only improve the real-time performance but also simulate different kinds of brush deformations. Inspired by the method from Chu and Tai [5], we represent the geometry (Figure 1) as a skeleton and a lofting surface.
The brush skeleton mainly controls the general deformation related to the virtual 3D brush. We describe the virtual brush skeleton as a series of line segments (spine segment) connected front and back, which becomes gradually shorter when approaching the tip of the 3D brush because the stiffness of the 3D brush tip is usually more softer than that of the 3D brush root and needs to be bent more easily. In fact, only the brush tip and brush belly are used to implement the painting. So, considering modeling efficiency and real brush features, progressively shorter spine segments towards the 3D brush tip are used and can also prompt higher resolution.
It is assumed that the brush skeleton is divided into n segments by n + 1 nodes, , with as the brush tip node and Pn as the brush root node. The length of the line segment Pi is defined as Li. L1, L2, …, Ln constitute an arithmetic progression, and the calculation formula is described as follows [15–17]:where (positive integer) is less than or equal to , denotes the length of each line segment (spine segment) from the node to the node , n denotes the total number of spine segments, and the value of L1 is related to the hardness of the brush head. For the same exerted force which is applied at the virtual brush head, the more softer the brush head is, the more easier to be deformed the virtual brush bristle close to the tip of the virtual brush is. Thus, the more smaller the L1 is, vice versa. The value of L1 is obtained by many experiments. And the general difference is controlled by the initial total length (Lspine) of the brush skeleton and the length of L1.
The whole spine is located in the same virtual plane (seen in Figure 2) when the brush is bent. The inclined angle of the virtual brush-holder (named the angle formed by the virtual brush-holder and the virtual painting plane) is denoted as . The angle between the spine segment and the paper plane (also named the virtual projective plane) is denoted as . If the brush is unbent, . Without considering the distortion of the brush head in the process of the real-time painting, all the control nodes () of the center skeleton of the brush which is bent are located in the same space plane that is denoted as A0. To restrain the deformation of the virtual brush, a cross section is defined as the brush outline controlling plane, which is perpendicular to the space plane A0, passes through the control node Pi, and bisects the angle which is formed through the adjacent two spine segments. And the virtual brush surface is expressed abstractly as the triangular mesh model which is formed through a skin defined via the center skeleton of the brush and a series of the outline-controlling plane from top to bottom. The shape of the virtual hair brush is expressed as an inverted cone when the virtual hair brush is not exerted by external forces, so the outline-controlling planes are initially a series of circles with different diameters along the brush skeleton.
The outline controlling plane changes from a circle to an ellipse in the virtual interactive painting process. The change of the position of control nodes cause the deformation of the center skeleton, and each outline controlling plane will also change the position and shape accordingly, thereby to produce the deformation of the triangular mesh surface of the brush.
2.2. Mechanical Model of the Brush
We employ a variable stiffness mass spring [17] (Figure 3) to mimic the brush head’s dynamic deformation, bristle spreading, and the brush flattening when real-time force (or bending moment) is applied at the virtual brush.
(a)
(b)
In the interactive painting process using our virtual brush, a bending spring, that is deformed in the spatial plane, is set to place among the node Pm and the contact point . That is to say that the bending spring is inclined or perpendicular to the virtual paper plane (the virtual projective plane) and is deformed in that virtual spatial plane, formed by the spring and the projection of spring on the paper plane (along the normal direction of virtual paper plane). And when the virtual hair brush head just contacts the virtual paper plane, the brush head is not bent. However, the variable stiffness spring begins to bend elastically and plastically by applying bending moment at the brush holder. Thus, the position of each brush spine node will change while the real-time length of every spine segment will still remain as unchanged constant.
Considering real-shape characteristics of the actual round brush, the brush is in contact with the paper under the action of external moment after dipping in ink when the artists use the brush to implement the painting and calligraphy creation, and the bending deformation of the brush head is occurred. Due to the artist’s continuous movement action applied to the brush, various brush strokes are formed on the paper or external surface of the actual object. Based on the abovementioned analysis, we can conclude that the mechanical model of the brush can be described by a bending spring or a bending beam with variable cross section and stiffness. And the spring will undergo an elastic bending deformation under the combined action of external moment and paper reaction force. Therefore, it is necessary to study the complex mechanical behavior of the brush head in detail.
2.2.1. Basic Equation
First, we can abstract the bending spring into a rectangular variable stiffness beam, as shown in Figure 4.
(a)
(b)
The width of the cross section of the beam is , and the height varies in the form of an arbitrary function along the axis of the beam. The material of the beam is an isotropic and ideal elastoplastic material, whose properties for tensile and compressive are the same. The cross-section height and bending stiffness of the beam can be expressed as follows [18]:where and are constants representing the reference values of the section height and elastic modulus of materials, respectively. is the reference value of inertia moment of the beam section. and represent the variation of elastic modulus and cross-section height along the axis of the beam, respectively. It is further assumed that the yield strength of the material varies arbitrarily along the axis of the beam as follows:
When the variable stiffness beam is in the state of elastic deformation, the deflection of any point on the beam can be expressed by , then the approximate differential equation of the deflection curve can be expressed as follows:where is the cross-section moment. Integrating the abovementioned equation twice can obtainwhere and are the integral constants determined by the boundary conditions at both ends of the beam. If , , and the section moment are some special functions, the analytical solutions of the bending elastic deformation of the beam can be obtained by solving the abovementioned equation. For other complex cases, the numerical methods can be used to solve the equation.
With the increase of the external force on the beam, the bending moment of the beam section will be gradually increased. The beam will enter the state of elastic limit when the bending moment of a certain section is equal to elastic limit of section bending moment. The elastic ultimate moment can be obtained as follows:
2.2.2. The Solution of the Specific Problems
In this paper, we abstract the virtual bending spring into a rectangular variable stiffness beam with exponential change of elastic modulus and linear change of section height.
It is assumed that the elastic modulus of the material is varied exponentially along the axis of the beam, and the cross-section height of the beam is varied linearly as follows:
For the brush mechanical model in this paper, we can abstract its mechanical model as a cantilever beam fixed at the left end according to the actual characteristics of calligraphy and painting. Ignoring the self-weight of the beam, when a concentrated load P is applied to the free end of the beam with exponential change of elastic modulus and linear change of section height, the deflection of the virtual beam can be specifically simplified by substituting equations (13) and (14) into equation (5) and using direct integration as follows:wherewhere represents the exponential integral function.
2.2.3. Force Feedback Analysis
Based on the abovementioned analysis of dynamic deformation of the brush skeleton (variable stiffness beam), we can establish a basic two-dimensional query table about the relation between the bending deformation and real-time force exerted on the brush to achieve fast force feedback calculation, as shown in Table 1.
In Table 1, M denotes the bending moment exerted on the brush holder; Pmb denotes the basic central coordinates of the brush holder, whose initialization value is set as (0, 0, 0); LE denotes the length of the brush head; P0b denotes the basic coordinates of the brush tip with respect to Pmb; θb denotes the relative rotation angle at the brush tip with respect to the brush holder; Pib denotes the basic coordinates of the ith brush skeleton node with respect to Pmb; θib denotes the relative rotation angle at the ith brush skeleton node with respect to the brush holder; mi denotes the bending moment exerted on the ith brush skeleton node Pi. M, Pmb, and LE are provided by the Phantom Desktop input device, and the rest parameters are obtained by our algorithm.
Many important parameters can be derived based on the abovementioned basic query table. The basic principles are named Fixed Shape Method. The Fixed Shape Method can be divided into two steps. First, considering the actual painting process, the basic shape of the bending deformation for the brush skeleton is fixed when given the bending moment M, the actual central coordinates of brush holder Pm, and the length of brush head LE, as shown in Figure 5.
Then, the coordinates of the basic virtual brush tip Pm0 () is fixed after producing deformation of the brush skeleton and can be determined by using the basic query table. Second, in order to obtain the actual coordinates of brush tip P0, the basic shape of the bending deformation for the brush skeleton will rotate a clockwise angle with as the center of rotation (Figure 6), then the basic virtual brush tip Pm0 () is just moved to the actual brush tip P0 () which is located in the x-axis (painting plane). Based on the knowledge of geometry shown in Figure 6, we can obtain
Therefore, . The posture angle of the brush holder is inferred as , which is obtained as follows:
The absolute rotation angle θ at the brush tip with respect to the vertical line (the normal vector of the painting plane) (shown in Figure 5) can be obtained as follows:
Similarly, both Pi which denotes the absolute coordinates of the ith brush skeleton node and θi which denotes the absolute rotation angle of the ith brush skeleton node with respect to the vertical line (the normal vector of the painting plane) are easy to be obtained.
The reaction force Fre of the painting plane exerted on the brush tip (Figure 6) can be obtained as follows:where is the adjustment factor for reaction force with a range of , which is obtained by many painting experiments; denotes the feedback force.
Friction force could enable the user to feel the roughness on the surface of the virtual 3D object so that the 3D brush can be stably controlled. And the friction can be calculated as follows:where is the angle which is formed by the normal of the virtual projective plane (painting plane) and the normal vector of the 3D object surface, whose range is ; is the friction coefficient which is obtained by actual experiments.
2.2.4. Brush Surface Deformation
The brush tip begins to contact with the drawing plane to form a brush stroke under the action of pressure in the actual painting process, and the brush head was gradually flattened from the brush root to brush tip. Thus, the shape of each outline controlling plane of the brush is changed from the standard circle to the variable ellipse due to the exerted pressure and real-time friction of the drawing plane, as shown in Figure 7 [14]. With the increasing of pressure exerted on the brush, the short axis for the ellipse is becoming shorter and shorter, while the long axis for the ellipse is becoming more longer. Thus, the entire ellipse is becoming more flatter.
When the brush is not subjected to the external force, the can be obtained. With the gradual increase of the external force exerted on the brush, the can be calculated as follows:where is the bending moment at the section of node ,· and are adjustment factors of ellipses determined via actual painting experiment to mimic the very realistic bending deformation of the hair brush in different painting conditions, and is the pressure factor that is expressed as the ratio about the exerted pressure F (reaction force on the paper plane) to actual maximum output force for the haptic feedback device. In fact, the maximum output force which is provided by our Phantom Desktop haptic device is 7.9 N in our painting system, so , whose range is in . We assume that the real-time area of the outline-controlling surface remains as unchanged before and after the bending deformation; thus, the short axis could be obtained by the following equation:
3. Collision Detection between the 3D Brush and the 3D Object
In this paper, we implement collision detection between the brush head and the object surface based on an algorithm of the OBB bounding box [19]. In order to confirm the interrelated position between the brush head and the 3D model (virtual clay), our system imports a contact distance which floats within a small tolerance. is represented as the distance from the brush head to the surface of the virtual clay model. If is within a stated tolerance limits, a small tactile force (0.1 N) will be generated and transferred to users via the Phantom Desktop device, thus the collision between the brush and 3D object can be perceived quickly. Our system presents the surface of the brush head and the virtual clay model in the form of several triangular meshes, thus is also defined as the weighted average distance from the brush head tip P0 to a set of several triangular mesh vertices of the surfaces of the virtual clay model which are closest to P0, and whose amount is usually set to . So, can be calculated quantitatively as follows:where denotes weighted coefficient, denotes the normal vector of 3d object exterior, denotes the coordinate vector of triangle mesh vertexes on clay exterior, and is the coordinate vector of brush tip P0·, is the maximum value of .
When is in the given tolerance range, the collision among the 3D brush and the 3D object is felt by users. Then, the button on the stylus of the Phantom Desktop device is pressed down and contact virtual force is set to be zero; then, the brush tip control node P0 is constrained to the surface of the 3D object to carry out subsequent simulation of haptic painting. The coordinate vector of the virtual brush tip P0 is updated as below:where the weighted average vector can be expressed as below:
4. Virtual 3D Painting on the Surface of the 3D Model
In our system, a local mapping method is proposed to map the 2D brush footprint onto the 3D model exterior in real-time. Thus, the 3D brush stroke is obtained via superimposing the 3D brush footprint along the direction of the painting. Then, a Kubelka Munk-based algorithm is used to render the 3D brushstroke in real time.
4.1. Formation of a 2D Footprint of the Virtual Brush
The brush head begins to produce deformation under the action of external force (bending moment) and gets in touch to the painting plane, and then to form a “raindrop” [13] type of footprint. We consider to choose the orthographic projection for the real-time contacted zone, formed among the 3D brush head and painting plane, to describe the virtual brush footprint. In the procedure of the virtual painting, it is assumed that the actual footprint profile are reduced to be formed by two symmetrical B-spline curves, as shown in Figure 8. Thus, the size and shape for the footprint profile could be varied through adjusting the position and the numbers of the related key point on the profile of the virtual brush footprint as a different force (moment) is exerted onto the brush.
(a)
(b)
Considering geometric relations, the line is the intersection between outline control circle and the plane (with the center of the circle ). The coordinates of the point can be obtained via intersecting the left outline of the hair brush with the plane that the brush stroke is located in. Hence, the coordinates for the control node of the hair brush tip can be calculated easily, and the coordinates for the control node of the brush outline central line can be obtained via the coordinates of , thus the coordinates of control points for the hair brush stroke outline can be easily obtained. The specific and detailed algorithms are described as follows. First, the coordinates of are as follows:where the range of is . And the coordinates of outline control point can be calculated as follows:
Hence, the outline control points for the hair brush stroke and are the intersection points between the control circle of the hair brush surface and the painting plane at node ; thus, the coordinates for point can be obtained as follows:where and are symmetrical about ; therefore, the coordinate formulas for can be expressed as follows:
Then, the B-spline fitting method is applied to produce a 2D brush footprint via , , , and . A 2D brush footprint generated by our system is shown in Figure 8(b).
4.2. The Construction of Projective Plane
Illuminated by the smallest bounding sphere theory [14, 15], all the vertices on the surface of the curved brush are included in the bounding sphere through an effective ball expansion operation, and the bounding sphere of the curved brush can be obtained ultimately. In Figure 9, vertex A is an arbitrary vertex on the surface of the curved brush, and the vertex, farthest from vertex A, is denoted as B, which can be acquired by traversing the total vertices on the surface of the curved brush [20, 21]. Similarly, the farthest point C, away from point B can be acquired. The length and midpoint of the segment BC are represented as 2r0 and D0, separately. The incipient bounding sphere is then denoted as (D0, r0).
The vertex E is a point on the deformed brush surface outside the initial bounding sphere (D0, r0), and the length of segment D0E is denoted as l1. D1 is a point on the line segment D0E, whose distance from vertex E is denoted as r1. The newly obtained bounding sphere is defined as (D1, r1). Line segment FG is the diameter of the initial ball, which is perpendicular to line D0E. Then, the isosceles triangle ∆EFG, whose base is 2r0, is obtained through vertex E and the diameter FG. The radius r1 of the bounding sphere (D1, r1) is calculated by
Thus, the length for the line segment D0D1 (ld1) is calculated below:
The coordinates of D1 for the center of sphere are calculated by ld1 and the coordinates of D0. The smallest enclosing sphere (Ds, rs) for the bent brush head can be obtained by repeating the abovementioned iterative steps until all the vertices, that are on the bending brush surface, are all at the enclosing sphere. A vertex on the deformed brush surface, whose distance from the center of the smallest enclosing sphere (Ds, rs) is less than or equal to the radius rs, is defined as a valid vertex. And the average normal vector about these effective vertices is defined as the normal vector of the projective plane (also named the virtual 2D brush footprint plane). In Figure 10, the point on the surface of the 3D object is the projection point of the root node , and the projective plane pass through the point . Thus, the projective plane can be determined by its normal and . The 3D brush footprint profile is obtained by mapping the 2D brush footprint in the projective plane onto the 3D object exterior according to the normal vector of each effective vertex.
4.3. The Construction for Virtual 3D Brush Stroke on the 3D Object Surface
In the process of virtual 3D painting, the ink is transferred between the brush and the 3D object surface and filled into the 3D brush footprints on the 3D object surface in real time.
In general, the ink quantity of the 3D brush footprint at a sampling point is proportional to the bending moment M which is acted at the virtual brush and is also related to the current ink quantity contained in the moving brush head. It is assumed that the variable Z denotes the initial ink quantity contained in the 3D brush head, and denotes the bending moment factor for the ith real-time sampling point. Thus, the ink quantity of the 3D virtual brush footprint for the first real-time sampling point (Q1) can be calculated as below:where η is the ink quantity factor controlling the ink quantity contained in the virtual 3D hair brush footprint for different real-time painting cases, which is obtained via many experiments.
The ink quantity for the virtual 3D footprint at the ith sampling point (Qi) is calculated below:
The ink, which is contained in the virtual brush head is transferred onto the virtual 3D model surface, as the brush moves based on an arbitrary path trajectory. Then, the real-time ink quantity is translated into the value of color intensity by applying the KM theory (a theory of color optics) [14], then the 3D footprint is formed on the surface of 3D object. The specific principles are shown in Figure 11.
(a)
(b)
(c)
The 3D hair brush stroke is formed via real-time superimposing 3D footprint along the direction of the sampling point during the virtual 3D painting (Figure 12).
4.4. Brush Bifurcation Modeling
Brush bifurcation is also called bristle spreading of the brush head. The bifurcation of the brush head is a peculiar phenomenon when using real brush to create works of art. Based on practical painting experience, we observe that the bifurcation of the brush head is related to the degree of bending deformation and real-time ink quantity of the brush head. The tip of the brush head begins to exhibit a phenomenon of bifurcation when the bending deformation and real-time ink quantity of the brush head reach a certain threshold value. The local skeleton at the tip of the brush head will be differentiated from one skeleton into three subskeletons in our brush bifurcation model, and the length of the local skeleton accounts for one-third of the length of the total skeleton. Each subskeleton is further divided into several small segments by spine nodes similar to the brush modeling in Section 3, as shown in Figure 13. It should be emphasized that the bending deformation of the brush head is positively related to the bending moment M exerted on the brush holder. The threshold, which is used to describe a brush head that begins to emerge bifurcation, can be expressed by SP. SP can be calculated by the following formula:where is the bifurcation factor which is obtained by many painting experiments.
5. Experimental Verification
We use Visual Studio 2005 as the integrated development environment, based on Qt graphical user interface and Openinventor 3D geometric display engine, combined with our own algorithm, to independently develop a 3D virtual painting system, which is based on real-time force feedback technology. The painting system includes a six-DOF Phantom Desktop input device. The system was successfully run on HP XW8600 workstation and perfectly realized 3D virtual painting through the six-DOF input device.
Figure 14 shows a schematic diagram of the system of tactile decoration. The motion and position information of the brush to calculate the force applied to the brush is provided by the haptic device. Table 2 lists some commonly used input devices for tactile decoration.
The Phantom Desktop (Figure 15) uses an automatic calibration mechanism and controls the virtual model motion inside the Phantom Desktop via a motor. Compared to the Wacom Intuos tablet in Table 2, the phantom haptic device provides sense of resistance to the user under grammatical control. Figure 16 shows the interactive haptic decoration system.
In a virtual experiment about the mechanical behavior of the 3D Brush described in Table 3, we use the related 3D brush parameters to mimic the real brush behavior. When the different pressure (Fp) is exerted on the virtual brush, the deformations of the virtual brush are shown in Figure 17, respectively.
As an example of Chinese Calligraphy, the characters “七上八下” are painted on the inner surface of a porcelain bowl (Figure 18).
A vivid example for the virtual 3D haptic decorating system, which is painted at the surface of a virtual 3D model (plate) is as shown in Figure 19.
In another 3D painting experiment that apply our novel virtual 3D brush described in Table 2, the black Chinese characters “漫步人生” is drawn in a porcelain bowl (shown in Figure 20(a)). Contrasting with the common 2D painting such as “Moxi” painting system (seen in Figure 20(b)), the virtual 3D strokes using our novel method are shown in Figure 20(a), with the effect of the ink-water diffusion. Besides, the bifurcation effects of the brush head are shown in Figure 21 using our method.
(a)
(b)
Next, we invited eight volunteers to use our system for painting simulation experiments to verify the effectiveness of tactile feedback through the effects of their paintings. Three of the eight volunteers were art students with painting experience, and the other four were ordinary students who have no experience in painting. In the experiment, we asked each volunteer to draw four identical Chinese characters, “漫步”, using Device I (mouse and keyboard) and Device II (Phantom Desktop device) under the same painting conditions. Before conducting the experiment, we guarantee that each of them is the first time to use the system, and they will not go through any detailed instructions or read the user’s instructions before operation. After the experiment, we statistically sorted the painting time used by the two devices, as shown in Figure 22, and it can be seen from the figure that the drawing time of the mouse and keyboard is more than twice that of the Phantom Desktop device. We also asked each volunteer to check and compare the Chinese characters drawn by the two devices with a pen, to observe which way to draw is more perfect. The results of the experiment showed that seven of the eight volunteers (the seven volunteers included two art students) said that it is better to use Device II (virtual desktop device) than Device I (mouse and keyboard) to paint. They prefer to use Device II (virtual desktop device) to draw than device I (mouse and keyboard). So, we can conclude that paintings with tactile feedback are better than paintings without tactile feedback.
6. Conclusion
In this paper, a novel variable stiffness brush model and haptic painting technique at the surface of the 3D objects (such as the virtual clay) for automobile industry are proposed via real-time haptic feedback technique using a 6 DOFs input device, and the haptic behavior of virtual 3D brush based on variable stiffness and elastoplasticity is studied in detail for the first time. The related mechanism between the applied force (bending moment) and the real-time deformation for the 3D virtual brush is studied by employing a bending spring mass to represent the virtual 3D hair brush mechanical model. Then, the collision contact checking among the two objects is studied by employing a contact checking method named average-weighted distances. We then calculate the related bounding ball for deformed 3D brush using a fast ball-expanding search algorithm to determine the virtual projection plane. Based on the real-time deformation about the virtual brush head at a sampling point, the 2D painting footprints is calculated and rendered. Next, the 3D painting footprint could be easily produced via mapping the 2D painting footprints onto the surface of the 3D model in real time. Finally, the 3D painting strokes (or trace) are formed, via controlling the exerted force and overlapping the virtual 3D painting footprints with different shapes and sizes, following the moving direction of the 3D brush. A real-time algorithm for the ink quantity calculation contained in 3D painting footprints has also been improved to simulate the real-time ink transfer process between the brush and the 3D object surface. In our system, you could paint 3D brush strokes with 3D semidry and ink diffusion effect by using the Phantom Desktop haptic device, which greatly enhances the sense of reality of the user experience.
However, there are still some problems and challenges in our system, such as the softer brush modeling, the construction of the water-ink diffusion model, and the mechanism of the real-time plastic deformation of the Chinese brush. In the future, mutisubjects crossed research of computer graphics rendering technology, artificial intelligence technology, and MEMS technology should be considered comprehensively to further promote the development of computer haptic simulation of the 3D brush, for Chinese calligraphy or the haptic decoration of the clay model in the automobile industry.
Data Availability
The data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.
Acknowledgments
This work was supported by the National Natural Science Foundation of China (no. 51175058).