Abstract
Chinese calligraphy is a charismatic ancient art form with high artistic value in Chinese culture. Virtual calligraphy learning system is a research hotspot in recent years. In such system, a judging mechanism for user’s practice result is quite important. Sometimes, user’s handwritten character is not that standard, the size and position are not fixed, and the whole character may be even askew, which brings difficulty for its evaluation. In this paper, we propose an approach by using DBSCs (disk B-spline curves) vectorization and ICP (iterative closest point) algorithm, which cannot only evaluate a calligraphic character without knowing what it is, but also deal with the above problems commendably. Firstly we find the promising candidate characters from the database according to the angular difference relations as quickly as possible. Then we check these vectorized candidates by using ICP algorithm based upon the skeleton, hence finding out the best matching character. Finally a comprehensive evaluation involving global (the whole character) and local (strokes) similarities is implemented, and a final composited evaluation score can be worked out.
1. Introduction
Chinese character is one of the fundamental elements of Chinese culture. Chinese calligraphy is produced and developed based on Chinese characters, and it belongs to the domain of visual art. Calligraphy has brought the writing of characters into an aesthetic stage. It fuses the creator’s idea, thought, and spirit and can arouse aesthetic emotions of the appreciator. Calligraphy can even reflect one’s personality, temperament, knowledge, and self-cultivation. Practicing calligraphy is conductive to training learning abilities, developing minds, and cultivating physical and mental qualities.
After thousand years of development, a lot of calligraphy styles are established. Many famous calligraphers have formed their own unique writing characteristics and Mr. Qi Gong is one of them. Qi Gong is a great master of traditional Chinese painting and calligraphy. His calligraphy is slender, comely, and elegant and is known as “Qi font.” Mr. Qi Gong left behind a large amount of art treasure to the world and has gained worldwide reputation. He is a role model for later generations in terms of both artistic achievements and moral quality. In this paper, we choose Qi font as basic data to support our research.
With the rapid development of computer graphics and image processing technologies, calligraphy research and creation have been brought into a new field. Many researchers are devoting themselves to the study of computer-aided virtual calligraphy. Existing researches mainly involve the following aspects: modeling of calligraphy elements such as brush, stroke, dip, and paper; calligraphy retrieval; character recognition; calligraphy generation; calligraphy analysis and evaluation; and feature extraction and style synthesis. In addition, designing an interactive and user-friendly calligraphy learning system based on the above techniques is a hot issue now and very helpful for calligraphy learners. In such system, a judging mechanism is quite important and essential. With an evaluation result, users could know how well they write, and this helps to improve writing skills.
In this paper, we propose an evaluation approach based on DBSCs (disk B-spline curves) vectorized Qi Gong calligraphy database. Firstly, via angular difference relations, we can find the promising candidate characters from the database, thus reducing the search space. Then these vectorized candidates are registrated with user’s handwriting by using ICP (iterative closest point) algorithm based upon the skeleton; hence the best matching character is found. Finally the character and strokes similarities are calculated according to their shape features. A great advantage of this evaluation approach is that it provides a mechanism to deal with the different situation of position, size, and tilt of user’s handwritten character without knowing what this character is. Experiments prove this approach is feasible, effective, and objective.
The rest of this paper is organized as follows. In Section 2, related works on virtual calligraphy are presented. In Section 3, the database establishment of vectorized Qi Gong calligraphy is given. In Section 4, the preprocessing method of how to reduce the search space is described. In Section 5, character registration based on ICP algorithm is described. In Section 6, the comprehensive evaluation method is investigated. In Section 7, the experiment results are analyzed. Conclusions and our future work are summarized in the final section.
2. Related Work
Computer-aided virtual calligraphy research is a frontier and many research achievements on this subject have been published. Here we will introduce some research results which are closely related to our work in this paper.
In 1986, Strassmann [1] proposed a 2D virtual brush model to generate paintings at the SIGGRAPH conference for the first time. This model consists of four parts, that is, brush, stroke, dip, and paper. Although the simulation result of this model is simple, it is still a great reference for the future study. There are a lot of researches on the modeling of Chinese ink calligraphy and paintings afterwards. Chu’s [2] rendering of realistic brushwork is realized by responding input data captured from a device with six degrees of freedom; Seah’s [3] modeling and representation approach for brushstroke and animation is based on disk B-spline curves. The modeling method in this paper is like Strassmann, Chu, and Seah’s models.
Calligraphy analysis and evaluation are quite important and widely used in calligraphy style synthesis, calligraphy generation and calligraphy learning system, and so forth. Han et al. [4] proposed an interactive grading and learning system of Chinese calligraphy which uses the image processing and fuzzy inference techniques to evaluate characters based on the position, size, and projection features. Gao et al. [5] proposed a Chinese handwriting quality evaluation method based on the analysis of online handwriting recognition confidence. Shichinohe et al. [6] designed an augmented calligraphy system; by monitoring the calligraphy learner’s posture, it can give feedback to the learner and support the learner’s self-training process. Murata et al. [7] built a real-time measurement system of eye-hand coordination to extract the skilled elements in calligraphy. Xu et al. [8] introduced a numerical machine-learning approach to evaluate the visual quality of calligraphic writings from human aesthetic views. Han et al. [9] put forward a similarity assessment method which is based on the context of the skeleton to evaluate the similarity between two brush inks.
Calligraphy system is a synthesis of the calligraphy researches such as calligraphy modeling, feature extraction and style synthesis, calligraphy generation, and calligraphy analysis and evaluation, aiming at offering an integrated and systematic application for calligraphy learners and researchers. Henmi and Yoshikawa [10] developed a virtual calligraphy system that can display teacher’s skillful motion to students by recording the position and force trajectories of the teacher’s writing brush. Shin et al. [11] presented a calligraphy learning system by using Yongzi-Bafa which can also give an evaluation result of every stroke of user’s input.
The key issue of this paper is the evaluation of user’s handwritten calligraphy in a calligraphy learning system. So far, most of the evaluation researches are raster image processing based, but our work is carried out based on the vectorized calligraphy.
3. Database Establishment of Qi Gong Calligraphy
A good and complete calligraphy database is the basis of evaluation research, especially when evaluating a calligraphy character without knowing what it is. Our Qi Gong calligraphy database contains 3755 frequently used Qi font Chinese characters according to GB 2312 (Chinese Ideogram Coded Character Set), including both vector representation and binary image data.
3.1. Stroke Segmentation
Stroke is the minimum component of a Chinese character. There are mainly five kinds of strokes:(1)“heng (一),” which is a horizontal line.(2)“shu (丨),” which is a top-down vertical line.(3)“pie (),” which is a left-downward slope line.(4)“dian (丶),” or called “na,” which is a right-downward short pausing stroke.(5)“zhe (),” which is a turning stroke having large angular variations.
Stroke segmentation means decomposing a character into a number of strokes according to their correct stroke sequence, and each stroke is saved as a separate part. Corner detection method is used to mark crossover points, where curvature is large enough or gradient changes dramatically. When these corners are obtained, joinable corner point pairs are picked out by alternately human interaction. Hence complete and independent strokes can be separated from a character. For an optimized result, interpolation is applied to make the strokes smooth. Stroke segmentation work for Qi Gong calligraphy has been finished [12]. Figure 2 shows a segmentation example of character “她”.
3.2. Representing Characters by DBSCs
Raster character image, usually bitmap, requires large storage space and always causes anamorphosis and aliasing when they are zoomed or rotated, while, on the contrary, vector representation cannot only reduce the storage space, but also realize fast transformation and arbitrary scaling without anamorphosis. Disk B-spline curves are always used to represent Chinese characters in recent years.
3.2.1. Disk B-Spline Curve (DBSC) Definition
Let be the th B-spline basis of degree with knot vector . Here . Then the disk B-spline curve (DBSC) is defined as follows:where is control point and is control radii.
DBSC can be viewed as 2 parts: the center curve , which is a B-spline curve, and the radius function , which is a B-spline scalar function.
Owing to the perfect symmetry property of disks, the curve constructed from the centers of disks is exactly the skeleton of the 2D region represented by DBSC. Most of the properties and algorithms can be obtained by applying B-spline curve and function to the 2 parts of the disk B-spline curve, respectively. More details can be found in [3].
3.2.2. DBSC Based Stroke Representation
The DBSC is derived from B-spline curves. The difference between B-spline curves and DBSC is that instead of being defined by points, DBSC is defined by a set of disks. It derives nature of flexibility for transformation, deformation, and morphing.
A disk B-spline curve is a skeleton based parametric 2D representation, which represents the 2D region as well as the center curve (skeleton) of the 2D region explicitly. DBSC is a proper tool to describe a brushstroke. Therefore, we use DBSCs to represent the geometric data of each stroke in characters. The first stroke of a Qi font “她” is represented by a disk B-spline curve as shown in Figure 3.
Vectorized 3755 Qi font characters are included in our Qi Gong calligraphy database so that they can be used for the evaluation approach in the rest of our paper. Figure 4 is an example of some vectorized characters in our Qi Gong calligraphy database.
4. Preprocessing of the Search Space
In order to improve the whole efficiency of our method, some steps will be taken to reduce the search space at first. This reducing algorithm mainly involves two aspects: one is total stroke number, and the other is angular difference between adjacent strokes.
4.1. Reducing the Search Space according to Total Stroke Number
The input device of our system is digital tablet which is based on electromagnetic induction. It can perceive very subtle pressure changes, convert touch pressures and positions into pixels, and eventually form the handwriting. When the pressure between “brush” and “paper” changes from zero to a nonzero number, the system considers a stroke begins. When the pressure turns back to zero, system considers the stroke finishes. Positions of virtual pen form the stroke skeleton; pressures represent stroke radii. Hence, track and shape information of every stroke, namely, the skeleton and radii, as well as the total stroke number of user’s handwriting can be recorded.
Since our calligraphy database contains the total stroke number information of all characters, when user finishes input, the search space can then be reduced into a subspace in which the characters have the same total stroke number with user’s handwriting. And the next reducing step will be taken based on this subspace.
4.2. Reducing the Search Space according to Angular Difference between Adjacent Strokes
Every stroke in a character has a direction, namely, the correct writing direction. So there exists an angle relation between two strokes which are adjacent in writing order; we name this relation “angular difference.” No matter what kind of size or position the character has, or even if the character is written askew, the angular differences between adjacent strokes will still be almost invariable. So according to the similarities between user’s handwriting and standard Qi Gong calligraphy in terms of angular difference, we can exclude a large number of characters, or even recognize what user writes, thus reducing the search space.
But the actual situation of strokes is much more complicated. The direction of a stroke is not constant; it may change slightly or greatly. A most typical example is “zhe.” It can be horizontal in the beginning and then become vertical from the middle; and it can also be vertical in the beginning and then become horizontal from the middle. In the former situation, we call it “heng zhe (㇇),” and in the latter, we call it “shu zhe ().” So in order to avoid the direction deviation caused by large and small angular variation, we will distinguish the stroke’s classification in our approach.
The reducing process of the search space in this phase mainly includes four steps. Firstly, a curve fitting algorithm is used to obtain the line segment that can approximate the stroke in maximum. Secondly strokes are classified into straight or curving according to the approximating distance. Thirdly the fitting lines are used to represent strokes and calculate angular differences. Finally, a weighted algorithm is used to calculate similarities between user’s handwriting and standard Qi Gong characters according to stroke classifications and angular differences; hence we can find out the top several most similar characters.
4.2.1. Optimal Approximating Line Segment of Strokes
We use a least square method based curve fitting algorithm [13] to obtain the optimal approximating line segment of each stroke.
Figure 5 shows a fitting line example. The red points in the picture stand for the equidistant sampling data points of the stroke skeleton, and the green line stands for the calculated optimal approximating line segment of the stroke.
4.2.2. Classification of Strokes
The fitting line of a stroke whose direction changes greatly such as “zhe” may be quite similar to the fitting line of a stroke whose direction is almost unchanged. This situation may bring errors to our approach. So in order to be more accurate, it is necessary to classify the stroke into straight and curving according to the approximating distance. We use Euclidean distance to distinguish strokes.
Let be the Euclidean distance from sampling point to the optimal approximating line segment, . Let be the length of the approximating segment.
Then if , we judge the stroke to be straight, otherwise, curving. Here is a threshold value.
We denote as the stroke classification similarity between a standard Qi Gong character and user’s handwriting. Let be total stroke number of a character. If there are strokes whose classifications are different, then
Now we give an example. As shown in Table 1, there are two characters whose total stroke numbers are both 3. We can see that the strokes of character “工” are all straight, but in character “口”, the first and third stoke are straight and the second stroke is curving. So there is one stroke whose classification is different. According to (2), .
4.2.3. Calculation of Angular Differences
After figuring out the approximating line segments, we can use them to calculate angular differences. If the total stroke number of a character is , then there will be angular differences. For convenience, the value of angular difference ranges from 0 to .
Here we use characters “二” and “十” in Figure 5 to show how to calculate the angular differences. Assume , are the approximating line segments of the two strokes in character “二”, and , are the approximating line segments of the two strokes in character “十”. Segment directions are the correct writing directions of their corresponding strokes. Then the angular differences of “二” and “十” are
Clearly we can see that the angular differences of “二” and “十” in Figure 6 are quite unlike; they differ by almost 90°, so they can be judged to be two different characters qualitatively. In the next part we will give the quantitive evaluation criteria.
Let be the angular difference sequence of a standard Qi Gong character, and let be the sequence of user’s handwriting. here is equivalent to a character template. and are the angular difference between stroke and stroke . is the total stroke number and . Then the mean deviation of the two characters’ angular differences Dev can be calculated as follows:
The smaller the Dev is, the more similar the characters are.
4.2.4. Weighted Similarity Calculation Algorithm
Here we set two threshold values, and , to control the size of search space. is used to limit , namely, the number of strokes with different classifications, which has been introduced in Section 4.2.2. is used to limit Dev, namely, the mean deviation of the two characters’ angular differences, which has been introduced in Section 4.2.3.
We denote as the angular difference similarity between a standard Qi Gong character and user’s handwriting. Then it can be figured out according to the following equation:
If , or , we consider that the current two characters being compared are not the same, thus excluding a number of characters from the search space.
If and , we will calculate a compositive weighted similarity :
Here and are weight parameters, . Combining (2), (4), (5), and (6), we could get a final detailed similarity calculation equation
After scanning all the characters in the search space, we can find out the top several characters which have the highest according to the sorting result.
5. Character Registration by Using ICP Algorithm
Iterative closest point (ICP) algorithm [14] is one of the most commonly used registration algorithms based on point set to point set. The basic steps of this algorithm are [15] to find out the closest matching point pairs in the two point sets being processed; compute the transformation matrix that minimizes the sum of the squares between the paired points and then apply the transformation; iterate the above two steps until the distance satisfies a given convergence precision. When the iteration is stopped, we can get the final translation and rotation parameters. Therefore, we can consider the user’s handwritten character and the standard Qi font character candidates selected from the previous preprocessing step as point sets and match them via ICP algorithm.
Skeleton is an important descriptor for shape matching, and sometimes it performs better than contour or the whole pixel point set of an object. What is more, computation of skeleton-based ICP algorithm will be much faster than the computation based on whole points of the object. Since the Qi Gong calligraphy characters in our database are vectorized by DBSCs, the skeletons and shapes are easy to get. And as we said in Section 4.1, the skeleton and radii of user’s handwritten character are also recorded, so in this paper we will use the skeleton point set to realize ICP registration and find out the best matching character.
5.1. Scaling of the Standard Qi Gong Character
Our ICP registration is rigid, so before applying this algorithm, we must scale the standard character at first and make the two characters to be processed have the same size so that we can get a more accurate registration result in the following step.
Let be the skeleton point set of standard Qi Gong character, and let be the skeleton point set of user’s handwritten character. Here we take as the referenced point set. First the geometric centers and of and are calculated. Then the smallest disks which can cover the whole character are found, and the corresponding radii are denoted by and . With and , we can scale the vectorized standard Qi Gong character, thus making it have the same covering disk size with user’s handwritten character.
5.2. ICP Registration of Characters
5.2.1. ICP Algorithm
Registration operation of ICP algorithm actually means finding an optimal rigid transformation from one coordinate system to another which can minimize the sum of the squares between two point sets. This transformation can be represented by a rotation matrix and a three-dimensional translation vector .
Let and be two point sets to be registrated. Suppose is an arbitrary point in , and its coordinate value is . After transforming, the coordinate value of is . Then,
So the registration goal is to figure out the transformation of and which can minimize the value of the following function:where is iteration times, is the closest matching point of , , and is the total point number of . In this iteration process, and are not fixed, they are always changing. After each iteration, and the closest matching point pairs will be updated. will be the new in next iteration.
5.2.2. Character Registration Steps
In our approach, we use ICP registration algorithm to make standard Qi Gong character and user’s handwritten character match best, and the algorithm is carried out in two-dimensional space.
and are the two skeleton point sets of standard Qi Gong character and user’s handwritten character, and they have been scaled to the same size after the process in Section 5.1. Then the registration steps can be described as follows.(1)Search all the closest corresponding points of in .(2)Figure out the rigid transformation which can minimize the sum of the squares between the paired points above mentioned and then acquire rotation parameter and translation parameter .(3)Apply and to and get the transformed point set.(4)If the transformed point set of and the referenced point set can satisfy a given convergence precision of function in (9), namely, the sum of the squares of the two point sets being less than a given threshold value, then stop iterating. Otherwise set the transformed point set as the new , and iterate the above four steps until the function value is acceptable.
There are several commonly used methods of searching the closest corresponding point pairs in step (1), such as point to point [14], point to plane [16], and point to projection [17], as shown in Figure 7. Since our registration is based on skeleton point set, we use the point to point searching method. Figure 8 is a registration example of our experiment character “永”.
6. Comprehensive Evaluation
Skeleton represents the global topological information of a character, which can reflect the balance and arrangement of all the strokes. Local similarity of each stroke is also an important metric in calligraphy evaluation mechanism. With stroke evaluation score, learners could find out about which stroke they wrote well and which stroke they need to practice further more. So we compare both the global similarity and the local similarity in our approach.
However, skeleton distance is not easy to convert into a certain score based on the percentage grading system. What is more, it has lost the width information. So in this paper we use the whole character shape instead of skeleton to represent the topological feature. Similarly, we use stroke shape to calculate stroke similarity. Here the shape data is defined as the pixel distribution information of a character or stroke; that is the most direct and easy way to measure the distance between two characters or two strokes. With the skeleton and radii of a character, and the stroke segmentation data in our database, we can easily get the shape information of each stroke and the whole character.
6.1. Character Shape Similarity
When the registration step is finished, the best matching character will be found; then character shape similarity, denoted by , can be calculated according to the overlapping situation of user’s handwriting and transformed standard Qi Gong character as follows:where is the length of the smallest square which can cover the two registrated characters, is the pixel value of user’s character, is the pixel value of Qi Gong character, and we define
In (10), numerator represents the total number of pixels that both are black, namely, the overlapping black area; denominator represents the total number of pixels that are different or both black, namely, the remaining part after removing the overlapping white area.
6.2. Stroke Shape Similarity
In order to avoid the error brought by position and size of the strokes to be compared, we first normalize their “effective area” to in pixels. Here the “effective area” of a stroke is calculated as follows.
First, we find four boundaries of a stroke, that is, the top, the bottom, the left, and the right. Then this rectangle is turned into a square region according to its longer side, and we make sure that the rectangle is right in the middle of the square. This square region is called the effective area of a stroke. Figure 9 shows an example of finding the effective area of a stroke.
When the effective areas of user’s handwritten strokes and standard Qi Gong strokes are normalized to in pixels, the shape similarity of th stroke, denoted by , can be calculated according to the following equation, which is quite similar to (10):where, , is the total stroke number of the current character and . is the pixel value of user’s th stroke; is the pixel value of th standard stroke.
6.3. Composited Score
With character shape similarity and stroke shape similarities , we can also compute a composited single evaluation score Eva.
The mean similarity of all strokes is taken as the final stroke shape similarity, denoted by :
The maximal values of and are both 1. Hence,where and are weight parameters, .
So far, the introduction of the detailed steps of our evaluation approach has been finished. Figure 10 shows the whole architecture of the approach in this paper.
7. Experiment Results and Analysis
7.1. Validity of the Recognition Algorithm
Aiming at finding out the best matching character, our character recognition algorithm mainly consists of two parts: reducing the search space and ICP registration, namely, steps (1) to (5) in Figure 10. We have given one example as shown in Figure 8. In order to avoid the Type-I or Type-II error, we did several experiments by using a sample database containing 30 characters of 5 writers to validate the validity and make our algorithm more convincing. Table 2 shows the experiment result. We can see that, in this sample database, only one character is recognized as a wrong Qi Gong character. User wrote a character “己” (ji), but our algorithm recognized it as “已” (yi); their shape is quite similar. The proportion of two characters like “己” and “已” which not only are very similar in shape but also have the same total stroke number is quite small. So this error can be accepted. The experiment proves our algorithm is effective. In these 30 test characters, 21 of them are recognized at the stage of reducing the search space before ICP registration, which proves our method is efficient as well.
7.2. Composited Shape Evaluation
We take “永” as our experiment character. “永” is the first character of calligraphy “Preface to Orchid Pavilion” (蘭亭序) shown in Figure 1 and Table 3. The ICP registration of its skeleton has been shown in Figure 8, and we analyze its shape similarity. Likewise, other characters in calligraphy “Preface to Orchid Pavilion” or our vector Qi Gong calligraphy database can also be evaluated by our method.
(a) Point to point
(b) Point to plane
(c) Point to projection
(a)
(b)
(c)
(d)
In Table 3(a), we can see the user’s askew handwritten character (left), the original standard Qi Gong character (middle), and the overlapping comparison after registration (right). Global similarity is calculated. According to this similarity, users could know how well they wrote in terms of structure and shape of this character. Table 3(b) shows the comparison of each stroke; with these stroke similarities, users can get to know which stroke they wrote well and which stroke they need to practice more. For example, the best stroke of user’s handwriting is the first stroke, with a similarity of 0.906372, and the worst stroke is the third stroke, with a similarity of 0.690590. Table 3(c) gives the composited evaluation score, which can show the overall quality of user’s practice. Here the and in (14) are both set as 0.5 in our experiment.
We consulted several calligraphy teachers and asked them to evaluate experiment result. They concluded the result is relatively objective, which proves our approach is effective and satisfactory.
8. Conclusions
This paper presents the establishment of our vectorized Qi Gong calligraphy database, and we propose an effective evaluation approach by using angular difference relations, ICP algorithm, and shape features. In the proposed approach, character shape similarity can reflect the global whole structure and stroke arrangement of the character; and stroke shape similarity can reflect the local detail features. The proposed approach is comprehensive, and it is able to deal with the different situation of position, size, and tilt of user’s handwritten character without knowing what this character is. Experiment results show that this approach is feasible and effective. Furthermore, it can be extended to other calligraphy databases.
Conflict of Interests
The authors declare no conflict of interests.
Acknowledgment
This work is partially supported by National Natural Science Foundation of China (no. 61170170 and no. 61271366).