About this Journal Submit a Manuscript Table of Contents
International Journal of Computer Games Technology
Volume 2008 (2008), Article ID 135398, 7 pages
http://dx.doi.org/10.1155/2008/135398
Research Article

Auto Coloring with Enhanced Character Registration

1Interaction and Entertainment Research Center, School of Computer Engineering, Nanyang Technological University, 50 Nanyang Drive, Singapore 637553
2College of Information Science and Technology, Beijing Normal University, Beijing 100875, China

Received 27 July 2007; Accepted 2 October 2007

Academic Editor: Kevin Kok Wai Wong

Copyright © 2008 Jie Qiu et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

An enhanced character registration method is proposed in this paper to assist the auto coloring for 2D animation characters. After skeletons are extracted, the skeleton of the character in a target frame is relocated based on a stable branch in a reference frame. Subsequently the characters among a sequence are automatically matched and registered. Occlusion are then detected and located in certain components segmented from the character. Two different approaches are applied to color regions in components without and with occlusion respectively. The approach has been tested for coloring a practical animation sequence and achieved high coloring accuracy, showing its applicability in commercial animation production.

1. Introduction

Inking/coloring of the individual animated characters for every frame is one of the most time-consuming and labor-intensive procedures in cel animation production. Many systems and some algorithms have been proposed to assist the coloring of cel animation production. However, these systems or algorithms have significant limitation, in terms of not being able to automatically establish the correct region correspondences when there is a big change or occlusion of the character, as reviewed in [1]. In summary, computer-assisted auto coloring (CAAC) remains a tough issue in research.

To detect and handle occlusion, a novel character registration method was proposed in our previous work [2], as the first attempt for auto coloring 2D characters for some special cases. As indicated in [2], the auto coloring approach may be challenged if the character's position changes greatly in a sequence of frames. In this paper, a relocation method is proposed to enhance the character registration approach in [2], making the auto coloring approach more robust.

The rest of the paper is organized as follows. First, the enhanced character registration method is introduced in detail. Subsequently, our occlusion detection and auto coloring process is introduced. Experiments are then designed to test the algorithm and results analyzed. Conclusion and future work can be found at the end of this paper.

2. Character Registration

2.1. Skeleton Extraction

A skeleton is a useful shape abstraction that captures the essential topology of an object in both two and three dimensions. It represents a thinned version of the original object, which still retains the shape properties of the original object [3]. The skeleton of a 2D character can preserve the most essential geometric and topological information of it, which makes it suitable for our character registration purpose.

In practical animation production, some characters are complex featuring sharp convex or concave details in their outline, which introduce redundant branches when the skeleton extraction approach based on active contours [4, 5] is applied. As shown in Figure 1, the branches in red and green do not relate to the physical structure of the character. These are viewed as “noise” in this paper. The short redundant branches in Figure 1 can be easily pruned based on a threshold, but it is difficult to remove the long ones. To reduce the noise, Gaussian filtering is applied to smooth the outline and remove the details. The Gaussian function in 2D form is 1𝐺(𝑥,𝑦)=2𝜋𝜎2𝑒𝑥2+𝑦2/2𝜎2.(1) The smoothed outlines and corresponding skeletons with different 𝜎 are illustrated in Figure 2. The 𝜎 is empirically selected as 3 for the examples used in this paper.

35398.fig.001
Figure 1: Extracted skeletons.
fig2
Figure 2: Skeleton extraction after Gaussian smoothing.

After the outline is smoothed, the skeleton of each character is extracted using active contours approach [4, 5], and further thinned under SUSAN thinning rules [6], and pruned. Finally, it is segmented into L-branches and J-branches as introduced in [2]. Topology graphs of each character is also extracted using the approach introduced in [5]. Figure 3 shows the extracted skeleton and topology graph for the character in Frame 4 of the first test used in the paper. Junction nodes are illustrated in red, leaf nodes in blue, L-branches in green, and J-branches in black, respectively.

fig3
Figure 3: Extracted skeleton and topology graph.
2.2. Skeleton Relocation

Animation is an art of capturing a series of individual movements, whether on film or in digital form, and replaying them in rapid succession to give the illusion of movement [7]. Accordingly, even in two successive frames, the positions of the two characters may differ much. Besides affine transforms which are often used for depicting the character's global motion, deformation of the character parts usually exists for expressing the local motion like the movements of articulated parts. With these uncertainties, it is hard to find an accurate transform to locate the two characters in the same position with the most similarity. When drawing inbetweens, animators locate characters of two key frames by overlapping the most similar parts of the character in the same position. Mimicking this method, two skeletons can be relocated according to a stable branch which does not change much among the whole animation sequence.

The stable branch of each frame can be interactively selected by the user. Alternatively, user only needs to indicate the stable branch in the first reference frame to be registered. Given a reference frame and a target frame, a prediction and relocation method is proposed as follows to reduce user intervention.

Given a population of random vectors 𝑥, the principal components analysis (PCA) is defined as 𝑦=𝐴𝑥𝑚𝑥,(2) where 𝐴 is the matrix whose rows are formed from the eigenvectors of 𝑥's covariance matrix 𝐶𝑥, 𝑚𝑥 is 𝑥's mean vector [8].

(i)Compute the PCA transforming matrices 𝑇𝑖 and 𝑀𝑖 for the stable branch 𝐵𝑖 in the reference frame, where 𝑇𝑖 and 𝑀𝑖, respectively, correspond to 𝐴 and 𝑚𝑥 in (2).(ii)Transform the skeleton 𝑆 in the reference frame to the transforming coordinate system 𝑅2𝑡 according to 𝑇𝑖 and 𝑀𝑖: 𝑆𝑡𝑖=𝑇𝑖𝑆𝑀𝑖.(3)(iii)For each branch 𝐵𝑖 in the target frame, compute the dissimilarity value 𝐷𝑡(𝑖,𝑖).(a) Compute the PCA transforming matrices 𝑇𝑖 and 𝑀𝑖 for a branch 𝐵𝑖 in the target frame, where 𝑇𝑖 and 𝑀𝑖, respectively correspond to 𝐴 and 𝑚𝑥 in (2).(b) Transform the skeleton 𝑆 in the target frame to the transforming coordinate system 𝑅2𝑡 according to 𝑇𝑖 and 𝑀𝑖: 𝑆𝑡𝑖=𝑇𝑖𝑆𝑀𝑖.(4)(c) Compute the Hausdorff distance 𝐻𝑡(𝑆𝑡𝑖,𝑆𝑡𝑖) between 𝑆𝑡𝑖 and 𝑆𝑡𝑖: 𝑑(𝑎,𝑏)=𝑥𝑎𝑥𝑏2+𝑦𝑎𝑦𝑏2,(𝑥,𝑦)𝑅2𝑡,𝑆𝑡𝑖,𝑆𝑡𝑖=sup𝑡𝑖𝑎𝑆inf𝑏𝑆𝑡𝑖𝐻𝑑(𝑎,𝑏),𝑡𝑆𝑡𝑖,𝑆𝑡𝑖=max𝑆𝑡𝑖,𝑆𝑡𝑖,𝑆𝑡𝑖,𝑆𝑡𝑖.(5)(d) Compute the Hausdorff distance 𝐻𝑡(𝑆𝑡𝑖,𝑆𝑡𝑖) between 𝑆𝑡𝑖 and 𝑆𝑡𝑖.(e)The dissimilarity value 𝐷𝑡(𝑖,𝑖) between branches 𝐵𝑖 and 𝐵𝑖 is the minimum of 𝐻𝑡(𝑆𝑡𝑖,𝑆𝑡𝑖) and 𝐻𝑡(𝑆𝑡𝑖,𝑆𝑡𝑖): 𝐷𝑡(𝑖,𝑖𝐻)=min𝑡𝑆𝑡𝑖,𝑆𝑡𝑖,𝐻𝑡𝑆𝑡𝑖,𝑆𝑡𝑖.(6)(iv) If 𝐷𝑡(𝑖,𝑖) is the minimum, 𝐵𝑖 and 𝐵𝑖 are corresponded, the skeleton 𝑆 in the target frame is transformed and normalized to be 𝑆𝑡𝑖 or 𝑆𝑡𝑖 according to 𝑇𝑖 and 𝑀𝑖. The transforming coordinate system 𝑅2𝑡 for 𝑆𝑡𝑖 and 𝑆𝑡𝑖 or 𝑆𝑡𝑖 is treated as the global coordinate system 𝑅2𝑔.

Figure 4 shows the relocated skeletons and frames. Frames 1 and 2 are represented in blue and red, respectively.

fig4
Figure 4: Skeleton relocation.
2.3. Skeleton Registration

General topology models are predefined based on the character's physical structure. Figure 5 illustrates some common topology models, which represent the most essential topological information of the characters.

fig5
Figure 5: General topology model.

To assure registration accuracy, a frame is selected as the first reference frame from the sequence to be painted, which contains a stable branch as introduced in Section 2.2 and the maximum number of branches among the sequence. The skeleton and topology graph of the character in the selected frame is then registered to the general model, as illustrated in Figure 6. The topology graph is adjusted accordingly.

fig6
Figure 6: Skeleton registration.

For characters in the other frames in the sequence, their skeletons and branches are registered based on the branch correspondences established by skeleton matching introduced in the next section.

2.4. Skeleton Matching

As introduced in [2], our skeleton matching algorithm is based on both geometric and topological information of the skeleton, containing the following two steps.

2.4.1. Geometric Matching

Global dissimilarity due to the motion of branches is represented by the global Hausdorff distance 𝐻𝑔(𝐵𝑖,𝐵𝑖) and computed using the global coordinate system 𝑅2𝑔 of the two frames.

The local deformation of each branch is represented by the local Hausdorff distance 𝐻𝑙(𝐵𝑖,𝐵𝑖) and computed using the local coordinate system 𝑅2𝑙 of the two frames established based on PCA [2].

With the global and local Hausdorff distances between two branches, the dissimilarity value DV is defined as DV(𝑖,𝑖)=𝐻𝑔𝐵𝑖,𝐵𝑖𝐻+min𝑙𝐵𝑖,𝐵𝑖,𝐻𝑙𝐵𝑖,𝐵𝑖.(7)

Each branch 𝐵𝑖 in the reference frame is matched with all branches in the target frame first. If DV(𝑖,𝑖) is the minimum among all dissimilarity values, a matching (𝐵𝑖𝐵𝑖) is obtained. Subsequently, each branch 𝐵𝑖 obtains its best-matched branch in the reference frame. If a bidirectional matching is achieved, (𝐵𝑖𝐵𝑖)(𝐵𝑖𝐵𝑖), 𝐵𝑖 and 𝐵𝑖 are corresponded (𝐵𝑖𝐵𝑖), and 𝐵𝑖 is registered as the same branch with 𝐵𝑖 in the general topology model. The unregistered branches will be readjusted in later process.

2.4.2. Topological Readjustment

After geometric matching, some branches with big motion or deformation may be left unregistered. To match and register these branches and detect occlusion, a topological readjustment method is applied.

For each selected branch 𝐵𝑡𝑖 in the target frame, its merging candidates are obtained, and it is merged with the one related most closely [2]. If no merging candidate is found, geometric matching is applied to those unregistered branches in 𝐵𝑡𝑖's subgraph.

2.5. Component Segmentation

After skeleton registration, characters are segmented into several components corresponding to the branches in skeleton and topology graph, based on the explained area of each branch [2].

The segmented and registered components for Frames 4 and 5 are illustrated in Figure 7, where corresponding components are in the same color.

fig7
Figure 7: Segmentation result.

3. Enhanced Auto Coloring

3.1. Occlusion Detection

To detect and locate occlusion in components, a method based on the variation of region areas is advanced in [2].

(i)Each region in the reference frame is quantized and coded as an English character based on its area.(ii)All the regions in the target frame are quantized and coded based on the scale points computed in the reference frame.(iii) Regions in a component 𝐶𝑖 in the reference frame are coded as a string 𝑆𝑖 and compared with the string 𝑆𝑖 of its corresponding component 𝐶𝑖 in the target frame, and the least conversion cost between them is computed as 𝛾(𝑆𝑖𝑆𝑖).(iv) Occlusion is located in 𝐶𝑖 if 𝛾(𝑆𝑖𝑆𝑖) is bigger than an empirically defined threshold.

3.2. Auto Coloring

With occlusion detection, regions can be divided into two categories: the first category contains all regions which are in the components without occlusion, and the second consists of those in the components with occlusion. The former is supposed to be more stable than the latter and different matching methods are applied to the two categories.

The coloring process consists of three steps, and the coloring result after each step is illustrated in Figure 8.

fig8
Figure 8: Auto coloring process.

(i) For the first category, the hierarchical feature-based region matching approach as proposed in [1, 9] is applied.(ii) For the second category, the continuity of some region contours is broken because of occlusion. Hence, features such as area, curve length, character points, and relations with neighboring regions change greatly. But the rest of regions still have relatively stable features of area, curve length and character points. So a matching method similar to that for the first category is applied to match these regions. Each region is matched with those in corresponding components in the reference frame. The only difference is that the feature of relations with neighboring regions is ignored, so that the coded character string [1] is only composed of character points.(iii) Finally, each of the remaining regions in the relocated target frame inherits the color of the region that it overlaps mostly in the reference frame. If no such region is obtained, which means that the region fully overlaps the background, it is painted in the color which fills the majority of the component (major color) that it belongs to.

After skeleton registration for the first reference frame, the prior and subsequent uncolored frames are selected as new target frames iteratively. The nearest colored frame which contains a stable branch is selected as the target frame's reference frame.

4. Results and Analysis

To test the possibility of applying our approach in practical animation production, we use a Japanese-style sequence in practical production of the trailer of “Justeen” animation. The 8 frames as shown in Figure 9 compose 2/3 of a cut in the trailer, showing a boy jumping down into the control room of his robot. It can be noticed that occlusion arises due to the motion of the character's arms. To minimize the information loss due to occlusion, we select Frame 4 as the first reference frame. Then the prior and subsequent frames are colored one by one according to the color information of its next and prior frame, respectively. From the results we can see that most of the regions are correctly colored. The regions wrongly colored are mainly because no corresponding regions in the reference frame can be obtained due to the occlusion or information loss. Figure 10 illustrates the main errors among the sequence and the correct results. As illustrated in Figure 10(a), the right arm in Frame 5 changes greatly compared with the one in Frame 4, and the occluded part of the bracelet in Frame 4 is visible in Frame 5. Accordingly, the regions indicated in blue circles are wrongly matched as no correct correspondences exist in Frame 4. With more reference frames (which contain the corresponding regions) provided, this kind of error can be avoided. In Figure 10(c), the regions in blue circles in Frame 5 are created due to the occlusion, and they inherit the color of the regions they overlap mostly in Frame 4 as no corresponding regions are obtained. In Figure 10(e), the region in blue circle in Frame 7 is created as the hair is swaying to the position overlapping with the eyebrow. It is thus wrongly matched to a hair branch as they have great similarity. These errors can be solved if the hair is drawn in a separate layer. The mouth in Frame 1 is widely open so that the tongue appears only in this frame. It is not matched to any region, and thus inherits the color of the face in Frame 2. This kind of error due to information loss is hard to be avoided. In summary, the coloring accuracy for the total 7 uncolored frames is over 93%, which shows that our approach is applicable to practical animation production.

fig9
Figure 9: Test 1 (resolution: 2420×2420 pixels) (courtesy of Anime International Co., Inc., Japan).
fig10
Figure 10: Error Analysis for Test 1.

Figure 11 shows the other test, with the first frame selected as the first reference frames. The character is simple, but it is noticeable that large motions and deformations exist among the sequence. With the relocation method proposed in this paper, skeletons are accurately matched, ensuring the high coloring accuracy for the test.

fig11
Figure 11: Test 2 (resolution: 550×400 pixels).

5. Conclusion and Future Work

In this paper, our previous work on character registration is enhanced with a refined skeleton extraction procedure and a novel character relocation method. With the enhanced character registration, occlusion can be correctly detected and located in components segmented from the character. Subsequently, two different matching methods are applied to regions in components without and with occlusion, respectively. A practical animation sequence from a cut of “Justeen” animation trailer is applied to validate the approach. A high coloring accuracy is achieved. It shows that the proposed auto coloring approach with enhanced character registration is applicable to practical animation production.

Nevertheless, there are limitations. The proposed approach requires that each frame has a stable branch, which is not always true among a long sequence. For example, the cut we used in Test 1 contains 12 frames in total, and the first 4 frames are illustrated in Figure 12. Due to the information loss, no stable branch exists in these frames. So they cannot be automatically colored using our approach. Future research work will be focusing on the auto coloring for these kinds of special cases and new relocation methods.

fig12
Figure 12: Other frames in the cut used in Test 1 (courtesy of Anime International Co., Inc., Japan).

Acknowledgment

This work has been partially supported by the Science and Engineering Research Council (SERC) Grant no. 052 015 0024 Ref: 44, which is awarded by the Agency for Science and Technology Research (A*STAR) and administered through the Singapore National Grid Office.

References

  1. J. Qiu, H. S. Seah, F. Tian, Z. Wu, and Q. Chen, “Feature- and region-based auto painting for 2D animation,” The Visual Computer, vol. 21, no. 11, pp. 928–944, 2005. View at Publisher · View at Google Scholar
  2. J. Qiu, H. S. Seah, F. Tian, Q. Chen, Z. Wu, and M. Konstantin, “Auto coloring with character registration,” in Proceedings of the International Conference on Game Research and Development (CyberGames '06), vol. 223, pp. 25–32, Perth, Australia, December 2006.
  3. N. D. Cornea, D. Silver, X. Yuan, and R. Balasubramanian, “Computing hierarchical curve-skeletons of 3D objects,” The Visual Computer, vol. 21, no. 11, pp. 945–955, 2005. View at Publisher · View at Google Scholar
  4. P. Golland and W. E. L. Grimson, “Fixed topology skeletons,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '00), vol. 1, pp. 10–17, Hilton Head Island, SC, USA, June 2000. View at Publisher · View at Google Scholar
  5. J. Qiu, H. S. Seah, F. Tian, Q. Chen, and K. Melikhov, “Topology enhanced component segmentation for 2D animation character,” in Proceedings of International Workshop on Advanced Imaging Technology, pp. 30–35, Okinawa, Japan, January 2006.
  6. S. M. Smith, “Edge thinning used in the SUSAN edge detector,” Internal Technical Report TR95SMS5, Defence Research Agency, Surrey, UK, 1995.
  7. C. Patmore, The Complete Animation Course: The Principles, Practice and Techniques of Successful Animation, Thames & Hudson, London, UK, 2003.
  8. R. C. Gonzalez and R. E. Woods, Digital Image Processing, Addison-Wesley, Reading, Mass, USA, 1992.
  9. J. Qiu, H. S. Seah, F. Tian, Q. Chen, and Z. Wu, “Enhanced auto coloring with hierarchical region matching,” Computer Animation and Virtual Worlds, vol. 16, no. 3-4, pp. 463–473, 2005. View at Publisher · View at Google Scholar