Abstract
An enhanced character registration method is proposed in this paper to assist the auto coloring for 2D animation characters. After skeletons are extracted, the skeleton of the character in a target frame is relocated based on a stable branch in a reference frame. Subsequently the characters among a sequence are automatically matched and registered. Occlusion are then detected and located in certain components segmented from the character. Two different approaches are applied to color regions in components without and with occlusion respectively. The approach has been tested for coloring a practical animation sequence and achieved high coloring accuracy, showing its applicability in commercial animation production.
1. Introduction
Inking/coloring of the individual animated characters for every frame is one of the most time-consuming and labor-intensive procedures in cel animation production. Many systems and some algorithms have been proposed to assist the coloring of cel animation production. However, these systems or algorithms have significant limitation, in terms of not being able to automatically establish the correct region correspondences when there is a big change or occlusion of the character, as reviewed in [1]. In summary, computer-assisted auto coloring (CAAC) remains a tough issue in research.
To detect and handle occlusion, a novel character registration method was proposed in our previous work [2], as the first attempt for auto coloring 2D characters for some special cases. As indicated in [2], the auto coloring approach may be challenged if the character's position changes greatly in a sequence of frames. In this paper, a relocation method is proposed to enhance the character registration approach in [2], making the auto coloring approach more robust.
The rest of the paper is organized as follows. First, the enhanced character registration method is introduced in detail. Subsequently, our occlusion detection and auto coloring process is introduced. Experiments are then designed to test the algorithm and results analyzed. Conclusion and future work can be found at the end of this paper.
2. Character Registration
2.1. Skeleton Extraction
A skeleton is a useful shape abstraction that captures the essential topology of an object in both two and three dimensions. It represents a thinned version of the original object, which still retains the shape properties of the original object [3]. The skeleton of a 2D character can preserve the most essential geometric and topological information of it, which makes it suitable for our character registration purpose.
In practical animation production, some characters are complex featuring sharp convex or concave details in their outline, which introduce redundant branches when the skeleton extraction approach based on active contours [4, 5] is applied. As shown in Figure 1, the branches in red and green do not relate to the physical structure of the character. These are viewed as “noise” in this paper. The short redundant branches in Figure 1 can be easily pruned based on a threshold, but it is difficult to remove the long ones. To reduce the noise, Gaussian filtering is applied to smooth the outline and remove the details. The Gaussian function in 2D form is The smoothed outlines and corresponding skeletons with different are illustrated in Figure 2. The is empirically selected as 3 for the examples used in this paper.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
After the outline is smoothed, the skeleton of each character is extracted using active contours approach [4, 5], and further thinned under SUSAN thinning rules [6], and pruned. Finally, it is segmented into L-branches and J-branches as introduced in [2]. Topology graphs of each character is also extracted using the approach introduced in [5]. Figure 3 shows the extracted skeleton and topology graph for the character in Frame 4 of the first test used in the paper. Junction nodes are illustrated in red, leaf nodes in blue, L-branches in green, and J-branches in black, respectively.
(a)
(b)
2.2. Skeleton Relocation
Animation is an art of capturing a series of individual movements, whether on film or in digital form, and replaying them in rapid succession to give the illusion of movement [7]. Accordingly, even in two successive frames, the positions of the two characters may differ much. Besides affine transforms which are often used for depicting the character's global motion, deformation of the character parts usually exists for expressing the local motion like the movements of articulated parts. With these uncertainties, it is hard to find an accurate transform to locate the two characters in the same position with the most similarity. When drawing inbetweens, animators locate characters of two key frames by overlapping the most similar parts of the character in the same position. Mimicking this method, two skeletons can be relocated according to a stable branch which does not change much among the whole animation sequence.
The stable branch of each frame can be interactively selected by the user. Alternatively, user only needs to indicate the stable branch in the first reference frame to be registered. Given a reference frame and a target frame, a prediction and relocation method is proposed as follows to reduce user intervention.
Given a population of random vectors , the principal components analysis (PCA) is defined as where is the matrix whose rows are formed from the eigenvectors of 's covariance matrix , is 's mean vector [8].
(i)Compute the PCA transforming matrices and for the stable branch in the reference frame, where and , respectively, correspond to and in (2).(ii)Transform the skeleton in the reference frame to the transforming coordinate system according to and : (iii)For each branch in the target frame, compute the dissimilarity value .(a) Compute the PCA transforming matrices and for a branch in the target frame, where and , respectively correspond to and in (2).(b) Transform the skeleton in the target frame to the transforming coordinate system according to and : (c) Compute the Hausdorff distance between and : (d) Compute the Hausdorff distance between and .(e)The dissimilarity value between branches and is the minimum of and : (iv) If is the minimum, and are corresponded, the skeleton in the target frame is transformed and normalized to be or according to and . The transforming coordinate system for and or is treated as the global coordinate system .
Figure 4 shows the relocated skeletons and frames. Frames 1 and 2 are represented in blue and red, respectively.
(a)
(b)
(c)
(d)
2.3. Skeleton Registration
General topology models are predefined based on the character's physical structure. Figure 5 illustrates some common topology models, which represent the most essential topological information of the characters.
(a) Human and human-like characters
(b) Tailed human-like animals
(c) quadruped
To assure registration accuracy, a frame is selected as the first reference frame from the sequence to be painted, which contains a stable branch as introduced in Section 2.2 and the maximum number of branches among the sequence. The skeleton and topology graph of the character in the selected frame is then registered to the general model, as illustrated in Figure 6. The topology graph is adjusted accordingly.
(a)
(b)
For characters in the other frames in the sequence, their skeletons and branches are registered based on the branch correspondences established by skeleton matching introduced in the next section.
2.4. Skeleton Matching
As introduced in [2], our skeleton matching algorithm is based on both geometric and topological information of the skeleton, containing the following two steps.
2.4.1. Geometric Matching
Global dissimilarity due to the motion of branches is represented by the global Hausdorff distance and computed using the global coordinate system of the two frames.
The local deformation of each branch is represented by the local Hausdorff distance and computed using the local coordinate system of the two frames established based on PCA [2].
With the global and local Hausdorff distances between two branches, the dissimilarity value is defined as
Each branch in the reference frame is matched with all branches in the target frame first. If is the minimum among all dissimilarity values, a matching is obtained. Subsequently, each branch obtains its best-matched branch in the reference frame. If a bidirectional matching is achieved, , and are corresponded , and is registered as the same branch with in the general topology model. The unregistered branches will be readjusted in later process.
2.4.2. Topological Readjustment
After geometric matching, some branches with big motion or deformation may be left unregistered. To match and register these branches and detect occlusion, a topological readjustment method is applied.
For each selected branch in the target frame, its merging candidates are obtained, and it is merged with the one related most closely [2]. If no merging candidate is found, geometric matching is applied to those unregistered branches in 's subgraph.
2.5. Component Segmentation
After skeleton registration, characters are segmented into several components corresponding to the branches in skeleton and topology graph, based on the explained area of each branch [2].
The segmented and registered components for Frames 4 and 5 are illustrated in Figure 7, where corresponding components are in the same color.
(a)
(b)
3. Enhanced Auto Coloring
3.1. Occlusion Detection
To detect and locate occlusion in components, a method based on the variation of region areas is advanced in [2].
(i)Each region in the reference frame is quantized and coded as an English character based on its area.(ii)All the regions in the target frame are quantized and coded based on the scale points computed in the reference frame.(iii) Regions in a component in the reference frame are coded as a string and compared with the string of its corresponding component in the target frame, and the least conversion cost between them is computed as .(iv) Occlusion is located in if is bigger than an empirically defined threshold.
3.2. Auto Coloring
With occlusion detection, regions can be divided into two categories: the first category contains all regions which are in the components without occlusion, and the second consists of those in the components with occlusion. The former is supposed to be more stable than the latter and different matching methods are applied to the two categories.
The coloring process consists of three steps, and the coloring result after each step is illustrated in Figure 8.
(a) Frame 4
(b) Frame 5 (Step 1)
(c) Frame 5 (Step 2)
(d) Frame 5 (Step 3)
(i) For the first category, the hierarchical feature-based region matching approach as proposed in [1, 9] is applied.(ii) For the second category, the continuity of some region contours is broken because of occlusion. Hence, features such as area, curve length, character points, and relations with neighboring regions change greatly. But the rest of regions still have relatively stable features of area, curve length and character points. So a matching method similar to that for the first category is applied to match these regions. Each region is matched with those in corresponding components in the reference frame. The only difference is that the feature of relations with neighboring regions is ignored, so that the coded character string [1] is only composed of character points.(iii) Finally, each of the remaining regions in the relocated target frame inherits the color of the region that it overlaps mostly in the reference frame. If no such region is obtained, which means that the region fully overlaps the background, it is painted in the color which fills the majority of the component (major color) that it belongs to.
After skeleton registration for the first reference frame, the prior and subsequent uncolored frames are selected as new target frames iteratively. The nearest colored frame which contains a stable branch is selected as the target frame's reference frame.
4. Results and Analysis
To test the possibility of applying our approach in practical animation production, we use a Japanese-style sequence in practical production of the trailer of “Justeen” animation. The 8 frames as shown in Figure 9 compose of a cut in the trailer, showing a boy jumping down into the control room of his robot. It can be noticed that occlusion arises due to the motion of the character's arms. To minimize the information loss due to occlusion, we select Frame 4 as the first reference frame. Then the prior and subsequent frames are colored one by one according to the color information of its next and prior frame, respectively. From the results we can see that most of the regions are correctly colored. The regions wrongly colored are mainly because no corresponding regions in the reference frame can be obtained due to the occlusion or information loss. Figure 10 illustrates the main errors among the sequence and the correct results. As illustrated in Figure 10(a), the right arm in Frame 5 changes greatly compared with the one in Frame 4, and the occluded part of the bracelet in Frame 4 is visible in Frame 5. Accordingly, the regions indicated in blue circles are wrongly matched as no correct correspondences exist in Frame 4. With more reference frames (which contain the corresponding regions) provided, this kind of error can be avoided. In Figure 10(c), the regions in blue circles in Frame 5 are created due to the occlusion, and they inherit the color of the regions they overlap mostly in Frame 4 as no corresponding regions are obtained. In Figure 10(e), the region in blue circle in Frame 7 is created as the hair is swaying to the position overlapping with the eyebrow. It is thus wrongly matched to a hair branch as they have great similarity. These errors can be solved if the hair is drawn in a separate layer. The mouth in Frame 1 is widely open so that the tongue appears only in this frame. It is not matched to any region, and thus inherits the color of the face in Frame 2. This kind of error due to information loss is hard to be avoided. In summary, the coloring accuracy for the total 7 uncolored frames is over , which shows that our approach is applicable to practical animation production.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
Figure 11 shows the other test, with the first frame selected as the first reference frames. The character is simple, but it is noticeable that large motions and deformations exist among the sequence. With the relocation method proposed in this paper, skeletons are accurately matched, ensuring the high coloring accuracy for the test.
(a)
(b)
(c)
(d)
5. Conclusion and Future Work
In this paper, our previous work on character registration is enhanced with a refined skeleton extraction procedure and a novel character relocation method. With the enhanced character registration, occlusion can be correctly detected and located in components segmented from the character. Subsequently, two different matching methods are applied to regions in components without and with occlusion, respectively. A practical animation sequence from a cut of “Justeen” animation trailer is applied to validate the approach. A high coloring accuracy is achieved. It shows that the proposed auto coloring approach with enhanced character registration is applicable to practical animation production.
Nevertheless, there are limitations. The proposed approach requires that each frame has a stable branch, which is not always true among a long sequence. For example, the cut we used in Test 1 contains 12 frames in total, and the first 4 frames are illustrated in Figure 12. Due to the information loss, no stable branch exists in these frames. So they cannot be automatically colored using our approach. Future research work will be focusing on the auto coloring for these kinds of special cases and new relocation methods.
(a)
(b)
(c)
(d)
Acknowledgment
This work has been partially supported by the Science and Engineering Research Council (SERC) Grant no. 052 015 0024 Ref: 44, which is awarded by the Agency for Science and Technology Research (A*STAR) and administered through the Singapore National Grid Office.