An enhanced character registration method is proposed in
this paper to assist the auto coloring for 2D animation characters. After skeletons are extracted, the skeleton of the
character in a target frame is relocated based on a stable branch in a reference frame. Subsequently the characters among
a sequence are automatically matched and
registered. Occlusion are then detected and located in certain components segmented from the character.
Two different approaches are applied to color regions in components
without and with occlusion respectively. The approach has
been tested for coloring a practical animation sequence and
achieved high coloring accuracy, showing its applicability in
commercial animation production.
1. Introduction
Inking/coloring of the individual animated characters
for every frame is one of the most time-consuming and labor-intensive
procedures in cel animation production. Many systems and some algorithms have
been proposed to assist the coloring of cel animation production. However,
these systems or algorithms have significant limitation, in terms of not being
able to automatically establish the correct region correspondences when there
is a big change or occlusion of the character, as reviewed in [1]. In summary, computer-assisted auto coloring
(CAAC) remains a tough issue in research.
To detect and handle occlusion, a novel character
registration method was proposed in our previous work [2], as the first attempt for auto coloring 2D characters
for some special cases. As indicated in [2], the auto coloring approach may be challenged if the
character's position changes greatly in a sequence of frames. In this paper, a
relocation method is proposed to enhance the character registration approach in [2], making the auto coloring approach more robust.
The rest of the paper is organized as follows. First,
the enhanced character registration method is introduced in detail.
Subsequently, our occlusion detection and auto coloring process is introduced.
Experiments are then designed to test the algorithm and results analyzed.
Conclusion and future work can be found at the end of this paper.
2. Character Registration
2.1. Skeleton Extraction
A skeleton is a useful shape abstraction that captures
the essential topology of an object in both two and three dimensions. It
represents a thinned version of the original object, which still retains the
shape properties of the original object [3]. The skeleton of a 2D character can preserve the most
essential geometric and topological information of it, which makes it suitable
for our character registration purpose.
In practical animation production, some characters are
complex featuring sharp convex or concave details in their outline, which
introduce redundant branches when the skeleton extraction approach based on
active contours [4, 5] is applied. As shown in Figure 1, the branches in red and green do not relate to the
physical structure of the character. These are viewed as “noise” in this
paper. The short redundant branches in Figure 1 can be easily pruned based on a threshold, but it is
difficult to remove the long ones. To reduce the noise, Gaussian filtering is
applied to smooth the outline and remove the details. The Gaussian function in
2D form is
The smoothed
outlines and corresponding skeletons with different are illustrated
in Figure 2. The is empirically
selected as 3 for the examples used in this paper.
Figure 1: Extracted skeletons.
Figure 2: Skeleton extraction after Gaussian smoothing.
After the outline is smoothed, the skeleton of each
character is extracted using active contours approach [4, 5], and further thinned under SUSAN thinning rules [6], and pruned. Finally, it is segmented into L-branches and J-branches as introduced in [2]. Topology graphs of each character is also extracted
using the approach introduced in [5]. Figure 3 shows the extracted skeleton and topology graph for
the character in Frame 4 of the first test used in the paper. Junction nodes
are illustrated in red, leaf nodes in blue, L-branches in green, and J-branches
in black, respectively.
Figure 3: Extracted skeleton and topology graph.
2.2. Skeleton Relocation
Animation is an art of capturing a series of
individual movements, whether on film or in digital form, and replaying them in
rapid succession to give the illusion of movement [7]. Accordingly, even in two successive frames, the
positions of the two characters may differ much. Besides affine transforms
which are often used for depicting the character's global motion, deformation
of the character parts usually exists for expressing the local motion like the
movements of articulated parts. With these uncertainties, it is hard to find an
accurate transform to locate the two characters in the same position with the
most similarity. When drawing inbetweens, animators locate characters of two
key frames by overlapping the most similar parts of the character in the same
position. Mimicking this method, two skeletons can be relocated according to a stable
branch which does not change much among the whole animation sequence.
The stable branch of each frame can be interactively
selected by the user. Alternatively, user only needs to indicate the stable
branch in the first reference frame to be registered. Given a reference frame
and a target frame, a prediction and relocation method is proposed as follows
to reduce user intervention.
Given a population of random vectors , the principal components analysis (PCA) is defined
as
where is the matrix
whose rows are formed from the eigenvectors of 's covariance
matrix , is 's mean vector [8].
(i)Compute the PCA
transforming matrices and for the stable
branch in the
reference frame, where and , respectively, correspond to and in (2).(ii)Transform the
skeleton in the
reference frame to the transforming coordinate system according to and :
(iii)For each branch in the target
frame, compute the dissimilarity value .(a)
Compute the PCA transforming matrices and for a branch in the target
frame, where and , respectively correspond to and in (2).(b)
Transform the skeleton in the target
frame to the transforming coordinate system according to and :
(c)
Compute the Hausdorff distance between and :
(d)
Compute the Hausdorff distance between and .(e)The
dissimilarity value between
branches and is the minimum
of and :
(iv) If is the minimum, and are
corresponded, the skeleton in the target
frame is transformed and normalized to be or according to and . The transforming coordinate system for and or is treated as
the global coordinate system .
Figure 4 shows the relocated skeletons and frames. Frames 1
and 2 are represented in blue and red, respectively.
Figure 4: Skeleton relocation.
2.3. Skeleton Registration
General topology models are predefined based on the
character's physical structure. Figure 5 illustrates some common topology models, which
represent the most essential topological information of the characters.
Figure 5: General topology model.
To assure registration accuracy, a frame is selected
as the first reference frame from the sequence to be painted, which contains a
stable branch as introduced in Section 2.2 and the maximum number of branches among the
sequence. The skeleton and topology graph of the character in the selected
frame is then registered to the general model, as illustrated in Figure 6. The topology graph is adjusted accordingly.
Figure 6: Skeleton registration.
For characters in the other frames in the sequence,
their skeletons and branches are registered based on the branch correspondences
established by skeleton matching introduced in the next section.
2.4. Skeleton Matching
As introduced in [2], our skeleton matching algorithm is based on both
geometric and topological information of the skeleton, containing the following
two steps.
2.4.1. Geometric Matching
Global dissimilarity due to the motion of branches is
represented by the global Hausdorff distance and computed
using the global coordinate system of the two
frames.
The local deformation of each branch is represented by
the local Hausdorff distance and computed
using the local coordinate system of the two
frames established based on PCA [2].
With the global and local Hausdorff distances between
two branches, the dissimilarity value is defined as
Each branch in the
reference frame is matched with all branches in the target frame first. If is the minimum
among all dissimilarity values, a matching is obtained.
Subsequently, each branch obtains its
best-matched branch in the reference frame. If a bidirectional matching is
achieved, , and are
corresponded , and is registered
as the same branch with in the general
topology model. The unregistered branches will be readjusted in later process.
2.4.2. Topological Readjustment
After geometric matching, some branches with big
motion or deformation may be left unregistered. To match and register these
branches and detect occlusion, a topological readjustment method is applied.
For each selected branch in the target
frame, its merging candidates are obtained, and it is merged with the one
related most closely [2]. If no merging candidate is found, geometric matching
is applied to those unregistered branches in 's subgraph.
2.5. Component Segmentation
After skeleton registration, characters are segmented
into several components corresponding to the branches in skeleton and topology
graph, based on the explained area of each branch [2].
The segmented and registered components for Frames 4
and 5 are illustrated in Figure 7, where corresponding components are in the same
color.
Figure 7: Segmentation result.
3. Enhanced Auto Coloring
3.1. Occlusion Detection
To detect and locate occlusion in components, a method
based on the variation of region areas is advanced in [2].
(i)Each region in
the reference frame is quantized and coded as an English character based on its
area.(ii)All the regions
in the target frame are quantized and coded based on the scale points computed
in the reference frame.(iii) Regions in a
component in the
reference frame are coded as a string and compared
with the string of its
corresponding component in the target
frame, and the least conversion cost between them is computed as .(iv) Occlusion is
located in if is bigger than
an empirically defined threshold.
3.2. Auto Coloring
With occlusion detection, regions can be divided into
two categories: the first category contains all regions which are in the
components without occlusion, and the second consists of those in the
components with occlusion. The former is supposed to be more stable than the
latter and different matching methods are applied to the two categories.
The coloring process consists of three steps, and the
coloring result after each step is illustrated in Figure 8.
Figure 8: Auto coloring process.
(i) For the first
category, the hierarchical feature-based region matching approach as proposed
in [1, 9] is applied.(ii) For the second
category, the continuity of some region contours is broken because of
occlusion. Hence, features such as area, curve length, character points, and
relations with neighboring regions change greatly. But the rest of regions
still have relatively stable features of area, curve length and character
points. So a matching method similar to that for the first category is applied
to match these regions. Each region is matched with those in corresponding
components in the reference frame. The only difference is that the feature of
relations with neighboring regions is ignored, so that the coded character
string [1] is only composed of character points.(iii) Finally, each
of the remaining regions in the relocated target frame inherits the color of
the region that it overlaps mostly in the reference frame. If no such region is
obtained, which means that the region fully overlaps the background, it is
painted in the color which fills the majority of the component (major color)
that it belongs to.
After skeleton registration for the first reference
frame, the prior and subsequent uncolored frames are selected as new target
frames iteratively. The nearest colored frame which contains a stable branch is
selected as the target frame's reference frame.
4. Results and Analysis
To test the possibility of applying our approach in
practical animation production, we use a Japanese-style sequence in practical
production of the trailer of “Justeen” animation. The 8 frames as shown
in Figure 9 compose of a cut in the
trailer, showing a boy jumping down into the control room of his robot. It can
be noticed that occlusion arises due to the motion of the character's arms. To
minimize the information loss due to occlusion, we select Frame 4 as the first
reference frame. Then the prior and subsequent frames are colored one by one
according to the color information of its next and prior frame, respectively.
From the results we can see that most of the regions are correctly colored. The
regions wrongly colored are mainly because no corresponding regions in the
reference frame can be obtained due to the occlusion or information loss.
Figure 10 illustrates the main errors among the sequence and
the correct results. As illustrated in Figure 10(a), the right arm in Frame 5 changes greatly
compared with the one in Frame 4, and the occluded part of the bracelet in
Frame 4 is visible in Frame 5. Accordingly, the regions indicated in blue
circles are wrongly matched as no correct correspondences exist in Frame 4.
With more reference frames (which contain the corresponding regions) provided,
this kind of error can be avoided. In Figure 10(c), the regions in blue circles
in Frame 5 are created due to the occlusion, and they inherit the color of the
regions they overlap mostly in Frame 4 as no corresponding regions are
obtained. In Figure 10(e), the region in blue circle in Frame 7 is created as
the hair is swaying to the position overlapping with the eyebrow. It is thus
wrongly matched to a hair branch as they have great similarity. These errors
can be solved if the hair is drawn in a separate layer. The mouth in Frame 1 is
widely open so that the tongue appears only in this frame. It is not matched to
any region, and thus inherits the color of the face in Frame 2. This kind of
error due to information loss is hard to be avoided. In summary, the coloring
accuracy for the total 7 uncolored frames is over , which shows that our approach is applicable to
practical animation production.
Figure 9: Test 1 (resolution: pixels)
(courtesy of Anime International Co., Inc., Japan).
Figure 10: Error Analysis for Test 1.
Figure 11 shows the other test, with the first frame selected
as the first reference frames. The character is simple, but it is noticeable
that large motions and deformations exist among the sequence. With the relocation
method proposed in this paper, skeletons are accurately matched, ensuring the
high coloring accuracy for the test.
Figure 11: Test 2 (resolution: pixels).
5. Conclusion and Future Work
In this paper, our previous work on character
registration is enhanced with a refined skeleton extraction procedure and a
novel character relocation method. With the enhanced character registration,
occlusion can be correctly detected and located in components segmented from
the character. Subsequently, two different matching methods are applied to regions
in components without and with occlusion, respectively. A practical animation
sequence from a cut of “Justeen” animation trailer is applied to validate
the approach. A high coloring accuracy is achieved. It shows that the proposed
auto coloring approach with enhanced character registration is applicable to
practical animation production.
Nevertheless, there are limitations. The proposed
approach requires that each frame has a stable branch, which is not always true
among a long sequence. For example, the cut we used in Test 1 contains 12
frames in total, and the first 4 frames are illustrated in Figure 12. Due to the information loss, no stable branch exists
in these frames. So they cannot be automatically colored using our approach.
Future research work will be focusing on the auto coloring for these kinds of
special cases and new relocation methods.
Figure 12: Other frames in the cut used in Test 1 (courtesy of Anime International Co., Inc., Japan).
Acknowledgment
This work has been partially supported by the Science and Engineering Research Council (SERC) Grant no. 052 015 0024 Ref: 44, which is awarded by the Agency for Science and Technology Research (A*STAR) and administered through the Singapore National Grid Office.