[Paper Review] Alive Caricature from 2D to 3D
Q. Wu, J. Zhang, Y.-K. Lai, J. Zheng, and J. Cai, “Alive Caricature from 2D to 3D,” CVPR, 2018, pp. 7336–7345.
University of Science and Technology of China, China
Cardiff University, UK
Nanyang Technological University, Singapore
Overview
This paper proposes a 3D reconstruction method from an input 2D caricature image which has difficulties capturing such exaggerated styles that are quite out of boundaries among regular 3D mesh dataset. I believe the key contribution of this work lies in the new local deformation gradient that extends to the deformation space. This enabled the generation of natural-looking face mesh compared to other works with well-fitted landmarks.
Goal
Generation of 3D caricatures from 2D caricature images while sustaining face constraint with different exaggeration styles
Contributions
- Proposed a new intrinsic deformation representation that uses local deformation gradients
- Formulated an optimization problem based on landmark constraints to satisfy face constraint and exaggeration style
Related Work
- 3D face reconstruction
- 2D caricature generation
- 3D caricature generation
Methods
Intrinsic Deformation Representation
Defined deformation gradient that represents the energy of difference between two 3D face models by calculating the difference of a vertex sticking out or sink in with the affine transformation matrix that minimizes the energy
Defined deformation representation derived from the affine transformation matrix which is the set of rotation and scaling matrices from all vertices
Constructed deformation space with all of the deformation representations obtained from a chosen reference model by a linear combination of the basis deformations
With the reference model at the origin and two basis deformations at (1, 0) and (0, 1), we can generate new deformed meshes in the deformation space using different weights w
Then, with the deformation energy as follows:
By minimizing this energy, we are able to determine the position of each vertex p’ i ∈ R³ on the deformed mesh given weights w, or obtain the combination weights w given the deformed mesh P′
Parameter Optimization
Reconstructing the 3D model from a 2D image is the inverse process of observing a 3D object by projecting it to an imaging plane
The landmark fitting loss is defined as below:
Defined an optimization problem to generate a 3D caricature model that looks real and matches with the exaggerated 2D landmarks
To solve the above optimization problem, we initialize w by simply letting all weights be zero and then alternately solve for P′ and w using the following P′ -step and w step. The process continues until convergence or reaching the maximum number of iterations.
At each P’ or w step, the other variable is fixed. Then, the optimal parameters P’ and w are obtained after all the iterations which are fitted to an input 2D caricature image.
Results
Best quality in terms of landmarks alignment and shape deformation matching with different exaggeration styles
Quantitative comparison in the average fitting error of landmarks also showed a big difference over test data
Discussion
A unique advantage in using just normal face models to create 3D caricature models
Proposed new deformation representation which has the capacity of modeling caricature faces in a non-linear and more natural way
