3D Face Morphable Model Explained

In computer vision and computer graphics, the 3D Face Morphable Model (3DFMM) is a generative technique for modeling textured 3D faces. The generation of new faces is based on a pre-existing database of example faces acquired through a 3D scanning procedure. All these faces are in dense point-to-point correspondence, which enables the generation of a new realistic face (morph) by combining the acquired faces. A new 3D face can be inferred from one or multiple existing images of a face or by arbitrarily combining the example faces. 3DFMM provides a way to represent face shape and texture disentangled from external factors, such as camera parameters and illumination.^[1]

The 3D Morphable Model (3DMM) is a general framework that has been applied to various objects other than faces, e.g., the whole human body,^{[2] [3]} specific body parts,^{[4] [5]} and animals.^[6] 3DMMs were first developed to solve vision tasks by representing objects in terms of the prior knowledge that can be gathered from that object class. The prior knowledge is statistically extracted from a database of 3D examples and used as a basis to represent or generate new plausible objects of that class. Its effectiveness lies in the ability to efficiently encode this prior information, enabling the solution of otherwise ill-posed problems (such as single-view 3D object reconstruction).

Historically, face models have been the first example of morphable models, and the field of 3DFMM remains a very active field of research as today. In fact, 3DFMM has been successfully employed in face recognition,^[7] entertainment industry (gaming and extended reality,^{[8] [9]} virtual try on,^[10] face replacement,^[11] face reenactment^[12]), digital forensics,^[13] and medical applications.^[14]

Modeling

In general, 3D faces can be modeled by three variational components extracted from the face dataset:

• shape model - model of the distribution of geometrical shape across different subjects
• expression model - model of the distribution of geometrical shape across different facial expressions
• appearance model - model of the distribution of surface textures (color and illumination)

Shape modeling

The 3DFMM uses statistical analysis to define a statistical shape space, a vectorial space equipped with a probability distribution, or prior.^[15] To extract the prior from the example dataset, all the 3D faces must be in a dense point-to-point correspondence. This means that each point has the same semantical meaning on each face (e.g., nose tip, edge of the eye). In this way, by fixing a point, we can, for example, derive the probability distribution of the texture's red channel values over all the faces. A face shape $S$ of $n$ vertices is defined as the vector containing the 3D coordinates of the

n vertices in a specified order, that is S \in R ^{3n} . A shape space is regarded as a $d$-dimensional space that generates plausible 3D faces by performing a lower-dimensional ($d\; \backslash ll\; n$) parametrization of the database. Thus, a shape $S$ can be represented through a generator function $\backslash mathbf:\; \backslash mathbb^d\; \backslash rightarrow\; \backslash mathbb^$ by the parameters w \in R ^{d} , $\backslash mathbf(\backslash mathbf)\; =\; S\; \backslash in\; \backslash mathbb^$. The most common statistical technique used in 3DFMM to generate the shape space is Principal Component Analysis (PCA), that generates a basis that maximizes the variance of the data. Performing PCA, the generator function is linear and defined as $$\backslash mathbf(\backslash mathbf)\; =\; \backslash mathbf\; +\; \backslash mathbf\backslash mathbf$$where $\backslash mathbf$ is the mean over the training data and E \in R ^{3n} is the matrix that contains the $d$ most dominant eigenvectors.

Using a unique generator function for the whole face leads to the imperfect representation of finer details. A solution is to use local models of the face by segmenting important parts such as the eyes, mouth, and nose.^[16]

Expression modeling

The modeling of the expression is performed by explicitly subdividing the representation of the identity from the facial expression. Depending on how identity and expression are combined, these methods can be classified as additive, multiplicative, and nonlinear.

The additive model is defined as a linear model and the expression is an additive offset with respect to the identity $$\backslash mathbf(\backslash mathbf^s,\; \backslash mathbf^w)\; =\; \backslash mathbf\; +\; \backslash mathbf^s\backslash mathbf^s\; +\; \backslash mathbf^e\backslash mathbf^e$$where $\backslash mathbf^s$,$\backslash mathbf^$ and $\backslash mathbf^$,$\backslash mathbf^e$ are the matrices basis and the coefficients vectors of the shape and expression space, respectively. With this model, given the 3D shape of a subject in a neutral expression $\backslash mathbf\_$ and in a particular expression

c ^{exp} , we can transfer the expression to a different subject by adding the offset \Delta _{c} = c ^{exp} - c ^{ne} . Two PCAs can be performed to learn two different spaces for shape and expression.^[17]

In a multiplicative model, shape and expression can be combined in different ways. For example, by exploiting $d\_e$ operators

T _{j:} R ^{3n} → R ^{3n} that transform a neutral expression into a target blendshape we can write$$\backslash mathbf(\backslash mathbf^s,\; \backslash mathbf^e)\; =\; \backslash sum\_^w\_j^e\backslash mathbf\_j(\backslash mathbf(\backslash mathbf^s)\; +\; \backslash mathbf^s)\; +\; \backslash mathbf\_j^e$$where \delta ^{s} and $\backslash mathbf^s\_j$ are vectors to correct to the target expression.^[18]

The nonlinear model uses nonlinear transformations to represent an expression.^{[19] [20]}

Appearance modeling

The color information id often associated to each vertex of a 3D shape. This one-to-one correspondence allows us to represent appearance analogously to the linear shape model $$\backslash mathbf(\backslash mathbf^t)\; =\; \backslash mathbf\; +\; \backslash mathbf^\backslash mathbf^$$where

w ^{t} is the coefficients vector defined over the basis matrix E ^{t} . PCA can be again be used to learn the appearance space.

History

Facial recognition can be considered the field that originated the concepts that later on converged into the formalization of the morphable models. The eigenface approach used in face recognition represented faces in a vector space and used principal component analysis to identify the main modes of variation. However, this method had limitations: it was constrained to fixed poses and illumination and lacked an effective representation of shape differences. As a result, changes in the eigenvectors did not accurately represent shifts in facial structures but caused structures to fade in and out. To address these limitations, researchers added an eigendecomposition of 2D shape variations between faces. The original eigenface approach aligned images based on a single point, while new methods established correspondences on many points. Landmark-based face warping was introduced by Craw and Cameron (1991),^[21] and the first statistical shape model, Active Shape Model, was proposed by Cootes et al. (1995).^[22] This model used shape alone, but Active Appearance Model by Cootes et al. (1998)^[23] combined shape and appearance. Since these 2D methods were effective only for fixed poses and illumination, they were extended by Vetter and Poggio (1997)^[24] to handle more diverse settings. Even though separating shape and texture was effective for face representation, handling pose and illumination variations required many separate models. On the other hand, advances in 3D computer graphics showed that simulating pose and illumination variations was straightforward. The combination of graphics methods with face modeling led to the first formulation of 3DMMs by Blanz and Vetter (1999). The analysis-by-synthesis approach enabled the mapping of the 3D and 2D domains and a new representation of 3D shape and appearance. Their work is the first to introduce a statistical model for faces that enabled 3D reconstruction from 2D images and a parametric face space for controlled manipulation.

In the original definition of Blanz and Vetter, the shape of a face is represented as the vector

S = (X _{1,} Y _{1,} Z _{1,} ..., X _{n,} Y _{n,} Z ^{T} _{n)} \in R ^{3n} that contains the 3D coordinates of the n vertices. Similarly, the texture is represented as a vector T = (R _{1,} G _{1,} B _{1,} ..., R _{n,} G _{n,} B ^{T} _{n)} \in R ^{3n} that contains the three RGB color channels associated with each corresponding vertex. Due to the full correspondence between exemplar 3D faces, new shapes S _{models} and textures T _{models} can be defined as a linear combination of the $m$ example faces:$$\backslash mathbf\_\; =\backslash sum\_^m\; a\_i\; \backslash mathbf\_i\; \backslash qquad\; \backslash mathbf\_\; =\backslash sum\_^m\; b\_i\; \backslash mathbf\_i\; \backslash qquad\; \backslash text\; \backslash ;\; \backslash sum\_^m\; a\_i\; =\; \backslash sum\_^m\; b\_i\; =\; 1$$Thus, a new face shape and texture is parametrized by the shape a = (a _{1,} a _{2,...,} a ^{T} _{m)} and texture coefficients b = (b _{1,} b _{2,...,} b ^{T} _{m)} . To extract the statistics from the dataset, they performed PCA to generate the shape space of dimension to $d$ and used a linear model for shape and appearance modeling. In this case, a new model can be generated in the orthogonal basis using the shape and the texture eigenvector $s\_i$ and $t\_i$, respectively:S _{model} = \barS + \sum ^{m} _{i =1} a _{i} s _{i} T _{model} = \barT + \sum ^{m} _{i =1} b _{i} t _{i}

where

\bar{S } and $\backslash mathbf$ are the mean shape and texture of the dataset.

Publicly available databases

In the following table, we list the publicly available databases of human faces that can be used for the 3DFMM.

Publicly available databases of human faces!!Year!Geometry!Appearance!Size!Download!Institution

Basel Face Model 2009^[25]	2009	shape	per-vertex	100 individuals in neutral expression	Link	University of Basel
FaceWarehouse^[26]	2014	shape, expression	-	150 individuals in 20 different expressions	Link	Zhejiang University
Large Scale Facial Model (LSFM)^[27]	2016	shape	-	9,663 individuals	Link	Imperial College London
Surrey Face Model^[28]	2016	shape, expression (multi-resolution)	per-vertex	169 individuals	Link	University of Surrey
Basel Face Model 2017^[29]	2017	shape, expression	per-vertex	200 individuals and 160 expression scans	Link	University of Basel
Liverpool-York Head Model (LYHM)^[30]	2017	shape (full head - no hair, no eyes)	per-vertex	1,212 individuals	Link	University of York, Alder Hey Hospital
Faces Learned with an Articulated Model and Expressions (FLAME)^[31]	2017	shape (full head - no hair), expression, head pose	texture	3,800 individuals for shape, 8,000 for head pose, 21,000 for expression	Link	University of Southern California, Max Planck Institute for Intelligent Systems
Convolutional Mesh Autoencoder (CoMA)^[32]	2018	shape (full head - no hair), expression	-	2 individuals in 12 extreme expressions	Link	Max Planck Institute for Intelligent Systems
Morphable Face Albedo Model^[33]	2020	-	per-vertex diffuse and specular albedo	73 individuals	Link	University of York
FaceVerse^[34]	2022	shape	texture	128 individuals in 21 different expressions	Link	Tsinghua University

See also

• Eigenface
• 3D modeling
• Principal component analysis
• Statistical shape analysis

External links

• Web site: 3D Morphable Models . 2024-07-08.
• Web site: Curated List of 3D Morphable Model Software and Data . . 2024-07-11.
• Web site: Tutorial on 3D Morphable Models @ Symposium on Geometry Processing 2022 . . 28 June 2022 . 2024-07-12.
• Web site: What is a Linear 3D Morphable Face Model? . . 5 April 2020 . 2024-07-12.

Notes and References

1. Egger . Bernhard . Smith . William A. P. . Tewari . Ayush . Wuhrer . Stefanie . Zollhoefer . Michael . Beeler . Thabo . Bernard . Florian . Bolkart . Timo . Kortylewski . Adam . Romdhani . Sami . Theobalt . Christian . Blanz . Volker . Vetter . Thomas . 31 October 2020 . 3D Morphable Face Models—Past, Present, and Future . ACM Transactions on Graphics . 39 . 5 . 1–38 . 10.1145/3395208. 21.11116/0000-0007-1CF5-6 . free .
2. Allen . Brett . Curless . Brian . Popović . Zoran . 2003-07-01 . The space of human body shapes: reconstruction and parameterization from range scans . ACM Trans. Graph. . 22 . 3 . 587–594 . 10.1145/882262.882311 . 0730-0301.
3. Loper . Matthew . Mahmood . Naureen . Romero . Javier . Pons-Moll . Gerard . Black . Michael J. . 2015-10-26 . SMPL: a skinned multi-person linear model . ACM Trans. Graph. . 34 . 6 . 248:1–248:16 . 10.1145/2816795.2818013 . 0730-0301.
4. Khamis . Sameh . Taylor . Jonathan . Shotton . Jamie . Keskin . Cem . Izadi . Shahram . Fitzgibbon . Andrew . 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . June 2015 . Learning an efficient model of hand shape variation from depth images . https://ieeexplore.ieee.org/document/7298869 . IEEE . 2540–2548 . 10.1109/CVPR.2015.7298869 . 978-1-4673-6964-0.
5. Dai . Hang . Pears . Nick . Smith . William . 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) . May 2018 . A Data-Augmented 3D Morphable Model of the Ear . https://ieeexplore.ieee.org/document/8373858 . IEEE . 404–408 . 10.1109/FG.2018.00065 . 978-1-5386-2335-0.
6. Sun . Yifan . Murata . Noboru . 2020 IEEE Winter Applications of Computer Vision Workshops (WACVW) . March 2020 . CAFM: A 3D Morphable Model for Animals . https://ieeexplore.ieee.org/document/9096941 . IEEE . 20–24 . 10.1109/WACVW50321.2020.9096941 . 978-1-7281-7162-3.
7. Blanz . V. . Romdhani . S. . Vetter . T. . Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition . 2002 . Face identification across different poses and illuminations with a 3D morphable model . https://ieeexplore.ieee.org/document/1004155 . IEEE . 202–207 . 10.1109/AFGR.2002.1004155 . 978-0-7695-1602-8.
8. Lombardi . Stephen . Saragih . Jason . Simon . Tomas . Sheikh . Yaser . 2018-07-30 . Deep appearance models for face rendering . ACM Trans. Graph. . 37 . 4 . 68:1–68:13 . 10.1145/3197517.3201401 . 0730-0301. 1808.00362 .
9. Book: Weise . Thibaut . Li . Hao . Van Gool . Luc . Pauly . Mark . Face/Off: Live facial puppetry . 2009-08-01 . Proceedings of the 2009 ACM SIGGRAPH/Eurographics Symposium on Computer Animation . https://doi.org/10.1145/1599470.1599472 . SCA '09 . New York, NY, USA . Association for Computing Machinery . 7–16 . 10.1145/1599470.1599472 . 978-1-60558-610-6.
10. Bronstein . Alexander M. . Bronstein . Michael M. . Kimmel . Ron . September 2007 . Calculus of Nonrigid Surfaces for Geometry and Texture Manipulation . IEEE Transactions on Visualization and Computer Graphics . 13 . 5 . 902–913 . 10.1109/TVCG.2007.1041 . 1077-2626.
11. Blanz . Volker . Scherbaum . Kristina . Vetter . Thomas . Seidel . Hans-Peter . September 2004 . Exchanging Faces in Images . Computer Graphics Forum . en . 23 . 3 . 669–676 . 10.1111/j.1467-8659.2004.00799.x . 0167-7055.
12. Thies . Justus . Zollhöfer . Michael . Nießner . Matthias . Valgaerts . Levi . Stamminger . Marc . Theobalt . Christian . 2015-11-02 . Real-time expression transfer for facial reenactment . ACM Trans. Graph. . 34 . 6 . 183:1–183:14 . 10.1145/2816795.2818056 . 0730-0301.
13. Cozzolino . Davide . Rossler . Andreas . Thies . Justus . Niesner . Matthias . Verdoliva . Luisa . 2021 IEEE/CVF International Conference on Computer Vision (ICCV) . October 2021 . ID-Reveal: Identity-aware DeepFake Video Detection . https://ieeexplore.ieee.org/document/9710044 . IEEE . 15088–15097 . 10.1109/ICCV48922.2021.01483 . 978-1-6654-2812-5. 2012.02512 .
14. Mueller . A.A. . Paysan . P. . Schumacher . R. . Zeilhofer . H.-F. . Berg-Boerner . B.-I. . Maurer . J. . Vetter . T. . Schkommodau . E. . Juergens . P. . Schwenzer-Zimmerer . K. . December 2011 . Missing facial parts computed by a morphable model and transferred directly to a polyamide laser-sintered prosthesis: an innovation study . British Journal of Oral and Maxillofacial Surgery . 49 . 8 . e67–e71 . 10.1016/j.bjoms.2011.02.007 . 21458119 . 0266-4356.
15. Brunton . Alan . Salazar . Augusto . Bolkart . Timo . Wuhrer . Stefanie . November 2014 . Review of statistical shape spaces for 3D data with comparative analysis for human faces . Computer Vision and Image Understanding . 128 . 2 . 10.1016/j.cviu.2014.05.005 . 1077-3142. 1209.6491 .
16. Book: De Smet . Michaël . Van Gool . Luc . Optimal Regions for Linear Model-Based 3D Face Reconstruction . Lecture Notes in Computer Science . 2011 . 6494 . Kimmel . Ron . Klette . Reinhard . Sugimoto . Akihiro . Computer Vision – ACCV 2010 . https://link.springer.com/chapter/10.1007/978-3-642-19318-7_22 . en . Berlin, Heidelberg . Springer . 276–289 . 10.1007/978-3-642-19318-7_22 . 978-3-642-19318-7.
17. Blanz . V. . Basso . C. . Poggio . T. . Vetter . T. . September 2003 . Reanimating Faces in Images and Video . Computer Graphics Forum . en . 22 . 3 . 641–650 . 10.1111/1467-8659.t01-1-00712 . 0167-7055.
18. Bouaziz . Sofien . Wang . Yangang . Pauly . Mark . 2013-07-21 . Online modeling for realtime facial animation . ACM Trans. Graph. . 32 . 4 . 40:1–40:10 . 10.1145/2461912.2461976 . 0730-0301.
19. Ichim . Alexandru-Eugen . Kadleček . Petr . Kavan . Ladislav . Pauly . Mark . 2017-07-20 . Phace: physics-based face modeling and animation . ACM Trans. Graph. . 36 . 4 . 153:1–153:14 . 10.1145/3072959.3073664 . 0730-0301.
20. Koppen . Paul . Feng . Zhen-Hua . Kittler . Josef . Awais . Muhammad . Christmas . William . Wu . Xiao-Jun . Yin . He-Feng . 2018-02-01 . Gaussian mixture 3D morphable face model . Pattern Recognition . 74 . 617–628 . 10.1016/j.patcog.2017.09.006 . 0031-3203. free . 2018PatRe..74..617K .
21. Book: Craw . Ian . Cameron . Peter . Parameterising Images for Recognition and Reconstruction . 1991 . Mowforth . Peter . BMVC91 . https://link.springer.com/chapter/10.1007/978-1-4471-1921-0_52 . en . London . Springer . 367–370 . 10.1007/978-1-4471-1921-0_52 . 978-1-4471-1921-0.
22. Cootes . T.F. . Taylor . C.J. . Cooper . D.H. . Graham . J. . January 1995 . Active Shape Models-Their Training and Application . Computer Vision and Image Understanding . 61 . 1 . 38–59 . 10.1006/cviu.1995.1004 . 1077-3142.
23. Cootes . T.F. . Edwards . G.J. . Taylor . C.J. . June 2001 . Active appearance models . IEEE Transactions on Pattern Analysis and Machine Intelligence . 23 . 6 . 681–685 . 10.1109/34.927467.
24. Vetter . T. . Poggio . T. . July 1997 . Linear object classes and image synthesis from a single example image . IEEE Transactions on Pattern Analysis and Machine Intelligence . 19 . 7 . 733–742 . 10.1109/34.598230. 11858/00-001M-0000-0013-ECA6-4 . free .
25. Book: Paysan . Pascal . Lüthi . Marcel . Albrecht . Thomas . Lerch . Anita . Amberg . Brian . Santini . Francesco . Vetter . Thomas . Face Reconstruction from Skull Shapes and Physical Attributes . Lecture Notes in Computer Science . 2009 . 5748 . Denzler . Joachim . Notni . Gunther . Süße . Herbert . Pattern Recognition . https://link.springer.com/chapter/10.1007/978-3-642-03798-6_24 . en . Berlin, Heidelberg . Springer . 232–241 . 10.1007/978-3-642-03798-6_24 . 978-3-642-03798-6.
26. Chen Cao . Yanlin Weng . Shun Zhou . Yiying Tong . Kun Zhou . March 2014 . FaceWarehouse: A 3D Facial Expression Database for Visual Computing . IEEE Transactions on Visualization and Computer Graphics . 20 . 3 . 413–425 . 10.1109/TVCG.2013.249 . 1077-2626.
27. Booth . James . Roussos . Anastasios . Zafeiriou . Stefanos . Ponniah . Allan . Dunaway . David . 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . June 2016 . A 3D Morphable Model Learnt from 10,000 Faces . https://ieeexplore.ieee.org/document/7780967 . IEEE . 5543–5552 . 10.1109/CVPR.2016.598 . 978-1-4673-8851-1. 10871/31965 . free .
28. Huber . Patrik . Hu . Guosheng . Tena . Rafael . Mortazavian . Pouria . Koppen . Willem P. . Christmas . William J. . Rätsch . Matthias . Kittler . Josef . Proceedings of the 11th Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications . February 2016 . A Multiresolution 3D Morphable Face Model and Fitting Framework . https://www.scitepress.org/Link.aspx?doi=10.5220/0005669500790086 . 79–86 . 10.5220/0005669500790086 . 978-989-758-175-5.
29. Gerig . Thomas . Morel-Forster . Andreas . Blumer . Clemens . Egger . Bernhard . Luthi . Marcel . Schoenborn . Sandro . Vetter . Thomas . 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) . May 2018 . Morphable Face Models - An Open Framework . https://ieeexplore.ieee.org/document/8373814 . IEEE . 75–82 . 10.1109/FG.2018.00021 . 978-1-5386-2335-0. 1709.08398 .
30. Dai . Hang . Pears . Nick . Smith . William . Duncan . Christian . 2017 IEEE International Conference on Computer Vision (ICCV) . October 2017 . A 3D Morphable Model of Craniofacial Shape and Texture Variation . https://ieeexplore.ieee.org/document/8237597 . IEEE . 3104–3112 . 10.1109/ICCV.2017.335 . 978-1-5386-1032-9.
31. Li . Tianye . Bolkart . Timo . Black . Michael J. . Li . Hao . Romero . Javier . 2017-11-20 . Learning a model of facial shape and expression from 4D scans . ACM Trans. Graph. . 36 . 6 . 194:1–194:17 . 10.1145/3130800.3130813 . 0730-0301.
32. Book: Ranjan . Anurag . Bolkart . Timo . Sanyal . Soubhik . Black . Michael J. . Generating 3D Faces Using Convolutional Mesh Autoencoders . Lecture Notes in Computer Science . 2018 . 11207 . Ferrari . Vittorio . Hebert . Martial . Sminchisescu . Cristian . Weiss . Yair . Computer Vision – ECCV 2018 . https://link.springer.com/chapter/10.1007/978-3-030-01219-9_43 . en . Cham . Springer International Publishing . 725–741 . 10.1007/978-3-030-01219-9_43 . 978-3-030-01219-9.
33. Smith . William A. P. . Seck . Alassane . Dee . Hannah . Tiddeman . Bernard . Tenenbaum . Joshua B. . Egger . Bernhard . 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . June 2020 . A Morphable Face Albedo Model . https://ieeexplore.ieee.org/document/9156599 . IEEE . 5010–5019 . 10.1109/CVPR42600.2020.00506 . 978-1-7281-7168-5. 2004.02711 .
34. Wang . Lizhen . Chen . Zhiyuan . Yu . Tao . Ma . Chenguang . Li . Liang . Liu . Yebin . 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . June 2022 . FaceVerse: a Fine-grained and Detail-controllable 3D Face Morphable Model from a Hybrid Dataset . https://ieeexplore.ieee.org/document/9880411 . IEEE . 20301–20310 . 10.1109/CVPR52688.2022.01969 . 978-1-6654-6946-3. 2203.14057 .