Shape completion using Gaussian process regression
In this tutorial we will show how GP regression can be used to predict missing parts of a shape.
Preparation
As in the previous tutorials, we start by importing some commonly used objects and initializing the system.
We also load a dataset that we want to reconstruct. In this case, it is a face without nose:
Finally, we also load the face model.
Enlarging the flexibility of a shape model
The model, which we just loaded, was built from only a small dataset. Therefore, the chances that it manages to reconstruct the missing nose properly are rather slim.
To increase the shape variability of the model, we add smooth some additional smooth shape deformations, modelled by a GP with symmetric Gaussian kernel. The code should be familiar from the previous tutorials.
The new model should now contain much more flexibility, while still preserving the typical face-specific deformations.
Note: This step here is mainly motivated by the fact that we only have 10 face examples available to build the model. However, even if sufficient data is available, it might still be a good idea to slighly enlarge the flexibility of a model before attempting a reconstruction of missing parts. It gives the model some extra slack to account for bias in the data and explain minor shape variations, which have not been prominent in the dataset.
Equipped with our new model, we will perform the reconstruction in three steps:
- We fit the face model to the given partial face using Gaussian process regression.
- We restrict the model to the nose part by marginalizing and select a suitable nose shape.
- We choose a suitable nose from the model
As we saw previously, to perform GP regression we need observations of the deformation vectors at some points. We will discussed in Tutorial 10 how such observations can be obtained fully automatically. Here, we have done this already in a separate step and saved 200 corresponding points as landmarks, which we will now load and visualize:
These correspondences define how each selected point of the model should be deformed to its corresponding point on the target mesh. In other words, we observed a few deformation vectors at the selected model points. We use these deformation vectors and build a deformation field:
We can now perform GP regression and retrieve the rest of the deformations fitting our observations:
With this posterior model, we get a normal distribution of faces satisfying our observations by having the selected characteristic points at the indicated positions.
Finally, as we are interested in the nose region only, we marginalize our posterior to obtain a posterior nose model as we did in a previous tutorial:
We now have a probability distribution over fitting nose shapes. Our last task is to select one nose as a reconstruction. A simple and often reasonable choice is to use the mean of this model, as this is the reconstruction with the highest probability under the model.
Having a full posterior distribution over fitting nose shapes gives us, however, much more freedom in choosing a reconstruction. By introducing a utility function, which models the utility of each reconstruction for a given task, we can use this distribution as the basis for decision making by minimizing the expected posterior loss.