Hello Scalismo!
The goal in this tutorial is to present the most important data structures, as well as the visualization capabilities of Scalismo.
Related resources
The following resources from our online course may provide some helpful context for this tutorial:
- What is Scalismo (Video)
Initializing the system
Before we start, we need to initialize Scalismo by calling:
The call to scalismo.initialize
loads all the dependencies to native C++ libraries (such as e.g. vtk or hdf5).
The second call tells scalismo, which source
of randomness to use and at the same time seeds the random number generator appropriately.
Later on we would like to visualize the objects we create. This is done using Scalismo-ui - the visualization library accompanying scalismo.
We can load an instance of the GUI, which we name here simply ui
as follows:
Meshes (surface data)
The first fundamental data structure we discuss is the triangle mesh,
which is defined in the package scalismo.mesh
.
In the following we will need access to the following object, which we
now import:
Meshes can be read from a file using the method readMesh
from the MeshIO
:
To visualize any object in Scalismo, we can use the show
method of the ui
object.
We often want to organize different visualizations of an object in a group.
We start directly with this practice and
first create a new group, to which we then add the visualization of the mesh:
Now that the mesh is displayed in the "Scalismo Viewer's 3D view", you can interact with it as follows:
- to rotate: maintain the left mouse button clicked and drag
- to shift/translate: maintain the middle mouse button clicked and drag
- to scale: maintain the right mouse button clicked and drag up or down
Note: if you are a Mac user, please find out how to emulate these events using your mouse or trackpad Note also that you can use the RC, X, Y and Z buttons in the 3D view to recenter the camera on the displayed object.
Anatomy of a Triangle mesh
A 3D triangle mesh in scalismo consists of a pointSet
, which maintains a collection of 3D points and a
list of triangle cells. We can access individual points using their point id.
Here we show how we can access the first point in the mesh:
Similarly, we can access the first triangles as follows:
The first cell is a triangle between the first, second and third points of the mesh. Notice here that the cell indicates the identifiers of the points (their index in the point sequence) instead of the geometric position of the points.
Instead of visualizing the mesh, we can also display the points forming the mesh.
This should add a new point cloud element to the scene with the name "pointCloud".
Note: depending on your computer, visualizing the full point cloud may slow down the visualization performance.
Note that to clean up the 3D scene, you can delete the objects either from the user interface (by right-clicking on the object's name), or programmatically by calling remove
on the corresponding view object :
Points and Vectors
We are very often interested in modelling transformations of point sets. Therefore we need to learn how to manipulate point positions.
The two fundamental classes in this context are Point
and EuclideanVector
:
We define points by specifying their coordinates:
The difference between two points is a EuclideanVector
The sum of a point with a vector yields a new point:
Points can be converted to vectors:
and vice versa:
Remark: Observe that the type of the expression is a parametric type Point[_3D]
, where the type parameter _3D
encodes the dimensionality. This pattern holds true for most types in Scalismo. It allows us to write generic code, which is independent of the dimensionality of the space.
We put these concepts in practice, and illustrate how we can compute the center of mass, given a sequence of points:
In a first step, we treat all the points as displacement vectors (the displacement of the points from the origin)
The average displacement can be easily computed by averaging all the vectors.
And finally we treat the average displacement again as a point in space.
Scalar Images
The next important data structure is the (scalar-) image. A discrete scalar image (e.g. gray level image) in Scalismo is simply a function from a discrete domain of points to a scalar value.
We will need the following imports:
Let's read and display a 3D image (MRI of a human):
Note: depending on your view on the scene, it could appear as if the image is not displayed. In this case, make sure to rotate the scene and change the position of the slices as indicated below.
To visualize the different image slices in the viewer, select "Scene" (the upper node in the scene tree graph) and use the X,Y,Z sliders.
You can also change the way of visualizing the 3D scene under the
View -> Perspective menu.
Scalar Image domain
Let's inspect the domain of the image :
The discrete image domain is a 3-dimensional regular grid of points originating at point (92.5485, -121.926, 135.267), with regular spacing of 1.5 mm in each dimension and containing 171, 171, 139 grid slots in the x, y and z directions respectively.
To better see this, let's display the first 172 points of the image domain
Scalar image values
The other important part of a discrete image are the values associated with the domain points
This is an iterator of scalar values of type Short
as encoded in the read image.
Let's check the first value, which is the value associated with the origin :
The point origin corresponds to the grid point with index (0,0,0). Hence, the same value can be obtained by accessing the image at this index :
Naturally, the number of scalar values should be equal to the number of points
Notice that you can check the intensity value at a particular point position in the image, by maintaining the Ctrl key pressed and hovering over the image. The intensity value will then be displayed in the lower left corner of the Scalismo viewer window.
Creating scalar images
Given that discrete scalar images are a mapping between points and values, we can easily create such images programmatically.
Here we create a new image defined on the same domain of points with artificially created values: We threshold an MRI image, where all the values above 300 are replaced with 0.
Note: We need to write 0.toShort or 0 : Short in order to ensure that the threshValues
have type Short
and not Int
.
There is, however, also a more elegant way to write above code, namely using the map
method. The map
method applies
an operation to all values. Using this method, we can write instead
Statistical Mesh Models
Finally, we look at Statistical Shape Models.
Statistical models can be read by calling readStatisticalMeshModel
Sampling in the UI
Exercise: Sample random instances of faces by using the graphical tools in the scene pane : click on the "model" tree node and then the "Random" button
Exercise: click a landmark on a position of the face model, e.g. chin or eye corner.. (use the toggle button "LM" in the toolbar to activate landmark clicking). Rename this landmark and call it noseLM. Now continue sampling from the model. What happens to the selected point?
As you can see, a new instance of the face model is displayed each time along with the corresponding landmark point. Notice how the position of the landmark point changes in space while it keeps the same "meaning" on the face (eye corner, tip of nose ..)
Sampling programmatically
Sampling in the ui is useful for getting a visual impression of the variability of a model. But more often we want to
sample from a model programmatically. We can obtain a sample from the model, by calling the sample method
:
Exercise: Visualize a few randomly generated meshes in the ui.
Retrieving objects from Scalismo-ui
This is a good point to show how objects that we added manually in Scalismo-ui can be retrieved programmatically. A typical example is,
that we manually clicked a landmark, such as our noseLM
, on one of the visualized objects and would like to work with them in our
programs.
To achieve this we can use the filter
method of the ui object. It works as follows:
The filter
method is very general. The type parameter (the parameter inside []) indicates the type of view
object we want to
search for. Here we look only for landmarks, and consequently specify the type LandmarkView
. As a first we pass the group,
in which we want to search for an object. The second argument is a predicate, which is executed for all objects in the group, of the right type.
Here we specify, that filter
should match all objects whose name equals "noseLM". Calling the filter
method results in a sequence
of view objects, which match the predicate. To get the matching scalismo object, we call the method landmark
on the view object.
We can do this for all landmark view objects in the sequence using the familiar map
function.
Finally, we can get the id and position of the matched landmark as follows:
Remark: In exactly the same way we can retrieve all other types of objects, which we can visualize in in Scalismo-ui, such as images, meshes, pointClouds, etc.