Merging objects using machine learning

The ObjectMerger is a Random Forest Classifier part of the apoc library that can learn which labels to merge and which not. It allows post-processing label images after objects have been (intentionally or not) oversegmented.

import apoc
from skimage.io import imread, imshow
import pyclesperanto_prototype as cle
import numpy as np

A common example can be derived from an image showing intensities in cell membranes.

image = imread('../../data/membrane2d.tif')
cle.asarray(image)

cle._ image shape (256, 256) dtype float32 size 256.0 kB min 277.0 max 44092.0

As the membranes have different intensity depending on the region in the image, we need to correct for this first.

background_subtracted = cle.divide_by_gaussian_background(image, sigma_x=10, sigma_y=10)
background_subtracted

cle._ image shape (256, 256) dtype float32 size 256.0 kB min 0.15839748 max 11.448771

For technical reasons it is also recommeded to turn the intensity image into an image of type integer. Therefore, normalization might be necessary. It is important that images used for training and images used for prediction have intensities in the same range.

oversegmented = imread("../../data/membrane2d_oversegmented.tif")
cle.asarray(oversegmented).astype(np.uint32)

cle._ image shape (256, 256) dtype uint32 size 256.0 kB min 1.0 max 54.0

An annotation serves telling the algorithm which segmented objects should be merged and which not.

annotation = imread("../../data/membrane2d_merge_annotation.tif")
cle.asarray(annotation).astype(np.uint32)

cle._ image shape (256, 256) dtype uint32 size 256.0 kB min 0.0 max 2.0

For visualization purposes, we can overlay the annotation with the membrane image.

cle.imshow(background_subtracted, max_display_intensity=4, continue_drawing=True)
cle.imshow(annotation, alpha=0.6, labels=True)

To show more closely what needs to be annotated, we also overlay the label-edge image and the annotation. Note that the edges which are not supposed to be merged are small dots always carefully only annotating two objects (that should not be merged.

cle.imshow(cle.detect_label_edges(oversegmented), continue_drawing=True)
cle.imshow(annotation, alpha=0.6, labels=True)

Training the merger

The LabelMerger can be trained using three features:

• touch_portion: The relative amount an object touches another. E.g. in a symmetric, honey-comb like tissue, neighboring cells have a touch-portion of 1/6 to each other.

• touch_count: The number of pixels where object touch. When using this parameter, make sure that images used for training and prediction have the same voxel size.

• mean_touch_intensity: The mean average intensity between touching objects. If a cell is over-segmented, there are multiple objects found within that cell. The area where these objects touch has a lower intensity than the area where two cells touch. Thus, they can be differentiated. Normalizing the image as shown above is key.

• centroid_distance: The distance (in pixels or voxels) between centroids of labeled objects.

Note: most features are recommended to be used in isotropic images only.

feature_definition = "touch_portion mean_touch_intensity"

classifier_filename = "label_merger.cl"

apoc.erase_classifier(classifier_filename)
classifier = apoc.ObjectMerger(opencl_filename=classifier_filename)

classifier.train(features=feature_definition,
                 labels=oversegmented,
                 sparse_annotation=annotation,
                 image=background_subtracted) 

merged_labels = classifier.predict(labels=oversegmented, image=background_subtracted)
merged_labels

cle._ image shape (256, 256) dtype uint32 size 256.0 kB min 1.0 max 31.0