Inpainting using Dall-E

Inpainting is the task of replacing a selected part of an image by generated pixels that make the selection disappear. When working with scientific images, this could be seen as scientific misconduct. Be careful when applying this technique to your microscopy images.

In this notebook we will use Dall-E 2 to inpaint a region of an image. For convenience reasons, we will use the Darth-D library to process the image.

from darth_d import replace

from skimage.io import imread
import stackview
import numpy as np

In this example we aim to replace the cell in the lower middle of this image.

input_image = imread("../../data/hela-cells-8bit.tif")

stackview.insight(input_image)

shape (512, 672, 3) dtype uint8 size 1008.0 kB min 0 max 255

We mark this cell using a binary mask.

mask = np.zeros(input_image.shape[:2], dtype=np.uint8)
mask[300:500, 200:500] = 1

stackview.insight(mask)

shape (512, 672) dtype uint8 size 336.0 kB min 0 max 1

We then call the replace function to generate a new image.

new_image = replace(input_image, mask)

stackview.insight(new_image)

C:\Users\haase\mambaforge\envs\t310\lib\site-packages\darth_d\_replace.py:41: UserWarning: Using the replace function on scientific images could be seen as scientific misconduct. Handle this function with care.
  warn("Using the replace function on scientific images could be seen as scientific misconduct. Handle this function with care.")

shape (512, 672, 3) dtype uint8 size 1008.0 kB min 0 max 255

stackview.curtain(input_image, new_image, zoom_factor=0.5)

Exercise

Load ../../data/blobs.tif draw a binary mask sized 100x100 pixels in the center of the image and inpaint it.