{ "cells": [ { "cell_type": "markdown", "id": "790cc3e5-2360-4431-865e-fda11ea3af24", "metadata": {}, "source": [ "# Inpainting using Dall-E\n", "Inpainting is the task of replacing a selected part of an image by generated pixels that make the selection disappear. When working with scientific images, this could be seen as scientific misconduct. Be careful when applying this technique to your microscopy images.\n", "\n", "In this notebook we will use [Dall-E 2](https://openai.com/dall-e-2) to inpaint a region of an image. For convenience reasons, we will use the [Darth-D](https://github.com/haesleinhuepf/darth-d/) library to process the image." ] }, { "cell_type": "code", "execution_count": 1, "id": "869487e8-d526-4cc3-af1b-b9ac83c4f30b", "metadata": {}, "outputs": [], "source": [ "from darth_d import replace\n", "\n", "from skimage.io import imread\n", "import stackview\n", "import numpy as np" ] }, { "cell_type": "markdown", "id": "7d8c8d6e-7d28-493b-816c-0784c9526e09", "metadata": {}, "source": [ "In this example we aim to replace the cell in the lower middle of this image." ] }, { "cell_type": "code", "execution_count": 2, "id": "ce9f03ad-3f37-4940-b1ce-db67a96dc334", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\n", "\n", "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
shape(512, 672, 3)
dtypeuint8
size1008.0 kB
min0
max255
\n", "\n", "
" ], "text/plain": [ "StackViewNDArray([[[ 3, 6, 1],\n", " [ 3, 7, 0],\n", " [ 3, 6, 1],\n", " ...,\n", " [11, 8, 2],\n", " [11, 7, 2],\n", " [11, 11, 2]],\n", "\n", " [[ 3, 6, 1],\n", " [ 3, 8, 1],\n", " [ 3, 7, 1],\n", " ...,\n", " [11, 10, 2],\n", " [10, 10, 2],\n", " [11, 11, 2]],\n", "\n", " [[ 4, 6, 1],\n", " [ 3, 6, 1],\n", " [ 4, 6, 1],\n", " ...,\n", " [10, 10, 2],\n", " [11, 10, 2],\n", " [11, 10, 2]],\n", "\n", " ...,\n", "\n", " [[15, 14, 8],\n", " [14, 14, 8],\n", " [15, 14, 7],\n", " ...,\n", " [10, 11, 5],\n", " [10, 12, 4],\n", " [11, 14, 5]],\n", "\n", " [[14, 16, 7],\n", " [16, 15, 7],\n", " [15, 16, 8],\n", " ...,\n", " [10, 11, 4],\n", " [11, 13, 4],\n", " [11, 16, 5]],\n", "\n", " [[15, 18, 7],\n", " [14, 17, 8],\n", " [14, 17, 8],\n", " ...,\n", " [ 9, 12, 5],\n", " [10, 13, 5],\n", " [11, 15, 5]]], dtype=uint8)" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "input_image = imread(\"../../data/hela-cells-8bit.tif\")\n", "\n", "stackview.insight(input_image)" ] }, { "cell_type": "markdown", "id": "b67a7c6e-4f91-4fcc-bde6-63ce154c1a8c", "metadata": {}, "source": [ "We mark this cell using a binary mask." ] }, { "cell_type": "code", "execution_count": 3, "id": "252ee395-81cc-4bfd-a7e2-1cb6314941f3", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\n", "\n", "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
shape(512, 672)
dtypeuint8
size336.0 kB
min0
max1
\n", "\n", "
" ], "text/plain": [ "StackViewNDArray([[0, 0, 0, ..., 0, 0, 0],\n", " [0, 0, 0, ..., 0, 0, 0],\n", " [0, 0, 0, ..., 0, 0, 0],\n", " ...,\n", " [0, 0, 0, ..., 0, 0, 0],\n", " [0, 0, 0, ..., 0, 0, 0],\n", " [0, 0, 0, ..., 0, 0, 0]], dtype=uint8)" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mask = np.zeros(input_image.shape[:2], dtype=np.uint8)\n", "mask[300:500, 200:500] = 1\n", "\n", "stackview.insight(mask)" ] }, { "cell_type": "markdown", "id": "afe70a73-dd3e-4579-868f-89d87e301a16", "metadata": {}, "source": [ "We then call the `replace` function to generate a new image." ] }, { "cell_type": "code", "execution_count": 4, "id": "6a0375e9-8cd6-40e5-80bb-68412a314a9d", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "C:\\Users\\haase\\mambaforge\\envs\\t310\\lib\\site-packages\\darth_d\\_replace.py:41: UserWarning: Using the replace function on scientific images could be seen as scientific misconduct. Handle this function with care.\n", " warn(\"Using the replace function on scientific images could be seen as scientific misconduct. Handle this function with care.\")\n" ] }, { "data": { "text/html": [ "\n", "\n", "\n", "\n", "\n", "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
shape(512, 672, 3)
dtypeuint8
size1008.0 kB
min0
max255
\n", "\n", "
" ], "text/plain": [ "StackViewNDArray([[[ 3, 6, 0],\n", " [ 3, 6, 0],\n", " [ 3, 6, 0],\n", " ...,\n", " [10, 8, 2],\n", " [10, 7, 2],\n", " [10, 9, 2]],\n", "\n", " [[ 3, 6, 0],\n", " [ 3, 6, 0],\n", " [ 3, 6, 0],\n", " ...,\n", " [10, 9, 2],\n", " [10, 9, 2],\n", " [10, 9, 2]],\n", "\n", " [[ 3, 6, 0],\n", " [ 3, 6, 0],\n", " [ 3, 6, 0],\n", " ...,\n", " [10, 9, 2],\n", " [10, 9, 2],\n", " [10, 9, 2]],\n", "\n", " ...,\n", "\n", " [[13, 14, 6],\n", " [13, 14, 6],\n", " [14, 14, 7],\n", " ...,\n", " [ 8, 9, 3],\n", " [ 9, 11, 3],\n", " [ 9, 12, 3]],\n", "\n", " [[14, 15, 7],\n", " [14, 14, 7],\n", " [14, 15, 7],\n", " ...,\n", " [ 9, 10, 3],\n", " [ 9, 12, 4],\n", " [ 9, 14, 4]],\n", "\n", " [[14, 16, 7],\n", " [14, 16, 7],\n", " [14, 15, 7],\n", " ...,\n", " [ 9, 11, 3],\n", " [ 9, 12, 4],\n", " [10, 14, 4]]], dtype=uint8)" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "new_image = replace(input_image, mask)\n", "\n", "stackview.insight(new_image)" ] }, { "cell_type": "code", "execution_count": 5, "id": "13c1f4af-c8a6-4222-86b2-8298ac0a8c79", "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "fb936e220f8e4647b836df103051c785", "version_major": 2, "version_minor": 0 }, "text/plain": [ "VBox(children=(HBox(children=(VBox(children=(ImageWidget(height=256, width=336),)),)), IntSlider(value=256, de…" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "stackview.curtain(input_image, new_image, zoom_factor=0.5)" ] }, { "cell_type": "markdown", "id": "f52d50bd-fada-496e-a267-28a507908d25", "metadata": {}, "source": [ "## Exercise\n", "Load `../../data/blobs.tif` draw a binary mask sized 100x100 pixels in the center of the image and inpaint it." ] }, { "cell_type": "code", "execution_count": null, "id": "23e7ffad-667b-488c-8361-141cb9c9cd87", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.13" } }, "nbformat": 4, "nbformat_minor": 5 }