Generating images using DALL-E#

In this notebook we will demonstrate how to ask openAI’s DALL-E model to generate some images. Depending on what we request and how the model was trained, the results may be more or less appropriate. We will again use a small helper function to return the image directly.

Read more:

import openai
from skimage.io import imread, imshow
from numpy import random
from matplotlib import pyplot as plt

def prompt_image(message:str, width:int=1024, height:int=1024, model='dall-e-3'):
    client = openai.OpenAI()
    response = client.images.generate(
      prompt=message,
      model=model,
      n=1,
      size=f"{width}x{height}"
    )
    image_url = response.data[0].url
    image = imread(image_url)
    
    return image

When generating scientific images, they may, or may not be a proper representation. Presumably, DALL-E was trained on many natural images (showing cars, trees, houses, etc.) and not so many images of cells.

nuclei = prompt_image('an image of 100 nuclei in a fluorescence microscopy image')
imshow(nuclei)

<matplotlib.image.AxesImage at 0x19eb0e335e0>
../_images/6f6c1d8c3e8669904e49a658c2741fa82192219c3f97a38b5d32938b42e3a42c.png

Maybe, generating images of histological slices is more promising.

histology = prompt_image('a histology image of lung cancer cells and some healthy tissue')
imshow(histology)

<matplotlib.image.AxesImage at 0x19eb12ad930>
../_images/dd0b19d2a1c9ff86efc289a4866e9c92882c58b333a494d5190b02fbacdaa092.png

In some cases, Dall-E 2 produces more realistic output.

histology = prompt_image('a histology image of lung cancer cells and some healthy tissue',
                        model='dall-e-2')
imshow(histology)

<matplotlib.image.AxesImage at 0x19eb12f15a0>
../_images/75e431a539cf225a3d665851bc3ad1c3617aeb87a1d1408e33db1cb685dd79b7.png

Exercise#

Evaluating generated images is an active research field. One approach is to present readers real images and generated images asking them to identify which are real. In the following list of four images of cats sitting next to microscopes, one cat is real. Which one? Why?

cat_microscope_prompt = """
Image of a cat sitting behind a microscope.
Both are on a brown floor in front of a white wall.
The cat is mostly white and has some black dots.
The cat sits straight.
The cat is a bit larger than the microsope.
"""

images = [imread('../../data/real_cat.png')]
for _ in range(3):
    images.append(prompt_image(cat_microscope_prompt))

random.shuffle(images)

fix, ax = plt.subplots(1,len(images), figsize=(15,15))
for i, image in enumerate(images):
    ax[i].imshow(image)
../_images/df76d1162bcfef52c2145dc7f0ea581bf435e9f480e3f05ac68920ba2467565b.png

Run the same image generation using Dall-E 2 and compare the results visually.