Prompting bio-image analysis tasks using LangChain

Prompting bio-image analysis tasks using LangChain#

In this notebook we demonstrate how to prompt for executing bio-image analysis tasks using chatGPT and LangChain.

from langchain.memory import ConversationBufferMemory
from langchain.chat_models import ChatOpenAI
from langchain.agents import initialize_agent
from langchain.agents import AgentType
from langchain.tools import tool

from skimage.io import imread
from napari_segment_blobs_and_things_with_membranes import voronoi_otsu_labeling

import stackview

For accomplishing this, we need an image storage. To keep it simple, we use a dictionary.

image_storage = {}

To demonstrate bio-image analysis using English language, we define common bio-image analysis functions for loading images, segmenting and counting objects and showing results.

tools = []
@tools.append
@tool
def load_image(filename:str):
    """Useful for loading an image file and storing it."""
    print("loading", filename)
    image = imread(filename)
    image_storage[filename] = image
    return "The image is now stored as " + filename
@tools.append
@tool
def segment_bright_objects(image_name):
    """Useful for segmenting bright objects in an image that has been loaded and stored before."""
    print("segmenting", image_name)
    
    image = image_storage[image_name]
    label_image = voronoi_otsu_labeling(image, spot_sigma=4)
    
    label_image_name = "segmented_" + image_name
    image_storage[label_image_name] = label_image
    
    return "The segmented image has been stored as " + label_image_name
@tools.append
@tool
def show_image(image_name):
    """Useful for showing an image that has been loaded and stored before."""
    print("showing", image_name)
    
    image = image_storage[image_name]
    display(stackview.insight(image))
    
    return "The image " + image_name + " is shown above."
@tools.append
@tool
def count_objects(image_name):
    """Useful for counting objects in a segmented image that has been loaded and stored before."""
    label_image = image_storage[image_name]
    
    num_labels = label_image.max()
    print("counting labels in ", image_name, ":", num_labels)

    return f"The label image {image_name} contains {num_labels} labels."

We create some memory and a large language model based on OpenAI’s chatGPT.

memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True)
llm=ChatOpenAI(temperature=0)

Given the list of tools, the large language model and the memory, we can create an agent.

agent = initialize_agent(
    tools, 
    llm, 
    agent=AgentType.CHAT_CONVERSATIONAL_REACT_DESCRIPTION, 
    memory=memory
)

This agent can then respond to prompts.

agent.run("Please load the image ../../data/blobs.tif and show it.")
loading ../../data/blobs.tif
showing ../../data/blobs.tif
shape(254, 256)
dtypeuint8
size63.5 kB
min8
max248
'The image ../../data/blobs.tif is shown above.'
agent.run("Please segment the image ../../data/blobs.tif .")
segmenting ../../data/blobs.tif
'The segmented image has been stored as segmented_../../data/blobs.tif'
agent.run("Please show the segmented ../../data/blobs.tif image.")
showing segmented_../../data/blobs.tif
shape(254, 256)
dtypeint32
size254.0 kB
min0
max64
'The segmented image ../../data/blobs.tif is shown above.'
agent.run("How many objects are there in the segmented ../../data/blobs.tif image?")
counting labels in  segmented_../../data/blobs.tif : 64
'The segmented ../../data/blobs.tif image contains 64 objects.'

Chaining operations#

We can also chain these operations in a single sentence and the agent will figure out on it’s own how to do this.

# empty memory and start from scratch
image_memory = {}
agent.run("""
Please load the image ../../data/blobs.tif, 
segment bright objects in it, 
count them and 
show the segmentation result.
""")
loading ../../data/blobs.tif
segmenting ../../data/blobs.tif
counting labels in  segmented_../../data/blobs.tif : 64
showing segmented_../../data/blobs.tif
shape(254, 256)
dtypeint32
size254.0 kB
min0
max64
'The segmented image has been shown.'
agent.run("How many objects were there?")
counting labels in  segmented_../../data/blobs.tif : 64
'The segmented image contains 64 objects.'