This script takes a first step towards performing customized image processing by combining QuPath and ImageJ - via Groovy.

Here, we will look first at how to get the pixels from QuPath into ImageJ by scripting. Sending data back to QuPath will be the subject of a future post.

Problem

It’s already possible to perform custom image processing by calling ImageJ macros from QuPath.

But Groovy lets you do much more than is possible with the ImageJ macro language, with a lot more control.

Ideally it would be possible to grab image regions from QuPath, and convert them into ImageJ data structures for direct processing - all from the comfortable of a single Groovy script. This can be run directly through QuPath.

This is how I typically perform non-standard image processing within QuPath, e.g. to detect regions. I don’t personally use the macro runner much.

If you are using the Fiji distribution of ImageJ, you can run Groovy scripts from there. But currently Fiji does not connect easily with QuPath, and Fiji itself does not yet have strong whole slide image support.

Solution

QuPath gives lots of help to share data between QuPath and ImageJ.

It’s just a matter of knowing where to find it and how to use it… which is not always obvious.

To begin, we need to access two of the main QuPath data structures relating to the current image:

  • the ImageData, which includes the object hierarchy, image type and some other information related to the image, and
  • the ImageServer, which is used to request pixels, and is accessed from within the ImageData.

For an explanation of QPEx, see the wiki.

// Access the relevant QuPath data structures
import qupath.lib.scripting.QPEx
def imageData = QPEx.getCurrentImageData()
def server = imageData.getServer()

To define regions when accessing pixels, QuPath uses RegionRequest objects.

These have 3 main components:

  • a path to the image
  • a downsample value, indicating how much to the image region should be scaled down (1.0 indicates the full size image)
  • a bounding box for the requested region, composed of x, y, width and height values defined using coordinates from the full-resolution image, regardless of the downsample value that is specified in the request!

Why create RegionRequest objects and not just specify coordinates directly?

The reason is that the viewer keeps a cache of recently-accessed image tiles and uses the RegionRequest as the key. When a tile is needed, the corresponding RegionRequest is used to check this cache to see if it is already there - to avoid pestering the ImageServer to read the pixels again unnecessarily (since this is much slower than just reading from the cache).

Consequently, the use of RegionRequest makes lots more optimization possible.

The following code creates a RegionRequest corresponding to the entire image, using a fairly aggressive downsample value of 40.

// Define how much to scale down - 1.0 means requesting the image at the full resolution
// (I'm defaulting to a large downsample because a small one often won't work - on account of the image being much too large)
double downsample = 40.0

// Request the region
import qupath.lib.regions.RegionRequest
def region = RegionRequest.createInstance(server.getPath(), downsample, 0, 0, server.getWidth(), server.getHeight())

You can certainly change the downsample value above… but ought to do so cautiously.

When an image region is sent to ImageJ, all the pixels need to be read at once. This is in contrast to QuPath’s general approach, where only some of the pixels are stored in memory at any one time, because whole slide images are typically much too big to read in one go.

Therefore if you were to run the code above with a downsample value of 1, and a whole slide image, it would almost certainly not end well.

One reason is that there probably wouldn’t be enough memory available. But even if there is enough memory, it is still likely to fail because of an absolute limit to the number of pixels ImageJ can handle in a single 2D image - which is somewhere slightly below 231.

Rather than leaving QuPath to fail in its task without warning or explanation, we can build a sanity-check into the script that figures out roughly many pixels we expect the resulting image to have, and checks if it is anywhere close to the absolute limit.

// Check if we are likely to have too many pixels for a Java array
long w = (server.getWidth()/downsample) as int
long h = (server.getHeight()/downsample) as int
long maxPixels = Integer.MAX_VALUE - 5 // Approximate...
if (w * h > maxPixels) {
    downsample = Math.ceil(((long)server.getWidth() * (long)server.getHeight()) / maxPixels)
    print 'This image is going to be too big!  Requires ' + (w * h) + ' pixels, while I can handle at best ' + maxPixels
    print 'Try again setting the downsample to be at least ' + downsample + ' (and preferably more)'
    return
}

Now, having got this far we are ready to try to get the pixels from the ImageServer.

The ImageServer typically returns pixels in the form of a java.awt.image.BufferedImage - this is a common image representation used a lot in Java. If we wanted, we could just request a BufferedImage now, and perhaps write it to a file as shown here:

// Request a BufferedImage
def img = server.readBufferedImage(region)
import javax.imageio.ImageIO
ImageIO.write(img, 'PNG', new File('/path/to/some/output.png'))

But ImageJ has another way of representing images: ImagePlus.

An ImagePlus contains (potentially at least) a lot more important information than a BufferedImage, including pixel sizes and other metadata.

Fortunately, we can take advantage of some special helper functions within QuPath.

This applies to ImageJ1… for ImageJ2 the situation is different, and there are alternatives to ImagePlus. Helper functions are not yet available to link up ImageJ2 and QuPath.

To do so, we can wrap up our ImageServer in another ImageServer that not only knows how to return a BufferedImage, but also an ImagePlus.

This wrapper has the major benefit of being smart enough to know how to set the pixel size and location information. This means that the ImagePlus knows exactly where it came from in the original whole slide image. This will become important when we want to transfer the results of processing the ImagePlus back to QuPath afterwards.

Here’s the code to wrap the ImageServer, and get an ImagePlus for the region we have specified above.

// Request an ImagePlus
import qupath.imagej.images.servers.ImagePlusServerBuilder
server = ImagePlusServerBuilder.ensureImagePlusWholeSlideServer(server)
def imp = server.readImagePlusRegion(region).getImage()

At this point, we can really do whatever processing we like with the ImagePlus.

It’s just the same as if we were working in Java or Groovy with ImageJ directly. The plethora of documentation online about ImageJ can help here.

But for now we’ll restrict ourselves to showing the image - and leave processing for another day.

// Make sure that ImageJ is open (you can skip this, but then you may not have access to the full ImageJ user interface)
import qupath.imagej.gui.IJExtension
IJExtension.getImageJInstance()
// Show the image
imp.show()

Entire script

The script below may be slightly updated and cleaned up compared to what is described above - but does the same job.