MIKAIA Plugin API: “Plug in your own AI”

This app note describes the MIKAIA Plugin API. It can be used by bioinformaticians to plug in their own AI script, e.g., written in Python or any other language, into MIKAIA® and this way put their AI into the hands of a pathologist.

Motivation: It’s not easy to deploy an AI to a pathologist

When we visit computational pathology research groups, we often encounter this situation: The bioinformatician has created an awesome new AI and they would like to put it into the hands of a pathologists so that they can try it out and give feedback. But pathologists rarely have a linux computer, Python SDK, or programming skills (nor should they have to, by the way). The AI needs to be integrated into a viewer that is capable of triggering and executing the AI and visualizing the results. This is a big hurdle!

Solution: MIKAIA®’s “Plug-in your own AI” REST API

This motivated us to add a plug-in API to MIKAIA® that let’s user seamlessly plug their own AIs into MIKAIA®. This way, the pathologist can use MIKAIA® on their computer as a frontend. The plugin with the custom AI can either run on the same computer, but it could also run on a VM or linux computer in the local network, in case the plugin only works in Linux or MacOS. It could even run in a Docker. All data stays local. No cloud computing is involved. The user plugin will typically be a Python script, but it could also be a Matlab or R script, or anything else.

Benefits of plugging your AI into MIKAIA®

Using MIKAIA® as a frontend to your AI has many advantages:

• You don’t need to worry about programming a WSI viewer and annotation system.
• The user (the pathologist) does not need any Python SDKs or similar. MIKAIA® is a Windows software
• You are no longer limited to WSI formats supported by openslide. Instead, you read pixels through MIKAIA®, allowing you to support many WSI formats such as Zeiss CZI, Olympus, OME-TIFF, etc. The API can give you up to 16 bit pixels and supports accessing individual fluorescence channels. You can ask for pixels in a native resolution level or at any other arbitrary resolution (µm/px)
• While the analysis is still ongoing, you can visualize the results of your AI live. Simply pass annotation objects to the MIKAIA® annotation API as soon as they become available.
• It’s super simple to adapt your existing Python scripts to the MIKAIA® API. We have put a few examples on our GitHub, for instance using Meta’s Segment-Anything-Model for object segmentation.

We use this API ourselves: The Python developers in our team now also use this API in early development phases of new apps.

Software Architecture

The software architecture works as follows:

1. In MIKAIA®, you use the “Plug-in your own AI App” to connect a new plugin. The MIKAIA® Plugin Launcher sits between MIKAIA® and the plugin and must run on the computer where the plugin resides. You specify the computer name or IP of that computer and click “discover”.
2. MIKAIA® will then contact the Plugin Launcher and ask it to list all local scripts (default: *.py)
3. In MIKAIA®, you select which one is your plugin. You also assign an app title and pick an app cover image. Finally, you enable “show in app center”, and now your plugin is available in the app center.
4. Going forward, a user can choose to analyze a WSI (or FoV, or ROI) with your plugin. MIKAIA® will then again contact the Plugin Launcher and ask it to invoke (execute) the plugin. MIKAIA® will pass along a session ID and its own IP address. These will be provided to the plugin as command line parameters
5. The user script then loads the AI model and analyzes the requested region (or entire WSI). It reads pixels by contacting the MIKAIA® server and asking for the pixels in the required format for the required region. It then creates annotation classes (incl. styling) and generates annotation objects (rectangles, polygons, points, etc.) that are assigned to a class again via the MIKAIA® API.

We provide a convenience Python library that simplifies the API calls as an open source package on our GitHub. The Plugin Launcher, written in Python, is also open source.

https://www.github.com/mikaia-projects

Example Code

See: https://github.com/MIKAIA-projects/plug-in-your-own-ai/tree/main/examples

Drawings by Rosalie Kletzander. All images: Copyright Fraunhofer IIS.