Anaximander: Interactively Running Geospatial Deep Learning Models on Any Compute Backend

AI summary

Overview

Research area: Geospatial deep learning and remote sensing tooling, sitting at the intersection of computer vision, geographic information systems (GIS), and machine learning systems engineering.

Technical level: Advanced. The paper assumes familiarity with inference servers, RPC protocols, model formats (PyTorch, ONNX, HuggingFace), and GIS concepts such as tiling, coordinate reference systems, and geotransforms.

Scope: The paper presents Anaximander, an open-source client-server system that lets practitioners load geospatial deep learning models from four source families and run them on four compute backends from inside a QGIS plugin, demonstrated through a code-free three-model comparison on an agricultural field delineation task.

What This Paper Is About

Applying deep learning to satellite imagery inside a GIS is high-friction: models arrive in incompatible formats and target different compute environments, so every evaluation requires custom deployment, tiling, and georeferencing code before a single prediction reaches an analyst's map. This friction discourages systematic comparison of models in a domain where model choice directly affects operational outcomes such as field delineation, crop monitoring, and disaster response. Anaximander's goal is to unify model source and compute location behind one interactive interface so that evaluating a candidate model becomes a menu selection rather than an integration project.

Key Contributions

1. A source-and-compute-agnostic inference backend. The nxmndr server unifies four model source families (PyTorch, ONNX, HuggingFace, cloud APIs) with four compute backends (local CPU, local GPU, remote GPU, serverless cloud) behind one interface. A gRPC protocol serves the PyTorch, ONNX, and HuggingFace models, while cloud APIs are reached over HTTP, directly or through an nxmndr proxy.

2. A QGIS plugin that turns model evaluation into spatial navigation. The nxmndr-qgis plugin lets practitioners run any registered model on the current map extent without writing code, with live per-tile feedback and automatic georeferencing of results. It provides a Model browser, an Inference widget (task, bands, tiling, live status overlay), and a "Quick Labels" path that injects layer legends as prompts into vision-language models.

3. An open-source release. The inference backend and protocol are released under the MIT license at https://github.com/microsoft/nxmndr, with the release of the plugin pending. The design is extensible: users can register their own model sources and compute backends.

4. A three-model case study on agricultural field delineation. The study demonstrates code-free comparison across a cloud vision-language model API (gpt-image-1), a remote GPU running SAM 3, and a local CPU running DelineateAnything.

Main Findings

• One protocol separates model source from compute location. Model origin (PyTorch, ONNX, HuggingFace, cloud API) and execution target (local CPU, local GPU, remote GPU, hosted API) each become a per-run dropdown in a graphical interface rather than an integration project.

• Tiled inference is collapsed into a single long-lived stream. The server exposes a bidirectional-streaming channel in which the client opens one server connection, streams tile chunks, and receives one response per completed tile tagged with session and tile identifiers, per-tile timing, and the serving device. This avoids the connection setup cost of a unary-call-per-tile design.

• Session state outlives individual calls. Each session records negotiated transport capabilities (chunk size, in-flight tiles, and maximum tile size), the model it serves, per-tile counters for completed and failed tiles, and a creation timestamp, and concurrent sessions stay isolated from one another.

• Models are cached and dispatched centrally. Every loaded model sits in a thread-safe LRU cache with default capacity 10, keyed by its specification. At startup the server enumerates available GPUs (or CPU cores) into a device plan, and a round-robin cursor spreads requests across them.

• Model registration is declarative. A core-library ModelSpec names the model's format and source, so the LRU cache, session layer, and device dispatcher are agnostic to model family, and a new family is added by registering a new spec type.

• The plugin installs no machine learning dependencies into QGIS. Local compute runs in a dedicated worker process with its own Python environment (NumPy 2, PyTorch, ONNX Runtime, and the nxmndr package), started as a child process on the first local job or Quick Labels request, serving the same gRPC service on a loopback port.

• Results fill in on the map while inference is still running. Completed tiles are written into windows of a GeoTIFF on disk and flushed; the file is added to the active project as a raster layer as soon as the first tile is written and refreshed at most five times per second, with styling finalized when the job ends.

• Seams are handled by discarding tile context, not by blending. The model sees the full tile, but only a centered window of its output is composited. Written windows never overlap because stride equals the written window, so each output pixel comes from exactly one tile and no blending of class probabilities occurs. This means an object crossing a tile boundary is labeled consistently but not identified as one object; for field delineation, a parcel spanning two chips is delineated correctly as area while appearing as two instances.

• Client memory stays bounded. Input rasters are never fully loaded into memory; each chip is a windowed read and each prediction is written to disk as it arrives, so memory is bounded by chip size and the number of in-flight tiles rather than by scene extent.

• Case study outcome, reported qualitatively. On a 1,024 × 1,024 pixel Bing Maps crop (approximately 7.5 km × 7.5 km) over the Overberg District of Western Cape province in South Africa, with the default 512-pixel chip size and no overlap yielding four tiles per tiled run, gpt-image-1 (cloud API) produced field-like shapes with inaccurate boundaries; SAM 3 (remote GPU, run unprompted with score thresholding disabled) produced fragmented masks over roughly a third of the scene, many covering partial parcels; and DelineateAnything (local CPU, detection confidence threshold 0.1) captured most visible field parcels with tight boundaries, with visible artifacts where parcels cross tile edges. The paper states explicitly that this is a qualitative demonstration, not a benchmark, and makes no timing, accuracy, or statistical claims.

Methodology in Plain English

The researchers built a two-part system. The first part is a server (nxmndr) that knows how to load models from four places: local PyTorch weight files, ONNX graphs, HuggingFace repositories, and cloud APIs. The second part is a QGIS plugin that talks to that server.

The plugin works like this. A user picks a raster layer already loaded in QGIS, selects a model through a browser panel that can search HuggingFace, take a local file path (filtered to .onnx, .pt, and .pth files), or reuse a previously loaded model from a saved registry capped at twenty entries per inference provider. The user then chooses a task (segmentation, object detection, or embedding generation), maps the raster bands to the model's expected R/G/B inputs, sets a chip size and stride (default 512 pixels with no overlap), and selects a compute provider.

When the user clicks Run, the plugin cuts the visible map extent into a chip grid overlaid on the QGIS canvas and submits it as a background QGIS task. Each tile streams to the server, which routes it to a local CPU, a local GPU, or the GPU of a remote host. A tile-status overlay recolors in real time as tiles move through pending, in-flight, OK, error, or cancelled states (white, yellow, green, red, or dashed gray). Cancelling stops reading new tiles, ends the stream, sends a session-cancel call, and marks unfinished tiles as cancelled.

The plugin reassembles results into a GeoTIFF, georeferences it either on the input layer's own grid (coordinate reference system and full geotransform) when GDAL can read the layer, or on the canvas coordinate reference system and the rendered extent's grid for a layer it cannot, such as a web tile service. Partial tiles at the raster margin are padded to a full chip with the input's nodata value (zero if it has none), marked invalid, and cropped back after inference. A failed tile is masked as no data in the output so it stays distinguishable from a background prediction, and is also surfaced as an error in the live overlay.

A separate "Quick Labels" path bypasses tiling entirely. It extracts class names and RGB values from the renderer of a vector layer the user picks (or uses a default class list), injects them into a prompt template, and sends the prompt with the current extent's imagery to a cloud vision-language model such as gpt-image-1 hosted on Azure AI Foundry or the OpenAI platform. The returned mask is georeferenced and added as a raster layer, so both workflows converge on the same kind of QGIS layer.

For the case study, the researchers held tile size, band mapping (RGB), and input region constant so that any variation in output would reflect model behavior rather than configuration. All three models received exactly the same pixels.

Why This Matters

Impact on research. Existing QGIS-integrated tools each solve part of the problem but are constrained on at least one of supported model sources, inference task, or where inference executes. Geo-SAM is limited to the SAM family; SamGeo is confined to SAM variants and is stateless per request, requiring an independent request per tile; GeoAI reaches a remote model only for its vision-language path via a vLLM endpoint while its PyTorch, ONNX, and timm models run in-process; Deepness is tied to a single model format; IAMAP is scoped to feature extraction and runs only locally. The paper argues that no existing project couples source-agnostic remote delegation to a tile-streaming protocol with server-side session state, which is the combination Anaximander provides. This matters because reproducible, low-friction comparison is a precondition for systematic model selection in remote sensing.

Real-world applications. The paper names the operational domains that motivate the work:

• Agriculture, including field boundary delineation and crop monitoring.
• Disaster response.
• Environmental monitoring.
• Agentic Earth observation, in which a planner selects task-specific models for subtasks such as building-footprint extraction, crop-type classification, and change detection.

Industry relevance. The system deliberately builds tiling, georeferencing, and coordinate handling on top of the GDAL library that ships with QGIS, so the plugin adds no geospatial dependency to the user's QGIS installation. It also keeps machine learning dependencies out of QGIS's pinned Python environment by isolating local compute in a separate worker process. The paper argues that a thin GIS wrapper over generic inference servers such as Triton, TorchServe, or Ray Serve is insufficient because geospatial model inference is tied to session and tile semantics and requires maintaining geographic metadata (coordinate reference system, geotransform, chip grid position). The MIT-licensed release of the backend and protocol is intended to let the community extend the registry of model sources and compute backends.

Future Directions

• System benchmarks and non-RGB imagery. The paper names these as the most immediate extensions, since the current case study is qualitative and makes no timing or accuracy claims.
• Broadening the input assumption. Supporting multi-scale image pyramids, auxiliary non-image inputs (digital elevation models, temporal stacks, text metadata), and non-image geometries (point clouds, vector features) would expand the class of models that fit the streaming protocol beyond its current single-resolution-image assumption, requiring changes to the server's streaming message schema and the plugin's tile-generation logic.
• Input preprocessing. The server currently hands a torchvision or torchgeo model class the chip exactly as sent, unscaled and unnormalized, so pretrained weights from those catalogs need their training-time scaling and normalization exported into the model. A per-model preprocessing field, applied by the server and checked against the catalog's published statistics, is the proposed extension.
• Active learning and managed deployment. Integrating active-learning loops would turn the tile-status overlay into a human-in-the-loop labeling surface, and exposing the server through managed-deployment templates (Kubernetes operators, serverless functions) would simplify large-area mapping for teams without dedicated GPU infrastructure.
• Agentic use. The same gRPC surface that drives the QGIS plugin could drive an LLM-based planner that queries the nxmndr model registry and picks the appropriate model for each subtask, turning Anaximander into shared infrastructure for interactive and semi-autonomous Earth observation workflows.
• Open question on tiling artifacts. The paper leaves seam handling to the user rather than to a general stitching procedure: it expects a user to recenter an object split by a seam and run a smaller-scale inference call instead.

Target Audience

Remote sensing practitioners and GIS analysts who want to evaluate deep learning models without writing integration code; computer vision and machine learning engineers building or deploying geospatial inference infrastructure; researchers who need reproducible side-by-side model comparisons on satellite imagery; and developers interested in extending an open-source inference server and protocol to new model sources, compute backends, or input modalities. Readers without background in model serving, RPC protocols, or GIS raster handling will find the architecture and case-study sections demanding.

Authors’ abstract

Applying deep learning models to satellite imagery from within geographic information systems (GIS) remains high-friction for remote sensing practitioners. Models arrive in incompatible formats and target different compute environments, from local workstations to serverless cloud services. As a result, every evaluation demands custom deployment, tiling, and georeferencing code before a single prediction reaches the analyst's map. This friction discourages systematic comparison in a domain where model choice directly affects operational outcomes such as field delineation, crop monitoring, and disaster response. We present Anaximander, an open-source system that unifies model source and compute location choice behind one interactive interface. The system's backend is an inference server that loads models from multiple commonly-used sources and serves them on any accessible compute backend. The server provides session management and model caching, and streams results back per tile. The backend is paired with a QGIS plugin that drives tiling, result reassembly, georeferencing, and real-time per-tile status visualization. An additional user-interface path injects layer legends as prompts into vision-language models. We demonstrate the system in a code-free side-by-side comparison of three heterogeneous models on an agricultural field delineation task: gpt-image-1 via a cloud API, Segment Anything Model 3 (SAM3) on a remote GPU, and DelineateAnything on a local CPU. The inference backend and protocol are open-source and available at https://github.com/microsoft/nxmndr.