SkinMind: Equipping Agentic RAG with Human-Inspired Graph Probing for Reliable Dermatology AI

AI summary

Overview

Research area: Retrieval-augmented generation (RAG), agentic search, knowledge graphs, and multimodal clinical reasoning — applied specifically to dermatology (cs.IR).

Technical level: Intermediate. Readers should be comfortable with standard RAG pipelines and basic graph concepts; the paper explains its agentic loop and graph schema clearly enough that no prior dermatology or graph-database background is required.

Scope (one sentence): The paper introduces SkinMind, a multimodal agentic RAG framework that lets a policy model explore a purpose-built dermatology knowledge graph (SkinMind-KG) through eight hierarchical probing tools, and evaluates it on six dermatology visual question answering benchmarks with two policy models.

What This Paper Is About

Conventional RAG retrieves a fixed set of text chunks once, using only the original query, which is a poor fit for medical reasoning where the information need changes as the reasoning unfolds and where evidence is scattered across text, relations, and images. SkinMind reframes the knowledge graph not as a passive search index but as an interactive environment that an agent probes step by step — first finding a relevant entity, then inspecting its relations, following paths, and only then pulling full descriptions, source passages, or clinical images. The goal is reliable, traceable, evidence-grounded dermatology assistance rather than fluent but unverifiable answers.

Key Contributions

1. SkinMind framework. A multimodal agentic RAG framework that treats a knowledge graph as an interactive environment, with an orchestrator (serving as the policy model) that interleaves reasoning and evidence seeking across eight graph-probing tools, plus a slow–fast routing step that sends simple queries to direct answering and complex ones into the graph-probing loop.
2. Progressive-disclosure tool suite. Six tools for textual/graph exploration (find_node, read_node, read_chunks, probe_relations, probe_neighbors, probe_path) and two for visual evidence (search_image, describe_image), which return compact results first and expose detailed attributes, source text, or image metadata only on request.
3. SkinMind-KG. A large heterogeneous dermatology knowledge graph built from a 62M-token corpus of 227 clinical guidelines and 542 textbooks published since 2015 plus 423,016 images from Derm1M, using a dermatologist-informed schema of 23 entity types and 25 relation types, the PanDerm ontology (127 disease concepts) as a taxonomic backbone, and a detect-then-resolve conflict-resolution mechanism.
4. Extensive evaluation. Experiments across six dermatology VQA benchmarks under two policy-model settings showing consistent gains over direct answering, GraphRAG, LightRAG, Naive RAG, A-RAG, and RAGentA, plus ablations and a token-efficiency analysis.

Main Findings

• Gains over direct answering: SkinMind improves the six-benchmark average from 59.20 to 67.90 (+8.70 points) with GPT-5.4-mini, and from 60.53 to 69.42 (+8.89 points) with Gemini-2.5-Flash. Under Gemini-2.5-Flash, per-benchmark gains over direct answering range from 4.1 to 19.7 points.
• Gains over the strongest RAG competitor: Averaged across benchmarks, SkinMind beats LightRAG by 5.75 points with GPT-5.4-mini and 3.60 points with Gemini-2.5-Flash. Per-column deltas against the best baseline reach +11.67 (SNU134-4, GPT-5.4-mini) and +8.44 (SNU134-4, Gemini-2.5-Flash).
• Competitive with stronger standalone models: SkinMind with GPT-5.4-mini averages 67.90, comparable to GPT-5.5's 68.08, and exceeds DermoGPT-RL, SkinVL-PubMM, and Lingshu-32B by 1.36, 16.35, and 26.00 points respectively — though the authors state these standalone models are contextual references and not directly comparable.
• Graph exploration matters most for DDI: Removing probe_relations, probe_neighbors, and probe_path drops DDI from 65.09 to 62.65 (−2.44) and SNU134-4 from 85.00 to 82.50 (−2.50), the largest DDI degradation among the ablations.
• Image retrieval matters most for SNU134-4: Disabling search_image and describe_image drops SNU134-4 by 5.83 points (to 79.17) and DDI by 1.83 (to 63.26).
• Progressive disclosure helps: Making read_node return source chunks directly (bypassing read_chunks) costs 4.17 points on SNU134-4 and 0.61 on DDI.
• Fewer evidence tokens, better scores: On Morph-R with Gemini-2.5-Flash, SkinMind uses 1,380 retrieved evidence tokens per question versus 7,901 for LightRAG (5.7× more). SkinMind spends 21% of its tokens on outputs (5,576 tokens) versus 2% for LightRAG (291 tokens, with 11,802 input tokens); total usage is about 26.1k tokens for SkinMind versus 12.1k for LightRAG.
• Graph structure is highly skewed: SkinMind-KG has average degree 4.05, median/p90/p99 degree of 1/5/47, and a maximum degree of 64,345 — 52.7% of non-isolated nodes are degree-one — which the authors cite as motivation for relation-level rather than blanket neighbor expansion.
• Conflict resolution is largely automatic: Of 1,375 description-level conflicts processed, 1,366 (99.3%) were resolved automatically and nine were referred to expert review.
• Illustrative case: In the worked example, SkinMind correctly identifies lichen amyloidosis after iteratively retrieving similar images and comparing competing diagnoses, while Gemini-2.5-Flash and LightRAG answer incorrectly.

Methodology in Plain English

The system first decides whether a question is easy or hard. Easy questions are answered directly from the model's own knowledge. Hard ones — those needing external evidence, images, or multi-step reasoning — enter a loop where a "policy model" acts as an orchestrator that chooses one tool at a time.

The tools follow a need-to-know hierarchy. The agent can search the graph for entities by text, or search for images by text or by an uploaded image (encoded with DermLIP and matched by cosine similarity). Once it has a node, it can list which relation types exist around it, expand along a chosen relation to see neighbors, or search for a path between two nodes. Only when it needs detail does it read the node's description, and only when that is insufficient does it pull the original source passages. Images likewise expose captions, morphology, and disease labels on request. Each tool call returns an observation appended to the interaction history, and the loop ends when the agent calls a Finish action or hits the interaction budget. To keep context from ballooning, any single observation over 4,096 tokens is compressed, and if accumulated context exceeds 16,384 tokens, earlier interactions are summarized.

The graph itself was built in stages. Documents were parsed with MinerU, cleaned, and split into 1,200-token chunks (with 100-token overlap per the appendix). GPT-5.4-mini extracted entities and relations from each chunk according to the 23-entity / 25-relation schema, with every extraction kept linked to its source chunk so evidence remains traceable. A conflict detector flagged contradictory claims about the same fact in the same clinical context; Gemini-3.1-Pro searched for external evidence to adjudicate, resolving 1,366 of 1,375 cases automatically. Images were then attached: labels were matched exactly to disease node names, unmatched labels were matched to the top-10 nearest disease nodes using Qwen3-Embedding-4B, and GPT-5.4-mini made the final alignment decision, producing ILLUSTRATED_BY edges.

For evaluation, all RAG baselines share the same SkinMind-KG, the same retrieval backend (GraphML for topology, Qdrant for embeddings), the same answering model, and identical instructions. Image leakage was controlled by MD5 deduplication and by excluding retrieved images with DermLIP cosine similarity above 0.98 to the query image. Agentic methods were allowed 20 retrieval rounds per question, with k=5 as the default retrieval size, temperature 1.0, and a 32,768-token maximum context.

Why This Matters

Impact on research. The paper argues that the bottleneck in medical RAG is not access to knowledge but how that knowledge is explored. By exposing graph structure as an action space rather than a similarity index, it offers a concrete alternative to one-shot chunk retrieval and shows that an agent can reach better answers with far fewer retrieved evidence tokens. It also contributes a reusable, source-linked, multimodal dermatology graph and a leakage-controlled evaluation protocol over six benchmarks.

Real-world applications:

• Patient-facing dermatology triage and online consultation, where answers must cite authoritative guidelines rather than rely on parametric memory that cannot be updated.
• Clinician decision support for morphology and differential diagnosis, where the agent can surface visually similar cases alongside textbook descriptions and guideline text.
• Continually updatable medical knowledge bases, since the graph can be revised independently of the policy model as guidelines change — unlike retraining a model.
• Auditable, provenance-bearing medical QA, where every claim can be traced back to a source chunk, document identifier, or labeled image.

Industry relevance. The context-efficiency result is directly commercially relevant: SkinMind retrieves the most compact evidence set among evaluated methods while scoring highest, which matters for inference cost at scale. The slow–fast routing, which avoids graph exploration for simple queries, is a practical cost-control pattern. However, the higher total token usage (about 26.1k versus 12.1k for LightRAG) because of multi-round interaction is a latency consideration the paper reports rather than resolves.

Future Directions

• Reducing the cost of multi-round interaction. SkinMind uses roughly twice the total tokens of LightRAG despite using far fewer evidence tokens; cutting the multi-turn input overhead (history plus tool instructions) is an explicit open engineering question.
• Extending the framework beyond dermatology. The pipeline depends on a specialty-specific ontology (PanDerm, 127 disease concepts) and expert-defined schema; replicating this for other specialties with different evidence structures is untested here.
• Scalable human oversight for conflicts. Only nine of 1,375 conflicts were escalated to experts in this corpus — how the detect-then-resolve procedure behaves as source volume and disagreement grow is not addressed.
• Unlinked visual evidence. 121,455 image nodes (28.7% of image nodes) remain isolated from disease nodes because no reliable label alignment was found; recovering or evaluating the impact of this gap is a natural next step. Clinical validation and safety evaluation in real deployment settings are likewise not reported.

Target Audience

Researchers and practitioners working on retrieval-augmented generation, agentic tool-use, and knowledge-graph construction — especially those applying these methods to high-stakes medical domains. It is also useful for clinical informatics teams building evidence-grounded dermatology assistants, and for engineers who care about retrieval efficiency and provenance, since the paper reports both accuracy and token accounting against competing RAG designs.

Authors’ abstract

Reliable dermatology AI requires access to verifiable evidence distributed across heterogeneous textual and visual sources. However, conventional retrieval-augmented generation (RAG) typically relies on one-shot retrieval from flat corpora, limiting its ability to adapt retrieval granularity, explore multi-hop relations, and integrate multimodal clinical evidence. We introduce SkinMind, a multimodal agentic RAG framework that equips policy models with human-inspired graph…