SciGraphSelect: Auditable Scientific Data Selection through Knowledge--Reasoning Graphs

AI summary

Overview

Research area: AI for Science — specifically, training-data curation for scientific foundation models, combining knowledge-graph construction with document retrieval/ranking.

Technical level: Intermediate to Advanced. Readers should be comfortable with embedding similarity, precision/recall-style ranking metrics (P@k, AUPRC, AUROC), and basic graph concepts (typed nodes, directed edges, paths, DAGs).

Scope (one sentence): The paper introduces SciGraphSelect, a framework that builds a typed knowledge–reasoning graph from domain references and uses it to rank scientific documents for training corpora while exposing the reference-grounded evidence behind each selection.

What This Paper Is About

Most large-scale data-selection methods reduce each candidate document to a single relevance or utility score, so it is hard to say why a paper was kept. The authors argue that two documents can be equally "about Biology" or "about Chemistry" while differing substantially in the knowledge they cover, the scientific relations they support, and the reasoning operations they instantiate.

SciGraphSelect addresses this by extracting domain-specific knowledge concepts, reusable reasoning operations, and reference-attested relations into a typed directed acyclic graph, then scoring candidate documents against that structure using concept coverage, relation and path co-support, and directional alignment — all while keeping the document-side claims deliberately conservative.

Key Contributions

1. A reference-grounded knowledge–reasoning formulation of scientific data selection. The framework separates domain-specific knowledge concepts (mechanisms, structures, reactions, properties, laws) from reusable reasoning operations (calculation, comparison, mechanistic analysis, structure elucidation, evidence interpretation), while retaining reference provenance for every graph element.

2. A document-scoring method combining multiple structural signals. Scores aggregate knowledge and reasoning coverage, relation and path co-support, and directional alignment. The authors explicitly distinguish these signals from document-level relation entailment, so a graph match is never treated as proof that a document expresses a relation.

3. A "rich reference structure, conservative document evidence" design principle. Reference edges encode reference-supported semantic dependencies, but candidate documents are scored with a conjunctive bottleneck (min over the participating nodes' support) and a separate forward-versus-reversed margin, rather than an entailment check.

4. Evaluation across Biology and Chemistry with human verification. Beyond source-label proxy metrics, the authors collect human judgments of selection relevance and of attribution correctness for concepts, reasoning operations, and relations.

Main Findings

• Direction-aware structure wins overall. "Final direction-aware" scoring produced the strongest selection in both domains under both settings. On heterogeneous hard negatives, Chemistry P@1% rose from .7681 (embedding similarity) to .8753, and Biology from .2519 to .6701. On the PeS2o scientific-paper pool, Chemistry P@1% rose from .8229 to .9635 and Biology from .8780 to .9405. The paper reports only a minor exception at the broader Chemistry hard-negative budget.

• Directional alignment alone is a poor ranker. Used as an independent criterion, it collapsed — for example, Biology hard-negative P@1% of .3408 and AUPRC of .266727, below plain embedding similarity. The authors conclude directional consistency is useful only when anchored by reference-grounded relevance (their "Takeaway 1").

• Concept aggregation drives the largest initial gain. Moving from embedding similarity to concept-only scoring replaced 38.23% of the Chemistry top-1% set and 27.95% of the Biology set, with net target-domain gains of +10,459 and +4,997 documents respectively.

• Relation co-activation is small and strongly domain-dependent. It added 1,887 net Chemistry documents but only 58 Biology documents, with turnovers of 17.55% and 1.00%.

• Reasoning/path evidence changes what is selected, not purity. It yielded only +20 net Biology documents and −695 in Chemistry, so it reshapes content rather than uniformly raising target-domain precision.

• Directional fusion substantially reorders selections. Adding directional alignment to the structure-aware score gave +9,595 ΔTP in Chemistry and +4,369 in Biology, with turnovers of 31.85% and 74.31% — the Biology case showing heavy reordering of the top-ranked set.

• Humans prefer SciGraphSelect-only documents. Documents uniquely selected by SciGraphSelect scored higher on human relevance than embedding-only selections: .853 vs .821 in Chemistry (+.032) and .963 vs .915 in Biology (+.049).

• Attribution reliability is uneven, and relation-level support is the weak point. Biology: relevance .939, concept .890, reasoning .789, relation .503, κ = .719. Chemistry: relevance .837, concept .524, reasoning .521, relation .426, κ = .691. The authors read the weak relation scores as consistent with their conservative interpretation — co-support of linked nodes does not guarantee the relation is expressed.

• The two graphs differ enormously in shape. Biology: 38,217 references, 35,822 knowledge nodes, 20 reasoning nodes, 5,716 backbone edges, 6,320 attested paths. Chemistry: 146,871 references, 197,462 knowledge nodes, 23 reasoning nodes, 53,686 backbone edges, 56,711 attested paths. Chemistry's edges are overwhelmingly K→R (53,650), with only 12 K→K and 24 R→R edges.

Methodology in Plain English

Step 1 — Build a reference graph per domain. The authors collect scientific QA and reasoning resources (Chemistry: HLE Chemistry, ChemQuests, WildSci Chemistry, Nemotron Science Chemistry; Biology: GPQA Biology, HLE Biology, AA-Omniscience Biology, Nemotron Science). GPT-4o-mini tags each reference into reusable knowledge concepts and reasoning operations using domain-specific taxonomies and controlled vocabularies, plus relations the reference actually supports. Unsupported links are returned as empty arrays rather than inferred from co-occurrence.

Step 2 — Clean up the concept vocabulary. Extracted concepts are normalized and near-duplicates are merged conservatively, using Qwen3-Embedding-0.6B to retrieve candidates. Merging requires cosine similarity ≥ 0.99, mutual top-k neighborhood, compatible types, token Jaccard ≥ 0.80, sequence similarity ratio ≥ 0.85, matching numerical tokens, and complete-link cluster cosine ≥ 0.99; pairs above 0.92 that fail these tests are kept for review rather than collapsed. In Chemistry this reduced 198,214 extracted knowledge nodes to 197,462, with 764 automatically merged pairs.

Step 3 — Form a typed DAG. Nodes are knowledge (K) or reasoning (R); edges are K→K, K→R, or R→R. Cycles are removed, and only short paths attested within a single reference are retained rather than all graph-theoretically valid paths.

Step 4 — Score documents conservatively. Each graph node gets an activation score based on embedding cosine similarity against a threshold, rescaled to [0,1]. Reference edges get weights from mean annotation confidence, conditional support (n(u,v)/n(v))^ρ, and support frequency log(1+n(u,v)), normalized over incoming edges. Document-level edge evidence is the minimum of the two endpoint activations times the edge weight; path evidence is the minimum activation across the path — a conjunctive bottleneck requiring all parts to be supported.

Step 5 — Handle direction separately. Because co-support cannot distinguish u→v from v→u, the authors verbalize each edge in forward and reversed form and take the difference of the document's cosine similarity to each. A positive margin means better alignment with the reference direction.

Step 6 — Aggregate and fuse. Aggregation is domain-specific: Chemistry averages the top-16 knowledge nodes and top-5 reasoning nodes, uses top-16 weighted edge means and unweighted top-16 path means for a base group (K→K ∪ K→R) and a reasoning-only group (R→R), then combines structure-aware scores with directional margins after ECDF normalization. Biology's exact composition is given in Appendix B, which is truncated in the supplied text.

Step 7 — Evaluate. Ranking is measured with P@1%, P@5%, P@10%, AUPRC, and AUROC against PeS2o source-domain labels as a proxy, plus a human study of relevance and attribution support.

Why This Matters

Impact on research. The paper shifts interpretability upstream — from explaining model outputs to explaining the composition of the training corpus itself. It also draws a careful line between structural co-support and entailment, which is a useful methodological discipline for anyone tempted to read graph matches as evidence of reasoning. It positions itself as complementary to scientific claim verification systems such as SciTrue, and as a structural successor to MASS's skill-graph selection for mathematics.

Real-world applications:

• Pharmaceutical and biotech literature curation: selecting papers that cover specific mechanisms plus the reasoning operations needed to interpret them, with an audit trail for each inclusion.
• Enterprise scientific LLM training pipelines: building corpora where each document's inclusion can be justified to reviewers, compliance teams, or downstream model owners.
• Domain expansion into new fields: swapping in domain-matched references to build graphs for materials science, physics, or clinical literature using the same machinery.
• Data provenance and governance: exposing which reference-attested concept and relation evidence supported each selection, as a basis for documented curation decisions.

Industry relevance. The work comes from KT Corporation, and the framing is practical: a modular pipeline where the graph is reusable and the scorer is swappable. The reported hard-negative gains (+.107 P@1% in Chemistry, +.418 in Biology over embedding similarity) speak directly to the problem of separating genuine target-domain science from superficially similar or reasoning-heavy distractors in a mixed web-and-scientific corpus.

Future Directions

1. Finer-grained document evidence. The authors note that because reference structure and document evidence are separated, the whole-document embedding scorer can be replaced by sentence-, span-, or relation-level models while reusing the same graph — the natural fix for weak relation-level attribution (.503 Biology, .426 Chemistry).

2. Strengthening relation-level attribution. Relation support is explicitly identified as the primary remaining weakness; closing it likely requires evidence extraction rather than embedding similarity.

3. Validating downstream utility. SciGraphSelect measures relevance and structural coverage, not factual correctness or training benefit. Whether these selections actually improve scientific foundation models remains unmeasured — the evaluation relies on PeS2o source labels as a proxy.

4. Extending beyond Biology and Chemistry. The framework is designed to extend to new domains through domain-matched references, but the two domains studied produce very different graph shapes (20 vs 23 reasoning nodes, 5,716 vs 53,686 backbone edges), so generality is untested.

Target Audience

Machine-learning researchers and engineers working on pretraining data curation; AI-for-Science teams building scientific corpora; knowledge-graph and retrieval researchers interested in provenance-aware ranking; and practitioners at organizations building domain-specific foundation models who need selection decisions that can be audited rather than only scored. Readers focused on model architecture or training objectives will find little here — the contribution is entirely in the data-selection layer.

Authors’ abstract

Scientific data selection is typically driven by relevance or utility scores, providing limited insight into why a document is retained. We introduce SciGraphSelect, a reference-grounded framework for auditable scientific data selection. SciGraphSelect organizes domain-specific knowledge, reusable reasoning operations, and reference-attested relations into a typed knowledge--reasoning graph, then scores candidate documents using concept coverage, relation and path co-support, and directional alignment. To avoid over-interpreting document embeddings, rich relational structure is preserved on the reference side while candidate-side evidence is treated conservatively without assuming relation entailment. By exposing the reference-grounded evidence behind each selection, SciGraphSelect brings traceability to the upstream construction of scientific training corpora and complements downstream scientific verification.