AI-Assisted Authoring for Transparent, Data-Driven Documents

AI summary

Overview

• Research area: Human-Computer Interaction (cs.HC), at the intersection of LLM-based code synthesis, data provenance, and scholarly publishing/NLP.
• Technical level: Intermediate. The paper assumes familiarity with large language models, functional programming concepts, and provenance-tracking runtimes, but explains the motivation and workflow accessibly.
• Scope in one sentence: The paper presents a proof-of-concept LLM-based authoring tool that turns static quantitative claims in scholarly text into interactive, data-linked Fluid expressions, and evaluates how well language models can synthesise those expressions on a subset of SciGen.

What This Paper Is About

Scholarly articles make most of their claims in natural language, but readers and reviewers have no direct way to trace a phrase such as "system X is faster than system Y" back to the data that supports it. Existing provenance and data-visualisation techniques can link computed outputs (like charts) to source data automatically, but they do not extend to natural language, where claims are actually made. The paper's goal is to combine LLM query synthesis with the provenance-tracking runtime of the open source language Fluid to produce "transparent documents" — web articles where hovering over a text fragment reveals the underlying data.

Key Contributions

1. A proof-of-concept LLM-based tool for iteratively transforming a preexisting opaque document and its associated dataset into a transparent, data-driven counterpart, organised as a human-in-the-loop workflow with two agents (a SuggestionAgent and an InterpretationAgent).
2. A summary of the natural language idioms studied — including data retrieval, ratio, average, min/max, rank, sum, comparison, and generalised quantifiers — each paired with a gold Fluid solution (for example, "the Energy Sector accounts for total methane emissions of 52.80% in 2030", "The average methane emissions for the year 2030 is 13.51", and "3-layer stacked CNN gives an accuracy of 81.46%, which is the lowest compared with BiLSTM, and S-LSTM").
3. An empirical evaluation of how well state-of-the-art models solve the associated interpretation and code synthesis problems, using a sample of the SciGen dataset extended with hand-generated counterfactual test cases.
4. A characterisation of failure modes and unsupported linguistic forms, including problem categories such as false comparison, wrong numerical value, and ambiguous referent.

Main Findings

• Overall synthesis accuracy: With target-value sharing, the InterpretationAgent produced correct Fluid expressions in 74.9% (S.D. 3.0%) of cases, dropping to 57.1% when the target was withheld — indicating the system is robust when given sufficient guidance but degrades when underspecified.
• Category-level differences: Success rates exceeded 68% for comparison, 77.3% for data retrieval, and 97% for min/max search tasks. Accuracy fell to 20% for expressions requiring differences and 0% for ranking tasks.
• Complexity is non-monotonic: Success rates were 62% for single-category expressions, rose to 91% when two categories were combined, and collapsed to 0% when three categories were involved — suggesting moderate composition provides structural cues, while greater complexity overwhelms synthesis.
• Counterfactual robustness failures: Across 300 test executions, 121 contained at least one counterfactual error (an average of 3.8 per case), of which 42 ultimately still succeeded. These are cases where an expression coincidentally yields the correct output on the original data but is not extensionally equivalent under perturbation — for example, a query intended to retrieve the LSTM execution time that incorrectly referred to LSTM2, a mismatch only counterfactual testing exposed.
• Model capability: With models such as gpt-4o and gpt-5, the system could resolve basic table lookups, percentages, averages, minima and maxima, and totals, mapping them onto aggregations like sum, maximumBy, and explicit rank computations. The headline evaluation is measured over 5 runs with gpt-4o, and the paper reports that gpt4o is often able to synthesise compound expressions extensionally compatible with the gold solutions.
• Coverage gaps identified: The paper does not cover approximate quantitative terms ("around 50%", "roughly 100 instances"), interval descriptions ("between 30 and 40%", "within 5–10 seconds"), or graded modal adverbs modifying comparatives ("slightly better", "significantly higher", "steadily increasing", "sharply declining"). The exact size of the SciGen subset used is not reported.

Methodology in Plain English

The authors start from two observations: provenance-tracking runtimes can connect computed outputs to their data automatically, and LLMs can read technical language and write queries. They combine the two in Fluid, an open source programming language whose runtime tracks how data flows through a computation, which is what makes hovering interactions on output possible.

The authoring tool has two agents. The SuggestionAgent scans a paragraph and marks fragments that could be computed from data rather than left as literal text, and the InterpretationAgent takes such a fragment (chosen by the author or suggested automatically) and writes a Fluid expression that evaluates exactly to that string. The agent is steered by a structured system prompt that frames the job as replacing a [REPLACE …] placeholder, and its output must be nothing but a valid Fluid expression.

Synthesis is closed-loop: the candidate expression is run through the Fluid command-line interface to check it evaluates without error, coerces to a string, and matches the target. Failures trigger error-guided iterative re-prompting with the error message attached, up to a maximum number of retries. Outcomes are one of: success, failure with no expression, or an expression that produces a mismatched string — in which case the author may revise their goal (if their original claim was simply wrong) or abort. After synthesis, the author manually validates the proposal by republishing the page and interacting with it before accepting or rejecting the change.

For evaluation, the researchers took a sample of SciGen (an open source dataset of tables from scientific articles with their descriptions), grouped results by linguistic category and by the number of query sub-expressions in the gold solution, and added hand-crafted counterfactual modifications of the tables to test whether generated expressions remain correct when the data changes or when the input contains misleading phrasing.

Why This Matters

The paper targets a concrete problem in research integrity: empirical claims in papers typically lack author-supplied links to data, making them hard for reviewers to check, and retractions are often attributable not to fraud but to errors in data management or analysis. By making the evidential basis of a claim directly inspectable in the text, the approach offers a transparency mechanism that automated fact-checking or chart interpretation tools do not currently provide.

Real-world applications:

• Peer review: Reviewers could trace claims such as "system X is faster than system Y" directly to the supporting records rather than taking them on trust.
• Science journalism and publishing: A journalist or scientific publisher authoring an online article could link textual claims to the underlying open data so readers can explore the evidence base from the article itself.
• Reader-side reconstruction: Someone reading a paper or climate report built on open data could retroactively interpret parts of the text as queries over that data, gradually reconstructing the relationship between claims and evidence for their own comprehension.
• Error and inconsistency detection: Because expressions are computed rather than typed, counterfactual testing can expose mismatches (such as a query referring to the wrong model) that would otherwise go undetected, and could reveal inconsistent use of technical language by authors.

Industry relevance: the work is relevant to scholarly publishers, peer-review platforms, science communication outlets, and vendors of AI-assisted writing environments, since a natural next step is embedding this workflow into developer and authoring environments such as VSCode or Cursor.

Future Directions

• Reducing dependence on predefined helpers: The system currently relies on helper functions such as growShrink and trendWord; the authors want the system to work in their absence, for instance by turning "definition not found" errors into augmented prompts that trigger automatic generation of missing definitions.
• Broadening artifact and idiom coverage: Extending interpretation to visualisations and intermediate datasets derived from cleansing or aggregation, and covering additional idioms such as cardinals, multiplicatives, rounding, and graded adjectives.
• Integrating validation into authoring: Automatically generating counterfactual test cases, which are currently used only as an evaluation device, so robustness checks become part of the authoring workflow; and embedding the system into IDEs for a more seamless experience.
• Handling false statements: Distinguishing referential terms with fixed denotations from queries with data-dependent values, which may help repair false or inconsistent statements so generated expressions stay aligned with both the data and the author's intent.
• Open question — harder idioms: Graded modal adverbs ("slightly better", "significantly higher") and generalised quantifiers ("generally", "usually") remain unaddressed because their interpretation is subjective and context-dependent; the authors note such difficulties also confront human readers.

Target Audience

Researchers and practitioners in human-computer interaction, computational linguistics, and natural language processing; developers of scholarly publishing and peer-review tools; and anyone working on data provenance, interactive documents, or AI-assisted writing. The paper is also relevant to readers interested in transparency and reproducibility in scientific communication, since it frames transparent documents as a way to expose the evidential basis of scholarly claims.

Authors’ abstract

We introduce _transparent documents_, interactive web-based scholarly articles which allow readers to explore the relationship to the underlying data by hovering over fragments of text, and present an LLM-based tool for authoring transparent documents, building on recent developments in data provenance for general-purpose programming languages. As a target platform, our implementation uses Fluid, an open source programming language with a provenance-tracking runtime. Our agent-based tool supports a human author during the creation of transparent documents, identifying fragments of text which can be computed from data, such as numerical values selected from records or computed by aggregations like sum and mean, comparatives and superlatives like _better than_ and _largest_, trend-adjectives like _growing_, and similar quantitative or semi-quantitative phrases, and then attempts to synthesise a suitable Fluid query over the data which generates the target string. The resulting expression is inserted into the article's web page, turning the static text fragment into an interactable data-driven element able to reveal the data that underwrites the natural language claim. We evaluate our approach on a subset of SciGen, an open source dataset consisting of tables from scientific articles and their corresponding descriptions, which we extend with hand-generated counterfactual test cases to evaluate how well machine-generated expressions generalise. Our results show that gpt4o is often able to synthesise compound expressions extensionally compatible with our gold solutions.