What Git Does Not Record: Provenance Legibility in Two AI-Assisted Research Repositories

AI summary

Overview

Research area: Digital libraries and research provenance — specifically, what an ordinary Git repository can attest about AI involvement in computational research, and how AI-trace artifacts (standing rule files, agent scratchpads) behave as records.

Technical level: Intermediate. The argument is conceptual, but following the measurements requires comfort with Git concepts such as commits, git blame, trailers, path-limited logs and renames.

Scope: A two-case audit of two single-investigator environmental-science repositories (466 and 103 commits) by the same author, measuring how legible each repository's AI-trace artifact is and how much the commit log's Co-Authored-By trailer actually discloses.

What This Paper Is About

Journals, conferences and funders now require authors to disclose how generative AI contributed to a piece of work, but policies say where the statement goes, not where its content comes from. In computational research the obvious source is the project's version-control repository, which is already kept, timestamped and treated as the authoritative record of how the work was built. The authors ask what such a repository can actually supply, by auditing two repositories that carry structurally different classes of "agent residue."

Key Contributions

1. A field-by-field taxonomy of what a normally kept Git repository can, cannot, and only apparently records about the provenance of a change, separating fields that are unrecoverable from fields that are recoverable but misleading.
2. A legibility measurement over two AI-trace artifacts: what share of the lines ever committed to each is visible at the current revision, which revisions survive into it, and how each artifact compares with every other tracked text file in its own repository.
3. A measurement of the commit log's disclosure channel: its sensitivity against an explicit within-repository proxy for agent activity, and the instability of its disclosure rate over time, with the alternative explanations the authors cannot rule out.
4. Six tool-agnostic requirements for in-situ provenance capture, derived field by field from the taxonomy's gaps, supported by an auditable evidence ledger with one row per artifact revision, disclosing commit, tracked text file and blame cohort.

Main Findings

• Two artifacts, opposite legibility. The prescriptive trace (Project A's directory of standing rule files, 466 commits, Claude Code and Cursor) keeps 85.6% of the lines ever committed visible at the current revision, but a standing rule states a conclusion and discards the episode behind it. The narrative trace (Project B's Cursor-assisted scratchpad, 103 commits) records episodes but shows only 21.8%. The failures are complementary: the artifact that records the process does not show it, and the artifact that shows itself does not record the process.

• By revision the gap is far smaller. 15 of 41 prescriptive revisions leave no visible line, against 18 of 41 narrative revisions. Both are top-heavy: Project A's newest 3 revisions (7.3% of its history) supply half of everything visible, largely because one revision contributes 853 of the 1,915 visible lines; Project B's newest 7 (17.1%) do the same there.

• Raw line counts behind the shares. Project A committed 2,236 lines, removed 321, and shows 1,915 now (14.4% no longer visible). Project B committed 1,537, removed 1,202, shows 335, so 78.2% is history-only; across 39 revisions it grew by a net 335 lines while absorbing 2,739 lines of change.

• Both artifacts are unusual next to their neighbours. Both repositories' median tracked text file is 0% history-only, and 58.0% of Project A's files and 67.0% of Project B's have no history-only content at all. Project A's rule directory sits at the 87th percentile of 174 files (22 above it); Project B's scratchpad at the 99th of 88, with exactly one file above it.

• The commit log's audited field covers little. Co-Authored-By naming an AI identity appears on 66 of 466 Project A commits (14.2%) and 1 of 103 Project B commits (1.0%) — a naive reading of roughly fourteen times more AI use in A.

• The choice of trailer key is part of the result. Project A carries 3 AI trailer keys: Co-Authored-By on 66, Made-with on 55 (naming the other tool), and Claude-Session on 2. No commit carries both Made-with and Co-Authored-By, so the keys divide the record rather than corroborate it; 121 of 466 commits (26.0%) carry one. An audit reading Co-Authored-By misses 55 disclosures, 45.5% of those in the repository.

• Low sensitivity against a within-repository proxy. In Project B, 39 commits revise the scratchpad and 1 (2.6%) carries an AI trailer — the remaining 38 do not. In Project A the same cross-tabulation gives 5 of 15 (33.3%).

• The rate is unstable in time. Project A's disclosure rate runs at 3 of 111 commits (2.7%) through 2025-12 and 63 of 355 (17.7%) afterwards, a factor of 6.6. The boundary was chosen after inspecting the monthly series, and the repository has no commits in 2026-01, so this is a resumption rather than a clean adjacent-month change. Using all trailer keys instead moves the step from 6.6 to 12.3.

• The query decides part of the answer. Three defensible history queries for the same artifact move the history-only share by at most 4.2 points in Project A and 2.1 in Project B, but Project A's revision count varies by a factor of 5.9 and its apparent start date by 228 days. git blame attributes 686 of 1,915 visible lines (35.8%) to 26 commits the path-limited log does not list, the earliest 2025-09-01, because the rule directory's first thirteen files entered it by rename on 2026-04-06.

• The trailer's free text is thin. Project A's 66 disclosing trailers take 6 distinct values, giving six answers for at most four models and 3 commits answering nothing (a bare vendor name). Two of the five model-version strings differ only by a context-window suffix and denote the same model.

• A trailer is not an involvement rate. The authors cannot rule out that the author simply used an agent more after the boundary; rejecting that would require a tool-settings history or a record of which process authored each commit, and neither repository contains either. The undercount also varies with the key read (45.5% of Project A's disclosures lost, none of Project B's), with the commit (33.3% vs 2.6%), and with the date (6.6 within one project).

• Reconstruction from the prescriptive trace fails uniformly. Coding every rule on three separate dimensions, origin evidence is unknown for every rule, human review state is not observed, and evidence strength reaches tool metadata at best and inference only for why a rule exists.

• One mechanically checkable row works. Comparing dates each artifact asserts about itself against the commit that introduced the line returned exactly one disagreement: a section header reading 2025-01-XX on a line written on 2026-01-20 — an unfilled template placeholder. Nothing in the workflow runs this check.

• Everything here depends on a non-default choice. Neither artifact is git-ignored (verified, not assumed). Had either author excluded it before first committing it, every number in the paper would be zero.

• Figure/table discrepancy, as reported. Figure 2's panels show 85.5% and 19.7% rather than the text's 85.6% and 21.8%, because panel percentages are shares of 41 blame cohorts (with 1,703 lines for Project B) rather than of the path query's totals.

Methodology in Plain English

The unit of analysis is not the project but the AI-trace artifact: the file or directory an agentic tool maintains in the repository to carry state between sessions, and the thing an auditor would actually open. Taking the artifact as the unit is what makes two projects with different research content comparable.

For each artifact the authors walk every commit touching any path it has ever occupied, record per-revision insertions and deletions, count the lines still present at a fixed census commit (2026-08-21 for Project A, 2026-02-03 for Project B), check whether the artifact is git-ignored, extract commit-message trailers, and cross-check dates the artifact asserts about itself against the commits that wrote those lines. Four scripts run over the repositories, a fifth converts their output into plotted coordinates, and a sixth checks every printed number against them.

Three measurement decisions are defended explicitly. First, the history query is stated: the authors use the path-union form (a path-limited log is what an audit would run) rather than --follow, and show all three forms side by side. Second, disclosure is counted from structured trailers, not from AI vocabulary matched as prose — a substring sweep reports 40.8% for Project A against the trailer's 14.2%, and 124 of the 190 matched commits carry no Co-Authored-By at all. Third, the census is closed: Project A is a live repository and the paper was written inside it, so its commits would raise the rate they report; pinning to a census commit prevents that.

Distributional context comes from measuring every other tracked text file with at least 20 lines ever inserted in the same way, and revision-level survival comes from git blame at the census. The rule coding in Section 7 keeps origin, human response and evidential basis as separate dimensions, on the argument that collapsing them into one label makes such coding unfalsifiable.

Why This Matters

The paper reframes AI disclosure from a writing task into a record-keeping problem, and shows that the record most institutions would reach for — the Git repository — constrains a provenance claim without establishing one. A disclosure rate read off a commit log is a rate of commits that say so; no repository evidence converts it into a rate of commits a model helped write, and the error is not a constant that pooling over a corpus would average away.

Real-world applications:

• Journal, conference and funder disclosure review. The taxonomy tells a reviewer which claims a submitted repository can and cannot back, and flags fields that answer confidently and wrongly.
• Research-integrity and reproducibility audits. The six requirements define what an auditor should demand, and the evidence ledger shows how to make each number regenerable.
• AI coding-tool design. Default trailer behaviour matters: one widely used editor switched its AI co-author trailer on by default in one release, reverted it two releases later, and is reconsidering the key name. Competing key names (Co-Authored-By, Made-with, Claude-Session) already split one repository's record.
• Electronic lab notebooks and provenance-capture systems. The requirements are stated tool-agnostically, so they transfer to systems meant to record process as it happens rather than reconstruct it later.

Industry relevance: vendors of agentic development tools, institutional compliance and research-integrity offices, and anyone building attestation or auditing infrastructure on top of version control — supply-chain attestation work already binds identities to artifacts, but presupposes that the events worth attesting were identified first.

Future Directions

• Capture the accept event, not just the result. Whether a human modified model output is the distinction that separates assistance from authorship, and it has to be recorded in the editor or the agent, at the moment of acceptance, because a repository only ever sees the post-decision state.
• Record what the repositories here do not. A tool-settings history and a record of which process authored each commit would let the authors test the rival explanation for the 2.7%-to-17.7% step, which they explicitly decline to attribute to a mechanism.
• Repeat the audit beyond two cases. The authors state the design supports existence claims only — a repository can hold a large record and present a small one, and a disclosure field can vary sixfold within one author's history — and that conditions (tool, trace format, subject, duration, year) are confounded, so a multi-author or controlled design is needed before any contrast can be attributed to trace format.
• Make provenance queries rename-safe and self-checking. Path queries without rename following returned less than half of one artifact's revisions with no sign of partiality, and the date cross-check that caught the 2025-01-XX placeholder is implementable from data the repository already holds but is run by no tool. The Section 7 coding also needs independent coders, since it was done by an author who created both repositories and no inter-rater agreement can be reported.

Target Audience

Researchers and research-software engineers who use agentic coding tools and keep Git repositories; journal editors, conference program chairs and funders drafting or enforcing AI-disclosure policy; research-integrity officers and auditors; and designers of provenance-capture, lab-notebook and AI coding-tool infrastructure. The paper is also directly useful to anyone who assumes a "well-kept repository" is sufficient evidence of how a computational result was produced.

Authors’ abstract

Publishers and funders increasingly ask authors to disclose how AI tools contributed to a piece of research. In computational work the repository is where an author or an auditor would look for the answer, and we ask what it can actually supply. We audit two single-investigator repositories by the same author in the same field, each carrying a different class of agent residue: a prescriptive trace of standing rule files and a narrative trace of a running agent scratchpad, in repositories of 466 and 103 commits. The two are legible in opposite ways. 85.6% of the lines ever committed to the prescriptive trace are visible at the current revision, and a standing rule states a conclusion and discards the episode behind it; the narrative trace records episodes and shows 21.8%. Counted by revision the gap is far smaller: 15 of the prescriptive trace's 41 revisions leave no visible line, against 18 of the narrative trace's 41. The Co-Authored-By trailer, the field an automated audit reads, is one of three AI trailer keys Project A carries and covers 66 of the 121 commits carrying any. In Project B it has low sensitivity against an explicit within-repository proxy: 39 commits revise a file maintained to hold an agent's working notes and 1 (2.6%) carries the trailer. Project A's repository-level rate is unstable in time too, stepping from 2.7% to 17.7% across a boundary we did not specify in advance. Repository channels disagree, and neither rate can be read as an involvement rate; we do not isolate the cause, and say so. We give a field-by-field account of which provenance claims such traces support, which they cannot, and which they answer confidently and wrongly, construct an auditable evidence ledger, and derive the requirements this places on any system meant to record research provenance as it happens.