Understanding Usage and Engagement in AI-Powered Scientific Research Tools: The Asta Interaction Dataset

AI summary

Overview

Research area: Human-Computer Interaction and information retrieval, specifically the study of how researchers use deployed AI-powered scientific research tools.

Technical level: Intermediate. The core analyses are descriptive (query distributions, interaction rates, session statistics), but the paper also fits binomial logistic regression models and reports odds ratios, so some familiarity with quantitative user-study methods helps.

Scope in one sentence: The paper releases and analyzes the Asta Interaction Dataset, 258,935 anonymized real-world queries and 432,059 clickstream interactions with two deployed LLM-based research assistants, to characterize how researchers query these systems and engage with AI-generated reports.

What This Paper Is About

AI-powered tools for literature discovery and scientific question answering are being adopted quickly, but there is no publicly available large-scale dataset of how researchers actually use them in practice, only aggregate statistics from proprietary logs. To close that gap, the authors release the Asta Interaction Dataset, collected between February and August 2025 from two interfaces inside the Asta platform (PaperFinder, a paper search tool, and ScholarQA, a scientific question-answering tool), alongside a comparative baseline of queries to the traditional academic search engine Semantic Scholar. Their goal is to characterize query patterns, engagement behavior, and how usage changes as users gain experience, and to introduce a new query intent taxonomy suited to AI research assistants rather than traditional search engines.

Key Contributions

1. A public large-scale interaction dataset. The authors release the Asta Interaction Dataset, an anonymized, opt-in collection of 258,935 queries and 432,059 clickstream interactions from deployed AI-powered research tools. They state this is the first public dataset of real-world user interactions with a deployed AI-powered scientific research tool.

2. A behavioral analysis of querying and engagement. They characterize query patterns and engagement behavior across PaperFinder, ScholarQA, and the Semantic Scholar baseline, and track how behavior evolves with experience across three progression stages.

3. A new multidimensional query taxonomy. They introduce a taxonomy with non-mutually exclusive labels for query intent (16 labels), phrasing style (7 labels), and criteria (6 labels), built through an iterative human-and-LLM process, plus 28 fields of study.

4. Identification of which query types the system handles well or poorly. Using logistic regression predicting click-through rate, they quantify which query attributes correlate with higher or lower odds of engagement, as a way of quantifying bias in the behavioral findings.

Main Findings

• Queries are longer and more complex than traditional search. Mean query length was 17.04 words for PaperFinder and 36.96 for ScholarQA versus 5.35 for Semantic Scholar. Mean entities were 4.00 (PF) and 5.14 (SQA) versus 2.25 (S2); mean relations were 2.17 and 2.68 versus 1.20; mean constraints were 0.60 and 0.82 versus 0.15.

• Users treat the system as a research collaborator. Query intents span traditional retrieval but also extend to methodological guidance, interpreting the user's own experimental results, research gap analysis, ideation, and delegating content generation such as drafting full manuscript sections.

• Keyword-style queries still dominate, even among experienced users. Keyword-style phrasing is the most common on both tools, and nearly all (98%) of S2 queries are keyword-style. Broad Topic Exploration and Keyword-style queries dominate the Asta query distribution, a pattern the authors describe as functional fixedness.

• Abstract phrasing replaces jargon. 66% of PaperFinder queries include at least one abstract concept versus 38% in Semantic Scholar, with a median of 1 abstract concept per query in PF versus 0 in S2. Query length correlates positively with the number of abstract concepts (Pearson r = 0.519).

• Generated reports are treated as persistent artifacts. 50.5% of SQA users and 42.1% of PF users revisit previous reports, substantially more than the 18.8% and 14.8% who submit near-duplicate queries. Near-duplicate queries occur on shorter timescales (median under 16 minutes) than revisits (median 4–6 hours).

• Reading of SQA reports is non-linear. Users skip the introduction 43% of the time, and over half of reports (52.4%) involve non-consecutive section expansions; the authors also observe backward traversal and frequent returns to the introduction.

• Behavior shifts with experience. Broad Topic Exploration drops from 61.23% in the single-query stage to 53.48% in the experienced stage, while Citation & Evidence Finding rises from 6.25% to 9.65% and Methodology-Specific Criteria rises from 41.84% to 47.17%.

• Engagement deepens or shifts with experience, depending on the tool. The rate of clicking inline evidence in SQA increases by 27% between a user's first query and their 4th, while PF link clicks drop by 24% over the same period, which the authors attribute to experienced PF users consuming information directly from the result list.

• Latency tolerance differs by tool. Median session duration was 4 minutes for PF and 8 minutes for SQA; median response time was 34 seconds for PF and 129 seconds for SQA. SQA churn stays near 11% for response times under 5 minutes, whereas PF churn increases by 10% relative if responses exceed 1 minute.

• Errors are costly for first impressions. First-time users who encounter an error have only a 10% chance of returning, compared to 53% for users whose initial experience succeeded.

• Explicit feedback is rare. Thumbs up/down feedback appears on fewer than 2% of reports and is less predictive of return than link clicks, which is why the authors use click-through rate as their success surrogate.

• Success varies by query type. For PF, Citation/Evidence Finding (OR = 1.17) and Broad Topic Exploration (OR = 1.12) have higher click odds, while Content Generation and Expansion (OR = 0.47), Data Resource Availability (OR = 0.61), and Temporal Constraint (OR = 0.82) have lower odds. For SQA, Concept Definition and Explanation (OR = 1.29) and Complex Contextual Narrative (OR = 1.47) have higher click odds, while Citation Format Specification (OR = 0.62) has lower odds.

• Report quality correlates with engagement. LLM-assessed high-quality reports have an average CTR of 5.5% versus 3.8% for low-quality reports (a 44% relative increase), and a return rate of 62.3% versus 54.7%.

• Semantic Scholar queries have themselves grown more complex. Comparing S2 queries between 2022 and 2025, queries with at least one constraint rose from 7% to 10%, queries with at least one relation grew from 65% to 78%, and average query length increased from 4.8 to over 6 words.

Methodology in Plain English

The authors log opt-in interactions from two interfaces in the Asta platform: PaperFinder, which returns a ranked list of papers with short generated summaries and links to Semantic Scholar pages, and ScholarQA, which produces a multi-section literature summary report with inline citations that open evidence cards. They compare both against queries issued to the traditional Semantic Scholar search site.

Preprocessing removes bots and canned queries, filters out queries flagged by an LLM as possibly containing personally identifiable information (less than 1% of queries), identifies sessions using a 45-minute UI action timeout, and applies debouncing so a page counts as revisited only after 5 minutes. The released data contains only hashed report identifiers and no user identifiers; internal pseudonymous identifiers were used for cohort and retention analysis but are not released.

For labeling, they sample 30,000 single-turn queries and prompt GPT-4.1 with structured decoding to assign intent, phrasing style, criteria, and field-of-study labels, which are non-mutually exclusive. The taxonomy itself was built iteratively, starting with manual inspection, having Gemini-2.5-pro propose additional labels, and manually consolidating until convergence.

They track four actions (S2 link clicks, section expansions, evidence clicks, and thumbs feedback) and derive click-through rate, churn rate, and return rate. To study learning, they define three stages by cumulative query count: single-query (first query), inexperienced (queries 2 through 10), and experienced (beyond the 10th), tracking the same users over time. Statistical comparisons use two-sided t-tests at α = 0.05 with 95% confidence intervals, and binomial logistic regression models predicting click-through rate, with the Benjamini-Hochberg procedure controlling false discovery across p-values. Because PaperFinder is a multi-turn chat experience while ScholarQA and Semantic Scholar did not support follow-up queries during collection, PF analysis is restricted to the first query in each conversation.

Why This Matters

Most evidence about how people use AI research tools comes from proprietary logs that are never released, so researchers cannot reproduce findings, build realistic evaluations, or compare systems. This paper makes a large real-world interaction log public and pairs it with a taxonomy designed for AI assistants rather than keyword search, which lets others study and benchmark these tools against actual usage rather than synthetic queries. It also shows that user expectations have shifted: people bring general-purpose chatbot habits, such as pasting draft paragraphs or issuing multi-part instructions, to tools built primarily for retrieval.

Real-world applications:

• Interface design for research tools. The finding that reading is non-linear, with 43% of reports having their introduction skipped and 52.4% involving non-consecutive section expansions, supports section-level navigation, TL;DR summaries, and user control over content ordering.

• Query understanding and clarification. Because users often discover unmet requirements only after seeing initial results, as shown by near-duplicate refinements, designers can justify clarifying intent before running long queries.

• Evaluation datasets. The released queries and taxonomy can seed realistic benchmarks that reflect genuine researcher information needs instead of keyword-style test sets.

• Reliability engineering. The sharp drop in first-time return rate after an error, from 53% to 10%, is a concrete argument for graceful degradation and clear error recovery.

Industry relevance: commercial AI search engines, deep research agents, and science-focused platforms face the same latency, error, and retention trade-offs documented here, and the paper's comparative finding that users tolerate SQA's roughly 2-minute latency but penalize PF for exceeding 1 minute gives product teams a concrete expectation to design against.

Future Directions

• Follow-up query and long-term journey analysis. The authors explicitly leave multi-turn follow-up behavior for future work, since PF analysis here was restricted to first queries and SQA and S2 did not support follow-ups during collection.

• Cross-tool usage patterns. They plan to study how users move between PaperFinder and ScholarQA within research workflows, and what triggers transitions between search-oriented and report-oriented tools.

• How mental models evolve. Tracking how users' understanding of system capabilities changes with experience, including why keyword-style queries persist even among experienced users.

• Addressing underserved query types. Content generation, temporal constraints, data resource requests, and citation format specifications all show lower satisfaction; determining how to serve users who expect a general-purpose agent rather than a task-specific retrieval tool remains open.

Target Audience

This paper is most useful for HCI and information retrieval researchers studying human-AI interaction and search behavior; designers and product managers building AI research assistants or LLM-based retrieval systems; and NLP researchers who need realistic, large-scale query data and a domain-specific intent taxonomy for evaluation. Researchers studying scientific workflows and scholarly communication will also find the behavioral findings relevant, though the authors caution that results come from one representative system and may not generalize to tools with substantially different retrieval scope, interaction modality, or optimization objectives.

Authors’ abstract

AI-powered scientific research tools are rapidly being integrated into research workflows, yet the field lacks a clear lens into how researchers use these systems in real-world settings. We present and analyze the Asta Interaction Dataset, a large-scale resource comprising over 200,000 user queries and interaction logs from two deployed tools (a literature discovery interface and a scientific question-answering interface) within an LLM-powered retrieval-augmented generation platform. Using this dataset, we characterize query patterns, engagement behaviors, and how usage evolves with experience. We find that users submit longer and more complex queries than in traditional search, and treat the system as a collaborative research partner, delegating tasks such as drafting content and identifying research gaps. Users treat generated responses as persistent artifacts, revisiting and navigating among outputs and cited evidence in non-linear ways. With experience, users issue more targeted queries and engage more deeply with supporting citations, although keyword-style queries persist even among experienced users. We release the anonymized dataset and analysis with a new query intent taxonomy to inform future designs of real-world AI research assistants and to support realistic evaluation.