Gricea: An Open Science Platform for Conversational AI Research

AI summary

Overview

Research area: Human-Computer Interaction (HCI), specifically conversational AI (CAI) user studies, controlled human-subject experiments, and open-science research infrastructure. The paper's CCS categories include HCI design and evaluation methods, user studies, empirical studies in HCI, collaborative and social computing, field studies, usability testing, user models, natural language interfaces, web-based interaction, and interaction design process and methods.

Technical level: Intermediate. The paper is a systems-and-evaluation HCI contribution rather than a modeling paper; it assumes familiarity with controlled experiment design (between- and within-subject structures, condition assignment, counterbalancing) but explains the platform's architecture in accessible terms.

Scope: The paper presents Gricea, an open-science platform that represents conversational AI studies as configurable, deployable, inspectable, and reusable research artifacts, and evaluates it through a formative analysis of 57 papers, a replication of CUI 2026 study configurations, and an authoring study with N = 10 researchers and practitioners.

What This Paper Is About

Controlled studies of conversational AI are needed at scale, but the systems, interfaces, prompts, contexts, and procedures behind those studies are reported in fragmented ways, so later researchers must reconstruct an interaction before they can reproduce or extend it. Gricea addresses this by representing a study as a shared, executable artifact: the study procedure (Study Flow) and the behavior inside each conversational task (Task Flow) are authored visually, executed by the same runtime, versioned, and shared. The goal is to lower the engineering cost of building these studies while making each study a reusable resource that supports cumulative knowledge building.

Key Contributions

1. A shared, executable representation of conversational AI studies that connects study procedure, task behavior, participant-facing conditions, and instrumentation, so researchers can inspect, reproduce, and extend experimental designs.
2. Gricea, a configurable no-code research platform that operationalizes this representation through visual authoring, participant-facing execution, immutable study versions, shareable configurations, and reusable community templates.
3. Empirical findings from an authoring study with researchers and practitioners from diverse disciplinary backgrounds, who completed an assisted authoring walkthrough and then independently designed studies around their own research questions, producing valid, runnable studies that varied across procedures, interfaces, models, surveys, and outcomes.
4. Reproductions of CUI 2026 study configurations (none of which informed Gricea's design), showing that published designs can be reconstructed as executable artifacts for inspection and reuse, while also identifying remaining needs for guidance, validation, and workflow support.

Main Findings

• Formative analysis of 57 papers. From 100 candidate papers collected via keyword combinations around conversational AI, agent, or LLM together with users, humans, and studies (across venues and repositories such as CHI, UIST, and CUI), the team filtered to 57 papers that centered participant-facing conversational or agentic AI systems with sufficient detail about the study design, system configuration, or evaluated interaction condition. Coding covered study type, focus area, participant count, independent and dependent variables, between- and within-subject structure, procedural stages, system or pipeline components, analysis methods, and overall procedure structure.

• The experimental object is a "configured interaction condition," not just a prompt or model. Studies vary far more than prompts or underlying models; the condition is the combination of agent behavior, interface, context, and procedure. The analysis synthesizes recurring configurations into six interacting dimensions: Interface Condition, Agent Condition, Context & Grounding, Task & Modality, Domain & Audience, and Study Procedure.

• Two coupled levels are required. A study couples a study procedure (instructions, condition assignment, branching, measurement timing, cross-session continuity) with an interactive runtime (what participants see and do, what the system receives, how it responds and adapts, what is logged). This coupling motivates Gricea's connected Study Flow and Task Flow.

• Five design desiderata. The paper derives D1: explicit representation of study conditions; D2: coupled support for Study Flow and Task Flow; D3: lower technical barriers to controlled study authoring; D4: reproducibility through inspectable and reusable artifacts; D5: extensibility across tasks, modalities, and study settings.

• Replication of published designs. Of 29 eligible CUI 2026 full papers, Gricea replicated designs from 27 as executable artifacts: 10 completely and 17 partially. The abstract also reports that Gricea replicated configurations of 93% of eligible CUI 2026 papers, and that it flagged missing information in 96% of papers that hinders faithful replication, which the authors say further motivates Gricea's need.

• Authoring study with N = 10. Researchers and practitioners from diverse disciplinary backgrounds completed an assisted authoring walkthrough before independently designing studies around their own research questions. Participants implemented valid, runnable studies that varied across procedures, interfaces, models, surveys, and outcomes, and the authors report that the no-code interface reduced barriers for researchers without systems backgrounds.

• Supported tasks and instrumented signals. The runtime supports chat, web-based search tasks, voice interaction, human–AI coding, annotation workflows, image generation, and split-view configurations. Instrumentation records timestamps, clicks, focus events, scroll depth, input timing, text-edit counts, and task-specific interaction traces according to the study's collection settings; where enabled, additional capture includes voice recordings, masked browser-session replay, screen recordings, file-upload traces, and eye-tracking data.

• Remaining gaps. The authors report that their evaluations identify remaining needs for guidance, validation, and workflow support alongside demonstrated support for diverse research questions and published study designs.

Methodology in Plain English

The work proceeded in four stages.

1. Formative design-space analysis. The team gathered 100 candidate papers using keyword combinations around conversational AI, agent, or LLM together with terms related to users, humans, and studies across venues and repositories such as CHI, UIST, and CUI, then kept 57 that centered participant-facing conversational or agentic AI systems and described the study design, system configuration, or evaluated condition in enough detail. They coded study type, focus, participant count, independent and dependent variables, between/within-subject structure, procedural stages, system components, and analysis methods, then clustered the codes into recurring outcome areas, manipulation dimensions, procedural structures, and infrastructural demands. They also examined how papers visually represented their designs, finding staged diagrams, branching structures, and flowcharts—evidence that researchers already think in graph-like representations, which Gricea makes executable.

2. Deriving design desiderata. The recurring patterns were turned into five requirements (D1–D5) for what a study representation must make explicit and what infrastructure must reduce.

3. Building the platform. Gricea represents study designs as directed graphs: a Study Flow graph governs participant progression, and nested Task Flow graphs specify behavior within each interactive condition. Three decoupled layers operate on this representation—a researcher-facing visual authoring environment, a participant-facing execution runtime, and a publication layer for versioning, reuse, and community sharing. Publication validates the authored definition and serializes graphs, configurations, and assets into an immutable Study Version that the runtime executes directly, so the representation researchers inspect is the definition the runtime runs. Researchers can preview a draft, then publish, with further development continuing in new drafts.

4. Evaluation. Two evaluations: a replication exercise coding 29 eligible CUI 2026 full papers (none of which informed Gricea's design) and attempting to rebuild their configurations as executable artifacts, and an authoring study with N = 10 researchers and practitioners using an assisted walkthrough followed by independent study design.

Why This Matters

Impact on research. When prompts, task materials, interface behavior, and procedural logic live inside one-off implementations, publishing findings does not preserve the condition needed to reproduce or extend a study. Gricea keeps the configured study as an inspectable, executable artifact, giving the community a shared frame of reference for comparing implementations, adapting procedures, and accumulating findings. Because the same definition both configures and executes the study, the artifact is a by-product of running the study rather than extra documentation effort. This also matters for experimental control: documented changes in model behavior show why the same prompt and model name do not establish an equivalent condition, so preserving model and context settings alongside the record of what participants actually encountered is essential.

Real-world applications (as studied or supported in the paper):

• Co-writing and writing assistants where participants compose text with generated suggestions and the interface exposes provenance, highlighting, or editability of shared artifacts.
• Conversational search and recommendation where retrieval policy, citation constraints, grounded evidence, and persistent memory are the manipulations of interest, alongside dietary recommendation or other personalized-advice settings.
• Learning and tutoring interactions, including voice and multimodal setups such as a real-time voice conversation alongside a driving task.
• Human–AI collaboration at work, including conversational coding with an editable code workspace, annotation workflows, image generation, and studies with two conversational agents shown in separate panels.
• Longitudinal studies such as daily planning and reflection that link repeated conversations to daily surveys and an exit interview, using scheduled check-ins and cross-session continuity.

Industry relevance. Commercial conversational systems offer limited experimental control, and changes to their interfaces, retrieval policies, or models can alter the conditions under investigation. Teams that need to evaluate assistant behavior, uncertainty communication, initiative, or verification support under controlled conditions can use a platform like Gricea to keep conditions fixed and inspectable across product iterations rather than rebuilding bespoke interfaces and model pipelines for each study. Separating experimental assignment, participant interaction, and measurement also gives product and policy researchers a way to document what a deployed configuration actually was.

Future Directions

• Guidance and validation within authoring. The authors explicitly identify remaining needs for guidance, validation, and workflow support. Open questions include how much structural or semantic checking the platform should perform before a study is published, and how errors in experimental logic (as distinct from graph validity) should be surfaced.

• Extending to a broader design space. D5 calls for extensibility across tasks, modalities, and study settings—text chat, coding, multimodal interaction, voice, longer-running procedures, and community-facing workflows—through extensions that reuse the platform's execution and data-collection mechanisms rather than reimplementing the surrounding system.

• Supporting AI agents as consumers of research artifacts. The paper frames the community and AI agents as able to build on shared executable artifacts across successive studies; how agents would inspect, extend, or generate these study definitions is left as an open direction rather than a demonstrated result.

• Cumulative knowledge across versions. Immutable study versions and explicit change tracking are the mechanism proposed for reuse and extension. Open questions remain about how successive versions should be compared, how findings should be attributed across versions, and how community templates and document collections accumulate over time; the paper's description of publication, reuse, and community workflow mechanisms is not fully available in the provided content.

Target Audience

• HCI and conversational AI researchers who design and run controlled human-subject studies and need to reproduce, extend, or compare against prior work.
• Researchers without systems or engineering backgrounds who want to build participant-facing conversational interfaces and complex task conditions without bespoke code, and for whom the no-code interface and reusable components lower the barrier.
• Research infrastructure and open-science developers interested in executable representations of experiments, study versioning, and artifact preservation.
• Evaluation and trust-and-safety teams studying human–AI interaction in industry settings, where controlled manipulations of agent behavior, interface support, and contextual grounding are needed for internal or published evaluations.
• Methods-focused readers who want a worked example of deriving platform design requirements from a formative analysis of prior papers and validating them through both reconstruction of published designs and an authoring study.

Authors’ abstract

We need studies on conversational AI (CAI) at scale to understand human behavior and shape CAI design. However, fragmented reporting of systems and study configurations hinders replication, extension, and knowledge accumulation. We present Gricea, an open-science platform representing studies as configurable, deployable research artifacts that researchers can run, inspect, share, and reuse. Informed by a formative analysis of prior CAI research, Gricea couples study procedures, participant-facing systems, and conversational task behavior in. In a replication study using Gricea, we replicated configurations 93% of eligible CUI 2026 papers; while also flagging missing information in 96% of papers that hinder faithful replication --- further motivating Gricea's need. In a user study, researchers and practitioners from diverse backgrounds successfully constructed runnable studies addressing various open-ended research questions. Together, these findings demonstrate Gricea's support for constructing, reproducing, and extending CAI studies through shared research artifacts, enabling cumulative knowledge building through open science.