Responsible Institutional Analytics: Interpreting Bias with AI Support

AI summary

Overview

• Research area: Human-Computer Interaction, specifically Institutional Analytics (IA) and Learning Analytics in higher education, with a focus on responsible interpretation of dashboards and generative-AI scaffolding. The paper is also categorized under I.2, L.3, and L.3.1.
• Technical level: Intermediate. The work combines a conceptual framework, a retrieval-augmented-generation (RAG) chatbot, a qualitative think-aloud study, and transition network analysis (TNA); no advanced mathematics is required to follow the argument.
• Scope: The paper proposes the FACTRIA framework of biasing factors in institutional analytics and tests an LLM-based chatbot that uses FACTRIA to prompt stakeholders to reflect on those factors while interpreting four real institutional visualizations.

What This Paper Is About

Institutional Analytics dashboards are used in higher education to support decisions about teaching quality, student experience, and institutional effectiveness, but their indicators are shaped by data limitations, analytical choices, and context that users often cannot see. Existing governance and literacy frameworks (for example DELICATE, SHEILA, and CALVI) either operate at the institutional policy level or focus on graphical pitfalls, leaving a gap in support at the exact moment a person reads a visualization. The paper's goal is to close that gap by organizing previously fragmented knowledge about biasing factors into the FACTRIA framework and using it to drive a generative-AI chatbot that asks reflective questions rather than delivering interpretations or recommendations.

Key Contributions

1. The FACTRIA framework. A structured, extensible framework that organizes potential biasing factors into four areas: the analytics pipeline, institutional context, course-level characteristics, and demographics. Each area contains named sub-factors (for example data quality, types of data analysis and visualization, academic units, educational framework, modality of the university, course level and class size, learning design, students' workload, student's performance, gender, age, and language). The authors state it is intended to synthesize fragmented knowledge, support better-grounded hypotheses, and enable scaffolding tools.
2. A FACTRIA-aware chatbot. A web-based, LLM-based assistant grounded in FACTRIA through retrieval-augmented generation, integrated into the dashboard interface and driven by a deterministic state machine that cycles through relevant FACTRIA factors and prompts users to reflect on the data rather than supplying conclusions.
3. An empirical qualitative study. A think-aloud study with 11 stakeholders (4 male, 7 female professors, academic managers, and technicians at a Spanish university) working through four authentic IA cases, including System Usability Scale (SUS) responses and coded interaction transcripts (Cohen's k = 0.76).
4. Transition network analysis of human–chatbot interaction. TNA was applied per case and across all cases to visualize how chatbot moves (for example RI.Reflection, PI.Explanation, A.Validation) linked to user interpretive moves (for example PI.SharingInsight), plus paired-samples t-tests on the number of sub-factors addressed.

Main Findings

• Participants recognized overlooked factors after chatbot prompting. The transition network showed that assistant prompts and explanatory moves structured the interaction, after which participants generated their own interpretations. The most prominent chatbot-coded nodes were RI.Reflection (46 occurrences), A.Validation (40), PI.Explanation (26), and RI.Interpretation (20); the most prominent user-coded node was PI.SharingInsight (84), including a self-loop indicating users often stayed in an interpretive mode once they began reasoning aloud.
• Chatbot use increased the number of sub-factors considered. In the pooled scenario (Case X + Case Y, n = 22), the average number of sub-factors addressed rose from 1.41 without to 3.45 with the chatbot (standard deviations 0.80 and 0.86; medians 1.50 and 4.00), with p < .001 and Cohen's d = 2.1530.
• The effect held when learning effects were controlled. Restricting to each participant's first assigned case (n = 11), the average rose from 1.73 to 3.55 (standard deviations 0.65 and 0.82; medians 2.00 and 4.00), p < .001, Cohen's d = 4.4947. For second assigned cases (n = 11), the average rose from 1.09 to 3.36 (standard deviations 0.83 and 0.92; medians 1.00 and 4.00), p < .001, Cohen's d = 1.7866.
• Per-case factor coverage differed. Participants identified 3 relevant factors for Case 1, 5 for Case 2, 5 for Case 3, and 4 for Case 4.
• Usability and perceived value were largely positive but mixed. Most participants (10 out of 11) rated the chatbot 4 or 5 out of 5 for providing meaningful support in contextualizing the data. The overall SUS score was 70.84 ± 16.39. Reported confidence was moderate and varied ("I felt very confident using the system," 3.7 ± 1.23), and participants generally did not feel they needed substantial technical support (2.27 ± 1.45 on the item about needing a technical person).
• On-demand definitions mattered for less-experienced users. P6 noted that without immediate explanations of terms such as "weighted grade" or "Shannon index," they would have "lost five minutes just figuring out what to do."
• Some criticism of the prompts. P2 described the system as challenging and rated its usefulness for responsible interpretation 2 out of 5; P8 found it useful but called the prompts "too generic" and preferred more direct guidance. Participants with prior experience using chatbots for similar tasks (e.g., P1, P3, P9) reported positive interactions.
• Interpretations shifted from descriptive to conditional. In the narratives, P7 moved from reading gender differences in satisfaction scores as a direct portrayal of bias toward caution about sample sizes and student–professor gender interactions; P10 moved from concluding that lower-performing courses were "simply weaker" toward considering aggregation, workload, and class size; P6 connected LMS interaction indicators to learning design and workload while institutional and demographic factors did not enter their reasoning; P2 moved from a mistaken satisfaction-based reading of the Shannon Diversity visualization toward data definition and representation concerns.
• Coverage was incomplete. In all scenarios, time constraints prevented participants from addressing all sub-factors related to the visualization.

Methodology in Plain English

The researchers first reviewed prior literature and had two researchers identify and group biasing factors into the four FACTRIA categories, documented in a figure and appendix. They then built a web-based chatbot that uses retrieval-augmented generation over this framework, with a deterministic state machine controlling the conversational flow so that the bot walks users through relevant factors with open-ended prompts instead of giving answers.

For the study, 11 stakeholders from a Spanish university each signed a consent form, read a summary of FACTRIA for 5 minutes, and then worked through two of four real institutional analytics visualizations, drawn from previous institutional studies the participants had not seen. For each case, they first interpreted the visualization without the chatbot and then interpreted the same visualization with the FACTRIA-aware chatbot (20 minutes), always in that order, because the aim was to assess added value rather than order effects. Sessions were recorded in a distraction-free environment, capturing screen activity and audio, and participants completed the System Usability Scale at the end.

Two researchers independently coded the transcripts using a bottom-up coding scheme, reaching Cohen's k = 0.76 and resolving disagreements through discussion; quotations were translated into English and lightly edited for hesitations without changing meaning. The team applied transition network analysis per case and across all cases to visualize transitions between coded chatbot and user actions, and ran paired-samples t-tests on the number of sub-factors addressed under three data scenarios (all cases pooled, first assigned cases only, second assigned cases only). Four narrative accounts were then presented to illustrate variation across the four cases and different participant profiles.

Why This Matters

• Impact on research: The paper reframes responsible analytics from a governance problem (consent, transparency, policy) into an interpretation problem that occurs at the moment a stakeholder reads a chart. FACTRIA offers a shared vocabulary that connects literature on data quality, learning design, institutional context, and demographics, and the use of transition network analysis demonstrates a way to study how conversational scaffolding changes reasoning rather than only measuring satisfaction. It also connects to prior AI-for-visualization work such as VizChat and multi-agent RAG explanations for learning analytics dashboards.
• Real-world applications (bullets):
  • University teaching-quality dashboards that display student satisfaction aggregated by professor gender, student gender, and faculty, where small samples and departmental composition can distort comparisons.
  • Course-level reviews comparing teaching methodologies (lectures, project-based learning, problem-based learning, flipped classroom, case studies) against performance and workload indicators.
  • Learning-management-system analyses, such as comparing high-interaction courses against an institutional average or examining the diversity of LMS task configurations via the Shannon Diversity Index.
  • Staff training and quality-assurance processes where academic managers and lecturers need prompts to question comparability, aggregation, and indicator definitions before acting.
• Industry relevance: The approach is directly relevant to edtech and learning-platform vendors building analytics dashboards, to institutional data and quality offices, and more broadly to any organization whose decision-makers consume aggregated indicators. The design lesson is that AI assistance can be positioned as a reflective scaffold that asks about assumptions and comparability rather than as an oracle that supplies conclusions, which matters given the risk of automation bias noted in prior educational AI research.

Future Directions

1. Extend and refine the framework. FACTRIA is explicitly presented as flexible and extensible; the authors anticipate that additional biasing factors will be incorporated as IA practices evolve and new evidence emerges.
2. Address the prompt-quality gap. Some participants found the prompts "too generic" and preferred more direct guidance, while one rated usefulness at 2 out of 5, suggesting the need for prompts tuned to the specific visualization and user profile.
3. Resolve the time constraint on factor coverage. Participants could not address all sub-factors within the session, so future work could examine pacing, session length, or prioritization of factors per case.
4. Broaden and re-sequence the evaluation. The study used 11 stakeholders at one Spanish university, with conditions not counterbalanced and only four authentic cases; broader participant pools, other institutions and university modalities, and order-controlled designs would test whether the observed shifts generalize. The content provided does not report a follow-up study or deployment beyond this evaluation.

Target Audience

This paper suits learning analytics and institutional analytics researchers and practitioners, human-computer interaction researchers studying AI-assisted sensemaking and reflective prompting, university academic managers, quality-assurance and teaching-innovation staff, and designers of dashboard or chatbot products in education technology. It is also relevant to methodologists interested in transition network analysis as a way to study conversational interaction.

Note: The provided text is truncated mid-sentence in the Case 4 narrative, and the Discussion, Limitations, and Conclusions sections referenced in the paper's outline are not included in the supplied content, so any details from those sections are not reported here.

Authors’ abstract

Institutional Analytics (IA) dashboards inform decision-making in higher education, yet data limitations, constraints in analytical techniques, and missing contextual information often affect their interpretation. To support more responsible interpretation of IA, we introduce FACTRIA, a framework that organizes potential biasing factors across four areas: the analytics pipeline, institutional context, course-level characteristics, and demographics. We used the FACTRIA framework as input to a generative-AI chatbot designed to prompt users to reflect on these factors while analyzing IA. A qualitative study with stakeholders, drawing on four authentic IA cases, and a transition network analysis showed that the chatbot prompted participants to recognize how overlooked factors influenced their initial interpretation. Findings indicated that combining a structured framework with AI-based guidance can enhance context-aware, responsible interpretation of institutional data.