Modular AI-Powered Interviewer with Dynamic Question Generation and Expertise Profiling

AI summary

Overview

• Research area: Human-Computer Interaction — specifically AI-mediated qualitative interviewing, adaptive dialogue systems, prompt engineering, and locally hosted large language models (LLMs).
• Technical level: Intermediate. The paper is readable without deep ML expertise, but it assumes familiarity with LLMs, system/user prompts, and basic descriptive and regression statistics (R², ANOVA, standardized coefficients).
• Scope: The paper designs, implements, and pilot-tests a five-module, prompt-driven AI interviewer built on a locally hosted LLM that dynamically generates interview questions and profiles participant expertise in real time, then evaluates it with 41 testers.
• Publication details: Listed as arXiv:2601.11534v1 [cs.HC], dated 21 Nov 2025, licensed CC BY 4.0. Authors are affiliated with the Department of Computing, University of Turku, Turku, Finland.

What This Paper Is About

Chatbots and automated interviewers typically rely on fixed question lists, strict rules, and limited personalization, producing repetitive conversations that are poorly suited to complex qualitative research that needs flexibility, context awareness, and ethical sensitivity. The authors build an interviewer that instead generates questions dynamically, adapts their complexity to the participant's demonstrated expertise, and keeps the whole conversation on a locally hosted LLM so sensitive data never leaves the institution's infrastructure. The goal is to test whether prompt-driven modular logic can produce coherent, non-repetitive, expertise-aligned interviews that users actually find engaging and satisfying.

Key Contributions

1. A modular, prompt-driven AI interviewer that dynamically adjusts question complexity in real time based on the user's expertise level, organized into five independent, inspectable modules (system prompt generation, initial question, expertise profiling, iterative question generation, and question uniqueness validation).
2. A structured framework for generating contextually relevant, non-redundant, and semantically coherent interview dialogues using locally hosted LLMs, including explicit prompt outlines for each module.
3. A case-study instantiation of the framework on the topic "how employees interact with LLMs in the workplace context," with research areas assigned High, Medium, or Low priority and fixed question counts (4, 3, 3, 4, and 2 questions respectively; 16 total).
4. A pilot evaluation with 41 testers reporting three performance indicators (question relevance and coherence, cognitive and emotional engagement, and overall user satisfaction and comparative experience), along with a regression model linking engagement and relevance to satisfaction.

Main Findings

• Expertise profiling worked in real time: The system assigned one of four levels — Novice, Basic Knowledge, Advanced Knowledge, or Expert — after each answer, based on technical terminology, insight depth, and academic relevance. Some participants began as Novices and moved upward mid-interview, for example a participant who started by describing simple uses such as summarizing reports and later discussed differences between local and commercial LLMs and the need for organizational training.
• Question difficulty tracked expertise level: Every interview opened with a low-complexity, novice-friendly question ("Can you describe a situation where you've heard of or used a Large Language Model (LLM) before?"). Novice-level questions were simple ("What do you think are some benefits or drawbacks of using AI tools in your daily work?"), Advanced Knowledge questions were more technical ("How do you think your organization should train employees to adapt to the presence of LLMs in daily operations?"), and Expert questions were scenario-driven, ethics- or governance-oriented ("Can you describe a situation where a public LLM could pose a data security risk within your organization, and how you would mitigate it?").
• No semantic duplication or tone violations detected: The uniqueness module checked each new question against prior ones for conceptual overlap, and the paper reports no semantic mismatches or tone violations across all 41 interview transcripts. Reviewers cited response messages such as "That's a valid concern about job displacement." and "That's a great use-case for efficiency." as evidence of accurate, tone-appropriate acknowledgements.
• High descriptive ratings across the three indicators (Table 2, N = 41):
  • Question Relevance and Coherence: mean 4.61, SD 0.4825, variance 0.23, range 3.67–5.00, skewness −0.519, kurtosis −1.615.
  • Cognitive and Emotional Engagement: mean 4.33, SD 0.3496, variance 0.12, range 3.33–5.00, skewness −0.410, kurtosis 0.986.
  • Overall User Satisfaction and Comparative Experience: mean 4.45, SD 0.2917, variance 0.09, range 4.00–5.00, skewness 0.003, kurtosis 0.080.
  • Skewness SE was 0.369 and kurtosis SE 0.724 for all three variables; the authors state these values fall within acceptable ranges for parametric analysis.
• Engagement, not question relevance, predicted satisfaction: The regression model yielded R = 0.499, R² = 0.249, adjusted R² = 0.209, and standard error of the estimate 0.2594, meaning roughly 24.9% of variation in satisfaction is explained by the two predictors. ANOVA confirmed overall significance (F = 6.289, p = 0.004; regression sum of squares 0.846, df 2, mean square 0.423; residual 2.556, df 38, mean square 0.067; total 3.402, df 40). Cognitive and Emotional Engagement was a significant positive predictor (B = 0.402, SE = 0.118, Beta = 0.481, t = 3.407, p = 0.002), while Question Relevance and Coherence was not (B = −0.055, SE = 0.085, Beta = −0.091, t = −0.645, p = 0.523). The constant was B = 2.965, SE = 0.676, t = 4.384, p < .001.
• Participant preferences differed by profile: Older participants preferred traditional interviews for personal connection and naturalness; younger participants were more open to AI-powered interviews; introverted participants preferred the AI interviewer because they felt more comfortable and less pressured. Several participants considered the AI format well suited to sensitive topics such as racism and discrimination because of its neutral, non-judgmental tone.
• Inconsistent averages are reported in the paper: The abstract states satisfaction of 4.45 and engagement of 4.33 (matching Table 2), whereas the Conclusion reports satisfaction 4.52, engagement 4.39, and question relevance 4.41 — the latter two figures differ from Table 2's 4.33 and 4.61. The paper does not reconcile this discrepancy, so both sets of numbers should be treated with care.
• Acknowledged limitations: The interviewer depends heavily on well-crafted prompts and could fail if prompts are not re-tuned when the underlying LLM is swapped; the smaller LLaMA model has context memory limitations, so an interview lasting several hours may not be fully remembered.
• Not reported: The paper does not report demographic breakdowns numerically (they appear only in a figure), baseline comparisons against other interview systems or human interviewers with matched measures, inter-rater reliability for the transcript evaluation, or statistical power analysis.

Methodology in Plain English

The authors built the interviewer as five cooperating modules, each performing one job and passing its output to the next:

1. Generate System Prompt: Creates one reusable JSON system prompt that fixes the model's behavior, ethics, and context for the entire session.
2. Generate Initial Question: Picks a high-priority research area and asks a simple, accessible opening question to encourage early engagement.
3. Expertise Profiling: Reads the participant's previous answer and classifies them as Novice, Basic Knowledge, Advanced Knowledge, or Expert using a four-level rubric.
4. Generate Iterative Questions: Produces a short acknowledgement (under 10 words), a smooth transition, and a context-aware open-ended follow-up aligned to the participant's expertise, plus a justification of up to 25 words.
5. Validate Question Uniqueness: Compares each new question against all previous ones to catch conceptual duplication even when wording differs — for example, treating "What are LLMs?" and "What do you understand by Large Language Models?" as the same question.

The system was implemented with LLaMA v3.2 (3B parameters) running locally, and prompts for all modules are linked to a public GitHub repository. The case study topic was "how employees interact with LLMs in the workplace context," covering awareness, skills, usage, and privacy concerns. A pilot study recruited 41 testers from diverse demographic, educational, and technical backgrounds; the authors examined interview transcripts for expertise adaptation, question complexity, response message accuracy, and redundancy, and collected ratings on three performance indicators that were then analyzed with descriptive statistics and a regression model.

Why This Matters

Impact on research: The work shows that a modest, locally hosted 3B-parameter model combined with disciplined prompt engineering can support semi-structured qualitative interviewing without sending sensitive participant data to commercial cloud services. It also proposes a way to standardize and audit adaptive question generation, which matters for reproducibility in qualitative data collection.

Real-world applications:

• Qualitative research and academic studies that need adaptive, privacy-preserving interview instruments across large participant pools.
• Recruitment and candidate screening, where structured interviewing currently relies on rigid scripts or commercial video-interview platforms.
• Workplace and organizational assessment — the paper's own case study covers employee awareness, skills, usage, and privacy concerns around LLMs.
• Sensitive-topic research, such as studies of discrimination or racism, where participants reported the neutral, non-judgmental AI tone made them more comfortable.

Industry relevance: Organizations with data-privacy or regulatory constraints (healthcare, public sector, regulated industries) can deploy this design on local infrastructure rather than depending on cloud LLMs. Vendors of automated interviewing and survey tools get a concrete modular blueprint for expertise-aware questioning, and the finding that engagement predicts satisfaction — while question relevance alone does not — suggests where product teams should invest.

Future Directions

• Real-time response analysis and summarization: Integrate an automated tool that analyzes and summarizes participant responses as the interview proceeds, delivering immediate insight while keeping data private.
• Expanded context memory: Extend the model's memory so long interviews (the paper mentions sessions lasting several hours) do not lose earlier conversational content.
• Multilingual and cross-cultural capability: The modular design is presented as easy to adapt to different languages and cultures, but this remains to be implemented and evaluated.
• Robustness across model swaps: Since prompt quality is currently a single point of failure, future work could test whether the prompt pipeline transfers cleanly to larger or different local LLMs, and address the inconsistency between the reported engagement and relevance averages in the abstract, tables, and conclusion.

Target Audience

This paper is most useful to HCI and qualitative-research methodologists exploring AI-assisted data collection, applied NLP and prompt-engineering practitioners building dialogue systems on local LLMs, privacy and compliance officers in institutions that cannot use cloud models for sensitive data, and product teams in recruitment, education, or customer research evaluating automated interviewing tools. Readers looking for a statistical benchmark study will find only a single 41-participant pilot with no baseline comparison, while readers looking for a practical, modular architecture and prompt-design blueprint will find concrete, reusable detail.

Authors’ abstract

Automated interviewers and chatbots are common in research, recruitment, customer service, and education. Many existing systems use fixed question lists, strict rules, and limited personalization, leading to repeated conversations that cause low engagement. Therefore, these tools are not effective for complex qualitative research, which requires flexibility, context awareness, and ethical sensitivity. Consequently, there is a need for a more adaptive and context-aware interviewing system. To address this, an AI-powered interviewer that dynamically generates questions that are contextually appropriate and expertise aligned is presented in this study. The interviewer is built on a locally hosted large language model (LLM) that generates coherent dialogue while preserving data privacy. The interviewer profiles the participants' expertise in real time to generate knowledge-appropriate questions, well-articulated responses, and smooth transition messages similar to human-like interviews. To implement these functionalities, a modular prompt engineering pipeline was designed to ensure that the interview conversation remains scalable, adaptive, and semantically rich. To evaluate the AI-powered interviewer, it was tested with various participants, and it achieved high satisfaction (mean 4.45) and engagement (mean 4.33). The proposed interviewer is a scalable, privacy-conscious solution that advances AI-assisted qualitative data collection.