Adobe Summit Concierge Evaluation with Human in the Loop

AI summary

Overview

Research area: Applied generative AI / enterprise conversational assistants, with a focus on human-in-the-loop evaluation and development (VLDB 2025 Workshop on Data Science with Human in the Loop).

Technical level: Intermediate. The paper assumes familiarity with retrieval-augmented generation, natural language-to-SQL, LLM-as-judge evaluation, and intent routing, but presents them at a system-design and process level rather than a mathematical one.

Scope: A deployment case study describing the architecture, human-in-the-loop development workflow, evaluation methodology, and production outcomes of Summit Concierge, a domain-specific AI assistant built for the Adobe Summit event, authored by researchers at Adobe Inc.

What This Paper Is About

Enterprises want generative AI assistants that answer domain-specific questions reliably, but building one for a specific event is hard because there is little or no historical query data to train or evaluate against, LLMs hallucinate on time-sensitive details, and the event's fixed date leaves no time for lengthy training pipelines. The paper's goal is to show how a team bootstrapped, evaluated, and deployed Summit Concierge — an assistant handling Adobe Summit session, speaker, sponsor, and logistics queries — by keeping human reviewers inside the development and evaluation loop rather than relying on large-scale supervised training.

Key Contributions

1. A human-in-the-loop workflow for quality assurance. A lightweight feedback loop in which human reviewers continuously validate and refine responses to ensure factual accuracy and contextual appropriateness in user interactions.

2. Techniques to overcome data sparsity in cold-start scenarios. Synthetic query generation, documentation-grounded retrieval, and prompt engineering used to bootstrap an assistant without historical usage data.

3. A multi-faceted evaluation framework. Three complementary strategies for unstructured-data responses (correctness-based scoring, side-by-side comparison, brand compliance verification), plus templated structured-data query evaluation, autocomplete pool evaluation, and multi-turn evaluation — all combining LLM judges with human review.

4. Lessons learned from real-world deployment at scale. Empirical insights and design decisions from the production deployment during Adobe Summit, including operational challenges, user feedback, and opportunities for improvement.

Main Findings

• Synthetic questions can bootstrap evaluation in a cold-start setting. Starting from 10 high-value seed questions curated with product managers and the Adobe Summit marketing team, LLM-driven paraphrase expansion produced more than 17K queries in a concise "smartphone-typing" style.
• SQLSynth guarantees in-scope test queries. Using 269 auto-generated queries produced in less than 30 minutes, the intent router achieved 100% accuracy while NL2SQL missed 2.6% of cases; these results drove prompt instruction improvements.
• LLM-as-judge sharply reduces annotation load for structured templated queries. Out of 3,000 templated queries, only 220 required human review; the rest were automatically verified as containing the necessary key facts.
• Autocomplete quality improved consistently across three chronological question pools. The ratio of relevant completions rose from 27.00% (Pool_1) to 48.85% (Pool_2) to 58.02% (Pool_3), and average keystroke savings rose from 6.09 to 8.85 to 11.45.
• Prompt improvements for multi-turn rewriting produced measurable gains. Rewrite error rate dropped from 4.35% to 1.45% and routing accuracy rose from 89.1% to 96.1%. LLM-as-judge auto-selection of uncertain samples reduced manual annotation needs from 1500 queries to 276.
• Daily triage surfaced clear error taxonomies. Across a four-day internal evaluation period, 624 interactions were annotated (174 pre-event/Day 1 retro, 245 Day 2, 123 Day 3, 82 Day 4). 436 (69.9%) required no further action; the remaining 188 were actionable errors.
• Actionable error breakdown (N = 624). Product Knowledge Error 79 (12.7%); Canned Response needing improvement 31 (5.0%); Rewrite Error 21 (3.4%); Intent Detection Error 20 (3.2%); Operational Insights Error 30 (4.8%); Tone of Response 5 (0.8%); Hyperlinking Issue 2 (0.3%).
• Out-of-scope misrouting was reduced. The rate of queries incorrectly routed as out-of-scope (OOS) fell from 4% to 3%.
• Not reported. The paper does not report end-to-end answer accuracy for free-form unstructured queries, latency, or total user traffic volumes.

Methodology in Plain English

The team built a standard conversational QA pipeline: a user query plus attendee-specific context passes through autocomplete and a query-rewriting component that uses chat history; an intent detection module then routes the rewritten query to either a retrieval-augmented generation branch (for guidebook-style content) or a natural-language-to-SQL branch (for structured data), and a unified answer generation module produces the final response.

Structured data — sessions, speakers, sponsors — comes from the event organizer via RainFocus and is ingested into relational tables by a periodically running transformation pipeline. Attendee-specific schedules are deliberately not ingested into the Snowflake database; instead the system queries the RainFocus API with the attendee ID and authentication token at query time, to protect privacy and keep schedule data fresh. SQL results and schedule data are rendered as Markdown tables and passed to the LLM.

Unstructured content centers on the ABC Guide, an internal event document covering logistics, navigation, onsite services, attendee support and safety, plus Adobe product summaries and live-authored content drafted during the event based on chat logs and search traffic.

For evaluation, the team generated questions two ways. Structured questions came from SME seed templates expanded by an LLM plus SQLSynth, a human-in-the-loop tool that reverse-engineers executable SQL into natural language questions, offering scope assurance and diversity. Unstructured questions were LLM-generated from extracted text passages and vetted by humans for fluency, naturalness, and answerability, with each question linked to a source passage as a gold answer. Follow-up questions were generated from an existing question, its answer, and retrieved documents, then reviewed. Autocomplete questions were assembled from curated expert/LLM questions, event-management-team questions based on historical data, and queries logged during the live event, with start and expiration timestamps assigned for time-aware re-ranking. Seed questions were validated during internal bug bashes, and a human-in-the-loop clustering and popularity analysis using semantic embeddings plus HDBSCAN grouped similar queries for manual review.

Evaluation mixed LLM judges with chain-of-thought reasoning and human reviewers, who handled only uncertain or ambiguous cases. Multi-turn evaluation used a reasoning-oriented LLM for prompt rewriting with chain-of-thought prompting and few-shot step-by-step examples, and human annotators reviewed sampled out-of-scope questions and cases where rewrites might alter user intent.

Why This Matters

Impact on research. The paper argues that public benchmarks (it cites Chang et al. 2024 as listing 46 public benchmark datasets) are often not applicable to domain-specific assistants, and that there is no single static benchmark that suits all needs. It contributes a concrete, reproducible-style account of how synthetic question generation, in-scope text-to-SQL question synthesis, and LLM-as-judge protocols can substitute for labeled data in a cold-start deployment, and it shows how evaluation benchmarks must evolve alongside product features and user needs.

Real-world applications:

• Event assistants for conferences, trade shows, and multi-day festivals, where content changes daily and attendees need session, speaker, and logistics answers on mobile.
• Internal knowledge access, turning support-staff documentation such as the ABC Guide into a conversational interface for employees.
• Customer service triage, using intent routing plus retrieval grounding to reduce support-staff load while keeping human reviewers on high-risk responses.
• Privacy-sensitive personalization, demonstrated by fetching attendee schedules via API per user rather than ingesting private data into the analytics warehouse.

Industry relevance. The paper's central operational claim is that agile, feedback-driven development with lightweight human oversight can yield scalable and reliable assistants without extensive pre-collected data or long training cycles — a practical template for teams facing fixed launch deadlines and no usage logs.

Future Directions

• Generalizing the methodology to other enterprise domains, particularly event support, internal knowledge access, and customer service.
• Continuing to refine the benchmark itself, since domain assistant evaluation sets must evolve with product features, user needs, and task definitions.
• Strengthening handling of ambiguous or underspecified multi-turn queries, building on the rewrite improvements and the ongoing monitoring of incorrect rewrites flagged by users or system performance monitoring.
• Addressing the recurring error categories surfaced in annotation — product knowledge errors, canned responses needing improvement, intent detection errors, and operational insights errors — as targets for further improvement.

Target Audience

Applied AI and ML engineers building enterprise conversational assistants; evaluation and data-quality practitioners interested in LLM-as-judge and human-in-the-loop protocols; product and program managers responsible for domain-specific assistants with fixed deadlines; and researchers studying cold-start evaluation, text-to-SQL question synthesis, and RAG grounding in production systems.

Authors’ abstract

Generative AI assistants offer significant potential to enhance productivity, streamline information access, and improve user experience in enterprise contexts. In this work, we present Summit Concierge, a domain-specific AI assistant developed for Adobe Summit. The assistant handles a wide range of event-related queries and operates under real-world constraints such as data sparsity, quality assurance, and rapid deployment. To address these challenges, we adopt a human-in-the-loop development workflow that combines prompt engineering, retrieval grounding, and lightweight human validation. We describe the system architecture, development process, and real-world deployment outcomes. Our experience shows that agile, feedback-driven development enables scalable and reliable AI assistants, even in cold-start scenarios.