CleverBirds: A Multiple-Choice Benchmark for Fine-grained Human Knowledge Tracing

AI summary

Overview

Research area: Computer vision benchmark datasets for knowledge tracing (KT) — specifically, modeling how humans acquire fine-grained visual recognition skills.

Technical level: Intermediate. The paper is readable for those familiar with machine learning benchmarks and evaluation metrics, but the knowledge tracing framing and the range of baseline models assume some background in sequential modeling.

Scope: The paper introduces CleverBirds, a large-scale dataset and benchmark built from an online bird identification quiz, and evaluates a range of computational models on the task of predicting which answer a human participant will choose.

What This Paper Is About

Existing benchmarks for knowledge tracing — the task of modeling what a learner knows and predicting how they will answer future questions — are concentrated in a small number of subjects such as mathematics, programming, and language learning, and typically contain relatively few learnable concepts. There is little data for domains where learning means acquiring visual identification skills, such as distinguishing similar species, medical conditions, or artworks. The authors build a benchmark from a real citizen-science bird identification quiz, in which participants repeatedly attempt to match bird images to species names, and use it to test whether current models can predict human responses and mistakes.

Key Contributions

1. A new large-scale visual knowledge tracing benchmark. CleverBirds contains 17,859,392 user interactions from over 40,000 unique participants, spanning 10,779 different bird species, with long-range learning patterns across an average of 400 questions per participant. The authors describe it as among the largest benchmarks of its kind, with a substantially higher number of learnable concepts.

2. A quantitative evaluation of many computational approaches. The authors evaluate transformer-based knowledge tracing models, a confusion prior classifier, a one-layer MLP, image-feature MLPs, and simple heuristics on multiple-choice response prediction, plus probabilistic models and KT baselines on binary correct/incorrect prediction.

3. An analysis of how context type affects prediction. Models are conditioned on different combinations of user context, species context, and image context (denoted U, S, and Img), showing which kinds of information help and where.

4. Release of dataset and code at a public project page, with images replaced by extracted features (DINOv2 ViT-B/14 and ImageNet-pretrained ResNet50) to respect image licenses.

Main Findings

• Feature-engineered context gives strong baselines. On the binary task of predicting whether a participant answers correctly, a random forest using both user and species context performs best overall, achieving over 80% average precision in predicting participant errors. On the multiple-choice task, a one-layer MLP with user and species context matches much larger transformer models.

• User context matters more than species context. Models receiving both user and species context (U+S) perform best overall, with user-only models close behind. Species context models still beat simple heuristics on the binary task and on the full-dataset multiple-choice task, but user-specific context appears necessary for strong predictions.

• Binary-trained classifiers beat multiple-choice-trained ones on the binary task. Multiple-choice classifiers are consistently outperformed on the binary objective, since they must allocate capacity to the structure of incorrect responses as well as to correctness. The authors suggest future models could use binary classifier outputs as auxiliary signals or gating mechanisms.

• Image features help most on incorrectly answered questions. On the full dataset, the MLP with user, species, and DINOv2 image context reaches 76%, slightly above other multiple-choice trained models. On the subset of questions participants got wrong, it reaches around 25%, while the same MLP without user and species context (MLP-Img) achieves only around 11%.

• Predicting incorrect choices is hard. Trained models reach approximately 70% accuracy on the full multiple-choice task, but all trained models achieve less than 25% accuracy on the subset of questions participants answered incorrectly. The random classifier attains the expected 20%, and the always-correct classifier scores 0% by design. The confusion prior restricted to never answer the correct species (Conf Prior Inc) scores below 9% on the full dataset (also described elsewhere as about 10%) but over 23% on the incorrect subset — nearly matching the best MLP. The authors note this marginal improvement suggests substantial room for improvement.

• Knowledge tracing baselines show little variance. The tested KT methods all perform similarly, with average precision on predicting user mistakes around 0.35 and average accuracy around 54%. The authors call this surprising given the diversity of architectures, and suggest these models focus disproportionately on the positive class, exploiting a shortcut — which they present as evidence of the benchmark's value in exposing limitations of existing approaches.

• Human learners improve over time. On average, within the first 20 times a user sees a particular species, their accuracy increases by 20%. Over half of users show measurable gains over sliding windows of 20 questions. User and species accuracy distributions are broad and centered around 60-70%. Users achieve an average accuracy of over 50% even on images rated low quality.

• The task has subtle confusions. The five most frequently confused species pairs among species with over 1,000 interactions are American Crow vs Fish Crow, Pin-tailed Snipe vs Common Snipe, Redpoll (Hoary) vs Redpoll (Common), Ross's Goose vs Snow Goose, Sharp-shinned Hawk vs Cooper's Hawk, and Short-tailed Shearwater vs Sooty Shearwater.

Methodology in Plain English

The data comes from a public bird identification quiz created by the citizen-science project eBird and first published online in March 2018, intended partly to gather image quality ratings. Players choose a location, time of year, and species prevalence, then answer sets of 20 multiple-choice questions. Each question shows a bird image with five options: four species names drawn from a sliding window over the taxonomic list centered on the true species (so distractors are taxonomically similar), plus "None of the above." After answering, the correct species is revealed and the user rates image quality on a scale of 1 to 5.

The authors use all quizzes completed from March 14th 2018 to October 8th 2024. They split by user ID into training, validation, and test sets, aiming for a 70/15/15 split, resulting in 28,100 training, 6,021 validation, and 6,023 test users, corresponding to 70.6%, 14.6%, and 14.8% of interactions respectively.

Instead of distributing the original images, they provide extracted embeddings: DINOv2 with a ViT-B/14 backbone (final-layer patch tokens average-pooled after LayerNorm, excluding special tokens) and a ResNet50 pretrained on ImageNet (global average pooling output before the classifier). ResNet50 features cover 14,753,114 distinct images and DINOv2 covers 14,747,840, the discrepancy of 5,274 arising because images get deleted by owners over time. Unavailable images are treated as zero inputs.

The prediction task: given a participant's interaction history plus the current question's image, options, and correct answer, predict the participant's response. The model is a shared parametric function with no learner-specific parameters, so learner behavior emerges only from conditioning on recent history. Models are grouped by context type: user context (interaction history plus location and time-of-year focus indicators), species context (aggregated species-level features such as average difficulty), and image context (extracted image features). These are labeled U, S, and Img, with combinations like U+S or U+S+Img.

Evaluation uses held-out user IDs. For the binary task, they report binary macro accuracy (macro averaged over correct vs incorrect) and binary AP for predicting mistakes (average precision with the incorrect class as positive). For multiple choice, they report multiple choice accuracy over labels 1 to 5 (where 5 denotes "None of the above") and multiple choice incorrect set accuracy, computed only on questions participants got wrong.

Baselines include: an all-knowing "Always Correct" heuristic; a confusion prior classifier that masks training-set confusion between the correct species and distractors and re-normalizes; a variant constrained to predict only incorrect choices; a one-layer MLP with a learned 250-dimensional species embedding and optional context, masked to the five presented options (NOTA chosen if probability mass outside the options exceeds that of every presented choice); logistic regression, XGBoost, and random forests for the binary task; an "Avg Species" heuristic; KT baselines (simpleKT, KQN, DKT, DKT+, ATKT, SAKT, AKT, DKVMN); and two language-modeling paradigms — sequence-to-sequence generation (LM-Seq2seq, fine-tuned T5-style encoder-decoder) and multiple-choice classification (LM-MCC, fine-tuned Bert-style encoder-only). All transformer models use a custom tokenizer with one token per possible species (11,142 tokens) plus special tokens, so each question occupies exactly 8 tokens. The transformer models use W = 50 for history length; other models use the complete history.

Privacy measures: all user-related identifiers, quiz IDs, question IDs, and image asset identifiers are anonymized; quiz locations are aggregated using the H3 geospatial index at resolution 3, averaging 12,393 km² per cell. Participants are registered Cornell Lab account holders who agreed to the Terms of Use, and the project was approved by the School of Informatics Ethics Committee (project number 954242).

Why This Matters

Impact on research. CleverBirds provides a benchmark with a much larger concept space than prior image-based knowledge tracing datasets. The authors contrast it with prior work containing three datasets (five butterfly species, three retina condition classes, three synthetic "greebles" classes) totaling only 6,750 interactions and less than one thousand images each, and with a gravitational wave spectrogram dataset covering 21 "glitch" types that does not explicitly target learning. CleverBirds' scale, the demonstrated improvement of its participants over time, and the difficulty of the task make it a testbed for methods that claim to model human knowledge acquisition.

Real-world applications.

• Intelligent tutoring and machine teaching systems that adapt to a learner's evolving knowledge state.
• Citizen science platforms, where participants must be trained to identify species reliably.
• Expert training in domains where identification is a core skill, such as medical image reading, art attribution, and biological taxonomy.
• Diagnostic tools that estimate which concepts a learner has not yet mastered, so instruction or feedback can be targeted.

Industry relevance. The benchmark is relevant to organizations building adaptive learning products, to platforms that depend on volunteer or crowd labelers for image data, and to research groups working on personalized modeling of human behavior. The finding that simple probabilistic models with well-chosen user and species context outperform much larger knowledge tracing architectures is directly useful to practitioners deciding how much model complexity is warranted.

Future Directions

• Better prediction of incorrect choices. All trained models score below 25% on the subset of questions participants answered incorrectly, barely above the 20% chance level, and one confusion prior heuristic restricted to predicting errors nearly matches the best neural model. Closing this gap would indicate genuine approximation of a learner's internal knowledge state.

• Longer temporal context and cross-species generalization. The authors explicitly suggest exploring longer temporal contexts and better generalization across species, since success or failure on one species could provide information for others.

• Combining binary and multiple-choice objectives. The paper proposes that multiple-choice models could benefit from binary classifier outputs as auxiliary signals or gating mechanisms, or from additional binary supervision, since binary-trained models consistently outperform multiple-choice-trained ones on the binary task.

• Addressing dataset scope and bias. The authors acknowledge the dataset is limited to bird species identification, which restricts immediate applicability to other domains, and that participants from eBird skew toward the Global North, introducing location and selection bias. Broadening coverage and testing transfer to other fine-grained domains remain open questions. The authors also caution that inaccurate models developed from such data could negatively bias future human learning.

Target Audience

Researchers in knowledge tracing and educational data mining; computer vision researchers interested in fine-grained recognition and human-in-the-loop benchmarks; machine learning practitioners building adaptive tutoring or expert-training systems; and citizen science or ecology researchers interested in how volunteers develop identification skill. Readers primarily interested in model architecture design may find it less relevant, since the paper focuses on benchmarking and context analysis rather than proposing a new architecture.

Authors’ abstract

Mastering fine-grained visual recognition, essential in many expert domains, can require that specialists undergo years of dedicated training. Modeling the progression of such expertize in humans remains challenging, and accurately inferring a human learner's knowledge state is a key step toward understanding visual learning. We introduce CleverBirds, a large-scale knowledge tracing benchmark for fine-grained bird species recognition. Collected by the citizen-science platform eBird, it offers insight into how individuals acquire expertize in complex fine-grained classification. More than 40,000 participants have engaged in the quiz, answering over 17 million multiple-choice questions spanning over 10,000 bird species, with long-range learning patterns across an average of 400 questions per participant. We release this dataset to support the development and evaluation of new methods for visual knowledge tracing. We show that tracking learners' knowledge is challenging, especially across participant subgroups and question types, with different forms of contextual information offering varying degrees of predictive benefit. CleverBirds is among the largest benchmark of its kind, offering a substantially higher number of learnable concepts. With it, we hope to enable new avenues for studying the development of visual expertize over time and across individuals.