Investigating the Presence and Development of Student Instructor Preferences in a Large-Scale CS1 Course

AI summary

Overview

Research area: Computer science education (CS1 / introductory programming), student-instructor relationships, and learning analytics using clickstream log data from an asynchronous online learning platform. Listed on arXiv under cs.CY and tagged in the AI Safety & Ethics category by the source listing.

Technical level: Intermediate. The conceptual framing is accessible, but the results rely on non-parametric statistics (Kruskal-Wallis tests, chi-square tests, Wilcoxon pairwise comparisons, Bonferroni correction, eta-squared and Cramer's V effect sizes) and on a custom preference-score metric.

Scope: A two-semester, log-based study of 662 students (Fall 2021) and 559 students (Fall 2022) in a large multi-instructor online CS1 course, examining whether students arrive with instructor preferences, how those preferences change across a semester, and how they relate to gender and self-reported prior programming experience.

What This Paper Is About

Most research on whether students prefer one instructor over another relies on one-time surveys or end-of-course evaluations, which capture a single snapshot and cannot show how a preference forms. This is partly a structural problem: in typical courses students are assigned to one instructor and never get to compare alternatives. The authors study a CS1 course built on a platform where every lesson offers short interactive walkthroughs and video explanations recorded by multiple instructors, so students repeatedly choose whose explanation to watch. That repeated choice generates fine-grained temporal data, letting the researchers track preference from the beginning of the semester to the end and test which student characteristics are associated with it.

Key Contributions

1. A method for quantifying instructor preference from behavioral logs. The authors define a per-student preference score computed over weekly lessons as the difference between the proportion of times a student selected Instructor A when the default was Instructor B, and the proportion of times they selected Instructor B when the default was A. The score ranges from 1 (exclusive preference for Instructor A) through 0 (no preference) to -1 (exclusive preference for Instructor B), and can be recalculated weekly to trace change over time.

2. A distinction between initial and semester-long preference. Using the first 20% of each student's video-watching records as "initial preference" and all records across the semester as "semester preference," the study separates pre-existing inclination from preference shaped by experience.

3. Evidence that instructor preference is dynamic rather than fixed. The paper reports that preferences shift over the semester, with different rates of stability depending on which instructor a student initially favored, and that a majority of initially uncommitted students eventually settle on a preference.

4. An analysis linking preference to student characteristics. The study tests whether gender and self-reported prior programming experience correlate with initial versus semester preference, finding no significant associations with initial preference but significant — though small-effect — associations with semester preference.

Main Findings

• A majority of students showed an initial preference. More than half of students (56.5%) had an absolute preference score at or above the threshold of 0.2 based on the first 20% of their video-watching records. The initial distribution was 235 students (43.5%) with no preference, 211 (39.1%) preferring Instructor A, and 94 (17.4%) preferring Instructor B.

• Semester-long patterns differed from initial patterns. Across the whole semester, the distribution shifted to 351 students (53.0%) with no preference, 139 (21.0%) preferring Instructor A, and 172 (26.0%) preferring Instructor B.

• Initial preferences were not stable. Only 73 of 211 (35%) students with an early preference for Instructor A still held that preference at semester's end, compared with 64 of 94 (68%) of those with an early preference for Instructor B. Among students with no initial preference, 148 of 235 (63%) eventually developed a preference for one of the two instructors. The trajectory plot showed a decreasing trend for the early PrefA group and a flatter, more stable trajectory for the early PrefB group.

• Gender was unrelated to initial preference but related to semester preference. A Kruskal-Wallis test on initial preference scores across gender groups found no significant difference (H(2) = 0.85, p = .655, eta-squared = -.002). For semester preference scores the difference was significant (H(2) = 11.48, p = .003, eta-squared = .015), with a small effect size based on the conventions the authors cite (0.01 small, 0.06 medium, 0.14 large). Wilcoxon pairwise tests indicated female learners on average showed a stronger preference for Instructor A, who self-identified as female; this remained significant after Bonferroni correction (p = .003).

• Categorical preference patterns echoed the gender result. A chi-square test found a significant association between self-reported gender and semester preference patterns (chi-square(4) = 17.31, p = .002) with a small effect size (.06 by Cramer's V). Female students showed a higher proportion preferring the female instructor (25.1%) than the male instructor (19.0%), while male students showed the opposite trend, with 29.4% preferring the male Instructor B versus 18.9% preferring Instructor A. The corresponding initial-preference chi-square test was not significant (chi-square(4) = 2.94, p = .568).

• Prior programming experience was unrelated to initial preference but related to semester preference. The Kruskal-Wallis test on initial preference across experience levels was not significant (H(2) = 2.38, p = .667, eta-squared = -.003). Semester preference scores differed significantly (H(2) = 9.53, p = .049, eta-squared = .009), a small effect. Wilcoxon pairwise tests found significant differences between learners rating their prior experience 4 (p = .03) or 5 (p = .03) and those rating it 1, surviving Bonferroni correction. Learners who perceived themselves as having higher prior experience tended to prefer Instructor A.

• Categorical experience patterns differed only for semester preference. Chi-square tests were not significant for initial preference patterns across experience levels (chi-square(8) = 10.25, p = .248) but were significant for semester preference patterns (chi-square(8) = 18.89, p = .015), with lower- and higher-experience learners showing opposite trends.

• The preference threshold was empirically chosen and tested for robustness. The 0.2 cutoff came from inspecting the preference score distribution and selecting the point with the largest gap; thresholds of 0.15 and 0.25 were also tested and the authors report all three were comparable in the magnitude of statistical comparisons.

• Not reported: the paper does not report whether expressed preference relates to learning outcomes, grades, or retention. Popularity of instructors as measured by raw watch counts, per-topic preference differences, and the specific pedagogical differences between the two instructors are also not quantified.

Methodology in Plain English

The study drew on two data sources: a pre-course survey capturing self-reported gender and prior programming experience on a 5-point Likert scale (5 being the most experienced), and log traces of what students actually watched.

Two Fall semesters were included: 662 students from 2021 and 559 from 2022 who had sufficient exposure to both instructors; students with minimal participation were excluded. To keep the comparison clean, the authors looked only at video explanations from the two faculty instructors and excluded teaching assistants.

The platform assigns a default first instructor at random for each concept, and that assignment stays the same across page reloads and across an entire lesson. So when a student watches the non-default instructor instead, that is a deliberate deviation, which the authors treat as a signal of preference. The preference score compares how often a student switched toward Instructor A versus toward Instructor B.

"Initial preference" was computed from the first 20% of each student's video-watching records, on the reasoning that this window captures inclination before much familiarity has developed. "Semester preference" used all records from the whole semester. Scores were also computed week by week to plot trajectories.

Students were sorted into three categories: preferring A, preferring B, or no preference, using an absolute score threshold of 0.2.

For the statistical work, a Shapiro-Wilk test showed preference scores were not normally distributed, so the authors used Kruskal-Wallis tests to compare continuous scores across subgroups, Wilcoxon pairwise tests for follow-up comparisons, and chi-square tests for the categorical preference patterns. Bonferroni correction controlled the false discovery rate, and eta-squared and Cramer's V described effect sizes.

Why This Matters

The paper matters because it treats instructor preference as something that develops rather than a fixed attitude to be measured once. That shift is only possible because of the platform design: short, modular, conceptually consistent explanations from multiple instructors that students can freely mix and match. This suggests a concrete alternative to the standard model of splitting a large course into independently taught sections.

Real-world applications:

• Multi-instructor course design. Departments teaching large introductory courses could offer students a common lesson structure with explanations contributed by several instructors, letting students choose whose explanation to view rather than locking them into one section.
• Online learning platform features. Platforms could support side-by-side explanations from different instructors for the same concept, with a neutral default assignment, so students can compare and revisit content without penalty.
• Instructor onboarding and content guidelines. The paper describes a structured process in which contributors receive guidelines on which concept and example each explanation must cover, and can review existing content. This is a reusable template for keeping multi-contributor content consistent.
• Equity and representation in teaching staff. The gender findings — with preferences for same-gender instructors emerging over the semester but not initially — give institutions a reason to build diverse instructional teams, particularly in STEM fields where representation is uneven.

Industry relevance: Companies building educational technology, corporate training, and MOOC platforms face the same problem the paper addresses: how to let many subject-matter experts contribute to a single curriculum without producing an incoherent mess. The paper's design choices — short single-concept explanations, randomized-but-stable default ordering, structured authoring guidelines — and its logging approach for measuring which contributors learners actually choose are directly transferable to commercial instructional-content systems.

Future Directions

• Measure pedagogical style and pacing. The authors flag that differences in teaching style and pacing between the two instructors are hard to measure explicitly and were not accounted for; these could confound the gender and experience findings and should be measured in follow-up work.
• Test whether preference affects learning. The study addresses preference expression and selection behavior, not outcomes. The authors call for research connecting preference to learning motivation, learning outcomes, and other behaviors that might indicate preference.
• Extend beyond one course and one institution. The authors note the context-specific nature of the study limits generalizability, and recommend replicating across different settings, student populations, and more diverse instructor profiles.
• Collect qualitative and topic-level feedback. The paper recommends platform features that let students explain why they prefer a particular instructor for particular topics, which would give richer explanatory data than selection logs alone.

Target Audience

This paper is most useful to CS education researchers and learning-analytics researchers interested in student-instructor relationships and log-based modeling of affect or preference. It is also relevant to instructors and administrators who design or run large introductory courses with multiple teaching staff, and to learning platform designers who need to consolidate contributions from many content authors into one coherent curriculum. Readers without a statistics background can follow the framing, platform description, and discussion, but will need some familiarity with non-parametric tests and effect-size interpretation to engage with the results section.

Authors’ abstract

Prior research has established the importance of student instructor preferences and identified various influencing factors. However, the dynamics of how student instructor preferences develop and change are less well understood, due to the limitations of common course structures and reliance on one-time measurements. To bridge this gap, we utilize data from a novel learning platform that provides students with access to instructional content created by multiple instructors. This platform enables the quantification of preference emergence and evolution throughout an entire semester, as students repeatedly select content from different instructors. Examining both initial and final student instructor preferences suggests that preference is a dynamic construct continually shaped by experiences. Furthermore, our analysis of the associations between preferences and student characteristics reveals a nuanced picture: while student attributes did not significantly correlate with initial preferences, substantial differences emerged in final preferences across genders and self-reported prior programming experience. This analysis contributes to the existing body of knowledge by expanding our understanding of student instructor preferences and student-instructor relationships in computer science education. We also provide practical insights that institutions and instructors can draw on when multiple instructors collaborate on a course.