Showing versus Doing: Teaching by Demonstration

AI summary

Overview

Research area: Machine learning and computational cognitive science, specifically inverse reinforcement learning (IRL), Bayesian models of pedagogy, and human teaching behavior.

Technical level: Intermediate. The paper assumes familiarity with Markov Decision Processes, softmax/Boltzmann policies, and Bayesian inference, but its central idea is explained through everyday examples.

Scope: The paper builds a computational model of "showing" (intentional teaching by demonstration) as distinct from "doing" (efficient task performance), tests it against human behavior in two experiments, and asks whether standard IRL algorithms learn better from teaching demonstrations than from ordinary ones.

What This Paper Is About

Standard inverse reinforcement learning assumes that a demonstrator is simply performing a task as well as possible, and the observer infers the reward function behind that behavior. This paper asks what changes when the demonstrator knows they have an audience and deliberately chooses actions that communicate the reward function, and it formalizes that difference as a Bayesian model called Pedagogical Inverse Reinforcement Learning. The authors then test whether people actually behave this way, and whether such teaching demonstrations help ordinary IRL algorithms learn faster or more accurately.

Key Contributions

1. A computational model of teaching by demonstration. The authors combine inverse reinforcement learning with Bayesian models of teaching by example, defining a teacher that picks trajectories which maximize how strongly an observer would infer the true reward function. The resulting distribution depends on the teacher reasoning about a learner who is reasoning about an ordinary "doing" agent.

2. A direct formal contrast between showing and doing. "Doing" is modeled as standard optimal planning in an MDP with a Boltzmann (softmax) action-selection policy; "showing" is modeled as a softmax over trajectories weighted by the observer's posterior probability of the true reward. The model is spelled out as an algorithm (Pedagogical Trajectory Algorithm) taking parameters including α (teacher maximization), λ (demonstrator softmax), l_max (maximum trajectory length), p_min (minimum hypothetical doing probability), and γ (discount factor).

3. Two behavioral experiments with human participants (Amazon Mechanical Turk) testing model predictions: Experiment 1 on goal-based reward functions in a grid with three terminal goals, and Experiment 2 on feature-based reward functions in a colored-tile grid where three tile types are independently safe or dangerous.

4. A test of whether showing helps off-the-shelf IRL. Human and model trajectories from both conditions were fed into Maximum-Likelihood Inverse Reinforcement Learning (MLIRL), a standard algorithm that estimates continuous feature weights by gradient ascent, to check whether teaching trajectories are more informative than ordinary ones.

Main Findings

• People choose disambiguating paths when showing (Experiment 1). On & trials, entering from the left and on % trials, entering from above were coded as "showing" trajectories (out of 3 per goal). A 2x2 ANOVA with condition (Show vs. Do) and goal (% vs. &) found a main effect of condition, F(1,57) = 16.17, p < .001 (Show: M = 1.82, S.E. 0.17; Do: M = 1.05, S.E. 0.17), and a main effect of goal, F(1,57) = 4.77, p < .05 (%-goal: M = 1.73, S.E. 0.18; &-goal: M = 1.15, S.E. 0.16). There was no interaction, F(1,57) = 0.98, p = 0.32.

• An unpredicted "swerving" effect. For the # (lower right) goal, the model predicts no difference between conditions, but participants swerved (more than one change of direction) more often in Show than Do (Show: M = 0.83, Do: M = 0.26; t(44.2) = 2.18, p = 0.03). The authors suggest participants may swerve to signal an intention to move "directly" toward the goal.

• Showing changes how people teach feature-based rewards (Experiment 2). Show participants visited a wider variety of colored tiles (entropy of colored-tile visits: Show M = 0.50, SE = 0.03; Do M = 0.39, SE = 0.03; t(54.9) = -3.27, p < 0.01), spent a higher proportion of visits on colored tiles after the first colored tile was visited (Show M = 0.87, SE = 0.01; Do M = 0.82, SE = 0.01; t(55.6) = -3.14, p < .01), and revisited states ("looping") far more (Show M = 1.38, SE = 0.22; Do M = 0.10, SE = 0.03; t(28.3) = -2.82, p < .01).

• Showing did not cost task performance. There was no difference between conditions in total rewards won (t(46.2) = .026, p = 0.80).

• Model fits separate the two conditions. Fitting parameters by maximum median likelihood for each reward function and condition combination, the standard planning model better captured the Do condition while the pedagogical model better captured the Show condition. The pedagogical model fit Show even though the planning model could have been given a high λ and thus behave more randomly, indicating that showing is not merely noisy behavior.

• Standard IRL learns better from showing. When human and model trajectories were fed into MLIRL (with non-goal feature weights constrained to be non-positive), the algorithm recovered the true reward function better from showing trajectories than from doing trajectories.

• A formal proof for a simple case. In the supplementary materials, a 3x2 gridworld with two goals (X and Y), no step costs, γ = .99, and trajectories of length 2 is used to prove that a showing agent prefers the trajectory x_out over x_in when X is the goal, because x_out is less consistent with the alternative goal Y.

Methodology in Plain English

The authors start from a standard picture of goal-directed behavior: an agent in a Markov Decision Process picks actions that are likely to lead to high reward, with some noise (a Boltzmann policy). An observer watching such an agent can run Bayes' rule backwards to guess which reward function the agent is pursuing. This is the "doing" model.

The "showing" model adds one layer. A teacher who knows the true reward function chooses trajectories not because they are likely under the doing model, but because they make the correct reward function most probable in the observer's mind, raised to a parameter α. A teacher with high α behaves like a maximally helpful instructor; with low α, behavior becomes more random. Because the teacher's objective is the learner's posterior, it can favor roundabout routes, repeated visits to informative tiles, and looping between safe regions — behaviors that would be inefficient for an agent only doing the task.

To test this, the authors ran two online experiments with Amazon Mechanical Turk participants. Each experiment had a learning phase, then split participants into a Do condition (paid for task performance) or a Show condition (paid according to how well a matched partner performed after watching their behavior). The key manipulation is that only the Show participants were told their behavior would be used by someone else.

They then compared model predictions with human behavior in two ways. First, they checked for the specific behavioral signatures the models predict (path choice, tile variety, time on informative tiles, looping). Second, they fitted model parameters to maximize the median likelihood of the observed trajectories and compared normalized fit quality between models and conditions. Finally, they used the trajectories as training data for MLIRL, a continuous-weight IRL algorithm, and measured how close the learned reward weights came to the true ones.

Why This Matters

Impact on research. The paper reframes IRL: if demonstrators adapt their behavior to be informative, then the standard assumption that an expert is just acting optimally is systematically wrong, and models of teaching should be built into how we interpret demonstrations. It also connects machine learning with developmental and cognitive psychology work on pedagogical reasoning, and provides an experimental paradigm (colored tiles with independently safe/dangerous features) for studying how people communicate reward structure.

Potential real-world applications (not evaluated in this paper):

• Training reward functions for robots or software agents from a small number of human demonstrations, where the human can deliberately choose informative examples.
• Educational technology and instructional design, where the ordering and selection of worked examples matters for what students infer.
• Human-robot interaction, where a robot that recognizes "teaching" intent can weight demonstrations differently from casual ones.
• Designing interfaces or tutorials where an expert's screen recording is meant to convey rules, not just accomplish a task.

Industry relevance. Any pipeline that learns preferences or rewards from human demonstrations — imitation learning, RLHF-style reward modeling, apprenticeship learning — depends on the quality and informativeness of the demonstrations collected. The paper's finding that teaching demonstrations train a standard algorithm like MLIRL better than ordinary performance, at no cost in task reward, suggests that simply instructing data collectors to teach rather than perform is a cheap improvement.

Future Directions

• Extending the model to settings with richer state features, where teaching a generalizable rule is harder than teaching a small set of tile values.
• Testing whether more sophisticated observers can explicitly exploit a demonstrator's pedagogical intent — that is, learners who know they are being taught — for even better inference.
• Explaining behaviors the model does not predict, such as the swerving observed on the # goal in Experiment 1, which the authors attribute to participants signaling an intention to move directly toward the goal.
• Broadening the evaluation from MLIRL to other inverse reinforcement learning algorithms and to interactive settings where the teacher can adapt to the learner over multiple demonstrations.

Target Audience

Researchers and graduate students in reinforcement learning and inverse reinforcement learning who work on learning from demonstration; cognitive scientists and psychologists studying pedagogy, communication, and theory of mind; and practitioners building systems that learn reward functions or preferences from human data, who want to know how the framing given to demonstrators affects what can be learned. Readers without background in MDPs or Bayesian inference will find the experimental sections accessible but the modeling sections demanding.

Authors’ abstract

People often learn from others' demonstrations, and inverse reinforcement learning (IRL) techniques have realized this capacity in machines. In contrast, teaching by demonstration has been less well studied computationally. Here, we develop a Bayesian model for teaching by demonstration. Stark differences arise when demonstrators are intentionally teaching (i.e. showing) a task versus simply performing (i.e. doing) a task. In two experiments, we show that human participants modify their teaching behavior consistent with the predictions of our model. Further, we show that even standard IRL algorithms benefit when learning from showing versus doing.