IM HERE: Interaction Model for Human Effort Based Robot Engagement

AI summary

Overview

Research area: Human-Robot Interaction (HRI), social robotics, engagement modeling, attention/focus estimation, autonomous systems.

Technical level: Intermediate. The paper is conceptual and theoretical, grounded in definitions, axioms and inferences rather than machine learning benchmarks. Readers need familiarity with HRI terminology (engagement, F-formations, multi-party interaction) but not with heavy mathematics.

Scope: The paper proposes and formalizes IM HERE, a generalizable, effort-based framework for modeling engagement across human-human, human-robot and robot-robot interaction, and discusses how autonomous systems could use it to behave according to social norms while pursuing their own social goals.

What This Paper Is About

Existing definitions and models of engagement in HRI are either vague or bound to specific scenarios, and most work focuses only on detecting engagement rather than reflecting and revealing it. The authors argue that engagement fundamentally arises from bilateral focus, and that effort — the allocation of an entity's limited resources — is a sufficient indicator of it. Their goal is to provide a formal, generalizable structure that describes relational states between entities, identifies miscommunication, and translates social situations into directives for autonomous systems.

Key Contributions

1. A formal effort-based engagement framework (IM HERE). It defines entities, effort and focus, and reduces any pairwise relationship to a focus placement and one of four states: Engaged, Passive, Requested and Buildup. Engaged is defined as reciprocal focus (focus_A = B ∧ focus_B = A); Buildup and Requested are the one-sided intermediate cases; objects (which cannot interpret or modulate effort) are limited to Passive and Requested.

2. Explicit separation of subjective and objective perspectives. By combining a meta-level view of all entities' focus with each entity's subjective Effort Interpretation (EI), the framework can pinpoint miscommunication as divergence between objective and subjective state estimates, and can characterize effective and polite communication against social norms.

3. A modular four-stage pipeline with a closed feedback loop. Signal conversion → EI → focus estimation → state estimation → goal evaluation → effort (EE) modulation, which is then perceived by others. Stages can be bypassed, merged or replaced while retaining the core principles, so the framework can complement existing models rather than override them.

4. An open-source implementation and simulation environment. The authors state they provide an open-source implementation of the model (https://github.com/RoSA-Study/IM-HERE) plus a simulation environment for testing different scenarios, and they show how prior engagement definitions (e.g., Sidner et al.'s dynamic process model, Poggi et al.'s attribution-based view) can be represented within the framework.

Main Findings

• Engagement reduces to reciprocal focus: The framework's central claim is that an Engaged state exists between two entities exactly when each one's focus points at the other, and that every pairwise relationship can be described by a single state at a time.

• Effort is the single sufficient indicator: Effort is defined as the allocation of an entity's limited resources (gaze, body or head orientation, movement/change of proximity, speech, etc.), parametrized by alignment and magnitude and susceptible to contrast. The Engagement Effort (EE) sent by one entity is transformed into the Effort Interpretation (EI) of the receiver through five multiplied factors: Contrast (T), Magnitude (M), Contribution (C), Alignment (A) and Preference (P): EI = T · M · C · A · P.

• Focus follows the strongest interpreted signal: Focus is computed as the argmax of EI over all entities, and an entity's focus points to the entity whose EE produces the strongest EI.

• Objects and engageable entities differ: Entities that can differentiate between other entities and modulate EE are engageable and can occupy all four states; objects cannot focus and are limited to Passive or Requested.

• State estimation can be simple: Because focus is binary, engagement can be checked with a single condition — entity.engaged = if (entity.focus.focus == entity) — and in a larger environment only the current focus and whether it is reciprocal needs to be tracked. For engageable entities there is always exactly one relationship in Buildup or Engaged.

• Complexity scales steeply: Each entity must estimate the EE of others, giving a complexity of O(N^N) in signal interpretation and focus estimation, though the binary nature of focus simplifies state determination and complexity can be reduced by tracking only relevant relationships.

• Groups need no separate model: A group is characterized by a concatenation (chain) of focus — for example, focus resting on the current speaker while the speaker's focus alternates among group members. Group constellations and spatial arrangements can then be described with F-formations and their O-space (reserved for activity), P-space (participants) and R-space (buffer to the outside world), letting third parties avoid disturbing or deliberately join an interaction.

• Robot prerequisites: To be perceived as an engageable participant rather than an object, a robot must receive signals via sensors, maintain a virtual focus, and emit an EE via actuators (e.g., a simple acknowledgment of presence) to enable Buildup and Engaged states.

• No empirical validation is reported: The paper presents a theoretical framework, a state machine, formulas and a simulation environment; results, benchmark comparisons, datasets or user-study numbers are not reported. The authors explicitly list real-world validation as a limitation and future step.

Methodology in Plain English

The authors build the framework deductively, defining terms, stating foundational assumptions, then deriving axioms and inferences from them.

They begin with three definitions: an Entity is a self-contained physical object able to send and receive signals (people, animals, machines or objects); Effort is the allocation of an entity's limited resources; and Focus is a summary of an entity's cognitive or perceptual resources directed at one specific entity. The authors deliberately use "focus" instead of "attention," citing Hommel and colleagues' argument in "No one knows what attention is" that attention is not a useful scientific term.

Two foundations support the model: Watzlawick's Communication Theorem ("You can not not communicate"), meaning every entity constantly emits effort consciously or unconsciously; and Subjectivity, drawing on the Constructivist Theory of Communication, Schramm's Model, the Helical Model, Reception Theory, the Transactional Model and Von Thun's Four-Sides Model, meaning each receiver interprets signals in its own context.

From these, three axioms follow: the Focus Principle (one focus at a time, on the highest EI), the Engagement Principle (engagement requires reciprocal focus) and the Revelation Principle (focus must be observable through EE modulation).

Practically, signals sent by an entity are attenuated by the environment, then interpreted by the receiver as EI, computed as the product of contrast, magnitude, contribution, alignment and preference. For example, when Bob waves at Carla, alignment reflects their relative position and orientation, magnitude is the gesture's intensity, contrast is the gesture relative to background noise or brightness, contribution carries the message and preference is Carla's personal scaling. Multiple EI values from the same sender across channels (audio, visual, etc.) can be combined into a total EI, and focus is simply the sender with the highest total. If entity A's focus is B and B's focus is A, they are Engaged. A simulation example with an entity named Alex shows seven possible EE outputs — walking (footstep sounds), body, gaze, touch, gesture, talking and bumping — with Bob receiving Alex's focus.

The framework is designed to be modular: the four stages can be bypassed, modified or added to, and focus or engagement may be estimated directly if the full signal chain is not needed.

Why This Matters

Impact on research: The paper positions itself as a unifying structure that addresses the vagueness and ambiguity of earlier engagement models identified by Oertel et al. and Sorrentino et al. Most prior HRI work treats engagement as detection and prediction using multimodal indicators (physiological signals, body posture, facial expressions, gaze, head pose, speech patterns); IM HERE adds the reflection and revelation side needed to derive interaction strategies, and it claims to represent existing definitions (Sidner et al., Poggi et al., "with-me-ness," task-, social- and social-task-engagement, group engagement) within one formalism.

Real-world applications (as discussed or implied by the paper):

• Task-oriented interactions: a robot prioritizing engagement with entities relevant to its operational goals.
• Social interactions: modulating behavior for politeness and adherence to social norms.
• Group dynamics: recognizing focus chains and F-formations to join a multi-party conversation without disturbing it, or to avoid intruding on O-space and P-space.
• Long-term engagement: continuously optimizing focus and effort to sustain meaningful interaction beyond the initial novelty.

Industry relevance: The authors frame the work as laying the foundation for real-world integration while accounting for noise and bias, and target autonomous systems that must pursue their own social goals while remaining socially acceptable. It is directly applicable to social and service robots, but the paper reports no deployed system, product or commercial evaluation.

Future Directions

1. Developing optimization functions for EE modulation tailored to specific contexts and interaction goals — how an entity should best adjust its effort to gain or maintain another's focus.

2. Validating the framework in real-world HRI scenarios to refine parameters and test the key assumptions, since the theoretical robustness of the model has not yet been empirically demonstrated.

3. Extending the framework to other domains such as education and healthcare, where sophisticated engagement is described as particularly crucial.

4. Addressing the practical challenges of scalability and input quality — the O(N^N) complexity as entities increase, and dependence on robust sensor data for gaze, posture and environmental factors, since noise or ambiguity can cause incorrect state estimation, as can multimodal signal processing pipelines for speech and gestures.

Target Audience

This paper benefits HRI and social robotics researchers and graduate students interested in engagement, attention and multi-party interaction modeling; roboticists designing interaction strategies and state machines for social robots; researchers studying communication theory, politeness and social norms as applied to machines; and practitioners looking for an open-source implementation and simulation environment to test engagement scenarios. It is less suited to readers seeking empirical results, benchmark comparisons or validated performance numbers, since the paper reports a theoretical framework and does not present experiments.

Authors’ abstract

The effectiveness of human-robot interaction often hinges on the ability to cultivate engagement - a dynamic process of cognitive involvement that supports meaningful exchanges. Many existing definitions and models of engagement are either too vague or lack the ability to generalize across different contexts. We introduce IM HERE, a novel framework that models engagement effectively in human-human, human-robot, and robot-robot interactions. By employing an effort-based description of bilateral relationships between entities, we provide an accurate breakdown of relationship patterns, simplifying them to focus placement and four key states. This framework captures mutual relationships, group behaviors, and actions conforming to social norms, translating them into specific directives for autonomous systems. By integrating both subjective perceptions and objective states, the model precisely identifies and describes miscommunication. The primary objective of this paper is to automate the analysis, modeling, and description of social behavior, and to determine how autonomous systems can behave in accordance with social norms for full social integration while simultaneously pursuing their own social goals.