SIR-Transformer

SIR-Transformer

Language-Guided Semantic Interaction Reasoning for Pedestrian Trajectory Prediction

🚧 Code coming soon.

Overview

SIR-Transformer is a language-guided semantic interaction reasoning framework for pedestrian trajectory prediction. It introduces structured semantic priors into a numerical trajectory predictor to jointly model individual motion patterns and socially meaningful interactions.

The framework first constructs a target-centered candidate neighborhood from observed trajectories. A frozen large language model is then used offline to transform trajectory-derived evidence into per-agent motion priors and pairwise social relation priors with confidence scores. These semantic priors are cached and integrated with continuous trajectory representations through the Individual Semantic Motion Transformer (ISMT), Relation Adapter, and Relation-aware Social Motion Transformer (RSMT).

The language model is not used to decode future coordinates and is not invoked during online inference.

Highlights

• Target-centered interaction reasoning: Candidate neighborhoods provide a compact interaction space for semantic reasoning.
• Offline language priors: A frozen LLM generates structured motion and social relation priors before model training and inference.
• Explicit motion semantics: Motion priors describe speed state, speed trend, and direction state.
• Explicit social relations: Relation priors cover none, parallel, following, approaching, and collision risk.
• Relation-aware prediction: ISMT, the Relation Adapter, and RSMT combine semantic priors with continuous trajectory and geometric representations.
• Efficient deployment: Cached priors preserve the online efficiency of a numerical trajectory predictor.

Framework

The SIR-Transformer pipeline consists of four main stages:

1. Trajectory and kinematic cue encoding
2. Target-centered candidate neighborhood construction
3. Offline language-guided semantic prior generation
4. Relation-aware trajectory prediction

The prediction module follows:

Observed trajectories
        ↓
Candidate neighborhood
        ↓
Offline motion and relation priors
        ↓
ISMT → Relation Adapter → RSMT
        ↓
Future pedestrian trajectories

Benchmarks

The framework is evaluated on:

• ETH-UCY
• Stanford Drone Dataset (SDD)
• N