linbogenb9-coder/SIR-Transformer

AI 仓库情报摘要

FR-AI / ANALYSIS

为什么值得关注

该框架创新地将冻结大语言模型生成的结构化语义先验集成到数值轨迹预测器中，实现无需在线 LLM 开销的显式运动状态和社交交互推理。

适合谁使用

从事行人轨迹预测与行为建模的研究人员
自动驾驶系统工程师
社交机器人导航算法开发者
研究人机交互与人群分析的学者

典型使用场景

提升自动驾驶车辆的行人轨迹预测精度
实现拥挤环境中移动机器人的社交感知导航
为虚拟环境或游戏模拟逼真的行人运动
分析体育（如 NBA）中的群体动态与交互模式

项目优势

以目标为中心的候选邻域缩小了交互搜索空间
离线 LLM 先验能够捕捉显式运动语义（速度、方向）和社交关系（并行、跟随、碰撞风险）
关系适配器与 ISMT/RSMT 联合建模个体与社交交互
缓存先验保持在线推理效率，与传统数值预测器相当

使用前须知

源代码尚未发布（仓库正在准备中）
依赖离线生成先验的冻结 LLM，可能引入偏差或需要精心设计提示
仅在三个数据集上评估，尚未证明在其他环境中的泛化能力

README 快速开始

SIR-Transformer

Language-Guided Semantic Interaction Reasoning for Pedestrian Trajectory Prediction

🚧 Code coming soon.

Overview

SIR-Transformer is a language-guided semantic interaction reasoning framework for pedestrian trajectory prediction. It introduces structured semantic priors into a numerical trajectory predictor to jointly model individual motion patterns and socially meaningful interactions.

The framework first constructs a target-centered candidate neighborhood from observed trajectories. A frozen large language model is then used offline to transform trajectory-derived evidence into per-agent motion priors and pairwise social relation priors with confidence scores. These semantic priors are cached and integrated with continuous trajectory representations through the Individual Semantic Motion Transformer (ISMT), Relation Adapter, and Relation-aware Social Motion Transformer (RSMT).

The language model is not used to decode future coordinates and is not invoked during online inference.

Highlights

Target-centered interaction reasoning: Candidate neighborhoods provide a compact interaction space for semantic reasoning.
Offline language priors: A frozen LLM generates structured motion and social relation priors before model training and inference.
Explicit motion semantics: Motion priors describe speed state, speed trend, and direction state.
Explicit social relations: Relation priors cover none, parallel, following, approaching, and collision risk.
Relation-aware prediction: ISMT, the Relation Adapter, and RSMT combine semantic priors with continuous trajectory and geometric representations.
Efficient deployment: Cached priors preserve the online efficiency of a numerical trajectory predictor.

Framework

The SIR-Transformer pipeline consists of four main stages:

Trajectory and kinematic cue encoding
Target-centered candidate neighborhood construction
Offline language-guided semantic prior generation
Relation-aware trajectory prediction

The prediction module follows:

Observed trajectories
        ↓
Candidate neighborhood
        ↓
Offline motion and relation priors
        ↓
ISMT → Relation Adapter → RSMT
        ↓
Future pedestrian trajectories

Benchmarks

The framework is evaluated on:

ETH-UCY
Stanford Drone Dataset (SDD)
N

项目描述

Official repository for SIR-Transformer: offline LLM-guided semantic interaction reasoning for pedestrian trajectory prediction. Code coming soon.