liangnjupt/VisTouch - 仓库详情

AI 仓库情报摘要

FR-AI / ANALYSIS

为什么值得关注

它提供了大规模同步视觉-触觉-音频的真实传感器数据，并设计了跨压力泛化划分与可复现的轻量基线，是多模态触觉感知研究的重要公开资源。

适合谁使用

机器人触觉与灵巧操作研究人员
多模态机器学习研究者
跨模态语义通信领域人员
材质识别与触觉信号处理开发者

典型使用场景

基于音频、触觉和视频的材质识别模型训练与基准测试
低采样率或噪声触觉信号的超分辨率重建
触觉与声音之间的跨模态检索
仅从声音生成触觉力曲线

项目优势

2000 个真实采集的时间对齐音频/触觉/视频三元组，覆盖 8 类材料
基线效果显著优于随机或朴素方法：材质识别 76.0%，触觉超分辨率相关系数 0.996
按接触力划分训练/测试集，用于评估跨压力泛化能力
代码与数据许可宽松（MIT/CC-BY-4.0），基线可在 CPU 上快速复现

使用前须知

当前公开版本仅包含 47 类材料中的 8 类
数据需从外部网盘下载，仓库内 dataset 目录为空
基线刻意保持轻量，不代表当前最优结果

README 快速开始

A large-scale synchronized vision–touch–audio dataset of robotic sliding contact

🎬 What you can do with VisTouch

  🏷️ Material Recognition
  fused audio+tactile+video · 76.0% test acc (chance 12.5%)


  
  🧵 Tactile Super-Resolution
  clean 100Hz force curve from noisy low-rate input · −85% MSE, 0.996 corr




  
  🔍 Cross-Modal Retrieval
  touch query → matching sound clip · 2× over chance Recall@1/@5


  
  🔄 Cross-Modal Generation
  haptic recovery: tactile force curve generated from sound alone · 0.80 corr

📖 About

VisTouch was constructed for Cross-Modal Semantic Communications (IEEE Wireless Communications, 2022) by controlling a robot arm (UR3 + RH56BF3 dexterous hand) to press and slide across everyday materials while a camera, a microphone, and a tactile force sensor record the same contact event simultaneously.

The full VisTouch research corpus spans 47 material categories and contains million-scale raw sensor observations across video frames, audio waveform samples, and tactile measurements. The companion paper reports 1000+ synchronized video–audio–haptic signal pairs over all 47 categories, supporting cross-modal semantic encoding, retrieval, and haptic signal recovery research.

This repository is the first curated public benchmark release: 2000 timestamp-aligned audio/tactile/video triplets from 8 representative material categories used in the paper's evaluation — brass · linen · paper · polyester · silk · spandex · stone · wood — with a predefined cross-pressure train/test split. Every released sample is a genuine sensor capture; no synthetic data is included. Future updates will progressively release more of the 47-category corpus, additional paths, views, and benchmark tasks.

Capture rig: dexterous hand + microphone + tactile sensor, fixed camera view.

📥 Download

The data files are hosted externally — this repository ships with an empty dataset/ folder:

Source	Link
🌐 Google Drive	drive.google.com/drive/folders/1U2qW1Oqbkj-...
☁️ Baidu Netdisk	[pan.baidu.com/s/1W4cRtxgY9SL85HdnZhoFJA](https://pan.baidu.com/s/

项目描述

A large-scale synchronized vision–touch–audio dataset of robotic sliding contact

VisTouch