为什么值得关注

它展示了 Gemini Live 低延迟音频流与移动端计算机操作层的实用集成，既能自主执行应用操作，又能提供引导式教学覆盖层，且全部代码开放。

适合谁使用

构建实时 AI 助手的安卓开发者
研究移动设备多模态交互的研究人员
关注语音驱动手机控制的辅助功能倡导者
需要可运行的 Gemini Live 原型的黑客马拉松参与者

典型使用场景

免持应用操作（如打开 Uber、设置目的地），在敏感操作前要求确认
引导式手机使用教程（如高亮按钮、解释 UI 元素）
为偏好语音命令而非触摸的用户提供辅助功能
快速原型验证安卓上的实时多模态 AI 交互

项目优势

通过 Gemini Live WebSocket 和音频流实现低延迟、可中断的语音对话
基于 AccessibilityService 的完整移动计算机操作层——可点按、输入、滚动、滑动并绘制覆盖层
多种唤醒方式（悬浮手柄、助理意图、快捷设置），使用便捷
开源（AGPL-3.0），采用清晰的令牌代理架构，API 密钥仅在服务端保留

使用前须知

需要多项运行时权限（麦克风、通知）和系统设置（无障碍、媒体投影），无法静默授权
当前为演示阶段，多轮自主循环和 UI 空闲检测仍在加固中
仅支持安卓，未提及跨平台支持

README 快速开始

-->

Tappy Live

Tappy Live is a real-time Android voice assistant and phone-use teacher for Problem Statement 1: Real-Time Multimodal Interaction.

It combines two layers:

Gemini Live API for low-latency, interruptible voice conversation.
A mobile computer-use layer on Android, built with AccessibilityService, app snapshots, indexed UI elements, actions, and overlays.

The demo goal is to let a user summon Tappy from anywhere, ask for help in natural language, and either receive guided teaching with overlays or ask Tappy to operate apps on their behalf until a sensitive final action requires confirmation.

Hackathon focus

Tappy Live is built around Gemini Live API using:

gemini-3.1-flash-live-preview for real-time audio, interruption handling, and tool calls.
A Cloud Run token broker that mints short-lived Gemini Live ephemeral tokens.
Direct Android WebSocket connection to Gemini Live.
Native Kotlin Android with Compose.
Android microphone streaming through AudioRecord.
Gemini audio playback through AudioTrack.
Android AccessibilityService for the mobile computer-use execution environment.

Tappy is not just a chat UI with a microphone. It sees app UI through Accessibility snapshots, reasons over indexed visible elements, can tap and type, and can draw overlays directly on top of the phone screen.

What works now

The current app can:

Call the Cloud Run token broker from the phone.
Receive a Gemini Live ephemeral token without displaying token values.
Open a Gemini Live WebSocket from Android.
Send setup to gemini-3.1-flash-live-preview.
Receive Gemini Live audio chunks.
Play Gemini audio through the phone speaker.
Stream microphone audio to Gemini Live as 16 kHz mono PCM.
Keep the Live session and mic in a foreground service.
Capture screen frames through a foreground MediaProjection service.
Run an AccessibilityService for app control and visual overlays.
Draw a floating side handle that summons Tappy from any screen.
Register as an Android assistant target.
Register a Quick Settings tile for Tappy.
Show a bottom voice pill with a Stop action during a summoned session.
Inspect the current app using an indexed Accessibility snapshot.
Execute mobile computer-use actions:
- list installed launchable apps
- open app by package name
- open app by common name
- open link or deep lin

tappy