Why it is worth attention

It demonstrates a practical integration of Gemini Live’s low-latency audio streaming with a mobile computer-use layer, enabling both autonomous app operation and guided teaching overlays, all from a single open-source Kotlin project.

Who it is for

Android developers building real-time AI assistants
Researchers studying multimodal interaction on mobile devices
Accessibility advocates exploring voice-driven phone control
Hackathon participants seeking a working Gemini Live prototype

Use cases

Hands-free app operation (e.g., open Uber, set destination) with confirmation before sensitive actions
Guided phone usage tutorials (e.g., highlight buttons, explain UI elements)
Accessibility aid for users who prefer voice commands over touch
Rapid prototyping of real-time multimodal AI on Android

Strengths

Low-latency, interruptible voice conversation via Gemini Live WebSocket and audio streaming
Full mobile computer-use layer built on AccessibilityService—can tap, type, scroll, swipe, and draw overlays
Multiple summon paths (floating handle, assistant intent, Quick Settings) for seamless access
Open-source (AGPL-3.0) with a clear token broker architecture that keeps API keys server-side

Considerations

Requires multiple runtime permissions (microphone, notifications) and system settings (Accessibility, MediaProjection) that cannot be silently granted
Currently a demo—multi-turn autonomous loops and UI idle detection are still being hardened
Android-only; no cross-platform support mentioned

README quick start

-->

Tappy Live

Tappy Live is a real-time Android voice assistant and phone-use teacher for Problem Statement 1: Real-Time Multimodal Interaction.

It combines two layers:

Gemini Live API for low-latency, interruptible voice conversation.
A mobile computer-use layer on Android, built with AccessibilityService, app snapshots, indexed UI elements, actions, and overlays.

The demo goal is to let a user summon Tappy from anywhere, ask for help in natural language, and either receive guided teaching with overlays or ask Tappy to operate apps on their behalf until a sensitive final action requires confirmation.

Hackathon focus

Tappy Live is built around Gemini Live API using:

gemini-3.1-flash-live-preview for real-time audio, interruption handling, and tool calls.
A Cloud Run token broker that mints short-lived Gemini Live ephemeral tokens.
Direct Android WebSocket connection to Gemini Live.
Native Kotlin Android with Compose.
Android microphone streaming through AudioRecord.
Gemini audio playback through AudioTrack.
Android AccessibilityService for the mobile computer-use execution environment.

Tappy is not just a chat UI with a microphone. It sees app UI through Accessibility snapshots, reasons over indexed visible elements, can tap and type, and can draw overlays directly on top of the phone screen.

What works now

The current app can:

Call the Cloud Run token broker from the phone.
Receive a Gemini Live ephemeral token without displaying token values.
Open a Gemini Live WebSocket from Android.
Send setup to gemini-3.1-flash-live-preview.
Receive Gemini Live audio chunks.
Play Gemini audio through the phone speaker.
Stream microphone audio to Gemini Live as 16 kHz mono PCM.
Keep the Live session and mic in a foreground service.
Capture screen frames through a foreground MediaProjection service.
Run an AccessibilityService for app control and visual overlays.
Draw a floating side handle that summons Tappy from any screen.
Register as an Android assistant target.
Register a Quick Settings tile for Tappy.
Show a bottom voice pill with a Stop action during a summoned session.
Inspect the current app using an indexed Accessibility snapshot.
Execute mobile computer-use actions:
- list installed launchable apps
- open app by package name
- open app by common name
- open link or deep lin

Related repositories

Similar projects matched by category, topics, and programming language.

Sylvara-Lin

LianYu-app

An open-source Android framework for building AI companion apps, offering a modular architecture, multiple AI provider support, local model inference, and a complete product skeleton including chat, role management, memory, and group chat systems.

Kotlin

172

MarshallBear1

react-native-system-thumbnails

A React Native library that generates OS-native thumbnails for local files with a single API, falling back to file icons when needed.

Kotlin

114

doesthings

FreeFCC

FreeFCC is a free, open-source Android app that unlocks FCC mode on DJI smart controllers by sending raw DUML commands from JSON profiles, without servers or tracking.

Kotlin

113