How the ESP32-iPhone-Agent Project Turns a $7 Chip Into an AI Controller
A trending open-source project is teaching existing iPhones to operate autonomously using a simple ESP32 board, even as Apple launches its new foldable hardware. The onehumanbeing/esp32-iphone-agent project allows an AI model to control a real iPhone using a $7 ESP32-S3 board, requiring no jailbreak, no developer account, and no external computer to function.
The Hardware Setup: Using a $7 ESP32-S3 as a Bridge
The ESP32-S3 board serves as both an AirPlay receiver and a USB HID device. It opens its own Wi-Fi hotspot and appears in the iPhone’s Screen Mirroring list as “ESP32 Agent.” The board keeps the H.264 video stream from the phone and forwards it to the agent app over Bluetooth Low Energy (BLE). Meanwhile, the same board acts as a standard USB mouse and keyboard when connected via USB, allowing it to simulate input actions. The project maintainers advise users to follow only the official README instructions and avoid unofficial ‘quick install’ scripts, which are not from the maintainers and may pose risks.
How the Agent Sees and Interacts with the iPhone
The agent app runs directly on the iPhone, decoding the AirPlay stream and sending screenshots to an OpenAI-compatible vision model. With AssistiveTouch enabled, the on-screen pointer moved by the USB HID mouse simulates taps and swipes. The AI agent can perform tasks like opening apps, typing, tapping, and swiping by moving this pointer. The system includes a feedback loop where the app tracks the pointer’s position, corrects misses by wiggling it to relocate it on screen, and waits for the display to settle between actions. Every autonomous run is recorded on the phone, allowing users to replay steps or export traces for further analysis. The iPhone’s Dynamic Island indicates the agent’s current activity while it runs in the background.
The Role of Vision Models in Autonomous Navigation
The agent relies on any OpenAI-compatible vision model to interpret screenshots and decide on actions. The phone uses its cellular data connection to communicate with the model while connected to the ESP32’s hotspot. After each action, the app checks whether the screen has changed and reports this to the model, enabling iterative task completion. If the board, hotspot, or mirroring link is lost, the run pauses and resumes from the same step once connectivity is restored. This design allows the agent to handle complex sequences, such as setting a recurring alarm in the Clock app, without human intervention.
Safety, Privacy, and the Importance of Official Install Methods
The project emphasizes that no jailbreak, developer account, or external computer is needed, reducing barriers to use. However, the maintainers warn against bypassing the official installation process, noting that unverified scripts could compromise device security. Users are instructed to install the Arduino CLI, ESP32 core 3.3.x, and NimBLE-Arduino 2.5.x, then use the provided flash.sh script to ensure proper BLE buffer sizes. The agent’s ability to export traces supports transparency, letting users review exactly what the AI did during a run. As AI agents become more capable of interacting with personal devices, projects like this highlight both the potential and the need for careful, source-verified deployment.
Reviewing and Exporting Agent Traces
Every autonomous run is recorded on the phone, allowing users to replay steps or export traces for further analysis. The agent app stores each run locally, preserving the sequence of actions, screenshots, and annotations for later review. This feature enables users to verify the AI’s behavior, debug unexpected outcomes, or refine task performance over multiple iterations.
To export a trace, users open a saved run within the app, tap the menu (⋯), and select “Export trace for analysis.” The exported file includes the screenshots, action logs, and any saved annotations from the run. These traces can be shared with other agents for collaborative learning or analyzed externally to improve model prompts and decision-making logic.
For batch processing, the project supports exporting multiple runs at once via the “Export runs for agent review” function. This allows users to copy all trace data over USB or transfer files for bulk analysis, facilitating deeper inspection of agent behavior across different tasks or environmental conditions.
Installation Requirements and Verified Setup Steps
The project requires specific software versions to ensure proper functionality and security. Users must install Arduino CLI, the ESP32 core version 3.3.x, and NimBLE-Arduino version 2.5.x. After installation, the provided flash.sh script should be used to flash the firmware, as it applies the necessary BLE buffer sizes defined in hardware/hid_bridge/nimble_cfg.h. The maintainers explicitly advise against using unofficial ‘quick install’ scripts or commands from issues and comments, noting these are not from the maintainers and may pose risks to device security.
Official installation instructions are the only recommended method, with the README serving as the authoritative source. The flash.sh script is optional but strongly encouraged, as it handles critical configuration details that manual flashing might overlook. By following these verified steps, users ensure the ESP32-S3 board correctly operates as both an AirPlay receiver and a USB HID device, maintaining the integrity of the BLE connection and input simulation.
Project Context Amid Apple’s Foldable iPhone Launch
The ESP32-iPhone-Agent project gained attention as Apple launched its first foldable iPhone, the iPhone Duo, with preorders opening October 16, 2026. The device features a 7.6-inch inner display and 5.4-inch outer screen, powered by the A20 Pro chip and running iOS 27.1. Apple unveiled the iPhone Duo on September 9, 2026, with availability beginning October 23 in over 70 markets. The project operates independently of this hardware, requiring no modifications to the iPhone and functioning with existing models through standard AirPlay and USB HID capabilities.
Sources
- GitHub – onehumanbeing/esp32-iphone-agent: Let an AI model use a real iPhone with a $7 ESP32: AirPlay eyes, USB HID hands, any OpenAI-compatible vision model. No Mac, no developer account, no jailbreak.
- iPhone Duo preorders are live! Everything you need to know about Apple's foldable iPhone
- iPhone Duo preorders are live! Everything you need to know about Apple's foldable iPhone
Photo: Brian Krent (CC BY-SA 4.0)
