A $7 ESP32 Chip Allows Vision Models to Control iPhones

A new open-source automation workflow allows vision models to drive an unmodified iPhone using a $7 ESP32-S3 microcontroller. The host system captures the phone display, processes the frames using a vision model, and sends movement commands back to the ESP32-S3, which acts as a standard USB mouse and keyboard. In a reference local setup running Qwen3.8-Flash-Next on a dual DGX Spark system, the pipeline achieved an average speed of approximately 1.3 seconds per action.

To process visual feedback without specialized iOS tools, the system mirrors the screen to UxPlay over AirPlay, where a GStreamer pipeline converts the video feed into JPEG image frames. The platform supports any OpenAI-compatible vision API, as well as local inference engines like vLLM, SGLang, and Ollama. Because pixel-based control lacks semantic element IDs or fixed coordinates, the software implements a closed-loop pointer correction mechanism. It wiggles the AssistiveTouch screen pointer, detects its location, executes the tap or drag, and verifies that the interface updated before proceeding.

Standard iOS testing frameworks like XCTest and WebDriverAgent generally require a connected Mac, Xcode installation, continuous code signing, and a custom application on the target device. Free provisioning signatures expire every seven days, creating maintenance overhead. By utilizing native features like AirPlay and USB Human Interface Device inputs, this setup operates without developer accounts, Mac hardware, or jailbreaks, granting AI agents access to apps and games that lack accessibility metadata.

For practitioners, this hardware-assisted design offers a lightweight method to test or automate mobile software using purely visual intelligence. However, certain security restrictions remain. The vision model cannot interact with inputs beyond basic ASCII keyboard entries, nor can it view protected screens such as passcode entries, Apple Pay transactions, or DRM-restricted media content.