Record a desktop task once. Replay it forever.
LearnBot watches you perform a task on a Linux desktop (via VNC), uses Claude to generalize the recording into a reusable task definition, and replays it autonomously using a multi-layer element location engine. It's desktop RPA that learns from demonstration instead of manual scripting.
# On EC2 (Ubuntu + XFCE + VNC)
cd learnbot && uv sync
uv run learnbot setup # configure DISPLAY, AT-SPI, API key
uv run learnbot record -y # perform your task, Ctrl+C to stop
uv run learnbot list # see generated tasks
uv run learnbot play "My Task" # replay itRecord (watch user) --> Analyze (Claude) --> Replay (execute autonomously)
| | |
Screenshots TaskDefinition Multi-layer
+ AT-SPI data with steps element location
+ Input events (role, name, bbox) + sanity checks
1. Record — Captures screenshots, accessibility tree snapshots (AT-SPI), and input events as you perform a task. Uses a producer-consumer architecture: AT-SPI listeners on the main thread feed events to a recording loop that streams frames to disk.
2. Analyze — Sends interleaved screenshots and event data to Claude Sonnet 4.5, which extracts the clean execution path: filtering noise (false starts, retries, exploratory navigation) while preserving every prerequisite step needed to replay from a clean desktop. Prefers keyboard shortcuts (Ctrl+L, Ctrl+V) over visual clicks for reliability.
3. Replay — Executes each step using a 7-layer element location fallback chain, with mandatory sanity checks that detect failures even when --no-verify is set.
- Ubuntu 22.04+ with XFCE desktop
- TigerVNC or KasmVNC for remote access
- Python 3.11+
- uv package manager
- An Anthropic API key (for Claude)
# System dependencies
sudo apt install python3-pyatspi xdotool at-spi2-core
# Clone and install
git clone https://github.com/Prakshal-Jain/learnbot.git
cd learnbot
uv sync
# Interactive setup (configures DISPLAY, DBUS, AT-SPI, API key)
uv run learnbot setuplearnbot setup handles:
- Auto-detecting DISPLAY from
/tmp/.X11-unix/(prefers:1for VNC) - Inheriting DBUS and AT-SPI session vars from
xfce4-session - Setting
GTK_MODULES=gail:atk-bridge(mandatory for GTK app accessibility) - Prompting for your
ANTHROPIC_API_KEY - Symlinking system
pyatspiandgipackages into the venv (no PyPI wheel exists) - Writing all vars to
~/.learnbot/env.sh(auto-loaded on every CLI invocation)
After setup, restart any running GTK apps so they pick up the accessibility bridge:
thunar --quit && GTK_MODULES=gail:atk-bridge thunar &cd learnbot
uv sync --group dev
# Tests mock all system deps — works on macOS/Linux without VNC
uv run python -m pytest tests/ -q --ignore=tests/test_ai_analyze.pyInteractive environment configuration. Must be run once on a fresh EC2 instance before any other command.
uv run learnbot setupValidates: xdotool installed, pyatspi importable, AT-SPI daemon running, screenshots capturable, storage directory writable.
Record a desktop task. Start recording, perform the task in VNC, then press Ctrl+C to stop.
uv run learnbot record # interactive mode (prompts for description)
uv run learnbot record -y # non-interactive (auto-analyze, skip prompts)| Flag | Description |
|---|---|
-y |
Skip all interactive prompts. Auto-analyze after recording if API key is set. |
What gets captured per frame:
- Full-screen and window-cropped screenshots (PNG)
- AT-SPI focused element + siblings (role, name, bounding box)
- Input event (click coordinates, key pressed, scroll delta)
- Window title and geometry
- All coordinates stored as window-relative (screen-absolute minus window origin)
Keyboard handling:
- Printable characters accumulate in a buffer
- Buffer flushes after 300ms of silence or on a modifier key (Ctrl, Alt, Super)
- Shift+char = uppercase letter (not a hotkey)
- Modifier combos (Ctrl+S, Alt+F4) are captured as hotkey events
Recordings stream to ~/.learnbot/recordings/<session-id>/ as individual frame JSON files — no in-memory accumulation, safe against crashes.
Re-analyze an existing recording with Claude. Useful after prompt improvements or to re-generate a task definition.
uv run learnbot analyze <session-id>
uv run learnbot analyze <session-id> -y # skip review prompt| Flag | Description |
|---|---|
-y |
Auto-save without prompting for review |
For complex recordings (>10 events or multiple windows), prompts for a natural-language description and success criteria to give Claude better context.
Pipeline:
- Load frames from session directory
- Split at window-change boundaries into chunks
- Cap at 5 chunks (merge if rapid window switching)
- Send each chunk to Claude with screenshots + serialized events
- Consolidate per-chunk steps into a single TaskDefinition
- Validate against Pydantic schema (auto-retry on parse failure)
Display all saved tasks in a table.
uv run learnbot listShows: task name, creation date, last run date, run count, number of steps.
Replay a task step by step with a live Rich progress table.
uv run learnbot play "My Task" # by name
uv run learnbot play abc123-def456 # by task ID
uv run learnbot play "My Task" --no-verify # skip Claude verification
uv run learnbot play "My Task" --timeout 60 # 60s per step
uv run learnbot play "My Task" --confidence 0.9 # stricter image matching| Flag | Default | Description |
|---|---|---|
--no-verify |
off | Skip Claude Haiku verification and AT-SPI state polling. Mandatory sanity checks (screenshot diff, window title) still run. |
--timeout |
30 | Per-step timeout in seconds, including retries |
--confidence |
0.85 | Minimum confidence threshold for OpenCV image template matching |
--no-evolve |
off | (Planned) Disable task evolution — replay exactly as defined, no script graduation |
Exit codes: 0 = all steps passed, 1 = any step failed.
What happens per step:
- Take pre-execution screenshot
- Execute the step (locate element via fallback chain + inject input)
- Run mandatory sanity checks (screenshot diff for clicks, window title for switches)
- If sanity fails: retry up to 2 times, then mark step FAILED
- If
--verify(default): additionally run AT-SPI state polling + Claude Haiku screenshot check - Record step outcome (passed/failed, location method used, details)
AT-SPI GLib Main Loop (main thread)
|
InputListener (producer)
- Mouse clicks (pyatspi mouse:button)
- Keyboard events (pyatspi key events)
- Scroll events
- Window change detection (polling)
|
queue.Queue
|
RecordingLoop (consumer thread)
- Screenshot capture (mss)
- AT-SPI snapshot (focused + siblings)
- Window geometry (xdotool)
- Stream frame JSON to disk
AT-SPI runs on the main thread because GLib's event loop requires it. The recording loop runs on a dedicated thread, consuming events from a queue and writing frames to disk immediately.
VNC keyboard events are captured through AT-SPI listeners, not pynput's X RECORD extension (which doesn't work over VNC).
Claude receives interleaved screenshots and event text for each frame:
[Screenshot PNG — base64]
Frame 1: Action: click at (245, 130) button=left | target_role=push button | target_name=Save | window=document.txt - Mousepad
[Screenshot PNG — base64]
Frame 2: Action: keydown key=ctrl+s | window=document.txt - Mousepad
The system prompt instructs Claude to:
- Extract the clean path — filter mistakes, retries, and redundant actions
- Preserve every prerequisite — URL navigation, file opening, search queries. The test: "if you removed this step and replayed from scratch on a clean desktop, would later steps fail?"
- Prefer AT-SPI data — use
target_role+target_nameover coordinates for resilient targeting - Prefer keyboard shortcuts — Ctrl+L for address bar, Ctrl+V for paste, Ctrl+S for save
- Include every window switch — each app transition must be an explicit
window_changestep with the application name in the target
For multi-window recordings, frames are split into chunks at window boundaries and analyzed separately, then consolidated into a single task.
The replay engine receives the TaskDefinition produced by analysis, plus access to the original recording for template extraction:
From the TaskDefinition (Claude's output):
| Field | Used By | Purpose |
|---|---|---|
input_type |
Runner | Determines action type (click, type, key, window_change, etc.) |
input_value |
Executor | Text to type, key to press, "right" for right-click |
target_role |
AT-SPI, Playwright | Semantic element identity (e.g., "push button", "menu item") |
target_name |
AT-SPI, Playwright, OCR | Element name (e.g., "Save", "Copy link address") |
target_bbox |
Coordinate fallback | Window-relative [x, y, w, h] — last resort position |
expected_state |
Verifier | AT-SPI state to verify after action (e.g., "focused") |
action |
Logging, UI | Human-readable step description |
From the original recording (via source_session_id):
| Data | Used By | Purpose |
|---|---|---|
| Screenshots | Image template layer | Cropped at target_bbox to create per-step template PNGs for OpenCV matching |
The analysis steps are the primary input. The recording screenshots are only used for image template extraction — the replay engine does not re-read the raw events or frame JSON files.
When replaying a step that requires finding a UI element, the executor tries each layer in order:
| # | Layer | Confidence | How It Works | Best For |
|---|---|---|---|---|
| 1 | Playwright (CDP) | High | Connects to Chrome DevTools Protocol, queries DOM by ARIA role + name | Chrome in-page elements (links, buttons, inputs) |
| 2 | AT-SPI | High | Walks the desktop accessibility tree, matches by role + name | GTK/Qt native app elements (menus, buttons, text fields) |
| 3 | Computer Use (Claude) | Medium | Sends screenshot to Claude's Computer Use API, extracts click coordinates | Any visible element when AT-SPI/Playwright fail |
| 4 | EasyOCR | Medium | Runs text detection on screenshot, matches target_name against detected text | Text labels without accessibility info |
| 5 | Image Template (OpenCV) | Low | cv2.matchTemplate with TM_CCOEFF_NORMED against pre-recorded element crops |
Visually stable icons/buttons across runs |
| 6 | Coordinate | Low | Window-relative bbox from recording + current window origin | Last resort when all semantic methods fail |
For window_change steps, xdotool searches by window name. If the window isn't found, the executor auto-launches the application (Chrome, Terminal, Thunar, Mousepad, Firefox) and waits up to 30s for it to appear.
Location confidence is returned with each result:
- High (playwright, atspi, xdotool): Element found by semantic identity — resilient to layout changes
- Medium (computer_use, ocr): Element found by visual/text analysis — usually correct but not guaranteed
- Low (image, coordinate): Element found by pixel matching or recorded position — fragile
When coordinate fallback is used for a step that has semantic targets (role/name), a warning is logged.
After locating the target element, the executor injects the appropriate input:
| Input Type | Action |
|---|---|
click |
pyautogui.click(x, y) — or rightClick if input_value="right" |
double_click |
pyautogui.doubleClick(x, y) |
type |
Focus target (click or Ctrl+L for address bar) then pyautogui.write(text) |
key |
pyautogui.press(key) or pyautogui.hotkey(*parts) for combos like ctrl+s |
scroll |
pyautogui.scroll(amount) — positive=up, negative=down |
window_change |
xdotool search --name <target> && windowactivate with auto-launch |
Special handling:
- Browser address bar: Ctrl+L instead of clicking (position varies with window size)
- Text editors without a located target: click center-bottom of active window to ensure focus
- XFCE submenu headers (role="menu"): hover instead of click (submenus open on hover in GTK)
- Right-click context menus: coordinate fallback still enabled as last resort for menu items
Step Executed
|
┌──────────┴──────────┐
│ MANDATORY (always) │
│ │
│ click/double_click: │
│ screenshot diff │
│ (similarity check)│
│ │
│ window_change: │
│ title match │
│ (xdotool verify) │
└──────────┬──────────┘
│
Sanity OK?
/ \
No Yes
| |
Retry ┌───┴───────────┐
(up to 2x) │ --verify only │
│ │
│ AT-SPI state │
│ polling │
│ │
│ Claude Haiku │
│ screenshot │
│ check │
└───┬────────────┘
│
Step Passed
Mandatory sanity checks run on every step, regardless of --no-verify:
- Screenshot diff (click/double_click): Compares pre and post screenshots using OpenCV connected component analysis on the grayscale diff. Small noise (cursor blink, clock updates) is filtered by requiring changed regions to be at least 500px in area. Similarity thresholds are tiered by location confidence: 0.999 for high-confidence locations (AT-SPI, Playwright), 0.998 for low-confidence (coordinate fallback). If similarity exceeds the threshold, the click had no visible effect — retry up to 2 times, then mark the step as FAILED.
- Window title verification (window_change): Runs
xdotool getactivewindow getwindownameand checks if the expected window name appears in the active title. Handles split parts (e.g., "ubuntu - Thunar" matches "Thunar").
Optional verification (enabled by default, disabled with --no-verify):
- AT-SPI state polling: If the step has an
expected_state(e.g., "focused", "checked"), polls the accessibility tree with a timeout. Used for confirming state transitions after clicks. - Claude Haiku screenshot check: Sends the current screenshot to Claude Haiku and asks whether the step's action appears to have succeeded. Returns pass/fail/retry verdict. On "retry", re-executes the step (up to 2 retries).
| Check | --verify (default) |
--no-verify |
|---|---|---|
| Screenshot diff after click | YES | YES |
| Window title after switch | YES | YES |
| AT-SPI state polling | YES | NO |
| Claude Haiku screenshot | YES | NO |
Over repeated successful runs, LearnBot will detect step sequences that can be replaced with direct script commands and graduate them from UI automation to deterministic execution.
Example: The step sequence "open terminal → type curl https://api.example.com → press Enter" becomes a single subprocess.run(["curl", "https://api.example.com"]) call — faster, cheaper, and more reliable than clicking through the UI.
Scriptable patterns:
- Terminal commands (type + Enter) →
subprocess.run() - URL navigation (address bar + type URL + Enter) →
google-chrome "https://..." - File operations (Thunar navigation) → direct filesystem calls
- Clipboard operations →
xclip/xsel
--no-evolve flag: Disables all task evolution. The task replays exactly as originally defined — no script graduation, no step replacements, no modifications to the TaskDefinition. Use this when deterministic, unchanged replay matters more than optimization.
Evolution history will be tracked in SQLite with rollback capability — graduated steps retain both their UI automation path and script path.
When a window_change step targets an application that isn't running, the executor auto-launches it:
| Keyword in target_name | Launch Command |
|---|---|
terminal |
xfce4-terminal --default-working-directory $HOME |
chrome, google chrome |
google-chrome --no-first-run --force-renderer-accessibility --password-store=basic --remote-debugging-port=9222 |
thunar, file manager |
thunar |
mousepad, text editor |
mousepad |
firefox |
firefox |
chromium |
chromium-browser |
The executor waits up to 30s for browser windows or 15s for other apps, polling via xdotool search. Stale/zombie window IDs are tracked and excluded from matching.
Chrome is launched with --force-renderer-accessibility (required for AT-SPI to see web page elements) and --remote-debugging-port=9222 (required for Playwright CDP connection).
The output of AI analysis — a reusable, replayable task.
TaskDefinition(
task_id="a1b2c3d4-...", # UUID, auto-generated
name="Paste YouTube URL into editor", # short imperative phrase
description="Copy a video link...", # 1-2 sentence summary
source_session_id="e5f6g7h8-...", # recording this came from
success_criteria="...", # user-defined validation text
steps=[...], # ordered TaskStep list
run_count=5, # number of times replayed
last_run_at=1710500000.0, # timestamp of last replay
)A single action in a task.
TaskStep(
step_id="...", # UUID
order=1, # 1-based sequence number
action="Click the Save button", # human-readable description
target_role="push button", # AT-SPI role (semantic identifier)
target_name="Save", # AT-SPI accessible name
target_bbox=(120, 45, 80, 30), # window-relative (x, y, width, height)
input_type="click", # click|double_click|type|key|scroll|window_change
input_value=None, # "right" for right-click, text for type, key name
expected_state="focused", # AT-SPI state to verify after action
verified=False, # whether step has been deterministically verified
copilot_intervention=None, # human annotation (planned)
)input_type values:
| Type | input_value | target needed? | Description |
|---|---|---|---|
click |
None (left) or "right" |
Yes | Single click at target position |
double_click |
None |
Yes | Double click at target position |
type |
Text to type | Optional | Type text (focus target first if available) |
key |
Key name, e.g. "Return", "ctrl+s" |
Optional | Press key or hotkey combo |
scroll |
Amount as string, e.g. "-3" |
Optional | Scroll at target or current position |
window_change |
None |
No (uses target_name) | Switch to or launch application |
The outcome of a replay.
RunResult(
run_id="...", # UUID
task_id="...", # which task was replayed
status="passed", # passed|failed|running
started_at=1710500000.0,
ended_at=1710500012.3,
total_duration=12.3, # seconds
step_outcomes=[
{
"step_id": "...",
"order": 1,
"action": "Activate Chrome",
"status": "passed",
"location_method": "xdotool",
"details": "step completed successfully",
},
{
"step_id": "...",
"order": 2,
"action": "Right-click video thumbnail",
"status": "failed",
"location_method": "coordinate",
"details": "sanity checks failed after retries",
},
],
)A single captured moment during recording.
RecordingFrame(
frame_id="...", # UUID
timestamp=1710500000.123, # capture time
trigger="click", # what triggered this frame
full_screenshot_path="/path/to/full.png",
window_screenshot_path="/path/to/window.png",
window_title="ubuntu - Thunar",
window_geometry=(100, 50, 800, 600), # x, y, width, height
input_event=InputEvent(...), # the user action
a11y_focused=A11yNode(...), # AT-SPI focused element
a11y_siblings=[A11yNode(...)], # sibling elements
)~/.learnbot/
env.sh # Auto-loaded environment variables
db/
learnbot.db # SQLite database
recordings/
<session-id>/
session.json # RecordingSession metadata
frame_0.json ... frame_N.json # Per-frame capture data
screenshot_0.png ... # Full screenshots
window_0.png ... # Window-cropped screenshots
tasks/
<task-id>.json # TaskDefinition (from analysis)
templates/
<task-id>/
<step-id>.png # Cropped element images for template matching
SQLite tables:
recording_sessions— session metadata (ID, name, created_at, frame/event counts)task_definitions— task metadata + JSON file path pointerrun_results— run outcomes with step_outcomes JSON blob
- AT-SPI requires GTK_MODULES: Without
GTK_MODULES=gail:atk-bridge, GTK apps (Thunar, Mousepad) expose no accessibility tree.learnbot setupconfigures this automatically. - Chrome needs special flags:
--force-renderer-accessibilityfor AT-SPI visibility,--remote-debugging-port=9222for Playwright CDP. The auto-launcher includes both. - XFCE submenus open on hover: If AT-SPI matches a "menu" role (not "menu item"), the executor hovers instead of clicking to avoid dismissing the parent menu.
super+dis a toggle: In XFCE, pressing super+d in show-desktop mode restores windows. The executor minimizes all visible windows first to avoid this.- AT-SPI offscreen coordinates: Collapsed submenu items report coords at
-2147483648. Rejected via bounds check before any click. - Auto-launch opens clean state: Apps launch without file arguments (e.g.,
mousepadnotmousepad file.txt). Tasks requiring a specific file must include the file-opening steps. - VNC keyboard capture: AT-SPI listeners capture VNC keyboard input.
pynput's X RECORD extension does not work over VNC. - pyatspi is system-only: No PyPI wheel exists — installed via
aptand symlinked into the venv bylearnbot setup. - Coordinate fallback is fragile: When all semantic locators fail, the executor falls back to recorded coordinates. These break if window position or size changes. The sanity check catches this (no visible effect = retry + fail).
LearnBot uses Claude's Computer Use API as one layer in the fallback chain (Layer 3), but not as the primary execution engine. Here's why:
| LearnBot (AT-SPI + pyautogui) | Pure Computer Use | |
|---|---|---|
| Cost per step | Free (AT-SPI/Playwright) or ~$0.01 (vision fallback) | ~$0.01-0.05 per action |
| Latency per step | Milliseconds (AT-SPI) | 2-5 seconds (API round-trip) |
| Determinism | AT-SPI match = same result every time | May click slightly different spots |
| 11-step task | ~2s total (AT-SPI path) | ~30-60s (API time alone) |
| Offline capable | Yes (AT-SPI/Playwright/OCR/image/coordinate) | No |
The recording gives the system something Computer Use lacks: a concrete step-by-step plan derived from watching a human, with semantic element identifiers (AT-SPI role + name) that survive UI layout changes.
Computer Use is the fallback brain — it handles steps that AT-SPI and Playwright can't locate. The planned self-healing system will use it more deeply, sending failure context to Claude for adaptive recovery.
The architecture (record -> AI analyze -> replay) is straightforward to replicate. The moat develops from:
- Accumulated edge-case knowledge: Months of real-world debugging — keysym mapping, XFCE submenu hover behavior, Chrome AT-SPI quirks, xfdesktop activation, stale window tracking, coordinate overflow prevention. A competitor starts at zero.
- Data flywheel: Every replay failure teaches the system what goes wrong. Recordings + failure modes across diverse desktops and applications make the analysis and self-healing dramatically better over time.
- Robustness compounding: Each bug fix makes the system more reliable. Reliability begets trust begets usage begets more edge cases found and fixed.
- Task evolution: Tasks that start as UI automation gradually evolve into direct script execution — faster, cheaper, and more reliable with each successful run. This optimization loop is hard to replicate without the underlying recording + analysis infrastructure.
- Domain-specific task libraries: Pre-built, tested task definitions for common workflows become a network effect when shared across users.
The defensibility isn't in the architecture — it's in the accumulated knowledge of how desktop automation breaks in the real world, and the data loop that continuously improves it.
# Run all tests (217 tests, ~16s)
cd learnbot
uv run python -m pytest tests/ -q --ignore=tests/test_ai_analyze.py
# Run specific test files
uv run python -m pytest tests/test_runner.py tests/test_executor.py -v
# Deploy to EC2
git subtree push --prefix=learnbot origin main
ssh ubuntu@<EC2-IP> "cd ~/learnbot && git pull && uv sync"Pre-commit hooks enforce ruff-format and ruff linting on every commit.
Core: pydantic, pillow, click, rich, mss, anthropic, pyautogui, opencv-python-headless, python-xlib, pynput
Optional: playwright (Chrome CDP), easyocr (text detection)
Dev: pytest, ruff, pre-commit
src/learnbot/
ai/ # Claude API client, analysis prompts, cost tracking
capture/ # Screenshot (mss), AT-SPI snapshots, window geometry
cli/ # Click CLI: setup, record, analyze, list, play (lazy-loaded)
models/ # Pydantic v2: RecordingSession, TaskDefinition, RunResult
platform/ # AT-SPI availability checks
recording/ # Producer-consumer: InputListener + RecordingLoop
replay/ # Executor (7-layer location), Runner (retry + sanity), Verifier
storage/ # SQLite + filesystem CRUD for recordings, tasks, runs
tests/ # 217 pytest tests (mock all system deps — run anywhere)
scripts/ # E2E test suite, utility scripts