Claude Computer Use Guide 2025: Let Claude Control Your Computer

What Is Claude Computer Use?

Claude Computer Use is a beta capability released by Anthropic that lets Claude perceive a computer screen via screenshots and take actions — clicking, typing, scrolling, pressing keys, running shell commands — just like a human would. It is not a browser extension or an RPA recorder; it is a vision-language model reasoning about pixel-level UI state and issuing low-level input events.

Computer Use is available on three models:

• claude-3-5-sonnet-20241022 — original release, strong general performance
• claude-3-7-sonnet-20250219 — improved UI understanding and multi-step reasoning
• claude-sonnet-4 — best overall capability for complex automation tasks

To enable it you must pass the computer-use-2024-10-22 beta header with every API request. The feature is in public beta — Anthropic considers it experimental and recommends running it inside a sandboxed environment.

pip install anthropic

Available Tools

Computer Use exposes three built-in tool types. You declare them in your API request and Claude decides which to call at each step:

• computer — The primary interaction tool. Actions: screenshot (capture the screen), left_click / right_click / double_click at (x, y) coordinates, type text, scroll, key (press keyboard shortcuts), mouse_move.
• text_editor — View, create, and edit files on disk. Actions: view, create, str_replace, insert. Lets Claude write code files, edit configs, or update documents without going through the UI.
• bash — Run arbitrary shell commands. Claude uses this for installing packages, running scripts, checking process output, or any task better done from the terminal than the GUI.

tools = [
    {
        "type": "computer_20241022",
        "name": "computer",
        "display_width_px": 1280,
        "display_height_px": 800,
        "display_number": 1,
    },
    {
        "type": "text_editor_20241022",
        "name": "str_replace_editor",
    },
    {
        "type": "bash_20241022",
        "name": "bash",
    },
]

Python Quickstart

The minimal Computer Use pattern: send a task, receive a ToolUseBlock, execute the action, return the result, and repeat until Claude stops calling tools.

import anthropic
import base64
from PIL import ImageGrab  # pip install pillow

client = anthropic.Anthropic()

def take_screenshot() -> str:
    """Capture screen and return base64-encoded PNG."""
    img = ImageGrab.grab()
    img = img.resize((1280, 800))
    import io
    buf = io.BytesIO()
    img.save(buf, format="PNG")
    return base64.standard_b64encode(buf.getvalue()).decode()

# Initial request
response = client.beta.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=4096,
    tools=[
        {
            "type": "computer_20241022",
            "name": "computer",
            "display_width_px": 1280,
            "display_height_px": 800,
            "display_number": 1,
        }
    ],
    messages=[
        {
            "role": "user",
            "content": "Open a terminal and print the current date.",
        }
    ],
    betas=["computer-use-2024-10-22"],
)

print(response.content)

Set ANTHROPIC_API_KEY in your environment before running. Never hard-code your key.

The Agent Loop

Computer Use works through a feedback loop: Claude takes a screenshot, decides on an action, you execute that action, take another screenshot, and feed it back. Repeat until Claude returns a final text answer with no tool calls. This loop is the core of every Computer Use agent:

import anthropic, base64, io
from PIL import ImageGrab

client = anthropic.Anthropic()
messages = [{"role": "user", "content": "Go to https://example.com and copy the page title."}]

while True:
    response = client.beta.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=4096,
        tools=[{"type": "computer_20241022", "name": "computer",
                "display_width_px": 1280, "display_height_px": 800}],
        messages=messages,
        betas=["computer-use-2024-10-22"],
    )

    # Check stop condition
    if response.stop_reason == "end_turn":
        print("Done:", response.content[-1].text)
        break

    # Process tool calls
    tool_results = []
    for block in response.content:
        if block.type == "tool_use" and block.name == "computer":
            action = block.input["action"]

            if action == "screenshot":
                img = ImageGrab.grab().resize((1280, 800))
                buf = io.BytesIO()
                img.save(buf, format="PNG")
                screenshot_b64 = base64.standard_b64encode(buf.getvalue()).decode()
                tool_results.append({
                    "type": "tool_result",
                    "tool_use_id": block.id,
                    "content": [{"type": "image",
                                 "source": {"type": "base64",
                                            "media_type": "image/png",
                                            "data": screenshot_b64}}],
                })
            else:
                # Execute click/type/key/scroll here via xdotool or pyautogui
                execute_action(action, block.input)
                tool_results.append({
                    "type": "tool_result",
                    "tool_use_id": block.id,
                    "content": "Action executed.",
                })

    # Append assistant turn + tool results and continue
    messages.append({"role": "assistant", "content": response.content})
    messages.append({"role": "user", "content": tool_results})

Always append both the assistant's full response and the tool results before the next API call — the model needs the full history to reason correctly.

Docker Reference Implementation

Anthropic publishes an official reference Docker image at ghcr.io/anthropics/anthropic-quickstarts. It bundles a full Ubuntu desktop, Xvfb virtual display, VNC server, xdotool for mouse/keyboard control, and a Playwright-managed Chromium browser. Running it is the fastest way to get a safe, reproducible sandbox:

# Clone the quickstarts repo
git clone https://github.com/anthropics/anthropic-quickstarts.git
cd anthropic-quickstarts/computer-use-demo

# Build and run (exposes VNC on 5900, web UI on 8080)
docker build -t computer-use-demo .
docker run \
  -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
  -p 5900:5900 \
  -p 8080:8080 \
  computer-use-demo

# Open http://localhost:8080 in your browser to watch Claude work

The container has no access to your host filesystem, browser sessions, or credentials by default. Anthropic recommends this Docker setup as the baseline for all Computer Use experiments — never run it directly on your host machine with real account access.

Xvfb        # virtual framebuffer display (no physical monitor needed)
xdotool     # programmatic mouse / keyboard events
x11vnc      # VNC server so you can watch over the network
Playwright  # manages a Chromium browser instance
noVNC       # web-based VNC viewer at localhost:8080

Security Considerations

Computer Use gives Claude broad control over an environment. These risks are real and must be mitigated before deploying anything beyond local experiments:

• Never give Claude access to real accounts or passwords. The sandbox should contain only throwaway credentials. Assume anything in the environment can be read, sent, or deleted.
• Use an isolated environment. Run inside Docker or a dedicated VM with no network access to your internal systems. Mount only the files Claude absolutely needs.
• Prompt injection. Malicious content on a web page (hidden text, image watermarks, content in iframes) can instruct Claude to take unintended actions — exfiltrate data, click "confirm", fill in forms. Always review what pages Claude visits.
• Require human confirmation for irreversible actions. Pause the agent loop before actions like sending emails, submitting forms, making purchases, or deleting files. Ask the user to approve.
• Restrict permissions. Run the Docker container as a non-root user. Disable clipboard integration with the host. Use network namespaces to block access to internal services.

docker run \
  --network=none \          # no internet access
  --read-only \             # read-only root filesystem
  --cap-drop=ALL \          # drop all Linux capabilities
  --security-opt no-new-privileges \
  -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
  computer-use-demo

Supported Models

Not all Claude models support Computer Use. The capability is tied to specific model versions trained with computer interaction data:

All three require the betas=["computer-use-2024-10-22"] parameter. Haiku models do not currently support Computer Use.

Use Cases

Computer Use shines in scenarios where traditional automation is too brittle or too expensive to maintain:

• Automated browser testing. Navigate an app as a real user would — no selectors to maintain, no flakiness from CSS class renames. Claude adapts to UI changes automatically.
• Web scraping. Handle JavaScript-heavy SPAs, infinite scroll, login-walled content, and CAPTCHA-adjacent flows that headless browsers struggle with.
• Robotic Process Automation (RPA). Automate repetitive workflows inside legacy desktop apps (ERP systems, old HR portals) with no API — Claude reads the screen and drives the UI.
• UI testing. Describe the expected behavior in plain English; Claude explores the UI and reports deviations — no test script to write.
• Form filling and data entry. Extract structured data from documents and enter it into web forms or desktop GUIs automatically.
• Cross-application workflows. Copy data from a PDF, paste into a spreadsheet, summarize in a Slack message — tasks that span multiple apps with no shared API.

Computer Use vs Playwright / Selenium

Both approaches automate a browser, but they solve different problems. Here is how to choose:

A practical pattern: use Playwright for your core regression suite and Computer Use for exploratory or cross-app automation where writing deterministic scripts is impractical. See also: LangChain guide · Anthropic API guide