Rather than automating through code at all, computer-use models let an agent see a screenshot and directly output mouse/keyboard actions — the most general but slowest and most expensive way to control a browser or desktop.
Core concepts
- What computer use actually is: The model receives a screenshot, decides on a click/type/scroll action, executes it, and repeats — no DOM access or selectors involved at all, which makes it the most general-purpose but least efficient option.
- Provider models: OpenAI's
computer-use-preview, Anthropic's Claude computer use, and Google's Gemini computer-use-preview are the three current options, each integrable into Stagehand or a custom harness with a few lines of code. OpenAI Computer Use · Anthropic Computer Use · Google Gemini API - When to reach for it vs Stagehand/Browser Use: Computer use is the fallback for interfaces with no accessible DOM (legacy desktop apps, canvas-rendered UIs) — for anything that's a normal web page, a DOM-aware tool is faster and cheaper.
What Is It?
Computer use APIs from OpenAI, Anthropic, Google drive mouse/keyboard from pixels.
No DOM required — works on canvas-heavy or remote desktops.
Security and confirmation flows are critical.
Why Should I Care?
Enables agents on apps with no accessibility tree.
Frontier capability with frontier risk — treat as privileged mode.
Real World Analogy
Computer-use models are remote desktop pilots — they look at the screen like a human and move mouse/keyboard instead of reading HTML source.
Visual Mental Model
screenshot -> VLM -> action (x,y,type)
OpenAI / Anthropic / Google variantsBeginner Explanation
Computer-use models consume screenshots and emit UI actions — powerful for arbitrary apps, expensive and sensitive to resolution and latency.
Smallest Possible Example
# model input: screenshot bytes
# output: mouse/keyboard actions
print("vision-driven UI control")Common Mistakes
❌ Full-screen sensitive data in screenshots sent to models.
Redact windows; scope capture regions.
❌ No kill switch for runaway mouse loops.
Implement max steps and user abort hotkeys.
❌ Assuming pixel-perfect actions across DPI scales.
Test on target display resolutions.
Real Project Example
Anthropic computer use beta targets operator workflows.
Google Gemini demos include UI navigation scenarios.
Build This Mini Project
On a sandbox VM, list safety rules (max steps, allowed apps, redaction) before trying any computer-use API.
Interview Question
DOM automation vs computer use?
DOM/Playwright paths are cheaper and more precise when HTML is available; computer use handles arbitrary pixels but costs more tokens and needs stronger safety controls.
Key Takeaways
- Screenshot-driven control.
- Strong safety and redaction required.
- Set max steps and kill switches.
- Use when DOM is insufficient.
Resources
YouTube learning
Core concepts from video walkthroughs:
The 60-second story
Some software does not have a DOM. It has a picture. A legacy desktop app, a canvas, a remote screen. There is no #submit-btn to worship and no accessibility tree to read politely. Computer use is hiring someone who looks at a screenshot and moves the mouse. They click, type, scroll, look again. It is the most general way to drive a machine and the slowest, because every step is “see a picture, decide, act, see another picture.”
OpenAI, Anthropic, and Google each sell a version of that hire. You wrap them in a harness, sometimes Stagehand, sometimes your own loop. The model does not get a magic backdoor into the app. It gets eyes and a hand. That is why it works on weird UIs and why it misclicks a button that looks like another button.
You reach for this when the DOM-aware tools cannot see the control. For a normal website, a person who can read the HTML is faster and cheaper than a person who must squint at pixels. Computer use is the fallback, not the lifestyle. A canvas-rendered dashboard or a Windows dialog from 2009 is the lifestyle it was built for.
Map the desk.
- A screenshot is the eyes.
- A click or keypress is the hand.
- No selector means no nickname to rot, and no nickname to trust.
- Provider models are different hires with the same job.
- DOM tools win on ordinary web pages.
- Pixel control wins when the page is really a picture.
see screenshot
choose click(x, y) or type(text)
execute
see againEach loop is a round trip through a vision model. Chain them and you feel it in latency and in the invoice. Add asserts a human would accept: the title changed, the file appeared, the dialog closed. “The cursor moved” is not success.
This exists because the long tail of software is not clean HTML. The engineering limit is that vision plus action is a lossy remote control. Use it at the edge where structure is missing. The moment structure exists, step back down to it.
Mental model: eyes and a mouse, not a map of the page. General, slow, easy to fool with two similar buttons. Perfect for a picture. Wasteful for a form you could have queried.
You can explain the page if you know when you would refuse this approach, not only when you would brag about it.
