computer use agents on a mac

2026 was the year the assistants got hands.

claude computer use arrived on the mac through cowork on march 23, 2026, for pro and max subscribers, macos only at that point with windows and linux unsupported. openai shipped desktop computer use inside codex on april 16, 2026, also mac first, with parallel agent sessions, having started with openai operator for browser tasks in early 2025. google's gemini computer use grew out of project mariner and leans toward the browser, where the dom gives it more to work with than raw pixels.

so an ai that controls your computer stopped being a research demo and became a subscription feature, in about 14 months. here is what a computer use agent actually does, what it needs from your mac, and the two places it reliably falls over.

what a computer use agent actually does

it takes a screenshot, decides on an action, performs it, and looks again.

that is the entire loop, and its crudeness is the point. rather than integrating with each app's api, the agent operates the way you do: it looks at pixels and moves a pointer. that makes it universal, because it works with any app including ones with no api at all, and it makes it brittle, because a moved button is an unrecoverable surprise.

the best implementations mix these. pixels for the general case, the accessibility tree where it exists because it is faster and far more precise, the dom in a browser. an agent that only screenshots is doing the job on hard mode.

the permissions it needs, and what they mean

two macos permissions, both powerful, both granted per app.

screen recording. so it can see. this is the same permission a screen memory tool needs, and it means the app can read anything visible, including whatever is behind the window you are working in.

accessibility. so it can move the pointer and type. accessibility permissions are the bigger of the two, and it is worth being blunt about what they grant: an app with accessibility permission can drive every other app you have open. it can click through your email, your bank, your password manager if it is unlocked.

that is not a flaw in the permission, it is what the permission is for, and it is the same mechanism screen readers have used for decades. but people grant it in a dialog they read in two seconds, so it deserves saying plainly. check system settings, privacy and security, accessibility, and see what is already on that list.

where they break, part one: arithmetic

reliability compounds badly, and this is what demos hide.

a step that works 95% of the time is genuinely good. run ten in sequence and the chain succeeds about 60% of the time. run twenty and it is roughly 36%. that maths is why the useful pattern in 2026 is short supervised runs rather than long autonomous ones, and it is not a model quality problem that a better model quietly solves.

so the tasks that actually work share a shape:

1. repetitive, so the ui is familiar and stable.
2. verifiable, so success or failure is obvious at a glance.
3. short, ideally under ten steps.
4. reversible, or at least cheap to redo.
5. boring, because interesting tasks have edge cases.

filing expenses, renaming and sorting, pulling numbers out of a dashboard into a sheet, clicking through a form flow you have done a hundred times. those work. "handle my inbox" does not, yet.

this is also why classic desktop automation did not disappear. a shortcut or an applescript that does the same thing every time is 100% reliable and costs nothing per run. reach for an agent when the task varies enough that a script cannot express it, not because it is newer.

where they break, part two: an agent that reads can be told what to do

prompt injection with a mouse is a different category of problem.

an agent cannot reliably distinguish your instructions from instructions hidden in content it reads, because both arrive as text in the same context. a chatbot that falls for this says something wrong. an agent with pointer control does something wrong, in an app where you are already signed in.

the numbers are not comforting. the cloud security alliance's 2026 research note found that across roughly 2 to 3 billion crawled pages a month, the share carrying malicious injection grew 32% in relative terms between november 2025 and february 2026. benchmark work has measured strong models in tool calling loops vulnerable in the 24 to 47% range. a university of washington study published july 3, 2026 found 4 of 7 agentic browsers allowed same origin policy bypass.

practical rules that follow, none of them exotic:

• do not leave it unsupervised on the open web while you are signed into anything that matters.
• use a separate browser profile with no banking, no admin, no password manager.
• watch the run. these agents are slow enough to watch, which is a feature.
• never let it act on content it just read from a stranger without you in the loop.

the full version of this is in what is prompt injection, and the browser specific case in ai browsers and what they do with your history.

the thing they still cannot do

a computer use agent can operate your mac and has no idea what you did on it last month.

this is the gap that keeps surprising people. the agent sees the current screen. it does not know which vendor you ruled out in july, what the number on that slide was, or why the config looks like that. so it asks, and you explain, every session.

per the anthropic economic index for may 2026, the number one work task in sampled ai conversations is searching electronic sources for information at 4.95%, with reference searching second at 3.74%. retrieval dominates. now look at what stays small.

searching your own knowledge is 3.61% and anything about a meeting is 0.26%. hands do not fix that. an agent with a mouse and no memory is a very capable new hire on their first morning, every morning.

pairing hands with a memory

the archive is the part that persists between sessions, and it is a separate job from driving the ui.

remynd does the memory half on a mac. it captures the focused window, runs ocr locally so what was on screen becomes searchable text, and keeps the index on your machine. since august 2026 it exposes that archive to claude code and codex over a local connector, which is the setup in how to give an ai agent context.

the scope, stated precisely: capture, ocr, call transcription and storage stay on your mac by default, you can exclude specific apps or sites from capture entirely, and your history is read only everywhere in the app. asking a question sends the retrieved slices to a cloud model, and there is no local model option today. worth adding, since this post is about agents with permissions: remynd's in app chat does ship general purpose tools, so the caution above applies to it as much as to anything else. more in private ai on your mac.

how to try one without regretting it

five minutes of setup saves a bad afternoon.

start on a task you would happily do twice. give it a separate browser profile. keep the first runs under ten steps. watch the whole thing at least three times before you look away. and check what already holds accessibility permission on your mac, because the answer is usually more apps than you remember granting.

computer use agents are genuinely useful and genuinely early. treat the capability list as a risk list, which is the same advice as for ai agents generally, and the tool stops being scary and starts being handy.

download remynd for mac and give the hands a memory to work from.