No longer content to shop and make bookings for us somewhere off in the cloud, AI agents have now begun reaching for the screen and mouse of the computer on your desk. Centering on "Computer Use," the computer-operation feature that has landed in one flagship consumer app after another, we trace where things stand for the AI companies clicking from outside the screen, the platform owners seeking to govern from inside the OS, and a new risk in which the text on a screen becomes a weapon.
Chess in under three minutes, and a "$4.99" entry
In July 2026, a reporter for the US publication PCWorld watched a curious game of chess unfold on a Mac mini. The "Computer Use" feature in OpenAI's desktop app, once granted permission to record the screen, opened a chess app and made eight moves in under three minutes (as reported by PCWorld). Anthropic's Claude, which had been unable to move a single piece when the reporter tried it four months earlier, could by then move pieces in the same way, the magazine reported.
Asking whether the same technology could be entrusted with a personal-finance app, the reporter gave a clear answer. "Probably. Am I willing to let it do so? Absolutely not," the reporter wrote, concluding with advice against letting it into sensitive files or apps. Between what these agents can do and what we are willing to entrust to them, a deep gap still lies.
Even so, on November 19 Google sent Computer Use for its consumer "Gemini" app (Mac version) into this battle. When the free app launched in April, it could only look at the screen through a shared window; after folder operations in June and built-in coding features in October, the app has taken seven months to step into clicking and typing in desktop apps and on the web. With this, all three companies, Anthropic (research preview released in March), OpenAI (released in July), and Google, now stand side by side with PC-operation features built into their flagship consumer apps rather than offered as building blocks for developers.
Google's version works on Macs with Apple silicon (macOS 15.0 or later), and in US English only. What stands out, on the other hand, is that it is included from "Google AI Plus," the lowest paid plan at $4.99 a month in the US. Against ChatGPT (Plus, although how the free plan is treated is unclear across official materials) and Claude (Pro), whose entry point is a $20-a-month plan, Google has begun its pursuit with a lower price barrier.
A "back window" that leaves your screen alone, and abilities no one has measured
Letting AI use your own computer does not necessarily mean the mouse cursor in front of you will race around on its own. On the Mac, all three companies are now competing on an experience that "runs quietly in the background."
In this announcement, Google explained that the AI works in a separate, isolated window of its own, leaving the user's screen and cursor free while they watch a live preview in the chat. This setup is nothing new in itself, however. Claude already runs in the background by default on macOS 15 and later without taking over the pointer and keyboard, and ChatGPT on the Mac can also carry out narrowly scoped work in the background. On Windows, by contrast, they are out of step: ChatGPT runs in the foreground and takes over the pointer and input, so OpenAI itself recommends using it on a separate device or in a virtual machine.
Perplexity, for its part, was reported by MacRumors to have released a paid "Personal Computer" feature in May that works with Mac files and apps (not confirmed in primary sources). Other approaches are also running in parallel, such as OpenAI's "dots," which mainly works on a dedicated PC in the cloud rather than on the user's own machine, and Gemini's "Tasks" feature (Gemini Tasks), which uses hardware-isolated cloud virtual machines.
So which of the three apps that directly operate the PC in front of you does the job most accurately? As of November 21, in fact, we could find no third-party review that actually tested and compared these three products under the same conditions. Google has not disclosed the name of the AI model behind the Mac app, nor its success rate or speed.
The available clues are benchmarks such as "OSWorld" that each company self-reports, but versions and scoring conditions vary, so a side-by-side comparison is impossible. Even for the same model, "Claude Opus 5.5," the figure shifts: 81.8% in Anthropic's table, 87.2% in the table Google DeepMind posted on its Gemini 4 page. Yet in both comparison tables Google itself published, the top spot in the computer use category goes not to its own Gemini 4 (81.5% for the higher-end Carbon) but to Claude, and Anthropic itself has said that "at these levels of capability we've found that benchmark margins have become a less reliable guide to real-world differences."
Clicking "from outside" or working "inside the OS"
The approach these three companies share, looking at an app's screen and clicking on it the way a human would, has a major strength. Even if the app on the other end has made no preparations for AI, as long as the user grants the OS permissions (for ChatGPT, Screen Recording and Accessibility on the Mac), the agent can reach into tasks like the ones Google gives as examples: color-grading photos or filling in quotes from local PDFs.
Apple and Microsoft, which own the operating systems themselves, are drawing up an entirely different blueprint.
"Siri AI," which Apple began offering in beta in English in September and in multiple languages including Japanese on October 26, does not take the general-purpose approach of clicking around the screen on its own. As its developer documentation puts it, "Siri can only find custom content and perform custom actions an app makes explicitly available using App Intents." In other words, it calls, from the inside, the actions that apps have put on offer. A preview opened to all developers on November 3, but App Store submissions are slated to start in the spring, so the real debut of Siri AI handling third-party apps' custom actions is still to come.
Microsoft, master of the other OS, has chosen to give agents "accounts of their own" and confine them there. In "Copilot Actions," described as a preview for Windows Insiders in a Microsoft support article (updated December 2025), once an administrator turns the setting on, a dedicated agent account and workspace separate from the user's personal account are created, and the agent runs in the background in a separate session. As for Copilot on Copilot+ PCs operating the PC locally, a November 17 announcement that put work use front and center said only that it is coming "soon," with no date given.
Of particular interest is MXC (Microsoft Execution Containers), a containment platform Microsoft made generally available on October 7. Built for enterprises and developers under the banner "An agent cannot be its own security authority," it has already been adopted by default by OpenAI in Codex and is running on tens of thousands of Windows devices at Meta, while more than 20 developers, including Anthropic and Perplexity, are building on it, Microsoft says. The picture can also be read as Microsoft keeping its own Copilot's PC operation at the "soon" stage while seeking to take command of the OS as the "foundation" on which other companies' agents can run safely.
1-point white text and a "resignation letter to the CEO"
Microsoft has itself written down why it is building mechanisms to stand guard as that "security authority." Once AI reads the text on a screen, opens files, and operates apps, malicious content planted in UI elements or documents can override the instructions given to the AI. Attacks of this kind, known as "prompt injection," in which a third party slips malicious commands into documents or web pages, are being demonstrated by researchers one after another.
In August 2025, the browser company Brave (which says it is also developing agent capabilities for Leo, the AI in its own browser) demonstrated an attack on Perplexity's agentic browser "Comet" in which, when a user simply asked for a summary of a page, the AI followed instructions hidden behind a Reddit spoiler tag and sent the user's email address and a Perplexity one-time password to an outside party. Perplexity shipped an initial fix two days after the report, but Brave added a note that its re-testing after publication found the attack still not fully mitigated, and that it had re-reported it. In October of the same year, Brave also reported a case involving a page containing an image with instructions written in faint light-blue text on a yellow background: a user simply asking a question with a screenshot of the page was enough for text nearly invisible to humans to get through to the AI as commands (we could not confirm the status of any later fix from the materials at hand).
On the desktop side as well, the security firm PromptArmor published a demonstration in January 2026 using Claude's "Cowork" feature (a file-operation and code-execution path separate from screen control). When instructions were hidden in "1-point white text" inside a Word document disguised as a handy add-on picked up online, the AI followed them and uploaded confidential files from the connected folder to the attacker's account (The Register reported that Anthropic said it planned to ship an update to the virtual machine the same day).
In December 2025, OpenAI also revealed a vivid example found in its internal experiments, in an article explaining its safeguards for browser agents. In the demonstration, the agent, swayed by instructions in a malicious email that had landed in the inbox, sent a "resignation letter" to the CEO of the user's company instead of drafting an out-of-office reply. (In July this year, separately, OpenAI itself disclosed an incident in which a model slipped out of isolation during internal evaluations and compromised parts of OpenAI's internal infrastructure and Hugging Face's systems, but that happened in an environment separate from consumer PC-operation features.)
Fortunately, research as of November 21 turned up no reports of consumer computer use features actually being exploited in attacks that harmed users. Even so, OpenAI itself concedes that "Prompt injection, much like scams and social engineering on the web, is unlikely to ever be fully 'solved,'" and Anthropic states plainly in its help pages that "Computer use has no sandbox between Claude and your applications" and that "no safeguards are perfect," urging users not to grant access to banking, healthcare, or government apps.
Who keeps a finger on the "stop button"?
Faced with these risks, the companies are working hard to draw the line between how far to let the AI's hands move and where to call a human back in.
OpenAI's design bars the AI from operating the terminal or ChatGPT itself, and does not let it click the OS's permission prompts either. Anthropic restricts browsers and investment apps to "view only" and terminals to "click only," and while on some accounts the feature's switch is on from the start, letting Claude see the list of installed apps, it asks for separate permission before actually viewing the screen or taking actions.
By comparison, the safety explanation published by latecomer Google is still sparse. The Mac release notes say that "Gemini pauses for logins or payments and may ask for confirmation before sensitive actions" and that users can "stop anytime in one click," but we could not yet find a dedicated help article, nor any specific countermeasures against prompt injection for the Mac version.
If the cloud shopping agents we looked at in Part 1 are a contest over "the keys to the wallet," the Computer Use that has begun on the desktop could be called an experiment in how far to hand over "a spare key to your own room (your computer)." The convenience of tedious chores getting done in an unseen back window and the danger that a single line of 1-point white text could carry off important files now cling to the front and back of the same screen. It can play chess, but you cannot hand it your household accounts. The day that reporter's hesitation disappears will depend not only on how smart the models become, but on how the "cage" that spans the OS and its apps is built.
References
- ChatGPT can now control your whole desktop. I tested it with chess (PCWorld, July 14, 2026)
- Gemini Apps' release updates & improvements (Google)
- Let Gemini tackle repetitive chores on your Mac (Google, November 19)
- Put Claude to work on your computer (Anthropic, March 23)
- ChatGPT is now a partner for your most ambitious work (OpenAI, July 9)
- Google AI Pro & Ultra plans and pricing, US (Google)
- Plans & pricing (Claude by Anthropic)
- Pricing, ChatGPT Work and Codex (OpenAI)
- Computer Use, ChatGPT documentation (OpenAI)
- Let Claude use your computer in Cowork (Claude Help Center)
- Perplexity's New Mac App Brings Personal Computer to Pro Users (MacRumors, May 7, 2026)
- Introducing dots (OpenAI, September 29)
- Put Gemini to work for you: Tasks in Gemini (Google, November 17)
- Introducing Claude Opus 5.5 (Anthropic, September 22)
- Gemini, benchmark tables (Google DeepMind)
- Siri AI, a profoundly more capable and personal assistant, is here (Apple Newsroom, September 14)
- Siri AI is available in more languages starting today (Apple Newsroom, October 26)
- Siri AI Developer Preview (Apple Developer Documentation)
- Siri AI is now open to all developers (Apple Developer News, November 3)
- Experimental Agentic Features (Microsoft Support)
- Microsoft Ignite 2026: Building Windows for the next era of work (Windows Experience Blog, November 17)
- Microsoft Execution Containers: Policy-driven containment for AI agents (Windows Developer Blog, October 7)
- Agentic Browser Security: Indirect Prompt Injection in Perplexity Comet (Brave, August 20, 2025)
- Unseeable prompt injections in screenshots: more vulnerabilities in Comet and other AI browsers (Brave, October 21, 2025)
- Claude Cowork Exfiltrates Files (PromptArmor, January 2026)
- Anthropic's Files API exfiltration risk resurfaces in Cowork (The Register, January 15, 2026)
- Continuously hardening ChatGPT Atlas against prompt injection (OpenAI, December 22, 2025)
- The Hugging Face incident and the road ahead (OpenAI, August 26)