What is computer use?
Computer use is a model operating a computer directly by reading the screen and issuing clicks and keystrokes of its own. There is no recorded script and no locator list, so the model interprets the display afresh at each step. It reaches applications that offer no integration and no stable markup, at the cost of a far wider range of possible actions.
Also called: Computer-using agent
What computer use reaches that no other method can
Computer use becomes relevant at the point in an assessment where a system turns out to have no interface and no supported export. Every other execution method assumes a surface that can be addressed, and here there is none. What this method buys is reach. What it costs is a much larger set of actions the model could take, which is why the permitted action list and the capture of every step matter more here than anywhere else.
- Computer use reaches applications that expose no integration of any kind.
- The action space is wide, so constraint comes from policy rather than from code.
- Every click is a decision the model made, which the record has to hold.
- Latency and cost per step are materially higher than an integration call.
How computer use works
- The task is given to the model as an outcome with the permitted applications named.
- A capture of the current display is passed to the model as its input.
- The model proposes a single action, which is checked against the permitted set.
- The action is issued, and the resulting display is captured as evidence.
- Progress is verified against the outcome before the next action is proposed.
- Steps with irreversible effect pause for a named approver.
How computer use is evaluated
Computer use is judged on task completion and on how often it attempts something outside its mandate. Measure the following over a representative task mix:
- task completion rate without human intervention
- actions attempted outside the permitted set
- steps taken against the minimum the task requires
- cost and latency per completed task
- proportion of steps with a captured display in the record
Limitations, risks and common failure modes
- A model misreads a confirmation dialogue and proceeds as though it had cancelled.
- Captured screens hold personal data that no retention policy ever covered.
- Permitted application lists are scoped to the pilot and widened without review.
- An interstitial notice shifts the layout and the model selects the wrong control.