Playwright AI Agent Guide

Playwright is Microsoft’s browser automation library, and when paired with an LLM it becomes a powerful foundation for AI agents that can interact with websites autonomously. Unlike browser-use, which wraps Playwright with its own agent layer, building a Playwright AI agent directly gives you complete control over how the agent interacts with the browser, how it reads page state, and how it decides what to do next. This guide covers how to build a Playwright-based AI agent from scratch, the patterns that make browser agents reliable, and how to handle the common failure modes that affect web automation.

Why Build Directly with Playwright

High-level browser agent libraries like browser-use are excellent for standard automation tasks, but building directly with Playwright is the right choice when you need fine-grained control. Direct Playwright access lets you:

• Define exactly how the agent reads page state — raw HTML, accessibility tree, screenshots, or a custom representation you design for your specific use case
• Implement custom waiting strategies for dynamic pages that high-level libraries handle poorly
• Optimise the agent’s context window usage by providing compact, task-relevant page representations rather than full HTML dumps
• Integrate tightly with your existing Python codebase without a framework’s abstractions in the way
• Handle edge cases — CAPTCHAs, custom authentication flows, complex SPAs — with the full Playwright API available

If a high-level library handles your use case, use it. Build directly with Playwright when the abstraction gets in the way.

The Core Architecture

A Playwright AI agent has three main components working together. The browser interface manages the Playwright browser instance — opening pages, taking screenshots, executing actions like clicks and typing. The page reader converts the current page state into something useful for the LLM — an accessibility tree, a simplified DOM, or a screenshot with annotations. The agent loop sends the page state and task to the LLM, receives an action decision, executes it through the browser interface, and repeats until the task is complete.

The page reader is the most important design decision. Full HTML dumps are expensive (many tokens), noisy (lots of irrelevant markup), and harder for the model to reason about than structured representations. The accessibility tree — which Playwright exposes via page.accessibility.snapshot() — provides a structured, semantic view of the page’s interactive elements that the model can reason about efficiently. For visual tasks, a screenshot passed to a vision-capable model gives the most natural page representation, at the cost of higher token usage.

Defining the Action Space

The action space is the set of actions the LLM can decide to take. Keeping it small and well-defined produces more reliable agents than giving the model access to every Playwright API. A minimal but effective action space covers most web tasks:

• navigate(url) — go to a specific URL
• click(selector) — click an element identified by a CSS selector or accessible name
• type(selector, text) — type text into an input field
• select(selector, value) — select an option from a dropdown
• extract(description) — extract specific information from the current page
• done(result) — signal task completion with the final result

Describe each action clearly in the system prompt, including what its parameters should contain and when to use each. The model chooses actions based on these descriptions — precise descriptions produce correct choices; vague descriptions produce inconsistent ones.

Using the Accessibility Tree as Page Representation

The accessibility tree provides a structured, compact representation of the page’s interactive elements — buttons, links, inputs, headings — without the noise of styling and layout markup that pollutes raw HTML. Playwright’s page.accessibility.snapshot() returns a JSON tree that you can format and include in the LLM’s context. Filter the tree to include only interactive elements and their labels, which further reduces token usage while preserving everything the model needs to decide what to click or type.

When the model needs to reference an element for clicking or typing, ask it to reference elements by their accessible name (the label text) rather than a CSS selector — accessible names are stable across minor layout changes and more human-readable in the agent trace. Your action executor translates accessible names back to Playwright selectors using page.getByRole() or page.getByLabel() which Playwright resolves reliably.

Handling Dynamic Pages and Waiting

The most common source of Playwright agent failures is timing — the agent tries to interact with an element before it has appeared, or reads the page before an action’s effect has loaded. Playwright’s auto-waiting handles many cases automatically, but complex SPAs with asynchronous data loading require explicit waits. A robust pattern: after every click or form submission, wait for the network to become idle (no outstanding requests) before reading the page state again. This ensures that any data loaded as a result of the action is present before the model decides what to do next.

For pages with loading spinners or skeleton screens, wait for the loading indicator to disappear rather than waiting a fixed number of seconds — fixed waits are either too short (the page is still loading) or too long (the page loaded faster than expected). Playwright’s waitForSelector with the hidden state waits for an element to disappear, making it the right tool for loader-based waiting.

Screenshot-Based Agents

For vision-capable models like GPT-4o and Claude 3.5+ Sonnet, using screenshots as the page representation rather than the accessibility tree produces agents that reason more naturally about visual layouts — they see the page the way a human would. The tradeoff is token cost: a screenshot costs significantly more tokens than a filtered accessibility tree. For tasks where visual reasoning matters — identifying the right element on a visually complex page, reading content from non-standard layouts, verifying that an action had the expected visual effect — the vision approach is worth the cost. For tasks involving standard web forms and navigation where the accessibility tree captures everything relevant, the text-based approach is more efficient.

Session Management and Cookies

For agents that need to interact with authenticated services repeatedly, managing browser sessions reduces the overhead of re-authenticating every run. Save the browser storage state after a successful login using Playwright’s context.storageState() method and load it at the start of subsequent sessions. The saved state includes cookies and local storage, which is typically sufficient to restore an authenticated session without re-entering credentials. Set an expiry check — if the session is older than a configurable threshold, re-authenticate rather than attempting to use a potentially expired session.

Debugging with Traces

Playwright’s built-in tracing feature records every action, screenshot, network request, and console log from a browser session into a trace file that you can view in Playwright’s trace viewer. Enable tracing for your Playwright AI agent sessions during development and when debugging production failures. A trace file shows you exactly what the agent saw at each step and what actions it took, making it straightforward to identify where the agent went wrong — whether it clicked the wrong element, read a page state before it fully loaded, or encountered an unexpected page state that caused it to make a wrong decision. The trace viewer is accessible at trace.playwright.dev and requires no installation.

Building a Playwright AI agent directly gives you the control and transparency that higher-level browser automation libraries trade away for convenience. The investment in designing a good page reader, a clean action space, and robust waiting logic pays back in an agent that handles your specific use cases reliably and is straightforward to debug when things go wrong. For standard automation tasks, start with browser-use. For tasks where you need full control or the standard library’s abstractions do not fit, Playwright’s direct API is the right foundation.

Error Recovery and Stuck States

A Playwright AI agent will occasionally get stuck — it takes an action that does not produce the expected result, finds itself on an unexpected page, or encounters an error it does not know how to recover from. Building explicit error recovery into the agent loop is what separates agents that work in production from ones that work only in demos. The most effective recovery mechanism is a recovery prompt: when the agent has taken three or more steps without making progress toward the task goal, inject a message into the context describing the current page state and asking the agent to reconsider its approach. This often breaks the agent out of local loops where it keeps trying the same wrong action. For truly stuck states — the agent has tried multiple approaches and none have worked — fall back to reporting the current state and what was attempted, rather than looping indefinitely. A graceful failure that explains what happened is more useful than an infinite loop.

Multi-Tab and Multi-Window Tasks

Some web tasks naturally involve multiple browser tabs — opening a link in a new tab, comparing information across two pages simultaneously, or following an OAuth flow that redirects through multiple domains. Playwright manages multiple pages within a single browser context, and a Playwright AI agent can be extended to handle multi-tab workflows by tracking which page is currently active and providing the agent with an action to switch between open tabs. The implementation adds a switch_tab(index) action to the action space and maintains a list of open tabs in the agent context. For most practical automation tasks, single-tab workflows are sufficient; multi-tab support is worth adding when the target workflow inherently involves parallel pages, such as comparing prices across different vendor sites simultaneously or handling OAuth redirects that open new windows.