Build a Browser-Automation Agent with OpenAI Function Calling and Playwright

What you'll build

A Python script that lets an OpenAI model drive a headless Chromium browser through Playwright. You give it a task in plain English ("go to this site and tell me X"), and it decides which browser actions to call, executes them, reads the results, and loops until the task is done.

Prerequisites

• Python 3.9 or later
• An OpenAI API key with access to a function-calling-capable model (gpt-4o or gpt-4o-mini)
• macOS, Linux, or Windows (Playwright supports all three, but Linux needs extra system libraries, noted below)
• Comfort with basic async-free Python; we're using Playwright's synchronous API to keep the control flow simple

1. Set up your project

mkdir browser-agent && cd browser-agent
python3 -m venv venv
source venv/bin/activate   # Windows: venv\Scripts\activate
pip install --upgrade pip
pip install openai playwright
playwright install chromium

On Linux (including CI containers), Chromium also needs system libraries:

playwright install-deps chromium

Set your API key as an environment variable rather than hardcoding it:

export OPENAI_API_KEY="sk-..."          # macOS/Linux
# PowerShell: $env:OPENAI_API_KEY="sk-..."

2. Wrap Playwright in tool functions

The model never touches the browser directly. It calls named functions with JSON arguments, and your code executes the actual Playwright API. Keep each function narrow and single-purpose, that's what makes the model's job of choosing between them tractable.

Create agent.py:

import json
from openai import OpenAI
from playwright.sync_api import sync_playwright

client = OpenAI()
MODEL = "gpt-4o-mini"

class Browser:
    def __init__(self):
        self.playwright = sync_playwright().start()
        self.browser = self.playwright.chromium.launch(headless=True)
        self.page = self.browser.new_page()

    def navigate(self, url: str) -> str:
        self.page.goto(url, wait_until="domcontentloaded", timeout=15000)
        return f"Navigated to {self.page.url}"

    def click(self, selector: str) -> str:
        self.page.click(selector, timeout=5000)
        return f"Clicked {selector}"

    def fill(self, selector: str, text: str) -> str:
        self.page.fill(selector, text, timeout=5000)
        return f"Filled {selector} with '{text}'"

    def get_text(self, selector: str) -> str:
        el = self.page.query_selector(selector)
        if not el:
            return f"No element matched selector: {selector}"
        return el.inner_text()[:2000]

    def close(self):
        self.browser.close()
        self.playwright.stop()

Notice the 5-15 second timeouts. Without them, a bad selector hangs the whole agent instead of returning an error the model can react to.

3. Define the function-calling schema

Each tool needs a JSON schema describing its name, purpose, and parameters. The model uses the description fields to decide when to call what, so write them like you're briefing a new engineer, not just labeling arguments.

tools = [
    {"type": "function", "function": {
        "name": "navigate",
        "description": "Go to a URL in the browser",
        "parameters": {"type": "object", "properties": {
            "url": {"type": "string", "description": "Full URL including https://"}},
            "required": ["url"]}}},
    {"type": "function", "function": {
        "name": "click",
        "description": "Click the first element matching a CSS selector",
        "parameters": {"type": "object", "properties": {
            "selector": {"type": "string"}}, "required": ["selector"]}}},
    {"type": "function", "function": {
        "name": "fill",
        "description": "Type text into an input or textarea matching a CSS selector",
        "parameters": {"type": "object", "properties": {
            "selector": {"type": "string"}, "text": {"type": "string"}},
            "required": ["selector", "text"]}}},
    {"type": "function", "function": {
        "name": "get_text",
        "description": "Read the visible text of an element matching a CSS selector",
        "parameters": {"type": "object", "properties": {
            "selector": {"type": "string"}}, "required": ["selector"]}}},
]

4. Write the agent loop

This is the core pattern for any function-calling agent: send the conversation, check if the model wants to call a tool, run it, feed the result back as a tool message, and repeat until the model responds with plain text instead of a tool call.

dispatch = {}

def run_agent(task: str, max_steps: int = 10) -> str:
    browser = Browser()
    dispatch.update({
        "navigate": lambda a: browser.navigate(a["url"]),
        "click": lambda a: browser.click(a["selector"]),
        "fill": lambda a: browser.fill(a["selector"], a["text"]),
        "get_text": lambda a: browser.get_text(a["selector"]),
    })

    messages = [
        {"role": "system", "content": (
            "You control a headless Chromium browser through function calls. "
            "Use navigate, click, fill, and get_text to complete the user's task step by step. "
            "Call get_text to check what's on the page before deciding your next move. "
            "When the task is finished, reply with plain text summarizing the result and "
            "do not call any more functions.")},
        {"role": "user", "content": task},
    ]

    try:
        for step in range(max_steps):
            response = client.chat.completions.create(
                model=MODEL, messages=messages, tools=tools, tool_choice="auto")
            message = response.choices[0].message

            if not message.tool_calls:
                return message.content

            messages.append({
                "role": "assistant",
                "content": message.content,
                "tool_calls": [
                    {"id": tc.id, "type": "function", "function": {
                        "name": tc.function.name, "arguments": tc.function.arguments}}
                    for tc in message.tool_calls],
            })

            for tc in message.tool_calls:
                name = tc.function.name
                args = json.loads(tc.function.arguments)
                print(f"-> {name}({args})")
                try:
                    result = dispatch[name](args)
                except Exception as e:
                    result = f"Error: {e}"
                messages.append({"role": "tool", "tool_call_id": tc.id, "content": str(result)})

        return "Max steps reached without a final answer."
    finally:
        browser.close()

if __name__ == "__main__":
    print(run_agent(
        "Go to https://quotes.toscrape.com, read the first quote on the page, "
        "and tell me the quote text and its author."))

A few details matter here. The assistant message with tool_calls has to be appended to history exactly as shown, as a dict with type: "function", or the next API call will reject it. And every single tool_call_id needs a matching tool message before you send the next request; the API errors out otherwise. Wrapping errors as strings (instead of raising) matters too, since it lets the model see the failure and try a different selector instead of crashing your process.

Verify it works

Run it:

python agent.py

You'll see the tool calls print as they happen, something like:

-> navigate({'url': 'https://quotes.toscrape.com'})
-> get_text({'selector': '.quote'})

followed by a final plain-text answer naming the quote and author (the first quote on that site is from Albert Einstein). If the model calls get_text on a selector that doesn't exist yet, watch it recover: the error string comes back as the tool result, and the next completion usually tries body or a broader selector instead.

Troubleshooting

• playwright._impl._errors.Error: Executable doesn't exist: you skipped playwright install chromium. Run it inside the same virtualenv you're executing the script from.
• 400 error about tool_calls with no matching tool message: you either didn't respond to every tool_call_id from the previous assistant turn, or you sent them out of order. Each tool call needs exactly one corresponding role: tool message before the next create() call.
• Agent loops without making progress: tighten the system prompt (tell it explicitly to call get_text before guessing selectors), and lower max_steps while debugging so you're not burning tokens on a stuck loop.
• Timeouts on click/fill: the selector is probably right but the element isn't in the viewport or is inside an iframe. Playwright's page.frame_locator() handles iframes; that's a good next tool to add.

Next steps

Add more tools as the agent needs them: scroll, wait_for_selector, and screenshot (returning a description, not raw pixels, unless you're using a vision-capable model). For structured extraction instead of free text, pair this with OpenAI's structured outputs (response_format with a JSON schema) on the final answer. For production use, sandbox the browser to an allowlist of domains, run it inside Docker using Microsoft's official mcr.microsoft.com/playwright/python image, and cap wall-clock time per task, not just step count, since a single slow page load can otherwise stall the whole agent.