Build a Web Scraping Pipeline for LLMs with Crawl4AI

What you'll build

A Python script that crawls a list of pages with Crawl4AI, strips navigation and boilerplate, and writes two outputs per run: clean Markdown files you can drop into a RAG index, and a JSONL file of structured records pulled from the same pages with a CSS schema. No LLM calls, no API keys.

Prerequisites

Verified on macOS 26 (Apple Silicon) with:

• Python 3.11 (Crawl4AI requires 3.10+; PyPI classifiers go up to 3.13, so avoid 3.14 for now)
• Crawl4AI 0.9.3 (released 2026-08-31)
• Playwright 1.62.0, pulled in as a dependency; crawl4ai-setup downloads Chromium Headless Shell 151 (about 95 MB)

On Linux, Chromium needs system libraries. Run python -m playwright install --with-deps chromium after the install step if crawl4ai-setup complains. No accounts needed. The target site, quotes.toscrape.com, exists for scraping practice.

1. Install Crawl4AI and the browser

python3.11 -m venv .venv && source .venv/bin/activate
pip install -U crawl4ai
crawl4ai-setup
crawl4ai-doctor

crawl4ai-setup installs the headless browser and initialises a local SQLite cache under ~/.crawl4ai/. crawl4ai-doctor does a real test crawl of crawl4ai.com and should end with:

[COMPLETE] ● ✅ Crawling test passed!

2. Write the pipeline

Save this as pipeline.py. It uses one CrawlerRunConfig for both jobs: markdown_generator produces the cleaned Markdown, extraction_strategy produces the JSON. Both run against the same fetched HTML, so each page loads once.

import asyncio
import json
from pathlib import Path

from crawl4ai import (
    AsyncWebCrawler,
    BrowserConfig,
    CacheMode,
    CrawlerRunConfig,
    DefaultMarkdownGenerator,
    JsonCssExtractionStrategy,
    PruningContentFilter,
)

URLS = [
    "https://quotes.toscrape.com/page/1/",
    "https://quotes.toscrape.com/page/2/",
    "https://quotes.toscrape.com/page/3/",
]
OUT = Path("out")

SCHEMA = {
    "name": "quotes",
    "baseSelector": "div.quote",
    "fields": [
        {"name": "text", "selector": "span.text", "type": "text"},
        {"name": "author", "selector": "small.author", "type": "text"},
        {
            "name": "tags",
            "selector": "div.tags a.tag",
            "type": "list",
            "fields": [{"name": "tag", "type": "text"}],
        },
    ],
}


async def main():
    OUT.mkdir(exist_ok=True)

    run_cfg = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS,
        excluded_tags=["nav", "footer", "header", "aside"],
        exclude_external_links=True,
        markdown_generator=DefaultMarkdownGenerator(
            content_filter=PruningContentFilter(threshold=0.45, threshold_type="dynamic"),
            options={"ignore_links": True},
        ),
        extraction_strategy=JsonCssExtractionStrategy(SCHEMA),
    )

    records = []
    async with AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler:
        results = await crawler.arun_many(URLS, config=run_cfg)
        for r in results:
            if not r.success:
                print(f"[FAIL] {r.url}: {r.error_message}")
                continue
            slug = r.url.rstrip("/").split("/")[-1]
            (OUT / f"{slug}.md").write_text(r.markdown.fit_markdown)
            items = json.loads(r.extracted_content)
            for item in items:
                item["tags"] = [t["tag"] for t in item["tags"]]
                records.append({"source_url": r.url, **item})
            print(f"[OK] {r.url}: {len(r.markdown.fit_markdown)} chars, {len(items)} quotes")

    with (OUT / "quotes.jsonl").open("w") as f:
        for rec in records:
            f.write(json.dumps(rec, ensure_ascii=False) + "\n")
    print(f"Wrote {len(records)} records to {OUT/'quotes.jsonl'}")


if __name__ == "__main__":
    asyncio.run(main())

The cleaning happens in three layers. excluded_tags drops whole DOM regions before conversion, which is the cheapest win: on page 1 it cut raw Markdown from 4,375 to 1,619 characters. PruningContentFilter then scores the remaining blocks by text density and link ratio and drops low scorers; its output lands in result.markdown.fit_markdown, while the unfiltered version stays in raw_markdown. ignore_links strips anchor URLs from the Markdown so you don't pay tokens for them.

JsonCssExtractionStrategy takes a schema: baseSelector matches one element per record, and each field's selector is scoped inside it. A list field returns a list of dicts ([{"tag": "change"}, ...]), which is why the script flattens it to plain strings before writing.

3. Run it

python pipeline.py

arun_many runs the three URLs concurrently in one browser. The whole run takes about 1.5 seconds on a laptop.

Verify it works

Crawl4AI prints its own [FETCH], [SCRAPE] and [EXTRACT] progress lines. Below them you should see:

[OK] https://quotes.toscrape.com/page/1/: 1410 chars, 10 quotes
[OK] https://quotes.toscrape.com/page/3/: 1584 chars, 10 quotes
[OK] https://quotes.toscrape.com/page/2/: 3241 chars, 10 quotes
Wrote 30 records to out/quotes.jsonl

Order varies because the pages finish concurrently. Check the outputs:

ls out/
head -c 300 out/1.md
head -1 out/quotes.jsonl

1.md  2.md  3.md  quotes.jsonl
#  Quotes to Scrape
Login
“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.” by Albert Einstein (about)
Tags: change deep-thoughts thinking world
{"source_url": "https://quotes.toscrape.com/page/1/", "text": "“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”", "author": "Albert Einstein", "tags": ["change", "deep-thoughts", "thinking", "world"]}

The Markdown has no nav, footer, or link URLs; the JSONL has one record per quote with tags as a flat list. Each .md file is ready to chunk and embed, and each JSONL line is ready to load as tool output or a metadata-filtered document.

Troubleshooting

playwright._impl._errors.Error: BrowserType.launch: Executable doesn't exist at .../chromium_headless_shell-1234/... You skipped crawl4ai-setup, or ran it in a different virtualenv. Activate the venv you installed Crawl4AI into and run crawl4ai-setup again. If it still fails, run python -m playwright install chromium directly.

Host system is missing dependencies to run browsers (Linux only) Chromium needs shared libraries that minimal images and CI runners don't ship. Run python -m playwright install --with-deps chromium (needs sudo on most distros) and retry.

TypeError: the JSON object must be str, bytes or bytearray, not NoneType result.extracted_content is None because no extraction_strategy was set on the CrawlerRunConfig you passed. Check you're passing config=run_cfg to arun_many, not a fresh config.

fit_markdown is an empty string fit_markdown is only populated when DefaultMarkdownGenerator has a content_filter. Without one, use result.markdown.raw_markdown instead.

Next steps

Tune PruningContentFilter per site: raise threshold toward 0.6 on link-heavy pages, or set min_word_threshold to drop short nodes (on this site a value of 10 also drops the author lines, so test before trusting it). For query-driven trimming, swap in BM25ContentFilter(user_query="...") from the same module. To discover pages instead of listing them, add a deep_crawl_strategy such as BFSDeepCrawlStrategy(max_depth=2) to the run config. And when a site's markup is too messy for CSS selectors, LLMExtractionStrategy accepts a Pydantic model and does the same job with a model call per page. All of these are documented under docs.crawl4ai.com.