Master AI Web Scraping: Python Tools & Tutorial

Key takeaways:

• AI web scraping uses large language models to read a page and find elements by their meaning.

• Transforming HTML into Markdown significantly reduced AI token use in our tests.

• Using AI models when scraping doesn't prevent blocks, so you still need proxies, delays, and retries.

The main appeal of using AI for web scraping, or any other task really, is that you can simply describe what you want in plain words. While large language models (LLMs) handle much of the parsing code, Python is still the best choice because its ecosystem covers every step, from fetching pages to validating output.

A Python library exists for almost every AI web scraping tool you might need, but you must be strategic in your choices. Learning about libraries and how AI scraping differs from traditional data collection is crucial to building an AI scraper that actually extracts data.

What Is AI Web Scraping?

AI web scraping is a data extraction method where an LLM, such as OpenAI's GPT models or Anthropic's Claude, is used to read a page. The model then returns structured data, such as JSON, that fits your schema and helps you extract it.

Traditional web scraping tools find what's needed by its position in the HTML, while AI-powered web scraping finds it by reading the page. As such, data is not only identified but interpreted at once, so one prompt can help extract data across different web page layouts.

Python has the fetching libraries (requests, Playwright), the parsing tools (BeautifulSoup), the LLM SDKs, and the validation tool (Pydantic) in one ecosystem. That's why learning Python web scraping is a great start, but paired with the newest advances, such as AI agents, it has even more potential.

AI vs Traditional Web Scraping in Python

Both traditional and AI web scraping download pages using the Requests library or a headless browser like Playwright or Selenium. The differences lie in data extraction.

Both traditional and AI web scraping download pages using the Requests library or a headless browser like Playwright or Selenium. The differences lie in data extraction.When layouts vary across sites, change often, or hold unstructured web data, such as reviews or listings across many domains, AI gives the most benefit. However, traditional approaches are still best for scraping one stable site at high volume, for identical outputs every run, or when you have a tight budget.

How AI Web Scraping Works with Python

Generally, AI web scraping works in five stages, mirroring a traditional data extraction process while replacing CSS selectors with LLM capabilities.

1. Fetch. Requests downloads static HTML, and Playwright renders pages that load content with JavaScript.
2. Clean. Strip scripts, styles, navigation, and footers, then convert the rest to Markdown. The conversion helps to get cleaner data extraction and save on AI tokens.
3. Extract. Send the Markdown to an LLM with a schema. The model fills in the fields.
4. Validate. Pydantic checks types and required fields to avoid bad output.
5. Store. Save results to JSON, CSV, or a database.

Best Python Libraries for AI Web Scraping

Before you pick an AI web scraping tool, decide whether you want a ready-made library or you can build your own scraper on top of an LLM SDK.

Firecrawl (Python SDK)

Firecrawl is an API that renders JavaScript and returns pages as Markdown or structured JSON. It stands out from other options due to a generous free tier, with 1,000 credits per month. It's enough to try out AI-powered web scraping or even complete a small project.

ScrapeGraphAI

The open-source scrapegraphai library builds scraping pipelines as graphs. SmartScraperGraph lets you extract data from one page using a prompt, SmartScraperMultiGraph runs the same prompt across several pages, and SearchGraph scrapes the top search results. It works with OpenAI, Gemini, Groq, and local models through Ollama.

Crawl4AI

Crawl4AI is an open-source async crawler designed for LLMs and AI apps. It renders pages with Playwright and outputs clean Markdown. Its LLMExtractionStrategy takes a Pydantic schema for data extraction and returns JSON.

Browser Use

Browser Use runs AI agents that drive a real browser through multi-step tasks, such as logins, forms, and "Load more" buttons. Each step is an LLM call, so it's slower and costs more, but can complete complex tasks.

DIY: An LLM SDK Plus Pydantic

The leanest stack is to use OpenAI or Anthropic's SDK with structured outputs. You pass a Pydantic model, and the API returns JSON with exactly those fields and types. You get full control, but need to write the fetching, cleaning, and proxy code yourself.

How to Build an AI Web Scraper in Python (Step by Step)

We'll be using the DIY approach with Pydantic, since libraries like ScrapeGraphAI and Crawl4AI run the same fetch, clean, and extract steps behind the scenes. If you're new to AI-powered web scraping, build a DIY setup at least once so you know what each tool setting does.

We'll be scraping web data from Books to Scrape, a practice site built for scraping. Before starting, install Python 3.10 or newer. We tested the script on Python 3.14.4. We'll be using Claude Opus 5.5 (claude-opus-5-5), but the process is similar with models from OpenAI and other vendors.

Step 1: Set Up Your Environment

The code below creates a virtual environment, installs all the required libraries, and sets up a browser. Run the following commands one by one in your PowerShell:

mkdir ai-scraper
cd ai-scraper
python -m venv venv
venv\Scripts\activate
pip install requests playwright beautifulsoup4 markdownify anthropic pydantic
playwright install chromium
$env:ANTHROPIC_API_KEY="your-key"

Note that "your-key" is a placeholder. Placing it as an environment variable helps to keep the key out of your Python scripts.

On macOS or Linux, activate the environment with source venv/bin/activate and set the key with export ANTHROPIC_API_KEY="your-key". To route requests through a proxy, set PROXY_URL the same way, in the format http://user:pass@host:port.

Step 2: Fetch the Page

Books to Scrape is a static website with a simple layout, so using the Requests library and proxies is enough.

html = requests.get(url, proxies=PROXIES, timeout=30).text

For JavaScript-rendered pages, you'll need to use Playwright and wait for the network to go idle:

from playwright.sync_api import sync_playwright
 
with sync_playwright() as p:
    browser = p.chromium.launch(proxy={"server": "http://HOST:PORT", "username": "USER", "password": "PASS"})
    page = browser.new_page()
    page.goto(url, wait_until="networkidle")
    html = page.content()
    browser.close()

Step 3: Clean the HTML and Convert It to Markdown

Remove tags with no data and keep only the product list. Ratings sit in a CSS class that Markdown drops, so we copy them into text.

def clean(html):
    soup = BeautifulSoup(html, "html.parser")
    for tag in soup(["script", "style", "nav", "footer", "img", "form"]):
        tag.decompose()
    for p in soup.select("p.star-rating"):
        p.string = "Rating: " + p["class"][1]
    main = soup.select_one("ol.row") or soup.body
    return md(str(main))

Claude charges per token, so cleaning cuts costs. In our tests, page one shrank from 15,350 tokens of raw HTML to 1,663 of Markdown.

Step 4: Define the Output Schema with Pydantic

The Pydantic library helps us tell Claude exactly what we want it to return. You can write it as a Python class with one line per field and its type. We want four fields per book: title (text), price (a number), availability (text), and rating (a whole number). Fields can be missing, except for title.

class Book(BaseModel):
    title: str
    price: float | None
    availability: str | None
    rating: int | None

Step 5: Extract Structured Data with the LLM

The code sends Claude one request with three parts: cleaned Markdown, a BookList schema, and a system prompt. Claude reads the Markdown, finds each book, and fills in the fields. Structured outputs force the reply into JSON that matches the schema.

The system prompt is a fixed instruction sent with every data extraction request. Our prompt tells Claude to return null, an empty value, when the page doesn't show a field, instead of guessing.

resp = client.messages.parse(
    model=MODEL, max_tokens=16000, system=SYSTEM,
    messages=[{"role": "user", "content": markdown}],
    output_format=BookList)

The max_tokens parameter caps the reply length, limiting Claude's thinking. The extract() function in the full code pulls the JSON out of the reply.

Step 6: Validate and Save the Results

The BookList.model_validate_json() reads the JSON from Step 5 and checks it against the schema. If a field is missing or has the wrong type, Pydantic raises a ValidationError. The script logs that page and moves on:

try:
    books += BookList.model_validate_json(extract(markdown)).books
except ValidationError as e:
    print("Invalid output, page skipped:", e)

The save() function then writes every valid book to books.json and books.csv:

def save(books):
    rows = [b.model_dump() for b in books]
    with open("books.json", "w") as f:
        json.dump(rows, f, indent=2)
    with open("books.csv", "w", newline="") as f:
        writer = csv.DictWriter(f, fieldnames=list(Book.model_fields))
        writer.writeheader()
        writer.writerows(rows)

Full Code

Save the code below as scraper.py in your project (ai-scraper) folder.

import csv, json, os, time
from urllib.parse import urljoin
 
import requests
from anthropic import Anthropic
from bs4 import BeautifulSoup
from markdownify import markdownify as md
from pydantic import BaseModel, ValidationError
 
START_URL = "https://books.toscrape.com/"
MODEL = "claude-opus-5-5"
PROXY = os.getenv("PROXY_URL")  # http://user:pass@host:port
PROXIES = {"http": PROXY, "https": PROXY} if PROXY else None
SYSTEM = ("Extract every book. Use the link title as the full title. "
          "Rating is 1-5. Return null for missing values. Never guess.")
 
client = Anthropic()  # reads ANTHROPIC_API_KEY
 
class Book(BaseModel):
    title: str
    price: float | None
    availability: str | None
    rating: int | None
 
class BookList(BaseModel):
    books: list[Book]
 
def fetch(url):
    r = requests.get(url, proxies=PROXIES, timeout=30)
    r.raise_for_status()
    return r.text
 
def clean(html):
    soup = BeautifulSoup(html, "html.parser")
    for tag in soup(["script", "style", "nav", "footer", "img", "form"]):
        tag.decompose()
    for p in soup.select("p.star-rating"):
        p.string = "Rating: " + p["class"][1]
    main = soup.select_one("ol.row") or soup.body
    return md(str(main))
 
def tokens(text):
    return client.messages.count_tokens(
        model=MODEL, messages=[{"role": "user", "content": text}]
    ).input_tokens
 
def extract(markdown):
    resp = client.messages.parse(
        model=MODEL,
        max_tokens=16000,
        system=SYSTEM,
        messages=[{"role": "user", "content": markdown}],
        output_format=BookList,
    )
    return next(b.text for b in resp.content if b.type == "text")
 
def next_url(html, url):
    link = BeautifulSoup(html, "html.parser").select_one("li.next a")
    return urljoin(url, link["href"]) if link else None
 
def save(books):
    rows = [b.model_dump() for b in books]
    with open("books.json", "w") as f:
        json.dump(rows, f, indent=2)
    with open("books.csv", "w", newline="") as f:
        writer = csv.DictWriter(f, fieldnames=list(Book.model_fields))
        writer.writeheader()
        writer.writerows(rows)
 
def main(max_pages=3):
    url, books = START_URL, []
    for _ in range(max_pages):
        html = fetch(url)
        markdown = clean(html)
        print(f"{tokens(html)} -> {tokens(markdown)} tokens")
        try:
            books += BookList.model_validate_json(extract(markdown)).books
        except ValidationError as e:
            print("Invalid output, page skipped:", e)
        url = next_url(html, url)
        if not url:
            break
        time.sleep(1)
    save(books)
    print(f"Saved {len(books)} books to books.json and books.csv")
 
if __name__ == "__main__":
    main()

With the virtual environment active and ANTHROPIC_API_KEY set, run the code using the command below in your PowerShell:

python scraper.py

If everything is set right, it should give you books.json and books.csv files in your project folder with the needed web data from Books to Scrape. Here's what the output should look like:

[
  {
    "title": "A Light in the Attic",
    "price": 51.77,
    "availability": "In stock",
    "rating": 3
  },
  {
    "title": "Tipping the Velvet",
    "price": 53.74,
    "availability": "In stock",
    "rating": 1
  }
]

Scaling AI Scraping

Our Python script only runs three pages one after another, which works for a simple test at a low rate. In a real-world scenario with thousands of pages, consider these three strategies to scale your AI web scraping project.

• Batch processing. Run pages in parallel with asyncio, and cap concurrency with a semaphore. The cap protects the site and keeps you under LLM rate limits.

• Pagination. Follow "next" links in plain code, as next_url() does. It's cheaper and more reliable than using tokens.

• Dynamic content. Handle infinite scroll and "Load more" with Playwright actions like page.mouse.wheel() or page.click(). For flows that need a login, use Browser Use.

How to Avoid Getting Blocked When Scraping With AI

Using AI for scraping websites changes how data is parsed, not how requests look to the target site. Rate limits, IP bans, and CAPTCHAs still apply as they do to other web scraping tools. A few essential practices for successful scraping apply here as well.

Common web scraping rules apply when using AI. Respect robots.txt, avoid personal or copyrighted data, and implement exponential backoff. Don't forget to limit your request rates when a site answers with an error like 429 (too many requests) or 403 (forbidden). Even with these measures, you'll still need proxies.

Rotating residential proxies send your requests through IP addresses of real home devices and can give each request a new one. To the target site, your scraper looks like many normal visitors instead of one scraper.

With MarsProxies residential proxies, you get a pool of over 1 million IPs in 190+ countries. It's easy to add them to an AI web scraping workflow. Simply set your MarsProxies endpoint and credentials as PROXY_URL, and the script will route every request through them.

Besides quality proxies, it's also important to send a real browser User-Agent header, because the default one from Requests identifies your scraper as a Python script.

Best Practices for AI Web Scraping in Python

AI-powered web scraping works best when the LLM does what it can do best, and code handles the rest. The main rule of thumb is to use the LLM for reading meaning, and code to handle fetching, cleaning, pagination, and checks.

• Cut token costs. Send only cleaned Markdown from the main container, and cache processed pages.

• Prevent hallucinated data. Use a strict schema and a null-over-guessing prompt. Spot-check against the live page.

• Use AI to write selectors, then scrape traditionally. For one stable site, ask the LLM once for CSS selectors and run them with BeautifulSoup.

• Handle errors and retries. Retry with backoff, skip invalid pages, and log prompts and outputs for reproducibility.

• Choose the right model. Start with a smaller model such as Claude Haiku 4.5 for simple listings, and move to a larger one only where accuracy drops.

• Scrape responsibly. Respect robots.txt and site terms, keep request rates low, and leave personal data alone.

Real-World Use Cases of AI Web Scraping

There are many ways to make money with web scraping or apply it in a professional field. Here are some of the most popular web scraping projects for our customers.

• Price monitoring. Extract data on prices from stores with different layouts.

• Lead lists. Extract data such as company names and contacts from directories.

• Review and sentiment extraction. Pull reviews and tag sentiment in the same call.

• News monitoring. Collect headlines and summaries from many publishers.

• RAG pipelines. Feed clean web data into retrieval-augmented generation (RAG) systems and AI agents.

Conclusion

A basic AI web scraping Python workflow goes like this: fetch with Requests or Playwright, clean to Markdown, extract with an LLM, and validate with Pydantic. Add rotating residential proxies, and the same script collects web data from thousands of pages.

Frequently Asked Questions