Build Web Scrapers with BeautifulSoup

June 16, 2026 · 12 min read ·Updated June 16, 2026 ·intermediate

pythonbeautifulsoupweb-scrapingrequestshtml

What you’ll build

How to build web scrapers that actually finish the job: 50 pages, 1000 books, one Python script, CSV or JSON output, with politeness baked in. By the end of the guide you’ll have a working command-line tool that walks every catalog page of books.toscrape.com, extracts four fields per book (title, price, star rating, in-stock status), and writes the result to either CSV or JSON. The skeleton generalizes to most small static-site scrapes, and the politeness rules (custom User-Agent, one-second delay, robots.txt check) apply anywhere you point it.

This guide assumes you already know the BeautifulSoup basics covered in the web scraping basics guide. We won’t re-explain find and find_all; we’ll go straight to a working project you can extend.

Project setup

You need three packages: requests for HTTP, beautifulsoup4 for parsing, and lxml because it’s the fastest parser BS4 ships with. Note the package name carefully: pip install BeautifulSoup (capital B, no 4) installs the abandoned BS3 package, not what you want. Always pin the lowercased beautifulsoup4 package on the command line, then import bs4 in your code.

python -m venv .venv
source .venv/bin/activate
pip install requests beautifulsoup4 lxml

The lxml wheel is prebuilt for Linux, macOS, and Windows on common Python versions, so the install above is usually fast. If you ever see a long build step during pip install lxml, that means a wheel wasn’t available and pip is falling back to compiling from source, so check your Python version against the lxml release matrix before assuming a build issue. With a working lxml, the stdlib html.parser fallback is rarely needed.

We’ll keep the project to a single file so the article stays focused. Splitting it into a fetch.py / parse.py / pipeline.py trio is the natural next step once you’ve finished the guide.

web_scraper/
├── scraper.py
└── requirements.txt

The requirements.txt pins the three packages we just installed. The >= floor matches the versions this article was written and tested against, so anything from those versions upward should run without code changes. If you want a hard pin (e.g. for a reproducible CI build), use == instead; the article is correct against the floors shown.

requests>=2.32
beautifulsoup4>=4.12
lxml>=5.0

The HTTP layer

The fetch layer has one job: get a page’s HTML or fail loud. It sets a custom User-Agent (the default python-requests/2.32.3 is on most naive bot-blocker lists), passes a timeout (without one a stalled server can hang your script forever), and raises on HTTP errors via response.raise_for_status(). The full requests surface lives in the requests guide; we only use get, headers, and timeout here.

import requests

HEADERS = {
    "User-Agent": "pyguides-scraper/1.0 (+https://pyguides.dev) classroom-demo",
    "Accept-Language": "en",
}

def fetch_page(url: str, timeout: float = 10.0) -> str | None:
    for attempt in range(2):
        try:
            response = requests.get(url, headers=HEADERS, timeout=timeout)
            response.raise_for_status()
            return response.text
        except requests.RequestException as exc:
            print(f"[warn] {url} -> {exc!r} (attempt {attempt + 1})")
    return None

Two details worth noticing. The retry loop is short on purpose: one retry is plenty for a polite scrape, and you don’t want to hammer a flaky server. fetch_page returns None on failure so the caller decides whether to keep going or abort, which keeps the policy in one place instead of leaking it into the parser. The requests exceptions tree explains which subclasses land where.

How to parse pages with BeautifulSoup

The catalog page renders each book as an <article class="product_pod"> card. Inside the card, the title lives in h3 a, the price in p.price_color, and the rating in p.star-rating (where class is a list like ["star-rating", "Three"]). The ["Three"] piece is the gotcha worth a callout: class is multi-valued in HTML, so tag["class"] returns a list, not a string, and you index [1] to get the rating word.

from bs4 import BeautifulSoup

RATING_WORDS = {"One": 1, "Two": 2, "Three": 3, "Four": 4, "Five": 5}

def parse_books(html: str) -> list[dict]:
    soup = BeautifulSoup(html, "lxml")
    books = []
    for card in soup.select("article.product_pod"):
        title_a = card.select_one("h3 a")
        price = card.select_one("p.price_color").get_text(strip=True)
        rating_class = card.select_one("p.star-rating")["class"]
        rating_word = rating_class[1]  # ['star-rating', 'Three']
        books.append({
            "title": title_a.get("title") or title_a.get_text(strip=True),
            "price": price,
            "rating": RATING_WORDS.get(rating_word),
            "in_stock": "In stock" in card.get_text(),
        })
    return books

Two safety details. First, title_a.get("title") or title_a.get_text(strip=True) prefers the title= attribute (the full book name) and falls back to the visible link text, which the page truncates with an ellipsis on long titles. Second, .get(...) is the safer access pattern throughout this project. Reaching for tag["href"] on a card where the anchor might be missing raises KeyError, which is the most common crash in beginner scrapers. The BeautifulSoup documentation covers the rest of the API in more depth than fits here, including SoupStrainer for skipping large chunks of a page you don’t need.

How to walk pagination correctly

The site has one annoying quirk: page 1 lives at https://books.toscrape.com/index.html and pages 2 through 50 live under https://books.toscrape.com/catalogue/page-N.html. Don’t try to reconstruct those URLs by hand. Follow the next link the page gives you, and stop when the link isn’t there. Don’t use a counter here either, because pages-per-site changes between releases and the next link is the source of truth.

from urllib.parse import urljoin
import time

BASE = "https://books.toscrape.com/"
url = urljoin(BASE, "index.html")
all_books: list[dict] = []

while url:
    html = fetch_page(url)
    if not html:
        break
    all_books.extend(parse_books(html))
    soup = BeautifulSoup(html, "lxml")
    next_link = soup.select_one("li.next > a")
    url = urljoin(url, next_link["href"]) if next_link else None
    time.sleep(1)  # be polite

urljoin is what turns the relative href from each page into an absolute URL, regardless of how the current URL is shaped. The urllib.parse docs cover the edge cases around trailing slashes and absolute paths. The time.sleep(1) keeps you under most sites’ “no more than one request per second” expectations, and the loop terminates naturally on the last page because select_one returns None once the pager is gone.

Politeness: User-Agent, delays, and robots.txt

Three rules. First, set a real User-Agent identifying your scraper, so the site’s operator can reach you if something goes wrong. The default python-requests/2.32.3 string is also the first thing naive bot-blockers ban, so it pays to override it even on friendly sites. Second, sleep at least one second between requests, more if the site is small. Third, check robots.txt before crawling anything you don’t own.

from urllib.robotparser import RobotFileParser

def allowed(url: str, ua: str = "*") -> bool:
    rp = RobotFileParser()
    rp.set_url("https://books.toscrape.com/robots.txt")
    rp.read()
    return rp.can_fetch(ua, url)

books.toscrape.com doesn’t have a robots.txt (the request returns 404), and RobotFileParser treats a missing file as “everything is allowed”. For real sites the answer is often different; the check is cheap, so do it. The urllib.robotparser docs cover the rest of the parser surface, including crawl-delay parsing if you ever need to honor a site’s preferred throttle.

Saving results (CSV and JSON)

Two output formats cover most projects. CSV opens cleanly in any spreadsheet; JSON keeps the data structure intact (lists, nested dicts, None for missing values). The csv module reference and json module reference document the full surface; the snippets below are the parts that matter for a scraper.

import csv
import json
from pathlib import Path

def save_csv(rows: list[dict], path: str) -> None:
    if not rows:
        Path(path).write_text("")
        return
    with open(path, "w", newline="", encoding="utf-8") as fp:
        writer = csv.DictWriter(fp, fieldnames=list(rows[0].keys()))
        writer.writeheader()
        writer.writerows(rows)

def save_json(rows: list[dict], path: str) -> None:
    with open(path, "w", encoding="utf-8") as fp:
        json.dump(rows, fp, indent=2, ensure_ascii=False)

The newline="" on the CSV open is required on Windows to keep csv.DictWriter from doubling line endings. The ensure_ascii=False on json.dump is what keeps book titles like “Âge de bronze” readable instead of turning them into \u00c2ge de bronze. If you ever need to merge two outputs into a single report, the urllib.parse module has urljoin and urlparse which you’ll likely want for relative-to-absolute link cleanup across merged rows.

Error handling and resilience

A scraper lives on a hostile network. Pages disappear mid-crawl, servers return 503, your network blips. Wrap each request in try/except requests.RequestException, log the URL and the error, and decide a policy: skip-and-continue (recommended for transient errors) or abort (recommended for 4xx, since the site isn’t going to fix itself between requests). Distinguishing the two cases is the difference between a scraper that finishes the night and one that needs babysitting at 2 a.m.

For long-running jobs, swap print for logging so you get timestamps, log levels, and a logging.FileHandler for free. A minimal setup is logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s"), then logging.warning(...) instead of print(...). The logging guide walks through handlers, formatters, and rotation if you need more than the basic config.

How to build a web scraper CLI

A small argparse surface lets you cap the run and choose the output file. The full argparse walkthrough is in the argparse guide; we use just two flags here. Keeping the CLI thin is a deliberate choice: more flags means more docs, more tests, and more ways to misuse the tool. Two flags are enough to demonstrate the pattern, and the rest follow the same shape.

import argparse

def main() -> None:
    parser = argparse.ArgumentParser(description="Scrape books.toscrape.com")
    parser.add_argument("--max-pages", type=int, default=50)
    parser.add_argument("--output", default="books.json",
                        help="Path ending in .json or .csv")
    args = parser.parse_args()

    # ... run the loop above, then:
    rows = all_books
    if args.output.endswith(".csv"):
        save_csv(rows, args.output)
    else:
        save_json(rows, args.output)

    print(f"scraped {len(rows)} books -> {args.output}")
    for row in rows[:3]:
        print(row)

if __name__ == "__main__":
    main()

Running the full crawl is a one-liner. The 1-second delay we set in the pagination loop means the run takes about a minute, which is the right ballpark for a sandbox site and well under any rate limit a real site would impose. If you want to smoke-test the parser before the full run, drop --max-pages to 1 and inspect the output file by hand.

python scraper.py --max-pages 50 --output books.json

The first three rows of the produced JSON look like this. Note the truncated title in the second row is the link text from the catalog page, which is what title_a.get("title") or title_a.get_text(strip=True) falls back to only when the title= attribute is missing on the card. Real output will be the full titles because the sandbox always sets the attribute.

# output:
# scraped 1000 books -> books.json
# {'title': 'A Light in the Attic', 'price': '£51.77', 'rating': 3, 'in_stock': True}
# {'title': 'Tipping the Velvet', 'price': '£53.74', 'rating': 1, 'in_stock': True}
# {'title': 'Soumission', 'price': '£50.10', 'rating': 1, 'in_stock': True}

How to test your web scraper

Quick confidence check: save three HTML files to tests/fixtures/ (page 1, a middle page, the last page) and assert that parse_books returns 20 items for the first two and 20 for the last. Pin your expected counts so a site redesign shows up as a failing test, not as silent data drift. The --max-pages 1 flag is the fastest way to iterate locally before running the full 50-page crawl against the live site.

Extending the project

A few directions once the catalog crawl works:

Fetch detail pages. Each book card links to a product page that exposes UPC, tax, and review count. Add a second loop that visits each URL with a delay and merges the fields into the same row.
Switch to async. When 1 request per second isn’t fast enough, httpx gives you an async client with a near-identical API. A concurrent.futures.ThreadPoolExecutor is a smaller step if you want concurrency without rewriting the loop.
Store in SQLite. Replace the CSV/JSON branch with a single INSERT per row into an SQLite table. Use a parameterized query so titles with apostrophes don’t break the insert.
Add tests. Save a few sample pages to tests/fixtures/ and assert that parse_books returns the expected count and fields.

Frequently asked questions

Is it legal to scrape books.toscrape.com? The site is hosted specifically as a public scraping sandbox, so yes for educational use. For real sites, read the terms of service, respect robots.txt, and back off if the operator asks you to stop. The robots.txt specification is short and worth a read before you run anything against a production site.

Why is my User-Agent getting blocked? Some sites ban the default python-requests/X.Y.Z string outright. Set a real, descriptive User-Agent like the one in HEADERS above. If you’re still blocked, the site is probably using a fingerprinting technique that goes beyond UA; at that point you’re fighting a detection system, not a scraper pattern, and the right answer is usually to ask for an API.

Why does class return a list? HTML’s class attribute is space-separated, and BeautifulSoup reflects that by always returning a list. Use tag["class"][0], tag["class"][1], and so on. The rating trick (class[1]) is the most common case, but the same pattern shows up on rel and headers too. The multi-valued attributes section of the BS4 docs spells this out.

Should I use requests or httpx? For synchronous scrapers, requests is simpler and battle-tested. For async or HTTP/2, switch to httpx. Both have nearly identical get, headers, and timeout surfaces, so porting is mostly mechanical, so change the import and the rest of the code often runs unchanged.

Conclusion

You now have a complete shape to build web scrapers that walk multi-page sites, parse structured cards, and export clean CSV or JSON. The skeleton generalizes to most small static-site scrapes: a fetch function with User-Agent and timeout, a parse function that prefers .get(...) over [], a loop that follows next links instead of counting, and a writer that picks the right file format at the end. When the site stops being a friendly sandbox and you need concurrency, rate-limit headers, or a real User-Agent policy, swap the requests layer for httpx and keep everything else the same.