Web Scraping With Python: A Beginner Tutorial

What you need

Python 3.9 or newer and two libraries. Install them with:

pip install requests beautifulsoup4

requests downloads the page. BeautifulSoup reads the HTML so you can pick out elements with CSS selectors. This is enough for pages whose content is in the HTML source. Pages that build their content with JavaScript need a browser tool such as Playwright, which is a separate topic.

Practice on a site made for scraping

We use quotes.toscrape.com, a sandbox built for learning. It lists quotes with an author and tags and has a Next button across ten pages. Practising on a site that welcomes it means you learn the technique without any legal or ethical doubt.

The complete script

Save this as scrape.py and run python scrape.py. It collects up to five pages and writes quotes.csv.

import csv
import time
import requests
from bs4 import BeautifulSoup

BASE = "https://quotes.toscrape.com"
HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; learning-scraper/1.0)"}

def parse_page(html):
    soup = BeautifulSoup(html, "html.parser")
    rows = []
    for q in soup.select("div.quote"):
        rows.append({
            "text": q.select_one("span.text").get_text(strip=True),
            "author": q.select_one("small.author").get_text(strip=True),
            "tags": ", ".join(t.get_text(strip=True) for t in q.select("div.tags a.tag")),
        })
    nxt = soup.select_one("li.next a")
    return rows, (nxt["href"] if nxt else None)

def scrape_all(max_pages=5):
    url, results = "/", []
    for _ in range(max_pages):
        resp = requests.get(BASE + url, headers=HEADERS, timeout=20)
        resp.raise_for_status()
        rows, url = parse_page(resp.text)
        results.extend(rows)
        if not url:
            break
        time.sleep(1)  # be polite: one request per second
    return results

def save_csv(rows, path="quotes.csv"):
    with open(path, "w", newline="", encoding="utf-8") as f:
        w = csv.DictWriter(f, fieldnames=["text", "author", "tags"])
        w.writeheader()
        w.writerows(rows)

if __name__ == "__main__":
    data = scrape_all()
    save_csv(data)
    print(f"Saved {len(data)} rows")

How it works

1. requests.get downloads the HTML. raise_for_status() stops the script with a clear error if the site returns 404 or 500.
2. soup.select finds elements with CSS selectors. Right-click the data in your browser, choose Inspect, and copy the class names you see.
3. get_text(strip=True) returns the visible text without extra spaces.
4. Pagination: each page has a Next link. The script reads its href and requests it until there is no link left, or the page limit is reached.
5. csv.DictWriter saves the rows so you can open them in Excel.

The parsing function was tested on sample HTML: two quotes with and without tags returned the right text, author and tag strings, and a page with no Next link correctly returned no next URL.

Scrape politely and legally

• Read the site’s robots.txt and terms. If scraping is disallowed, do not scrape it.
• Add a delay between requests and never hammer a server.
• Identify your script with a User-Agent.
• Prefer an official API when one exists. It is more stable and clearly allowed.
• Be careful with personal data. Collecting it can bring privacy law into play.

Next: open the data in Excel

Open quotes.csv in Excel and clean it up with our guides on removing duplicates, or paste emails you gathered into the email extractor. If you need a scraper built for your own project, with scheduling and clean Excel output, get in touch.

Frequently asked questions

More free tools: All free tools · Email extractor · Invoice automation · Free Excel templates