Python course ยท Module 4: Data, APIs and Databases

Web scraping - collecting data from the web

11 min read ยท
In this lesson12

Welcome back! Darwin here with a fascinating lesson.

You've learned how to fetch data from REST APIs - elegant, structural interfaces. But what if a website doesn't have an API? What if species data is hidden in an HTML table on a webpage?

Then we use web scraping - automatically downloading and extracting data from websites!

Safari Analogy: A REST API is like an official report from a national park in JSON format. Web scraping is like reading a tourist brochure and manually extracting information: "Ah, it says here there are 120 lions, and here that the habitat is savanna"!

What is web scraping?

Web scraping is the automatic extraction of data from websites by:

  1. Downloading the page's HTML (like a browser)
  2. Parsing the HTML structure (DOM)
  3. Finding elements of interest (tables, lists, headings)
  4. Extracting data (text, links, images)

When to use web scraping?

Use when:

  • The website has no API
  • The API is paid, but the data is public
  • You need historical data (archives)
  • Aggregating data from multiple sources

Avoid when:

  • The site has an official API (use the API!)
  • Terms of service prohibit scraping
  • Data is legally protected
  • You might violate GDPR/privacy

Legality and ethics

IMPORTANT: Web scraping is legal when:

  • You scrape public data (accessible without login)
  • You respect robots.txt (https://example.com/robots.txt)
  • You don't overload the server (rate limiting!)
  • You don't violate copyright

Bad practices:

  • Ignoring robots.txt
  • Thousands of requests per second (DDoS)
  • Scraping personal data
  • Breaking login/paywall

Golden rule: Behave like a human - visit the site reasonably, don't overload the server!

BeautifulSoup4 - HTML parser in Python

BeautifulSoup4 (bs4) is the most popular library for parsing HTML/XML in Python.

Installation

pip install beautifulsoup4
pip install requests  # For downloading HTML
pip install lxml      # Fast parser (optional)

Basic usage

import requests
from bs4 import BeautifulSoup

# 1. Download HTML
response = requests.get("https://example.com/species")
html_content = response.text

# 2. Parse HTML
soup = BeautifulSoup(html_content, "html.parser")  # or "lxml"

# 3. Find elements
title = soup.find("h1")
print(title.text)  # Heading text

# 4. Find all elements
species_items = soup.find_all("div", class_="species-card")
for item in species_items:
    name = item.find("span", class_="name").text
    print(name)

HTML Anatomy - what are we parsing?

HTML is a tree of elements (tags):

<div class="species-card" id="lion" data-population="120">
    <h2>Lion</h2>
    <p class="description">Panthera leo</p>
    <img src="/images/lion.jpg" alt="Lion">
    <a href="/species/lion">More information</a>
</div>

Elements to extract:

  • Text: <h2>Lion</h2> -> "Lion"
  • Attributes: class="species-card", id="lion"
  • Links: <a href="/species/lion">
  • Images: <img src="/images/lion.jpg">
  • Data: data-population="120"

Finding elements - selectors

find() - first matching element

from bs4 import BeautifulSoup

html = """
<div class="container">
    <h1>Safari Data</h1>
    <div class="species">Lion</div>
    <div class="species">Elephant</div>
</div>
"""

soup = BeautifulSoup(html, "html.parser")

# Find first element
title = soup.find("h1")
print(title.text)  # "Safari Data"

# Find first element with class
species = soup.find("div", class_="species")
print(species.text)  # "Lion" (first one!)

find_all() - all matching elements

# Find all elements with class "species"
all_species = soup.find_all("div", class_="species")
print(len(all_species))  # 2

for species in all_species:
    print(species.text)
# Lion
# Elephant

# Limit results
first_two = soup.find_all("div", limit=2)

CSS selector - select()

# CSS selectors (like in CSS!)
all_species = soup.select(".species")  # Class
title = soup.select_one("#main-title")  # ID
links = soup.select("div.container a")  # Nested

CSS selectors:

  • .species - class
  • #lion - ID
  • div.species - tag + class
  • div > p - direct child
  • div p - any nesting level
  • [data-population] - attribute

Data extraction

Element text

html = '<div class="name">Lion <span>(Panthera leo)</span></div>'
soup = BeautifulSoup(html, "html.parser")

# .text - all text (including nested tags)
print(soup.find("div").text)  # "Lion (Panthera leo)"

# .get_text() - with separator
print(soup.find("div").get_text(strip=True))  # "Lion(Panthera leo)" (no spaces)

# .string - only direct text (None if nested)
print(soup.find("div").string)  # None (because of <span>)

Element attributes

html = '<img src="/lion.jpg" alt="Lion" data-population="120">'
soup = BeautifulSoup(html, "html.parser")
img = soup.find("img")

# Access like a dictionary
print(img["src"])              # "/lion.jpg"
print(img["alt"])              # "Lion"
print(img["data-population"])  # "120"

# .get() - safe (returns None if missing)
print(img.get("title"))        # None
print(img.get("src", "default.jpg"))  # "/lion.jpg"

# All attributes
print(img.attrs)  # {'src': '/lion.jpg', 'alt': 'Lion', 'data-population': '120'}
html = '<a href="/species/lion" class="link">See the lion</a>'
soup = BeautifulSoup(html, "html.parser")
link = soup.find("a")

# URL
print(link["href"])  # "/species/lion"

# Link text
print(link.text)     # "See the lion"

# All links on the page
for link in soup.find_all("a"):
    print(f"{link.text}: {link.get('href')}")
html = """
<div class="parent">
    <h2>Title</h2>
    <p class="child1">Paragraph 1</p>
    <p class="child2">Paragraph 2</p>
</div>
"""
soup = BeautifulSoup(html, "html.parser")

# Children
parent = soup.find("div", class_="parent")
for child in parent.children:
    if child.name:  # Skip text nodes
        print(child.name)  # h2, p, p

# Descendants - all nested elements
for descendant in parent.descendants:
    if hasattr(descendant, 'name') and descendant.name:
        print(descendant.name)

# Parent
p = soup.find("p")
print(p.parent.name)  # "div"

# Siblings
h2 = soup.find("h2")
for sibling in h2.next_siblings:
    if sibling.name:
        print(sibling.text)  # Paragraph 1, Paragraph 2

Safari example - scraping species data

import requests
from bs4 import BeautifulSoup
from typing import List, Dict
import time

class SafariWebScraper:
    """Scraper for species data from a Safari website"""

    def __init__(self, base_url: str):
        self.base_url = base_url
        self.session = requests.Session()
        self.session.headers.update({
            "User-Agent": "Safari-Research-Bot/1.0 (Educational purposes)"
        })

    def get_soup(self, url: str) -> BeautifulSoup:
        """Download and parse HTML"""
        response = self.session.get(url, timeout=10)
        response.raise_for_status()
        return BeautifulSoup(response.text, "html.parser")

    def scrape_species_list(self, url: str) -> List[Dict[str, str]]:
        """
        Scrape species list from an HTML table

        Example structure:
        <table class="species-table">
            <tr>
                <td class="name">Lion</td>
                <td class="scientific">Panthera leo</td>
                <td class="population">120</td>
                <td class="habitat">Savanna</td>
            </tr>
        </table>
        """
        soup = self.get_soup(url)
        species_list = []

        # Find the table
        table = soup.find("table", class_="species-table")
        if not table:
            print("Warning: Table not found")
            return []

        # Iterate over rows (skipping header)
        rows = table.find_all("tr")[1:]  # [1:] skips header

        for row in rows:
            cells = row.find_all("td")
            if len(cells) >= 4:
                species = {
                    "common_name": cells[0].text.strip(),
                    "scientific_name": cells[1].text.strip(),
                    "population": cells[2].text.strip(),
                    "habitat": cells[3].text.strip()
                }
                species_list.append(species)

        return species_list

    def scrape_species_cards(self, url: str) -> List[Dict[str, str]]:
        """
        Scrape species from HTML cards

        <div class="species-card">
            <h3>Lion</h3>
            <p class="scientific">Panthera leo</p>
            <span class="population" data-count="120">120 individuals</span>
            <img src="/images/lion.jpg" alt="Lion">
            <a href="/species/lion">More</a>
        </div>
        """
        soup = self.get_soup(url)
        species_list = []

        # Find all cards
        cards = soup.find_all("div", class_="species-card")

        for card in cards:
            # Safe extraction (elements may be missing)
            name_tag = card.find("h3")
            scientific_tag = card.find("p", class_="scientific")
            pop_tag = card.find("span", class_="population")
            img_tag = card.find("img")
            link_tag = card.find("a")

            species = {
                "common_name": name_tag.text.strip() if name_tag else "Unknown",
                "scientific_name": scientific_tag.text.strip() if scientific_tag else "Unknown",
                "population": pop_tag.get("data-count", "0") if pop_tag else "0",
                "image_url": img_tag["src"] if img_tag else None,
                "detail_url": link_tag["href"] if link_tag else None
            }
            species_list.append(species)

        return species_list

    def scrape_species_detail(self, url: str) -> Dict[str, any]:
        """
        Scrape species details

        <div class="species-detail">
            <h1>Lion</h1>
            <div class="info">
                <dl>
                    <dt>Scientific name:</dt>
                    <dd>Panthera leo</dd>
                    <dt>Status:</dt>
                    <dd class="endangered">Endangered</dd>
                    <dt>Habitat:</dt>
                    <dd>Savanna, steppe</dd>
                </dl>
            </div>
            <div class="description">
                <p>The lion is the largest cat in Africa...</p>
            </div>
        </div>
        """
        soup = self.get_soup(url)

        # Name
        title = soup.find("h1")
        common_name = title.text.strip() if title else "Unknown"

        # Definition list - dt/dd pairs
        info = {}
        dl = soup.find("dl")
        if dl:
            terms = dl.find_all("dt")
            definitions = dl.find_all("dd")

            for term, definition in zip(terms, definitions):
                key = term.text.strip().replace(":", "").lower()
                value = definition.text.strip()
                info[key] = value

        # Description
        description_div = soup.find("div", class_="description")
        description = description_div.get_text(strip=True) if description_div else ""

        return {
            "common_name": common_name,
            "scientific_name": info.get("scientific name", "Unknown"),
            "status": info.get("status", "Unknown"),
            "habitat": info.get("habitat", "Unknown"),
            "description": description
        }

    def scrape_with_pagination(self, base_url: str, max_pages: int = 5) -> List[Dict]:
        """
        Scrape multiple pages with pagination

        URL pattern: /species?page=1, /species?page=2, ...
        """
        all_species = []

        for page_num in range(1, max_pages + 1):
            url = f"{base_url}?page={page_num}"
            print(f"Scraping page {page_num}: {url}")

            soup = self.get_soup(url)

            # Check if there's a next page
            next_button = soup.find("a", class_="next-page")
            if not next_button:
                print(f"Last page: {page_num}")
                break

            # Scrape data from the page
            species = self.scrape_species_cards(url)
            all_species.extend(species)

            # Rate limiting - wait between requests!
            time.sleep(1)  # 1 second between pages

        return all_species


# === DEMONSTRATION ===

print("=== SAFARI WEB SCRAPER ===\n")

scraper = SafariWebScraper(base_url="https://example-safari.com")

# 1. Scrape list from table
print("1. Scraping species table...")
species_table = scraper.scrape_species_list("https://example-safari.com/species-table")
print(f"   Found {len(species_table)} species")
for species in species_table[:3]:
    print(f"   - {species['common_name']} ({species['scientific_name']})")

# 2. Scrape cards
print("\n2. Scraping species cards...")
species_cards = scraper.scrape_species_cards("https://example-safari.com/species")
print(f"   Found {len(species_cards)} cards")

# 3. Scrape details
print("\n3. Scraping lion details...")
lion_detail = scraper.scrape_species_detail("https://example-safari.com/species/lion")
print(f"   Name: {lion_detail['common_name']}")
print(f"   Status: {lion_detail['status']}")
print(f"   Habitat: {lion_detail['habitat']}")

# 4. Scrape with pagination
print("\n4. Scraping with pagination...")
all_species = scraper.scrape_with_pagination("https://example-safari.com/species", max_pages=3)
print(f"   Total: {len(all_species)} species from 3 pages")

print("\nScraping complete")

Selenium - scraping dynamic pages

Problem: Some pages load data via JavaScript - BeautifulSoup sees empty HTML!

Solution: Selenium - automates a real browser!

Installation

pip install selenium
# Download ChromeDriver: https://chromedriver.chromium.org/

Selenium basics

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time

# Launch browser
driver = webdriver.Chrome()  # Requires ChromeDriver

try:
    # Open page
    driver.get("https://example.com/species")

    # Wait for element to load (JavaScript!)
    wait = WebDriverWait(driver, 10)
    species_cards = wait.until(
        EC.presence_of_all_elements_located((By.CLASS_NAME, "species-card"))
    )

    # Extract data
    for card in species_cards:
        name = card.find_element(By.TAG_NAME, "h3").text
        population = card.find_element(By.CLASS_NAME, "population").text
        print(f"{name}: {population}")

    # Click "Load More" button
    load_more = driver.find_element(By.ID, "load-more-btn")
    load_more.click()
    time.sleep(2)  # Wait for loading

    # Scroll down (infinite scroll)
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    time.sleep(2)

finally:
    # Always close the browser!
    driver.quit()

Headless mode - without GUI

from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("--headless")  # No browser window
options.add_argument("--no-sandbox")
options.add_argument("--disable-dev-shm-usage")

driver = webdriver.Chrome(options=options)
# Rest as above

Web Scraping Best Practices

1. Check robots.txt

import requests

response = requests.get("https://example.com/robots.txt")
print(response.text)

# Check if your scraper is allowed:
# User-agent: *
# Disallow: /admin/
# Crawl-delay: 10  # Wait 10s between requests

2. Rate Limiting - don't overload the server!

import time
import random

for url in urls:
    scrape_page(url)

    # Wait between requests
    time.sleep(random.uniform(1, 3))  # 1-3 seconds randomly

3. User-Agent - identify yourself

headers = {
    "User-Agent": "MyResearchBot/1.0 (+https://mysite.com/bot-info; contact@mysite.com)"
}
response = requests.get(url, headers=headers)

4. Error handling

from requests.exceptions import RequestException

def safe_scrape(url):
    try:
        response = requests.get(url, timeout=10)
        response.raise_for_status()
        soup = BeautifulSoup(response.text, "html.parser")
        return soup

    except RequestException as e:
        print(f"Error: {e}")
        return None

5. Cache - don't download the same thing twice

import pickle
from pathlib import Path

def get_cached_html(url, cache_dir="cache"):
    Path(cache_dir).mkdir(exist_ok=True)
    cache_file = Path(cache_dir) / f"{hash(url)}.pkl"

    # Check cache
    if cache_file.exists():
        with open(cache_file, "rb") as f:
            return pickle.load(f)

    # Download and save
    response = requests.get(url)
    html = response.text

    with open(cache_file, "wb") as f:
        pickle.dump(html, f)

    return html

Common pitfalls

1. HTML structure changes

Websites change their layout - your scraper stops working!

Solution: Use multiple selectors, error handling

# Multiple attempts to find an element
name = (
    soup.find("h3", class_="species-name") or
    soup.find("span", class_="name") or
    soup.find("div", attrs={"data-type": "name"})
)

2. Dynamic JavaScript

BeautifulSoup only sees the initial HTML, not the result of JS!

Solution: Selenium, or analyze the API (check Network in DevTools)

3. Anti-scraping mechanisms

  • CAPTCHA
  • Rate limiting (429 Too Many Requests)
  • IP ban
  • Required cookies/sessions

Solution: Follow the rules, don't attack, consider using the API

Summary

In this lesson you learned:

  • What web scraping is and when to use it
  • Legality and ethics of scraping (robots.txt, rate limiting)
  • BeautifulSoup4: find(), find_all(), select()
  • Extraction: text, attributes, links
  • DOM navigation: children, parent, siblings
  • Selenium for dynamic pages (JavaScript)
  • Best practices: User-Agent, rate limiting, cache, error handling
  • Common pitfalls and how to avoid them

Checkpoint

Before moving on:

  • You can parse HTML with BeautifulSoup
  • You know how to find elements: find(), select()
  • You can extract text and attributes
  • You understand the difference between BeautifulSoup vs Selenium
  • You know the ethics of web scraping

Safari Analogy: Web scraping is like reading tourist brochures about national parks and manually collecting data into a spreadsheet - time-consuming, but effective when there are no official reports (APIs)!

In the next lesson, Darwin will teach you SQL databases - how to store and manage large datasets in a structured way!

Docs entries used in this lesson

Spotted a mistake in this lesson?

Check yourself

Answer the questions from this lesson. Pick an answer to see right away whether it is correct.

  1. 1. BeautifulSoup is used for:

  2. 2. The method soup.find_all('a') returns:

Hands-on tasks in the game

  • Horizontal ordering

    Arrange the elements in the correct order:

  • Code editor

    Fetch the page and find all <h1> elements

  • Code editor

    Find all links (href attribute) on the page

  • Vertical ordering

    Arrange the steps of web scraping:

  • Click in order

    Click in order: soup = BeautifulSoup(html, 'html.parser')

  1. QuestionBeautifulSoup
  2. Questionfind_all()
  3. Put in orderImporting CSV and JSON
  4. Coding taskScraping
  5. Coding taskExtracting Links
  6. Put in orderWeb Scraping Flow
  7. Click in orderBeautifulSoup

Useful articles