Web scraping - collecting data from the web

Welcome back, @name! Darwin here with a fascinating lesson.

You've learned how to fetch data from REST APIs - elegant, structural interfaces. But what if a website doesn't have an API? What if species data is hidden in an HTML table on a webpage?

Then we use web scraping - automatically downloading and extracting data from websites!

Safari Analogy: A REST API is like an official report from a national park in JSON format. Web scraping is like reading a tourist brochure and manually extracting information: "Ah, it says here there are 120 lions, and here that the habitat is savanna"!

What is web scraping?

Web scraping is the automatic extraction of data from websites by:

1. Downloading the page's HTML (like a browser)
2. Parsing the HTML structure (DOM)
3. Finding elements of interest (tables, lists, headings)
4. Extracting data (text, links, images)

When to use web scraping?

Use when:

• The website has no API
• The API is paid, but the data is public
• You need historical data (archives)
• Aggregating data from multiple sources

Avoid when:

• The site has an official API (use the API!)
• Terms of service prohibit scraping
• Data is legally protected
• You might violate GDPR/privacy

Legality and ethics

IMPORTANT: Web scraping is legal when:

• You scrape public data (accessible without login)
• You respect robots.txt (

  https://example.com/robots.txt

  )
• You don't overload the server (rate limiting!)
• You don't violate copyright

Bad practices:

• Ignoring robots.txt
• Thousands of requests per second (DDoS)
• Scraping personal data
• Breaking login/paywall

Golden rule: Behave like a human - visit the site reasonably, don't overload the server!

BeautifulSoup4 - HTML parser in Python

BeautifulSoup4 (bs4) is the most popular library for parsing HTML/XML in Python.

Installation

1pip install beautifulsoup4
2pip install requests  # For downloading HTML
3pip install lxml      # Fast parser (optional)

Basic usage

1import requests
2from bs4 import BeautifulSoup
3
4# 1. Download HTML
5response = requests.get("https://example.com/species")
6html_content = response.text
7
8# 2. Parse HTML
9soup = BeautifulSoup(html_content, "html.parser")  # or "lxml"
10
11# 3. Find elements
12title = soup.find("h1")
13print(title.text)  # Heading text
14
15# 4. Find all elements
16species_items = soup.find_all("div", class_="species-card")
17for item in species_items:
18    name = item.find("span", class_="name").text
19    print(name)

HTML Anatomy - what are we parsing?

HTML is a tree of elements (tags):

1<div class="species-card" id="lion" data-population="120">
2    <h2>Lion</h2>
3    <p class="description">Panthera leo</p>
4    <img src="/images/lion.jpg" alt="Lion">
5    <a href="/species/lion">More information</a>
6</div>

Elements to extract:

• Text:

  <h2>Lion</h2>

  -> "Lion"
• Attributes:

  class="species-card"

  ,

  id="lion"

• Links:

  <a href="/species/lion">

• Images:

  <img src="/images/lion.jpg">

• Data:

  data-population="120"

Finding elements - selectors

find()

- first matching element

1from bs4 import BeautifulSoup
2
3html = """
4<div class="container">
5    <h1>Safari Data</h1>
6    <div class="species">Lion</div>
7    <div class="species">Elephant</div>
8</div>
9"""
10
11soup = BeautifulSoup(html, "html.parser")
12
13# Find first element
14title = soup.find("h1")
15print(title.text)  # "Safari Data"
16
17# Find first element with class
18species = soup.find("div", class_="species")
19print(species.text)  # "Lion" (first one!)

find_all()

- all matching elements

1# Find all elements with class "species"
2all_species = soup.find_all("div", class_="species")
3print(len(all_species))  # 2
4
5for species in all_species:
6    print(species.text)
7# Lion
8# Elephant
9
10# Limit results
11first_two = soup.find_all("div", limit=2)

CSS selector -

select()

1# CSS selectors (like in CSS!)
2all_species = soup.select(".species")  # Class
3title = soup.select_one("#main-title")  # ID
4links = soup.select("div.container a")  # Nested

CSS selectors:

• .species

  - class

• #lion

  - ID

• div.species

  - tag + class

• div > p

  - direct child

• div p

  - any nesting level

• [data-population]

  - attribute

Data extraction

Element text

1html = '<div class="name">Lion <span>(Panthera leo)</span></div>'
2soup = BeautifulSoup(html, "html.parser")
3
4# .text - all text (including nested tags)
5print(soup.find("div").text)  # "Lion (Panthera leo)"
6
7# .get_text() - with separator
8print(soup.find("div").get_text(strip=True))  # "Lion(Panthera leo)" (no spaces)
9
10# .string - only direct text (None if nested)
11print(soup.find("div").string)  # None (because of <span>)

Element attributes

1html = '<img src="/lion.jpg" alt="Lion" data-population="120">'
2soup = BeautifulSoup(html, "html.parser")
3img = soup.find("img")
4
5# Access like a dictionary
6print(img["src"])              # "/lion.jpg"
7print(img["alt"])              # "Lion"
8print(img["data-population"])  # "120"
9
10# .get() - safe (returns None if missing)
11print(img.get("title"))        # None
12print(img.get("src", "default.jpg"))  # "/lion.jpg"
13
14# All attributes
15print(img.attrs)  # {'src': '/lion.jpg', 'alt': 'Lion', 'data-population': '120'}

Links and URLs

1html = '<a href="/species/lion" class="link">See the lion</a>'
2soup = BeautifulSoup(html, "html.parser")
3link = soup.find("a")
4
5# URL
6print(link["href"])  # "/species/lion"
7
8# Link text
9print(link.text)     # "See the lion"
10
11# All links on the page
12for link in soup.find_all("a"):
13    print(f"{link.text}: {link.get('href')}")

Navigating the DOM tree

1html = """
2<div class="parent">
3    <h2>Title</h2>
4    <p class="child1">Paragraph 1</p>
5    <p class="child2">Paragraph 2</p>
6</div>
7"""
8soup = BeautifulSoup(html, "html.parser")
9
10# Children
11parent = soup.find("div", class_="parent")
12for child in parent.children:
13    if child.name:  # Skip text nodes
14        print(child.name)  # h2, p, p
15
16# Descendants - all nested elements
17for descendant in parent.descendants:
18    if hasattr(descendant, 'name') and descendant.name:
19        print(descendant.name)
20
21# Parent
22p = soup.find("p")
23print(p.parent.name)  # "div"
24
25# Siblings
26h2 = soup.find("h2")
27for sibling in h2.next_siblings:
28    if sibling.name:
29        print(sibling.text)  # Paragraph 1, Paragraph 2

Safari example - scraping species data

1import requests
2from bs4 import BeautifulSoup
3from typing import List, Dict
4import time
5
6class SafariWebScraper:
7    """Scraper for species data from a Safari website"""
8
9    def __init__(self, base_url: str):
10        self.base_url = base_url
11        self.session = requests.Session()
12        self.session.headers.update({
13            "User-Agent": "Safari-Research-Bot/1.0 (Educational purposes)"
14        })
15
16    def get_soup(self, url: str) -> BeautifulSoup:
17        """Download and parse HTML"""
18        response = self.session.get(url, timeout=10)
19        response.raise_for_status()
20        return BeautifulSoup(response.text, "html.parser")
21
22    def scrape_species_list(self, url: str) -> List[Dict[str, str]]:
23        """
24        Scrape species list from an HTML table
25
26        Example structure:
27        <table class="species-table">
28            <tr>
29                <td class="name">Lion</td>
30                <td class="scientific">Panthera leo</td>
31                <td class="population">120</td>
32                <td class="habitat">Savanna</td>
33            </tr>
34        </table>
35        """
36        soup = self.get_soup(url)
37        species_list = []
38
39        # Find the table
40        table = soup.find("table", class_="species-table")
41        if not table:
42            print("Warning: Table not found")
43            return []
44
45        # Iterate over rows (skipping header)
46        rows = table.find_all("tr")[1:]  # [1:] skips header
47
48        for row in rows:
49            cells = row.find_all("td")
50            if len(cells) >= 4:
51                species = {
52                    "common_name": cells[0].text.strip(),
53                    "scientific_name": cells[1].text.strip(),
54                    "population": cells[2].text.strip(),
55                    "habitat": cells[3].text.strip()
56                }
57                species_list.append(species)
58
59        return species_list
60
61    def scrape_species_cards(self, url: str) -> List[Dict[str, str]]:
62        """
63        Scrape species from HTML cards
64
65        <div class="species-card">
66            <h3>Lion</h3>
67            <p class="scientific">Panthera leo</p>
68            <span class="population" data-count="120">120 individuals</span>
69            <img src="/images/lion.jpg" alt="Lion">
70            <a href="/species/lion">More</a>
71        </div>
72        """
73        soup = self.get_soup(url)
74        species_list = []
75
76        # Find all cards
77        cards = soup.find_all("div", class_="species-card")
78
79        for card in cards:
80            # Safe extraction (elements may be missing)
81            name_tag = card.find("h3")
82            scientific_tag = card.find("p", class_="scientific")
83            pop_tag = card.find("span", class_="population")
84            img_tag = card.find("img")
85            link_tag = card.find("a")
86
87            species = {
88                "common_name": name_tag.text.strip() if name_tag else "Unknown",
89                "scientific_name": scientific_tag.text.strip() if scientific_tag else "Unknown",
90                "population": pop_tag.get("data-count", "0") if pop_tag else "0",
91                "image_url": img_tag["src"] if img_tag else None,
92                "detail_url": link_tag["href"] if link_tag else None
93            }
94            species_list.append(species)
95
96        return species_list
97
98    def scrape_species_detail(self, url: str) -> Dict[str, any]:
99        """
100        Scrape species details
101
102        <div class="species-detail">
103            <h1>Lion</h1>
104            <div class="info">
105                <dl>
106                    <dt>Scientific name:</dt>
107                    <dd>Panthera leo</dd>
108                    <dt>Status:</dt>
109                    <dd class="endangered">Endangered</dd>
110                    <dt>Habitat:</dt>
111                    <dd>Savanna, steppe</dd>
112                </dl>
113            </div>
114            <div class="description">
115                <p>The lion is the largest cat in Africa...</p>
116            </div>
117        </div>
118        """
119        soup = self.get_soup(url)
120
121        # Name
122        title = soup.find("h1")
123        common_name = title.text.strip() if title else "Unknown"
124
125        # Definition list - dt/dd pairs
126        info = {}
127        dl = soup.find("dl")
128        if dl:
129            terms = dl.find_all("dt")
130            definitions = dl.find_all("dd")
131
132            for term, definition in zip(terms, definitions):
133                key = term.text.strip().replace(":", "").lower()
134                value = definition.text.strip()
135                info[key] = value
136
137        # Description
138        description_div = soup.find("div", class_="description")
139        description = description_div.get_text(strip=True) if description_div else ""
140
141        return {
142            "common_name": common_name,
143            "scientific_name": info.get("scientific name", "Unknown"),
144            "status": info.get("status", "Unknown"),
145            "habitat": info.get("habitat", "Unknown"),
146            "description": description
147        }
148
149    def scrape_with_pagination(self, base_url: str, max_pages: int = 5) -> List[Dict]:
150        """
151        Scrape multiple pages with pagination
152
153        URL pattern: /species?page=1, /species?page=2, ...
154        """
155        all_species = []
156
157        for page_num in range(1, max_pages + 1):
158            url = f"{base_url}?page={page_num}"
159            print(f"Scraping page {page_num}: {url}")
160
161            soup = self.get_soup(url)
162
163            # Check if there's a next page
164            next_button = soup.find("a", class_="next-page")
165            if not next_button:
166                print(f"Last page: {page_num}")
167                break
168
169            # Scrape data from the page
170            species = self.scrape_species_cards(url)
171            all_species.extend(species)
172
173            # Rate limiting - wait between requests!
174            time.sleep(1)  # 1 second between pages
175
176        return all_species
177
178
179# === DEMONSTRATION ===
180
181print("=== SAFARI WEB SCRAPER ===\n")
182
183scraper = SafariWebScraper(base_url="https://example-safari.com")
184
185# 1. Scrape list from table
186print("1. Scraping species table...")
187species_table = scraper.scrape_species_list("https://example-safari.com/species-table")
188print(f"   Found {len(species_table)} species")
189for species in species_table[:3]:
190    print(f"   - {species['common_name']} ({species['scientific_name']})")
191
192# 2. Scrape cards
193print("\n2. Scraping species cards...")
194species_cards = scraper.scrape_species_cards("https://example-safari.com/species")
195print(f"   Found {len(species_cards)} cards")
196
197# 3. Scrape details
198print("\n3. Scraping lion details...")
199lion_detail = scraper.scrape_species_detail("https://example-safari.com/species/lion")
200print(f"   Name: {lion_detail['common_name']}")
201print(f"   Status: {lion_detail['status']}")
202print(f"   Habitat: {lion_detail['habitat']}")
203
204# 4. Scrape with pagination
205print("\n4. Scraping with pagination...")
206all_species = scraper.scrape_with_pagination("https://example-safari.com/species", max_pages=3)
207print(f"   Total: {len(all_species)} species from 3 pages")
208
209print("\nScraping complete")

Selenium - scraping dynamic pages

Problem: Some pages load data via JavaScript - BeautifulSoup sees empty HTML!

Solution: Selenium - automates a real browser!

Installation

1pip install selenium
2# Download ChromeDriver: https://chromedriver.chromium.org/

Selenium basics

1from selenium import webdriver
2from selenium.webdriver.common.by import By
3from selenium.webdriver.support.ui import WebDriverWait
4from selenium.webdriver.support import expected_conditions as EC
5import time
6
7# Launch browser
8driver = webdriver.Chrome()  # Requires ChromeDriver
9
10try:
11    # Open page
12    driver.get("https://example.com/species")
13
14    # Wait for element to load (JavaScript!)
15    wait = WebDriverWait(driver, 10)
16    species_cards = wait.until(
17        EC.presence_of_all_elements_located((By.CLASS_NAME, "species-card"))
18    )
19
20    # Extract data
21    for card in species_cards:
22        name = card.find_element(By.TAG_NAME, "h3").text
23        population = card.find_element(By.CLASS_NAME, "population").text
24        print(f"{name}: {population}")
25
26    # Click "Load More" button
27    load_more = driver.find_element(By.ID, "load-more-btn")
28    load_more.click()
29    time.sleep(2)  # Wait for loading
30
31    # Scroll down (infinite scroll)
32    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
33    time.sleep(2)
34
35finally:
36    # Always close the browser!
37    driver.quit()

Headless mode - without GUI

1from selenium.webdriver.chrome.options import Options
2
3options = Options()
4options.add_argument("--headless")  # No browser window
5options.add_argument("--no-sandbox")
6options.add_argument("--disable-dev-shm-usage")
7
8driver = webdriver.Chrome(options=options)
9# Rest as above

Web Scraping Best Practices

1. Check robots.txt

1import requests
2
3response = requests.get("https://example.com/robots.txt")
4print(response.text)
5
6# Check if your scraper is allowed:
7# User-agent: *
8# Disallow: /admin/
9# Crawl-delay: 10  # Wait 10s between requests

2. Rate Limiting - don't overload the server!

1import time
2import random
3
4for url in urls:
5    scrape_page(url)
6
7    # Wait between requests
8    time.sleep(random.uniform(1, 3))  # 1-3 seconds randomly

3. User-Agent - identify yourself

1headers = {
2    "User-Agent": "MyResearchBot/1.0 (+https://mysite.com/bot-info; contact@mysite.com)"
3}
4response = requests.get(url, headers=headers)

4. Error handling

1from requests.exceptions import RequestException
2
3def safe_scrape(url):
4    try:
5        response = requests.get(url, timeout=10)
6        response.raise_for_status()
7        soup = BeautifulSoup(response.text, "html.parser")
8        return soup
9
10    except RequestException as e:
11        print(f"Error: {e}")
12        return None

5. Cache - don't download the same thing twice

1import pickle
2from pathlib import Path
3
4def get_cached_html(url, cache_dir="cache"):
5    Path(cache_dir).mkdir(exist_ok=True)
6    cache_file = Path(cache_dir) / f"{hash(url)}.pkl"
7
8    # Check cache
9    if cache_file.exists():
10        with open(cache_file, "rb") as f:
11            return pickle.load(f)
12
13    # Download and save
14    response = requests.get(url)
15    html = response.text
16
17    with open(cache_file, "wb") as f:
18        pickle.dump(html, f)
19
20    return html

Common pitfalls

1. HTML structure changes

Websites change their layout - your scraper stops working!

Solution: Use multiple selectors, error handling

1# Multiple attempts to find an element
2name = (
3    soup.find("h3", class_="species-name") or
4    soup.find("span", class_="name") or
5    soup.find("div", attrs={"data-type": "name"})
6)

2. Dynamic JavaScript

BeautifulSoup only sees the initial HTML, not the result of JS!

Solution: Selenium, or analyze the API (check Network in DevTools)

3. Anti-scraping mechanisms

• CAPTCHA
• Rate limiting (429 Too Many Requests)
• IP ban
• Required cookies/sessions

Solution: Follow the rules, don't attack, consider using the API

Summary

In this lesson you learned:

• What web scraping is and when to use it
• Legality and ethics of scraping (robots.txt, rate limiting)
• BeautifulSoup4:

  find()

  ,

  find_all()

  ,

  select()

• Extraction: text, attributes, links
• DOM navigation: children, parent, siblings
• Selenium for dynamic pages (JavaScript)
• Best practices: User-Agent, rate limiting, cache, error handling
• Common pitfalls and how to avoid them

Checkpoint

Before moving on:

• [ ] You can parse HTML with BeautifulSoup
• [ ] You know how to find elements:

  find()

  ,

  select()

• [ ] You can extract text and attributes
• [ ] You understand the difference between BeautifulSoup vs Selenium
• [ ] You know the ethics of web scraping

Safari Analogy: Web scraping is like reading tourist brochures about national parks and manually collecting data into a spreadsheet - time-consuming, but effective when there are no official reports (APIs)!

In the next lesson, Darwin will teach you SQL databases - how to store and manage large datasets in a structured way!