Python course ยท Module 4: Data, APIs and Databases
Web scraping - collecting data from the web
In this lesson12
Welcome back! Darwin here with a fascinating lesson.
You've learned how to fetch data from REST APIs - elegant, structural interfaces. But what if a website doesn't have an API? What if species data is hidden in an HTML table on a webpage?
Then we use web scraping - automatically downloading and extracting data from websites!
Safari Analogy: A REST API is like an official report from a national park in JSON format. Web scraping is like reading a tourist brochure and manually extracting information: "Ah, it says here there are 120 lions, and here that the habitat is savanna"!
What is web scraping?
Web scraping is the automatic extraction of data from websites by:
- Downloading the page's HTML (like a browser)
- Parsing the HTML structure (DOM)
- Finding elements of interest (tables, lists, headings)
- Extracting data (text, links, images)
When to use web scraping?
Use when:
- The website has no API
- The API is paid, but the data is public
- You need historical data (archives)
- Aggregating data from multiple sources
Avoid when:
- The site has an official API (use the API!)
- Terms of service prohibit scraping
- Data is legally protected
- You might violate GDPR/privacy
Legality and ethics
IMPORTANT: Web scraping is legal when:
- You scrape public data (accessible without login)
- You respect robots.txt (
https://example.com/robots.txt) - You don't overload the server (rate limiting!)
- You don't violate copyright
Bad practices:
- Ignoring robots.txt
- Thousands of requests per second (DDoS)
- Scraping personal data
- Breaking login/paywall
Golden rule: Behave like a human - visit the site reasonably, don't overload the server!
BeautifulSoup4 - HTML parser in Python
BeautifulSoup4 (bs4) is the most popular library for parsing HTML/XML in Python.
Installation
pip install beautifulsoup4
pip install requests # For downloading HTML
pip install lxml # Fast parser (optional)Basic usage
import requests
from bs4 import BeautifulSoup
# 1. Download HTML
response = requests.get("https://example.com/species")
html_content = response.text
# 2. Parse HTML
soup = BeautifulSoup(html_content, "html.parser") # or "lxml"
# 3. Find elements
title = soup.find("h1")
print(title.text) # Heading text
# 4. Find all elements
species_items = soup.find_all("div", class_="species-card")
for item in species_items:
name = item.find("span", class_="name").text
print(name)HTML Anatomy - what are we parsing?
HTML is a tree of elements (tags):
<div class="species-card" id="lion" data-population="120">
<h2>Lion</h2>
<p class="description">Panthera leo</p>
<img src="/images/lion.jpg" alt="Lion">
<a href="/species/lion">More information</a>
</div>Elements to extract:
- Text:
<h2>Lion</h2>-> "Lion" - Attributes:
class="species-card",id="lion" - Links:
<a href="/species/lion"> - Images:
<img src="/images/lion.jpg"> - Data:
data-population="120"
Finding elements - selectors
find() - first matching element
from bs4 import BeautifulSoup
html = """
<div class="container">
<h1>Safari Data</h1>
<div class="species">Lion</div>
<div class="species">Elephant</div>
</div>
"""
soup = BeautifulSoup(html, "html.parser")
# Find first element
title = soup.find("h1")
print(title.text) # "Safari Data"
# Find first element with class
species = soup.find("div", class_="species")
print(species.text) # "Lion" (first one!)find_all() - all matching elements
# Find all elements with class "species"
all_species = soup.find_all("div", class_="species")
print(len(all_species)) # 2
for species in all_species:
print(species.text)
# Lion
# Elephant
# Limit results
first_two = soup.find_all("div", limit=2)CSS selector - select()
# CSS selectors (like in CSS!)
all_species = soup.select(".species") # Class
title = soup.select_one("#main-title") # ID
links = soup.select("div.container a") # NestedCSS selectors:
.species- class#lion- IDdiv.species- tag + classdiv > p- direct childdiv p- any nesting level[data-population]- attribute
Data extraction
Element text
html = '<div class="name">Lion <span>(Panthera leo)</span></div>'
soup = BeautifulSoup(html, "html.parser")
# .text - all text (including nested tags)
print(soup.find("div").text) # "Lion (Panthera leo)"
# .get_text() - with separator
print(soup.find("div").get_text(strip=True)) # "Lion(Panthera leo)" (no spaces)
# .string - only direct text (None if nested)
print(soup.find("div").string) # None (because of <span>)Element attributes
html = '<img src="/lion.jpg" alt="Lion" data-population="120">'
soup = BeautifulSoup(html, "html.parser")
img = soup.find("img")
# Access like a dictionary
print(img["src"]) # "/lion.jpg"
print(img["alt"]) # "Lion"
print(img["data-population"]) # "120"
# .get() - safe (returns None if missing)
print(img.get("title")) # None
print(img.get("src", "default.jpg")) # "/lion.jpg"
# All attributes
print(img.attrs) # {'src': '/lion.jpg', 'alt': 'Lion', 'data-population': '120'}Links and URLs
html = '<a href="/species/lion" class="link">See the lion</a>'
soup = BeautifulSoup(html, "html.parser")
link = soup.find("a")
# URL
print(link["href"]) # "/species/lion"
# Link text
print(link.text) # "See the lion"
# All links on the page
for link in soup.find_all("a"):
print(f"{link.text}: {link.get('href')}")Navigating the DOM tree
html = """
<div class="parent">
<h2>Title</h2>
<p class="child1">Paragraph 1</p>
<p class="child2">Paragraph 2</p>
</div>
"""
soup = BeautifulSoup(html, "html.parser")
# Children
parent = soup.find("div", class_="parent")
for child in parent.children:
if child.name: # Skip text nodes
print(child.name) # h2, p, p
# Descendants - all nested elements
for descendant in parent.descendants:
if hasattr(descendant, 'name') and descendant.name:
print(descendant.name)
# Parent
p = soup.find("p")
print(p.parent.name) # "div"
# Siblings
h2 = soup.find("h2")
for sibling in h2.next_siblings:
if sibling.name:
print(sibling.text) # Paragraph 1, Paragraph 2Safari example - scraping species data
import requests
from bs4 import BeautifulSoup
from typing import List, Dict
import time
class SafariWebScraper:
"""Scraper for species data from a Safari website"""
def __init__(self, base_url: str):
self.base_url = base_url
self.session = requests.Session()
self.session.headers.update({
"User-Agent": "Safari-Research-Bot/1.0 (Educational purposes)"
})
def get_soup(self, url: str) -> BeautifulSoup:
"""Download and parse HTML"""
response = self.session.get(url, timeout=10)
response.raise_for_status()
return BeautifulSoup(response.text, "html.parser")
def scrape_species_list(self, url: str) -> List[Dict[str, str]]:
"""
Scrape species list from an HTML table
Example structure:
<table class="species-table">
<tr>
<td class="name">Lion</td>
<td class="scientific">Panthera leo</td>
<td class="population">120</td>
<td class="habitat">Savanna</td>
</tr>
</table>
"""
soup = self.get_soup(url)
species_list = []
# Find the table
table = soup.find("table", class_="species-table")
if not table:
print("Warning: Table not found")
return []
# Iterate over rows (skipping header)
rows = table.find_all("tr")[1:] # [1:] skips header
for row in rows:
cells = row.find_all("td")
if len(cells) >= 4:
species = {
"common_name": cells[0].text.strip(),
"scientific_name": cells[1].text.strip(),
"population": cells[2].text.strip(),
"habitat": cells[3].text.strip()
}
species_list.append(species)
return species_list
def scrape_species_cards(self, url: str) -> List[Dict[str, str]]:
"""
Scrape species from HTML cards
<div class="species-card">
<h3>Lion</h3>
<p class="scientific">Panthera leo</p>
<span class="population" data-count="120">120 individuals</span>
<img src="/images/lion.jpg" alt="Lion">
<a href="/species/lion">More</a>
</div>
"""
soup = self.get_soup(url)
species_list = []
# Find all cards
cards = soup.find_all("div", class_="species-card")
for card in cards:
# Safe extraction (elements may be missing)
name_tag = card.find("h3")
scientific_tag = card.find("p", class_="scientific")
pop_tag = card.find("span", class_="population")
img_tag = card.find("img")
link_tag = card.find("a")
species = {
"common_name": name_tag.text.strip() if name_tag else "Unknown",
"scientific_name": scientific_tag.text.strip() if scientific_tag else "Unknown",
"population": pop_tag.get("data-count", "0") if pop_tag else "0",
"image_url": img_tag["src"] if img_tag else None,
"detail_url": link_tag["href"] if link_tag else None
}
species_list.append(species)
return species_list
def scrape_species_detail(self, url: str) -> Dict[str, any]:
"""
Scrape species details
<div class="species-detail">
<h1>Lion</h1>
<div class="info">
<dl>
<dt>Scientific name:</dt>
<dd>Panthera leo</dd>
<dt>Status:</dt>
<dd class="endangered">Endangered</dd>
<dt>Habitat:</dt>
<dd>Savanna, steppe</dd>
</dl>
</div>
<div class="description">
<p>The lion is the largest cat in Africa...</p>
</div>
</div>
"""
soup = self.get_soup(url)
# Name
title = soup.find("h1")
common_name = title.text.strip() if title else "Unknown"
# Definition list - dt/dd pairs
info = {}
dl = soup.find("dl")
if dl:
terms = dl.find_all("dt")
definitions = dl.find_all("dd")
for term, definition in zip(terms, definitions):
key = term.text.strip().replace(":", "").lower()
value = definition.text.strip()
info[key] = value
# Description
description_div = soup.find("div", class_="description")
description = description_div.get_text(strip=True) if description_div else ""
return {
"common_name": common_name,
"scientific_name": info.get("scientific name", "Unknown"),
"status": info.get("status", "Unknown"),
"habitat": info.get("habitat", "Unknown"),
"description": description
}
def scrape_with_pagination(self, base_url: str, max_pages: int = 5) -> List[Dict]:
"""
Scrape multiple pages with pagination
URL pattern: /species?page=1, /species?page=2, ...
"""
all_species = []
for page_num in range(1, max_pages + 1):
url = f"{base_url}?page={page_num}"
print(f"Scraping page {page_num}: {url}")
soup = self.get_soup(url)
# Check if there's a next page
next_button = soup.find("a", class_="next-page")
if not next_button:
print(f"Last page: {page_num}")
break
# Scrape data from the page
species = self.scrape_species_cards(url)
all_species.extend(species)
# Rate limiting - wait between requests!
time.sleep(1) # 1 second between pages
return all_species
# === DEMONSTRATION ===
print("=== SAFARI WEB SCRAPER ===\n")
scraper = SafariWebScraper(base_url="https://example-safari.com")
# 1. Scrape list from table
print("1. Scraping species table...")
species_table = scraper.scrape_species_list("https://example-safari.com/species-table")
print(f" Found {len(species_table)} species")
for species in species_table[:3]:
print(f" - {species['common_name']} ({species['scientific_name']})")
# 2. Scrape cards
print("\n2. Scraping species cards...")
species_cards = scraper.scrape_species_cards("https://example-safari.com/species")
print(f" Found {len(species_cards)} cards")
# 3. Scrape details
print("\n3. Scraping lion details...")
lion_detail = scraper.scrape_species_detail("https://example-safari.com/species/lion")
print(f" Name: {lion_detail['common_name']}")
print(f" Status: {lion_detail['status']}")
print(f" Habitat: {lion_detail['habitat']}")
# 4. Scrape with pagination
print("\n4. Scraping with pagination...")
all_species = scraper.scrape_with_pagination("https://example-safari.com/species", max_pages=3)
print(f" Total: {len(all_species)} species from 3 pages")
print("\nScraping complete")Selenium - scraping dynamic pages
Problem: Some pages load data via JavaScript - BeautifulSoup sees empty HTML!
Solution: Selenium - automates a real browser!
Installation
pip install selenium
# Download ChromeDriver: https://chromedriver.chromium.org/Selenium basics
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
# Launch browser
driver = webdriver.Chrome() # Requires ChromeDriver
try:
# Open page
driver.get("https://example.com/species")
# Wait for element to load (JavaScript!)
wait = WebDriverWait(driver, 10)
species_cards = wait.until(
EC.presence_of_all_elements_located((By.CLASS_NAME, "species-card"))
)
# Extract data
for card in species_cards:
name = card.find_element(By.TAG_NAME, "h3").text
population = card.find_element(By.CLASS_NAME, "population").text
print(f"{name}: {population}")
# Click "Load More" button
load_more = driver.find_element(By.ID, "load-more-btn")
load_more.click()
time.sleep(2) # Wait for loading
# Scroll down (infinite scroll)
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(2)
finally:
# Always close the browser!
driver.quit()Headless mode - without GUI
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("--headless") # No browser window
options.add_argument("--no-sandbox")
options.add_argument("--disable-dev-shm-usage")
driver = webdriver.Chrome(options=options)
# Rest as aboveWeb Scraping Best Practices
1. Check robots.txt
import requests
response = requests.get("https://example.com/robots.txt")
print(response.text)
# Check if your scraper is allowed:
# User-agent: *
# Disallow: /admin/
# Crawl-delay: 10 # Wait 10s between requests2. Rate Limiting - don't overload the server!
import time
import random
for url in urls:
scrape_page(url)
# Wait between requests
time.sleep(random.uniform(1, 3)) # 1-3 seconds randomly3. User-Agent - identify yourself
headers = {
"User-Agent": "MyResearchBot/1.0 (+https://mysite.com/bot-info; contact@mysite.com)"
}
response = requests.get(url, headers=headers)4. Error handling
from requests.exceptions import RequestException
def safe_scrape(url):
try:
response = requests.get(url, timeout=10)
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")
return soup
except RequestException as e:
print(f"Error: {e}")
return None5. Cache - don't download the same thing twice
import pickle
from pathlib import Path
def get_cached_html(url, cache_dir="cache"):
Path(cache_dir).mkdir(exist_ok=True)
cache_file = Path(cache_dir) / f"{hash(url)}.pkl"
# Check cache
if cache_file.exists():
with open(cache_file, "rb") as f:
return pickle.load(f)
# Download and save
response = requests.get(url)
html = response.text
with open(cache_file, "wb") as f:
pickle.dump(html, f)
return htmlCommon pitfalls
1. HTML structure changes
Websites change their layout - your scraper stops working!
Solution: Use multiple selectors, error handling
# Multiple attempts to find an element
name = (
soup.find("h3", class_="species-name") or
soup.find("span", class_="name") or
soup.find("div", attrs={"data-type": "name"})
)2. Dynamic JavaScript
BeautifulSoup only sees the initial HTML, not the result of JS!
Solution: Selenium, or analyze the API (check Network in DevTools)
3. Anti-scraping mechanisms
- CAPTCHA
- Rate limiting (429 Too Many Requests)
- IP ban
- Required cookies/sessions
Solution: Follow the rules, don't attack, consider using the API
Summary
In this lesson you learned:
- What web scraping is and when to use it
- Legality and ethics of scraping (robots.txt, rate limiting)
- BeautifulSoup4:
find(),find_all(),select() - Extraction: text, attributes, links
- DOM navigation: children, parent, siblings
- Selenium for dynamic pages (JavaScript)
- Best practices: User-Agent, rate limiting, cache, error handling
- Common pitfalls and how to avoid them
Checkpoint
Before moving on:
- You can parse HTML with BeautifulSoup
- You know how to find elements:
find(),select() - You can extract text and attributes
- You understand the difference between BeautifulSoup vs Selenium
- You know the ethics of web scraping
Safari Analogy: Web scraping is like reading tourist brochures about national parks and manually collecting data into a spreadsheet - time-consuming, but effective when there are no official reports (APIs)!
In the next lesson, Darwin will teach you SQL databases - how to store and manage large datasets in a structured way!
Docs entries used in this lesson
Spotted a mistake in this lesson?
Check yourself
Answer the questions from this lesson. Pick an answer to see right away whether it is correct.
1. BeautifulSoup is used for:
2. The method soup.find_all('a') returns:
Hands-on tasks in the game
- Horizontal ordering
Arrange the elements in the correct order:
- Code editor
Fetch the page and find all <h1> elements
- Code editor
Find all links (href attribute) on the page
- Vertical ordering
Arrange the steps of web scraping:
- Click in order
Click in order: soup = BeautifulSoup(html, 'html.parser')