Python course ยท Module 4: Data, APIs and Databases
HTTP - the data exchange protocol
In this lesson14
Welcome back! Darwin here.
In the previous lesson you learned about data formats - how to store information (JSON, CSV, XML, YAML). Now it's time for HTTP - the protocol that lets you transmit that data over the internet!
Safari Analogy: Data formats are the language in which you record observations. HTTP is the courier system - the way you send those observations between research stations in different parts of Africa!
What is HTTP?
HTTP (HyperText Transfer Protocol) is a communication protocol - a set of rules by which computers exchange data over the internet.
How it works:
- Client (e.g., browser, Python app) sends a request
- Server (e.g., website, API) processes the request
- Server sends back a response with data or error information
[CLIENT] ---(HTTP REQUEST)---> [SERVER]
| |
| Processes
| |
| <---(HTTP RESPONSE)--- [SERVER]Safari Example: You send a request to the central database in Nairobi: "Give me the data on lions in Serengeti". The server responds: "Here are 120 lions in JSON format"!
HTTP Methods - types of requests
HTTP has different methods (sometimes called "verbs") - they specify WHAT you want to do:
| Method | Purpose | Safari Analogy |
|---|---|---|
| GET | Retrieve data | "Show me the species list" |
| POST | Create a new resource | "Add a new elephant observation" |
| PUT | Replace an entire resource | "Replace the entire lion profile with new data" |
| PATCH | Update part of a resource | "Change only the gorilla population" |
| DELETE | Delete a resource | "Remove an erroneous observation" |
Most important: GET and POST - used in 90% of cases!
GET - retrieving data
GET /api/species/lion HTTP/1.1
Host: safari-api.comCharacteristics:
- Does not modify data on the server
- Parameters in URL (
?name=lion&habitat=savanna) - Can be bookmarked
- Safe to call multiple times
POST - creating/sending data
POST /api/observations HTTP/1.1
Host: safari-api.com
Content-Type: application/json
{
"species": "Panthera leo",
"count": 12,
"location": "Serengeti"
}Characteristics:
- Sends data in the request body, not in the URL
- Can modify data on the server
- Not saved in browser history
- Can create new resources
HTTP Status Codes - server responses
The server always responds with a status code - a number indicating whether the operation succeeded:
2xx - Success
- 200 OK - request successful, here's the data
- 201 Created - resource was created
- 204 No Content - success, but no data to return
3xx - Redirection
- 301 Moved Permanently - resource moved permanently
- 302 Found - temporary redirect
4xx - Client Error
- 400 Bad Request - malformed request (e.g., bad JSON)
- 401 Unauthorized - authentication required
- 403 Forbidden - no access
- 404 Not Found - resource doesn't exist
- 429 Too Many Requests - too many requests
5xx - Server Error
- 500 Internal Server Error - server error
- 502 Bad Gateway - proxy server problem
- 503 Service Unavailable - server unavailable
Safari Analogy: Code 200 = "Found the lion!", 404 = "There's no lion here", 500 = "My radio broke down"
Python requests - HTTP library
requests is the most popular Python library for HTTP - simple, elegant, powerful!
Installation
pip install requestsBasic usage - GET request
import requests
# Fetch data
response = requests.get("https://api.github.com/users/octocat")
# Status code
print(response.status_code) # 200
# Data as JSON
data = response.json()
print(data["login"]) # "octocat"
print(data["name"]) # "The Octocat"
# Data as text
print(response.text) # Raw JSON as string
# Response headers
print(response.headers["Content-Type"]) # "application/json"Checking for success
import requests
response = requests.get("https://api.github.com/users/nonexistent")
# Method 1: Check status code
if response.status_code == 200:
print("Success!")
elif response.status_code == 404:
print("Not found")
# Method 2: Use response.ok (True for 2xx)
if response.ok:
data = response.json()
print(data)
else:
print(f"Error: {response.status_code}")
# Method 3: Raise exception on error
try:
response.raise_for_status() # Raises HTTPError for 4xx/5xx
data = response.json()
except requests.HTTPError as e:
print(f"HTTP Error: {e}")URL Parameters - query parameters
GET requests often use parameters in the URL:
import requests
# Manually in URL
response = requests.get("https://api.example.com/species?habitat=savanna&endangered=true")
# Better: params dictionary (automatic encoding)
params = {
"habitat": "savanna",
"endangered": "true",
"min_population": 100
}
response = requests.get("https://api.example.com/species", params=params)
# URL: https://api.example.com/species?habitat=savanna&endangered=true&min_population=100
print(response.url) # See the generated URLHeaders
Headers are request metadata - information about the client, expected format, authorization:
import requests
# Custom headers
headers = {
"User-Agent": "Safari-Data-Collector/1.0",
"Accept": "application/json",
"Authorization": "Bearer YOUR_API_KEY_HERE"
}
response = requests.get("https://api.example.com/species", headers=headers)
# Check response headers
print(response.headers["Content-Type"])
print(response.headers["Date"])Common headers:
User-Agent- client identificationAccept- expected response formatContent-Type- format of sent dataAuthorization- token/API key
POST request - sending data
JSON payload
import requests
# Data to send
new_observation = {
"species": "Panthera leo",
"location": "Serengeti",
"count": 12,
"date": "2024-01-15",
"observer": "Darwin"
}
# POST with JSON
response = requests.post(
"https://api.safari.com/observations",
json=new_observation # Automatically: Content-Type: application/json
)
if response.status_code == 201: # Created
created_data = response.json()
print(f"Created observation ID: {created_data['id']}")
else:
print(f"Error: {response.status_code}")Form data (application/x-www-form-urlencoded)
import requests
# Form data
form_data = {
"species": "Loxodonta africana",
"count": "35"
}
response = requests.post(
"https://api.safari.com/quick-report",
data=form_data # Content-Type: application/x-www-form-urlencoded
)PUT, PATCH, DELETE
import requests
# PUT - replace entire resource
updated_species = {
"scientific_name": "Panthera leo",
"population": 125, # Updated
"habitat": "savanna"
}
response = requests.put(
"https://api.safari.com/species/lion",
json=updated_species
)
# PATCH - change only part
partial_update = {
"population": 125 # Only population
}
response = requests.patch(
"https://api.safari.com/species/lion",
json=partial_update
)
# DELETE - remove resource
response = requests.delete("https://api.safari.com/observations/12345")
if response.status_code == 204: # No Content
print("Successfully deleted")Error handling
import requests
from requests.exceptions import RequestException, Timeout, ConnectionError
def fetch_species_data(species_id: str):
"""Fetch species data with error handling"""
try:
response = requests.get(
f"https://api.safari.com/species/{species_id}",
timeout=5 # Timeout after 5 seconds
)
# Check status
response.raise_for_status()
# Process data
return response.json()
except Timeout:
print("Timeout - server did not respond in time")
return None
except ConnectionError:
print("Connection error - check your internet")
return None
except requests.HTTPError as e:
print(f"HTTP Error {response.status_code}: {e}")
return None
except RequestException as e:
print(f"Request error: {e}")
return None
# Usage
data = fetch_species_data("lion")
if data:
print(f"Species: {data['name']}")Safari example - API client for observation management
import requests
from typing import List, Dict, Optional
from datetime import datetime
class SafariAPIClient:
"""API client for the Safari observation system"""
def __init__(self, base_url: str, api_key: str):
self.base_url = base_url.rstrip("/")
self.api_key = api_key
self.session = requests.Session() # Reusable connection
# Default headers for all requests
self.session.headers.update({
"User-Agent": "Safari-Data-Collector/2.0",
"Authorization": f"Bearer {self.api_key}",
"Accept": "application/json"
})
def get_species(self, species_id: str) -> Optional[Dict]:
"""Fetch species data"""
try:
response = self.session.get(
f"{self.base_url}/species/{species_id}",
timeout=10
)
response.raise_for_status()
return response.json()
except requests.RequestException as e:
print(f"Error fetching species: {e}")
return None
def search_species(self, habitat: Optional[str] = None,
endangered: Optional[bool] = None,
min_population: Optional[int] = None) -> List[Dict]:
"""Search species with filters"""
params = {}
if habitat:
params["habitat"] = habitat
if endangered is not None:
params["endangered"] = str(endangered).lower()
if min_population:
params["min_population"] = min_population
try:
response = self.session.get(
f"{self.base_url}/species",
params=params,
timeout=10
)
response.raise_for_status()
return response.json()
except requests.RequestException as e:
print(f"Search error: {e}")
return []
def create_observation(self, species: str, location: str,
count: int, notes: str = "") -> Optional[Dict]:
"""Create a new observation"""
observation_data = {
"species": species,
"location": location,
"count": count,
"date": datetime.now().isoformat(),
"notes": notes
}
try:
response = self.session.post(
f"{self.base_url}/observations",
json=observation_data,
timeout=10
)
response.raise_for_status()
if response.status_code == 201:
created = response.json()
print(f"Created observation ID: {created.get('id')}")
return created
else:
return response.json()
except requests.RequestException as e:
print(f"Error creating observation: {e}")
return None
def update_species_population(self, species_id: str,
new_population: int) -> bool:
"""Update species population (PATCH)"""
try:
response = self.session.patch(
f"{self.base_url}/species/{species_id}",
json={"population": new_population},
timeout=10
)
response.raise_for_status()
print(f"Updated population {species_id} -> {new_population}")
return True
except requests.RequestException as e:
print(f"Update error: {e}")
return False
def delete_observation(self, observation_id: str) -> bool:
"""Delete an observation"""
try:
response = self.session.delete(
f"{self.base_url}/observations/{observation_id}",
timeout=10
)
response.raise_for_status()
if response.status_code == 204:
print(f"Deleted observation {observation_id}")
return True
return False
except requests.RequestException as e:
print(f"Delete error: {e}")
return False
def get_statistics(self) -> Optional[Dict]:
"""Fetch statistics (with query parameters)"""
params = {
"include_endangered": "true",
"group_by": "habitat"
}
try:
response = self.session.get(
f"{self.base_url}/statistics",
params=params,
timeout=15
)
response.raise_for_status()
return response.json()
except requests.RequestException as e:
print(f"Error fetching statistics: {e}")
return None
def close(self):
"""Close session"""
self.session.close()
# === DEMONSTRATION ===
print("=== SAFARI API CLIENT ===\n")
# Initialize client
client = SafariAPIClient(
base_url="https://api.safari-data.org/v1",
api_key="demo_key_12345"
)
# GET - fetch species
print("1. Fetching lion data...")
lion_data = client.get_species("lion")
if lion_data:
print(f" Species: {lion_data.get('common_name')}")
print(f" Population: {lion_data.get('population')}")
# GET with parameters - search
print("\n2. Searching for endangered species in the savanna...")
endangered = client.search_species(habitat="savanna", endangered=True)
print(f" Found {len(endangered)} species")
# POST - create observation
print("\n3. Creating a new observation...")
new_obs = client.create_observation(
species="Panthera leo",
location="Serengeti North",
count=8,
notes="Pride with 2 cubs"
)
# PATCH - update
print("\n4. Updating population...")
client.update_species_population("lion", 125)
# DELETE - remove
print("\n5. Deleting erroneous observation...")
client.delete_observation("obs_12345")
# GET with statistics
print("\n6. Fetching statistics...")
stats = client.get_statistics()
if stats:
print(f" Total species: {stats.get('total_species')}")
# Close session
client.close()
print("\nSession closed")Session - efficiency for multiple requests
If you're making many requests to the same API, use Session - it reuses the TCP connection:
import requests
# WITHOUT Session - each request = new connection
for i in range(100):
response = requests.get("https://api.example.com/data")
# WITH Session - one connection, faster!
session = requests.Session()
session.headers.update({"Authorization": "Bearer token123"})
for i in range(100):
response = session.get("https://api.example.com/data")
session.close()Timeouts - don't wait forever
Always use timeout so a request doesn't hang indefinitely:
import requests
# 5-second timeout
response = requests.get("https://slow-api.com/data", timeout=5)
# Different timeouts: (connect_timeout, read_timeout)
response = requests.get("https://api.com/data", timeout=(3, 10))
# 3s to connect, 10s to read dataSummary
In this lesson you learned:
- What HTTP is and how it works (client -> server -> response)
- HTTP methods: GET, POST, PUT, PATCH, DELETE
- Status codes: 200 OK, 404 Not Found, 500 Server Error
- The
requestslibrary:requests.get(),requests.post() - URL parameters (
params=), headers (headers=) - Sending JSON (
json=) and form data (data=) - Error handling:
response.raise_for_status(), try/except - Session for efficiency
- Timeouts for safety
Checkpoint
Before moving on:
- You understand the difference between GET and POST
- You can send a GET request with parameters
- You can send a POST with JSON
- You know how to handle HTTP errors
- You understand status codes (2xx, 4xx, 5xx)
Safari Analogy: HTTP is a courier system between research stations. GET = "send me data", POST = "save these new observations", status codes = delivery confirmations!
In the next lesson, Darwin will show you REST API - API architecture standards that allow you to build professional data exchange systems!
Docs entries used in this lesson
Spotted a mistake in this lesson?
Check yourself
Answer the questions from this lesson. Pick an answer to see right away whether it is correct.
1. Which HTTP method is used to retrieve data?
2. How do you make a GET request using the requests library?
These are 2 of 4 questions for this lesson. Solve the rest in the game.
Hands-on tasks in the game
- Horizontal ordering
Arrange the elements in the correct order:
- Click in order
Click the elements in the correct order:
- Code editor
Fetch data from URL 'https://api.example.com/animals' using requests
- Code editor
Perform a GET request and parse the response as JSON
- Click in order
Click in order: response = requests.get(url)