HTTP - the data exchange protocol

Welcome back! Darwin here.

In the previous lesson you learned about data formats - how to store information (JSON, CSV, XML, YAML). Now it's time for HTTP - the protocol that lets you transmit that data over the internet!

Safari Analogy: Data formats are the language in which you record observations. HTTP is the courier system - the way you send those observations between research stations in different parts of Africa!

What is HTTP?

HTTP (HyperText Transfer Protocol) is a communication protocol - a set of rules by which computers exchange data over the internet.

How it works:

1. Client (e.g., browser, Python app) sends a request
2. Server (e.g., website, API) processes the request
3. Server sends back a response with data or error information

1[CLIENT]  ---(HTTP REQUEST)--->  [SERVER]
2   |                                  |
3   |                              Processes
4   |                                  |
5   |  <---(HTTP RESPONSE)---  [SERVER]

Safari Example: You send a request to the central database in Nairobi: "Give me the data on lions in Serengeti". The server responds: "Here are 120 lions in JSON format"!

HTTP Methods - types of requests

HTTP has different methods (sometimes called "verbs") - they specify WHAT you want to do:

Most important: GET and POST - used in 90% of cases!

GET - retrieving data

1GET /api/species/lion HTTP/1.1
2Host: safari-api.com

Characteristics:

• Does not modify data on the server
• Parameters in URL (?name=lion&habitat=savanna)
• Can be bookmarked
• Safe to call multiple times

POST - creating/sending data

1POST /api/observations HTTP/1.1
2Host: safari-api.com
3Content-Type: application/json
4
5{
6  "species": "Panthera leo",
7  "count": 12,
8  "location": "Serengeti"
9}

Characteristics:

• Sends data in the request body, not in the URL
• Can modify data on the server
• Not saved in browser history
• Can create new resources

HTTP Status Codes - server responses

The server always responds with a status code - a number indicating whether the operation succeeded:

2xx - Success

• 200 OK - request successful, here's the data
• 201 Created - resource was created
• 204 No Content - success, but no data to return

3xx - Redirection

• 301 Moved Permanently - resource moved permanently
• 302 Found - temporary redirect

4xx - Client Error

• 400 Bad Request - malformed request (e.g., bad JSON)
• 401 Unauthorized - authentication required
• 403 Forbidden - no access
• 404 Not Found - resource doesn't exist
• 429 Too Many Requests - too many requests

5xx - Server Error

• 500 Internal Server Error - server error
• 502 Bad Gateway - proxy server problem
• 503 Service Unavailable - server unavailable

Safari Analogy: Code 200 = "Found the lion!", 404 = "There's no lion here", 500 = "My radio broke down"

Python requests - HTTP library

requests is the most popular Python library for HTTP - simple, elegant, powerful!

Installation

1pip install requests

Basic usage - GET request

1import requests
2
3# Fetch data
4response = requests.get("https://api.github.com/users/octocat")
5
6# Status code
7print(response.status_code)  # 200
8
9# Data as JSON
10data = response.json()
11print(data["login"])  # "octocat"
12print(data["name"])   # "The Octocat"
13
14# Data as text
15print(response.text)  # Raw JSON as string
16
17# Response headers
18print(response.headers["Content-Type"])  # "application/json"

Checking for success

1import requests
2
3response = requests.get("https://api.github.com/users/nonexistent")
4
5# Method 1: Check status code
6if response.status_code == 200:
7    print("Success!")
8elif response.status_code == 404:
9    print("Not found")
10
11# Method 2: Use response.ok (True for 2xx)
12if response.ok:
13    data = response.json()
14    print(data)
15else:
16    print(f"Error: {response.status_code}")
17
18# Method 3: Raise exception on error
19try:
20    response.raise_for_status()  # Raises HTTPError for 4xx/5xx
21    data = response.json()
22except requests.HTTPError as e:
23    print(f"HTTP Error: {e}")

URL Parameters - query parameters

GET requests often use parameters in the URL:

1import requests
2
3# Manually in URL
4response = requests.get("https://api.example.com/species?habitat=savanna&endangered=true")
5
6# Better: params dictionary (automatic encoding)
7params = {
8    "habitat": "savanna",
9    "endangered": "true",
10    "min_population": 100
11}
12
13response = requests.get("https://api.example.com/species", params=params)
14# URL: https://api.example.com/species?habitat=savanna&endangered=true&min_population=100
15
16print(response.url)  # See the generated URL

Headers

Headers are request metadata - information about the client, expected format, authorization:

1import requests
2
3# Custom headers
4headers = {
5    "User-Agent": "Safari-Data-Collector/1.0",
6    "Accept": "application/json",
7    "Authorization": "Bearer YOUR_API_KEY_HERE"
8}
9
10response = requests.get("https://api.example.com/species", headers=headers)
11
12# Check response headers
13print(response.headers["Content-Type"])
14print(response.headers["Date"])

Common headers:

• User-Agent - client identification
• Accept - expected response format
• Content-Type - format of sent data
• Authorization - token/API key

POST request - sending data

JSON payload

1import requests
2
3# Data to send
4new_observation = {
5    "species": "Panthera leo",
6    "location": "Serengeti",
7    "count": 12,
8    "date": "2024-01-15",
9    "observer": "Darwin"
10}
11
12# POST with JSON
13response = requests.post(
14    "https://api.safari.com/observations",
15    json=new_observation  # Automatically: Content-Type: application/json
16)
17
18if response.status_code == 201:  # Created
19    created_data = response.json()
20    print(f"Created observation ID: {created_data['id']}")
21else:
22    print(f"Error: {response.status_code}")

Form data (application/x-www-form-urlencoded)

1import requests
2
3# Form data
4form_data = {
5    "species": "Loxodonta africana",
6    "count": "35"
7}
8
9response = requests.post(
10    "https://api.safari.com/quick-report",
11    data=form_data  # Content-Type: application/x-www-form-urlencoded
12)

PUT, PATCH, DELETE

1import requests
2
3# PUT - replace entire resource
4updated_species = {
5    "scientific_name": "Panthera leo",
6    "population": 125,  # Updated
7    "habitat": "savanna"
8}
9
10response = requests.put(
11    "https://api.safari.com/species/lion",
12    json=updated_species
13)
14
15# PATCH - change only part
16partial_update = {
17    "population": 125  # Only population
18}
19
20response = requests.patch(
21    "https://api.safari.com/species/lion",
22    json=partial_update
23)
24
25# DELETE - remove resource
26response = requests.delete("https://api.safari.com/observations/12345")
27
28if response.status_code == 204:  # No Content
29    print("Successfully deleted")

Error handling

1import requests
2from requests.exceptions import RequestException, Timeout, ConnectionError
3
4def fetch_species_data(species_id: str):
5    """Fetch species data with error handling"""
6    try:
7        response = requests.get(
8            f"https://api.safari.com/species/{species_id}",
9            timeout=5  # Timeout after 5 seconds
10        )
11
12        # Check status
13        response.raise_for_status()
14
15        # Process data
16        return response.json()
17
18    except Timeout:
19        print("Timeout - server did not respond in time")
20        return None
21
22    except ConnectionError:
23        print("Connection error - check your internet")
24        return None
25
26    except requests.HTTPError as e:
27        print(f"HTTP Error {response.status_code}: {e}")
28        return None
29
30    except RequestException as e:
31        print(f"Request error: {e}")
32        return None
33
34# Usage
35data = fetch_species_data("lion")
36if data:
37    print(f"Species: {data['name']}")

Safari example - API client for observation management

1import requests
2from typing import List, Dict, Optional
3from datetime import datetime
4
5class SafariAPIClient:
6    """API client for the Safari observation system"""
7
8    def __init__(self, base_url: str, api_key: str):
9        self.base_url = base_url.rstrip("/")
10        self.api_key = api_key
11        self.session = requests.Session()  # Reusable connection
12
13        # Default headers for all requests
14        self.session.headers.update({
15            "User-Agent": "Safari-Data-Collector/2.0",
16            "Authorization": f"Bearer {self.api_key}",
17            "Accept": "application/json"
18        })
19
20    def get_species(self, species_id: str) -> Optional[Dict]:
21        """Fetch species data"""
22        try:
23            response = self.session.get(
24                f"{self.base_url}/species/{species_id}",
25                timeout=10
26            )
27            response.raise_for_status()
28            return response.json()
29
30        except requests.RequestException as e:
31            print(f"Error fetching species: {e}")
32            return None
33
34    def search_species(self, habitat: Optional[str] = None,
35                      endangered: Optional[bool] = None,
36                      min_population: Optional[int] = None) -> List[Dict]:
37        """Search species with filters"""
38        params = {}
39        if habitat:
40            params["habitat"] = habitat
41        if endangered is not None:
42            params["endangered"] = str(endangered).lower()
43        if min_population:
44            params["min_population"] = min_population
45
46        try:
47            response = self.session.get(
48                f"{self.base_url}/species",
49                params=params,
50                timeout=10
51            )
52            response.raise_for_status()
53            return response.json()
54
55        except requests.RequestException as e:
56            print(f"Search error: {e}")
57            return []
58
59    def create_observation(self, species: str, location: str,
60                          count: int, notes: str = "") -> Optional[Dict]:
61        """Create a new observation"""
62        observation_data = {
63            "species": species,
64            "location": location,
65            "count": count,
66            "date": datetime.now().isoformat(),
67            "notes": notes
68        }
69
70        try:
71            response = self.session.post(
72                f"{self.base_url}/observations",
73                json=observation_data,
74                timeout=10
75            )
76            response.raise_for_status()
77
78            if response.status_code == 201:
79                created = response.json()
80                print(f"Created observation ID: {created.get('id')}")
81                return created
82            else:
83                return response.json()
84
85        except requests.RequestException as e:
86            print(f"Error creating observation: {e}")
87            return None
88
89    def update_species_population(self, species_id: str,
90                                 new_population: int) -> bool:
91        """Update species population (PATCH)"""
92        try:
93            response = self.session.patch(
94                f"{self.base_url}/species/{species_id}",
95                json={"population": new_population},
96                timeout=10
97            )
98            response.raise_for_status()
99            print(f"Updated population {species_id} -> {new_population}")
100            return True
101
102        except requests.RequestException as e:
103            print(f"Update error: {e}")
104            return False
105
106    def delete_observation(self, observation_id: str) -> bool:
107        """Delete an observation"""
108        try:
109            response = self.session.delete(
110                f"{self.base_url}/observations/{observation_id}",
111                timeout=10
112            )
113            response.raise_for_status()
114
115            if response.status_code == 204:
116                print(f"Deleted observation {observation_id}")
117                return True
118
119            return False
120
121        except requests.RequestException as e:
122            print(f"Delete error: {e}")
123            return False
124
125    def get_statistics(self) -> Optional[Dict]:
126        """Fetch statistics (with query parameters)"""
127        params = {
128            "include_endangered": "true",
129            "group_by": "habitat"
130        }
131
132        try:
133            response = self.session.get(
134                f"{self.base_url}/statistics",
135                params=params,
136                timeout=15
137            )
138            response.raise_for_status()
139            return response.json()
140
141        except requests.RequestException as e:
142            print(f"Error fetching statistics: {e}")
143            return None
144
145    def close(self):
146        """Close session"""
147        self.session.close()
148
149
150# === DEMONSTRATION ===
151
152print("=== SAFARI API CLIENT ===\n")
153
154# Initialize client
155client = SafariAPIClient(
156    base_url="https://api.safari-data.org/v1",
157    api_key="demo_key_12345"
158)
159
160# GET - fetch species
161print("1. Fetching lion data...")
162lion_data = client.get_species("lion")
163if lion_data:
164    print(f"   Species: {lion_data.get('common_name')}")
165    print(f"   Population: {lion_data.get('population')}")
166
167# GET with parameters - search
168print("\n2. Searching for endangered species in the savanna...")
169endangered = client.search_species(habitat="savanna", endangered=True)
170print(f"   Found {len(endangered)} species")
171
172# POST - create observation
173print("\n3. Creating a new observation...")
174new_obs = client.create_observation(
175    species="Panthera leo",
176    location="Serengeti North",
177    count=8,
178    notes="Pride with 2 cubs"
179)
180
181# PATCH - update
182print("\n4. Updating population...")
183client.update_species_population("lion", 125)
184
185# DELETE - remove
186print("\n5. Deleting erroneous observation...")
187client.delete_observation("obs_12345")
188
189# GET with statistics
190print("\n6. Fetching statistics...")
191stats = client.get_statistics()
192if stats:
193    print(f"   Total species: {stats.get('total_species')}")
194
195# Close session
196client.close()
197print("\nSession closed")

Session - efficiency for multiple requests

If you're making many requests to the same API, use Session - it reuses the TCP connection:

1import requests
2
3# WITHOUT Session - each request = new connection
4for i in range(100):
5    response = requests.get("https://api.example.com/data")
6
7# WITH Session - one connection, faster!
8session = requests.Session()
9session.headers.update({"Authorization": "Bearer token123"})
10
11for i in range(100):
12    response = session.get("https://api.example.com/data")
13
14session.close()

Timeouts - don't wait forever

Always use timeout so a request doesn't hang indefinitely:

1import requests
2
3# 5-second timeout
4response = requests.get("https://slow-api.com/data", timeout=5)
5
6# Different timeouts: (connect_timeout, read_timeout)
7response = requests.get("https://api.com/data", timeout=(3, 10))
8# 3s to connect, 10s to read data

Summary

In this lesson you learned:

• What HTTP is and how it works (client -> server -> response)
• HTTP methods: GET, POST, PUT, PATCH, DELETE
• Status codes: 200 OK, 404 Not Found, 500 Server Error
• The requests library: requests.get(), requests.post()
• URL parameters (params=), headers (headers=)
• Sending JSON (json=) and form data (data=)
• Error handling: response.raise_for_status(), try/except
• Session for efficiency
• Timeouts for safety

Checkpoint

Before moving on:

• You understand the difference between GET and POST
• You can send a GET request with parameters
• You can send a POST with JSON
• You know how to handle HTTP errors
• You understand status codes (2xx, 4xx, 5xx)

Safari Analogy: HTTP is a courier system between research stations. GET = "send me data", POST = "save these new observations", status codes = delivery confirmations!

In the next lesson, Darwin will show you REST API - API architecture standards that allow you to build professional data exchange systems!