Python course ยท Module 4: Data, APIs and Databases

HTTP - the data exchange protocol

10 min read ยท
In this lesson14

Welcome back! Darwin here.

In the previous lesson you learned about data formats - how to store information (JSON, CSV, XML, YAML). Now it's time for HTTP - the protocol that lets you transmit that data over the internet!

Safari Analogy: Data formats are the language in which you record observations. HTTP is the courier system - the way you send those observations between research stations in different parts of Africa!

What is HTTP?

HTTP (HyperText Transfer Protocol) is a communication protocol - a set of rules by which computers exchange data over the internet.

How it works:

  1. Client (e.g., browser, Python app) sends a request
  2. Server (e.g., website, API) processes the request
  3. Server sends back a response with data or error information
[CLIENT]  ---(HTTP REQUEST)--->  [SERVER]
   |                                  |
   |                              Processes
   |                                  |
   |  <---(HTTP RESPONSE)---  [SERVER]

Safari Example: You send a request to the central database in Nairobi: "Give me the data on lions in Serengeti". The server responds: "Here are 120 lions in JSON format"!

HTTP Methods - types of requests

HTTP has different methods (sometimes called "verbs") - they specify WHAT you want to do:

MethodPurposeSafari Analogy
GETRetrieve data"Show me the species list"
POSTCreate a new resource"Add a new elephant observation"
PUTReplace an entire resource"Replace the entire lion profile with new data"
PATCHUpdate part of a resource"Change only the gorilla population"
DELETEDelete a resource"Remove an erroneous observation"

Most important: GET and POST - used in 90% of cases!

GET - retrieving data

GET /api/species/lion HTTP/1.1
Host: safari-api.com

Characteristics:

  • Does not modify data on the server
  • Parameters in URL (?name=lion&habitat=savanna)
  • Can be bookmarked
  • Safe to call multiple times

POST - creating/sending data

POST /api/observations HTTP/1.1
Host: safari-api.com
Content-Type: application/json

{
  "species": "Panthera leo",
  "count": 12,
  "location": "Serengeti"
}

Characteristics:

  • Sends data in the request body, not in the URL
  • Can modify data on the server
  • Not saved in browser history
  • Can create new resources

HTTP Status Codes - server responses

The server always responds with a status code - a number indicating whether the operation succeeded:

2xx - Success

  • 200 OK - request successful, here's the data
  • 201 Created - resource was created
  • 204 No Content - success, but no data to return

3xx - Redirection

  • 301 Moved Permanently - resource moved permanently
  • 302 Found - temporary redirect

4xx - Client Error

  • 400 Bad Request - malformed request (e.g., bad JSON)
  • 401 Unauthorized - authentication required
  • 403 Forbidden - no access
  • 404 Not Found - resource doesn't exist
  • 429 Too Many Requests - too many requests

5xx - Server Error

  • 500 Internal Server Error - server error
  • 502 Bad Gateway - proxy server problem
  • 503 Service Unavailable - server unavailable

Safari Analogy: Code 200 = "Found the lion!", 404 = "There's no lion here", 500 = "My radio broke down"

Python requests - HTTP library

requests is the most popular Python library for HTTP - simple, elegant, powerful!

Installation

pip install requests

Basic usage - GET request

import requests

# Fetch data
response = requests.get("https://api.github.com/users/octocat")

# Status code
print(response.status_code)  # 200

# Data as JSON
data = response.json()
print(data["login"])  # "octocat"
print(data["name"])   # "The Octocat"

# Data as text
print(response.text)  # Raw JSON as string

# Response headers
print(response.headers["Content-Type"])  # "application/json"

Checking for success

import requests

response = requests.get("https://api.github.com/users/nonexistent")

# Method 1: Check status code
if response.status_code == 200:
    print("Success!")
elif response.status_code == 404:
    print("Not found")

# Method 2: Use response.ok (True for 2xx)
if response.ok:
    data = response.json()
    print(data)
else:
    print(f"Error: {response.status_code}")

# Method 3: Raise exception on error
try:
    response.raise_for_status()  # Raises HTTPError for 4xx/5xx
    data = response.json()
except requests.HTTPError as e:
    print(f"HTTP Error: {e}")

URL Parameters - query parameters

GET requests often use parameters in the URL:

import requests

# Manually in URL
response = requests.get("https://api.example.com/species?habitat=savanna&endangered=true")

# Better: params dictionary (automatic encoding)
params = {
    "habitat": "savanna",
    "endangered": "true",
    "min_population": 100
}

response = requests.get("https://api.example.com/species", params=params)
# URL: https://api.example.com/species?habitat=savanna&endangered=true&min_population=100

print(response.url)  # See the generated URL

Headers

Headers are request metadata - information about the client, expected format, authorization:

import requests

# Custom headers
headers = {
    "User-Agent": "Safari-Data-Collector/1.0",
    "Accept": "application/json",
    "Authorization": "Bearer YOUR_API_KEY_HERE"
}

response = requests.get("https://api.example.com/species", headers=headers)

# Check response headers
print(response.headers["Content-Type"])
print(response.headers["Date"])

Common headers:

  • User-Agent - client identification
  • Accept - expected response format
  • Content-Type - format of sent data
  • Authorization - token/API key

POST request - sending data

JSON payload

import requests

# Data to send
new_observation = {
    "species": "Panthera leo",
    "location": "Serengeti",
    "count": 12,
    "date": "2024-01-15",
    "observer": "Darwin"
}

# POST with JSON
response = requests.post(
    "https://api.safari.com/observations",
    json=new_observation  # Automatically: Content-Type: application/json
)

if response.status_code == 201:  # Created
    created_data = response.json()
    print(f"Created observation ID: {created_data['id']}")
else:
    print(f"Error: {response.status_code}")

Form data (application/x-www-form-urlencoded)

import requests

# Form data
form_data = {
    "species": "Loxodonta africana",
    "count": "35"
}

response = requests.post(
    "https://api.safari.com/quick-report",
    data=form_data  # Content-Type: application/x-www-form-urlencoded
)

PUT, PATCH, DELETE

import requests

# PUT - replace entire resource
updated_species = {
    "scientific_name": "Panthera leo",
    "population": 125,  # Updated
    "habitat": "savanna"
}

response = requests.put(
    "https://api.safari.com/species/lion",
    json=updated_species
)

# PATCH - change only part
partial_update = {
    "population": 125  # Only population
}

response = requests.patch(
    "https://api.safari.com/species/lion",
    json=partial_update
)

# DELETE - remove resource
response = requests.delete("https://api.safari.com/observations/12345")

if response.status_code == 204:  # No Content
    print("Successfully deleted")

Error handling

import requests
from requests.exceptions import RequestException, Timeout, ConnectionError

def fetch_species_data(species_id: str):
    """Fetch species data with error handling"""
    try:
        response = requests.get(
            f"https://api.safari.com/species/{species_id}",
            timeout=5  # Timeout after 5 seconds
        )

        # Check status
        response.raise_for_status()

        # Process data
        return response.json()

    except Timeout:
        print("Timeout - server did not respond in time")
        return None

    except ConnectionError:
        print("Connection error - check your internet")
        return None

    except requests.HTTPError as e:
        print(f"HTTP Error {response.status_code}: {e}")
        return None

    except RequestException as e:
        print(f"Request error: {e}")
        return None

# Usage
data = fetch_species_data("lion")
if data:
    print(f"Species: {data['name']}")

Safari example - API client for observation management

import requests
from typing import List, Dict, Optional
from datetime import datetime

class SafariAPIClient:
    """API client for the Safari observation system"""

    def __init__(self, base_url: str, api_key: str):
        self.base_url = base_url.rstrip("/")
        self.api_key = api_key
        self.session = requests.Session()  # Reusable connection

        # Default headers for all requests
        self.session.headers.update({
            "User-Agent": "Safari-Data-Collector/2.0",
            "Authorization": f"Bearer {self.api_key}",
            "Accept": "application/json"
        })

    def get_species(self, species_id: str) -> Optional[Dict]:
        """Fetch species data"""
        try:
            response = self.session.get(
                f"{self.base_url}/species/{species_id}",
                timeout=10
            )
            response.raise_for_status()
            return response.json()

        except requests.RequestException as e:
            print(f"Error fetching species: {e}")
            return None

    def search_species(self, habitat: Optional[str] = None,
                      endangered: Optional[bool] = None,
                      min_population: Optional[int] = None) -> List[Dict]:
        """Search species with filters"""
        params = {}
        if habitat:
            params["habitat"] = habitat
        if endangered is not None:
            params["endangered"] = str(endangered).lower()
        if min_population:
            params["min_population"] = min_population

        try:
            response = self.session.get(
                f"{self.base_url}/species",
                params=params,
                timeout=10
            )
            response.raise_for_status()
            return response.json()

        except requests.RequestException as e:
            print(f"Search error: {e}")
            return []

    def create_observation(self, species: str, location: str,
                          count: int, notes: str = "") -> Optional[Dict]:
        """Create a new observation"""
        observation_data = {
            "species": species,
            "location": location,
            "count": count,
            "date": datetime.now().isoformat(),
            "notes": notes
        }

        try:
            response = self.session.post(
                f"{self.base_url}/observations",
                json=observation_data,
                timeout=10
            )
            response.raise_for_status()

            if response.status_code == 201:
                created = response.json()
                print(f"Created observation ID: {created.get('id')}")
                return created
            else:
                return response.json()

        except requests.RequestException as e:
            print(f"Error creating observation: {e}")
            return None

    def update_species_population(self, species_id: str,
                                 new_population: int) -> bool:
        """Update species population (PATCH)"""
        try:
            response = self.session.patch(
                f"{self.base_url}/species/{species_id}",
                json={"population": new_population},
                timeout=10
            )
            response.raise_for_status()
            print(f"Updated population {species_id} -> {new_population}")
            return True

        except requests.RequestException as e:
            print(f"Update error: {e}")
            return False

    def delete_observation(self, observation_id: str) -> bool:
        """Delete an observation"""
        try:
            response = self.session.delete(
                f"{self.base_url}/observations/{observation_id}",
                timeout=10
            )
            response.raise_for_status()

            if response.status_code == 204:
                print(f"Deleted observation {observation_id}")
                return True

            return False

        except requests.RequestException as e:
            print(f"Delete error: {e}")
            return False

    def get_statistics(self) -> Optional[Dict]:
        """Fetch statistics (with query parameters)"""
        params = {
            "include_endangered": "true",
            "group_by": "habitat"
        }

        try:
            response = self.session.get(
                f"{self.base_url}/statistics",
                params=params,
                timeout=15
            )
            response.raise_for_status()
            return response.json()

        except requests.RequestException as e:
            print(f"Error fetching statistics: {e}")
            return None

    def close(self):
        """Close session"""
        self.session.close()


# === DEMONSTRATION ===

print("=== SAFARI API CLIENT ===\n")

# Initialize client
client = SafariAPIClient(
    base_url="https://api.safari-data.org/v1",
    api_key="demo_key_12345"
)

# GET - fetch species
print("1. Fetching lion data...")
lion_data = client.get_species("lion")
if lion_data:
    print(f"   Species: {lion_data.get('common_name')}")
    print(f"   Population: {lion_data.get('population')}")

# GET with parameters - search
print("\n2. Searching for endangered species in the savanna...")
endangered = client.search_species(habitat="savanna", endangered=True)
print(f"   Found {len(endangered)} species")

# POST - create observation
print("\n3. Creating a new observation...")
new_obs = client.create_observation(
    species="Panthera leo",
    location="Serengeti North",
    count=8,
    notes="Pride with 2 cubs"
)

# PATCH - update
print("\n4. Updating population...")
client.update_species_population("lion", 125)

# DELETE - remove
print("\n5. Deleting erroneous observation...")
client.delete_observation("obs_12345")

# GET with statistics
print("\n6. Fetching statistics...")
stats = client.get_statistics()
if stats:
    print(f"   Total species: {stats.get('total_species')}")

# Close session
client.close()
print("\nSession closed")

Session - efficiency for multiple requests

If you're making many requests to the same API, use Session - it reuses the TCP connection:

import requests

# WITHOUT Session - each request = new connection
for i in range(100):
    response = requests.get("https://api.example.com/data")

# WITH Session - one connection, faster!
session = requests.Session()
session.headers.update({"Authorization": "Bearer token123"})

for i in range(100):
    response = session.get("https://api.example.com/data")

session.close()

Timeouts - don't wait forever

Always use timeout so a request doesn't hang indefinitely:

import requests

# 5-second timeout
response = requests.get("https://slow-api.com/data", timeout=5)

# Different timeouts: (connect_timeout, read_timeout)
response = requests.get("https://api.com/data", timeout=(3, 10))
# 3s to connect, 10s to read data

Summary

In this lesson you learned:

  • What HTTP is and how it works (client -> server -> response)
  • HTTP methods: GET, POST, PUT, PATCH, DELETE
  • Status codes: 200 OK, 404 Not Found, 500 Server Error
  • The requests library: requests.get(), requests.post()
  • URL parameters (params=), headers (headers=)
  • Sending JSON (json=) and form data (data=)
  • Error handling: response.raise_for_status(), try/except
  • Session for efficiency
  • Timeouts for safety

Checkpoint

Before moving on:

  • You understand the difference between GET and POST
  • You can send a GET request with parameters
  • You can send a POST with JSON
  • You know how to handle HTTP errors
  • You understand status codes (2xx, 4xx, 5xx)

Safari Analogy: HTTP is a courier system between research stations. GET = "send me data", POST = "save these new observations", status codes = delivery confirmations!

In the next lesson, Darwin will show you REST API - API architecture standards that allow you to build professional data exchange systems!

Docs entries used in this lesson

Spotted a mistake in this lesson?

Check yourself

Answer the questions from this lesson. Pick an answer to see right away whether it is correct.

  1. 1. Which HTTP method is used to retrieve data?

  2. 2. How do you make a GET request using the requests library?

These are 2 of 4 questions for this lesson. Solve the rest in the game.

Hands-on tasks in the game

  • Horizontal ordering

    Arrange the elements in the correct order:

  • Click in order

    Click the elements in the correct order:

  • Code editor

    Fetch data from URL 'https://api.example.com/animals' using requests

  • Code editor

    Perform a GET request and parse the response as JSON

  • Click in order

    Click in order: response = requests.get(url)

  1. QuestionHTTP Methods
  2. QuestionThe requests Library
  3. QuestionHTTP Status Codes
  4. Put in orderRequests GET
  5. QuestionStatus Code 404
  6. Click in orderBuilding import json, csv
  7. Coding taskGET Request
  8. Coding taskParsing JSON from Response
  9. Click in orderGET Request

Useful articles