Python course ยท Module 4: Data, APIs and Databases
NoSQL - MongoDB in the jungle
In this lesson7
Congratulations! This is the last lesson of Module 4. Darwin here for the last time in this module!
You've learned SQL - relational databases with tables, rows, columns, and rigid schemas. Now it's time for NoSQL - flexible document databases without rigid structure!
Safari Analogy: SQL is like a catalog with drawers and index cards - everything has its place. MongoDB is a field notebook - you write observations in any form, without a rigid format!
What is NoSQL?
NoSQL (Not Only SQL) is a family of databases that don't use the relational table model.
Types of NoSQL databases:
- Document (MongoDB, CouchDB) - JSON-like documents
- Key-value (Redis, DynamoDB) - simple key: value pairs
- Columnar (Cassandra, HBase) - columns instead of rows
- Graph (Neo4j, ArangoDB) - nodes and relationships
In this lesson: MongoDB - the most popular document database!
MongoDB - document database
MongoDB stores data as BSON documents (Binary JSON):
{
"_id": ObjectId("507f1f77bcf86cd799439011"),
"scientific_name": "Panthera leo",
"common_name": "Lion",
"population": 120,
"habitat": "savanna",
"endangered": true,
"observations": [
{"date": "2024-01-15", "location": "Serengeti", "count": 12},
{"date": "2024-01-20", "location": "Masai Mara", "count": 8}
],
"tags": ["carnivore", "big cat", "africa"]
}Characteristics:
- Flexible schema - each document can have different fields
- Nested structures - objects and arrays
- Horizontal scaling
- Fast for certain use cases
MongoDB vs SQL
| Concept | SQL | MongoDB |
|---|---|---|
| Database | Database | Database |
| Table | Table | Collection |
| Row | Row | Document |
| Column | Column | Field |
| Primary Key | Primary Key | _id (automatic) |
| JOIN | JOIN | Embedded docs / $lookup |
| Schema | Rigid | Flexible |
PyMongo - MongoDB in Python
PyMongo is the official MongoDB driver for Python.
Installation
pip install pymongoNote: You need a running MongoDB server:
- Install MongoDB Community: https://www.mongodb.com/try/download/community
- Or use MongoDB Atlas (cloud, free tier): https://www.mongodb.com/cloud/atlas
Connecting to MongoDB
from pymongo import MongoClient
# Local connection
client = MongoClient("mongodb://localhost:27017/")
# Or MongoDB Atlas (cloud)
# client = MongoClient("mongodb+srv://username:password@cluster.mongodb.net/")
# Select database
db = client["safari_database"]
# Select collection (like a table in SQL)
species_collection = db["species"]CRUD in MongoDB
CREATE - adding documents
from pymongo import MongoClient
from datetime import datetime
client = MongoClient("mongodb://localhost:27017/")
db = client["safari_database"]
species = db["species"]
# Add a single document
lion = {
"scientific_name": "Panthera leo",
"common_name": "Lion",
"population": 120,
"habitat": "savanna",
"endangered": True,
"created_at": datetime.utcnow(),
"observations": [
{"date": "2024-01-15", "location": "Serengeti", "count": 12},
{"date": "2024-01-20", "location": "Masai Mara", "count": 8}
],
"tags": ["carnivore", "big cat", "africa"]
}
result = species.insert_one(lion)
print(f"Added document ID: {result.inserted_id}")
# Add multiple documents
many_species = [
{
"scientific_name": "Loxodonta africana",
"common_name": "Elephant",
"population": 450,
"endangered": True
},
{
"scientific_name": "Gorilla gorilla",
"common_name": "Gorilla",
"population": 230,
"endangered": True
}
]
result = species.insert_many(many_species)
print(f"Added {len(result.inserted_ids)} documents")READ - retrieving documents
# All documents
all_species = species.find()
for doc in all_species:
print(doc["common_name"], doc["population"])
# Single document
lion = species.find_one({"common_name": "Lion"})
print(lion)
# Filtering - WHERE
endangered = species.find({"endangered": True})
savanna = species.find({"habitat": "savanna"})
# Multiple conditions (AND)
results = species.find({
"endangered": True,
"population": {"$gt": 100} # $gt = greater than (>)
})
# OR
results = species.find({
"$or": [
{"habitat": "savanna"},
{"habitat": "forest"}
]
})
# Comparison operators
# $gt - greater than (>)
# $gte - greater than or equal (>=)
# $lt - less than (<)
# $lte - less than or equal (<=)
# $ne - not equal (!=)
# $in - in list
large_population = species.find({"population": {"$gte": 200}})
specific_habitats = species.find({"habitat": {"$in": ["savanna", "forest"]}})
# Sorting
sorted_species = species.find().sort("population", -1) # -1 = descending
# Limit
top_5 = species.find().limit(5)
# Projection - select only certain fields
names_only = species.find({}, {"common_name": 1, "population": 1, "_id": 0})
# Count
count = species.count_documents({})
endangered_count = species.count_documents({"endangered": True})UPDATE - updating documents
# Update a single document
species.update_one(
{"common_name": "Lion"}, # Filter
{"$set": {"population": 125}} # Update
)
# Update multiple documents
species.update_many(
{"habitat": "savanna"},
{"$set": {"endangered": True}}
)
# Update operators:
# $set - set value
# $inc - increment
# $push - add to array
# $pull - remove from array
# Examples
species.update_one(
{"common_name": "Lion"},
{
"$inc": {"population": 5}, # Increase by 5
"$push": { # Add observation
"observations": {
"date": "2024-01-25",
"location": "Ngorongoro",
"count": 10
}
}
}
)DELETE - deleting documents
# Delete a single document
species.delete_one({"common_name": "Test Species"})
# Delete multiple documents
species.delete_many({"population": 0})
# Delete all (BE CAREFUL!)
# species.delete_many({})Safari example - complete MongoDB system
from pymongo import MongoClient
from datetime import datetime
from typing import List, Dict, Optional
class SafariMongoDB:
"""Safari database management in MongoDB"""
def __init__(self, connection_string: str = "mongodb://localhost:27017/"):
self.client = MongoClient(connection_string)
self.db = self.client["safari_database"]
self.species = self.db["species"]
# === SPECIES ===
def create_species(self, scientific_name: str, common_name: str,
population: int = 0, habitat: str = "",
endangered: bool = False, tags: List[str] = None) -> str:
"""Add a species"""
species_doc = {
"scientific_name": scientific_name,
"common_name": common_name,
"population": population,
"habitat": habitat,
"endangered": endangered,
"tags": tags or [],
"observations": [],
"created_at": datetime.utcnow(),
"updated_at": datetime.utcnow()
}
result = self.species.insert_one(species_doc)
return str(result.inserted_id)
def get_species(self, species_id: str) -> Optional[Dict]:
"""Get species by ID"""
from bson import ObjectId
return self.species.find_one({"_id": ObjectId(species_id)})
def get_species_by_name(self, common_name: str) -> Optional[Dict]:
"""Get species by name"""
return self.species.find_one({"common_name": common_name})
def list_species(self, endangered: Optional[bool] = None,
habitat: Optional[str] = None,
min_population: int = 0) -> List[Dict]:
"""List species with filters"""
query = {"population": {"$gte": min_population}}
if endangered is not None:
query["endangered"] = endangered
if habitat:
query["habitat"] = habitat
return list(self.species.find(query).sort("common_name", 1))
def update_species(self, common_name: str, **kwargs) -> bool:
"""Update a species"""
if not kwargs:
return False
kwargs["updated_at"] = datetime.utcnow()
result = self.species.update_one(
{"common_name": common_name},
{"$set": kwargs}
)
return result.modified_count > 0
def delete_species(self, common_name: str) -> bool:
"""Delete a species"""
result = self.species.delete_one({"common_name": common_name})
return result.deleted_count > 0
# === OBSERVATIONS ===
def add_observation(self, common_name: str, observation_date: str,
location: str, count: int, notes: str = "") -> bool:
"""Add an observation to a species"""
observation = {
"date": observation_date,
"location": location,
"count": count,
"notes": notes,
"recorded_at": datetime.utcnow()
}
result = self.species.update_one(
{"common_name": common_name},
{"$push": {"observations": observation}}
)
return result.modified_count > 0
def get_observations(self, common_name: str) -> List[Dict]:
"""Get observations for a species"""
species = self.species.find_one(
{"common_name": common_name},
{"observations": 1, "_id": 0}
)
return species.get("observations", []) if species else []
# === STATISTICS ===
def get_statistics(self) -> Dict:
"""Database statistics"""
pipeline = [
{
"$group": {
"_id": "$habitat",
"count": {"$sum": 1},
"total_population": {"$sum": "$population"}
}
},
{"$sort": {"count": -1}}
]
by_habitat = list(self.species.aggregate(pipeline))
return {
"total_species": self.species.count_documents({}),
"endangered_count": self.species.count_documents({"endangered": True}),
"by_habitat": by_habitat
}
def close(self):
"""Close connection"""
self.client.close()
# === DEMONSTRATION ===
print("=== SAFARI MONGODB SYSTEM ===\n")
db = SafariMongoDB()
# 1. Add species
print("1. Adding species...")
lion_id = db.create_species(
"Panthera leo", "Lion", 120, "savanna", True,
tags=["carnivore", "big cat", "africa"]
)
elephant_id = db.create_species("Loxodonta africana", "Elephant", 450, "savanna", True)
gorilla_id = db.create_species("Gorilla gorilla", "Gorilla", 230, "forest", True)
print(f" Added 3 species")
# 2. Fetch species
print("\n2. Fetching species...")
lion = db.get_species_by_name("Lion")
print(f" {lion['common_name']}: {lion['population']} individuals")
print(f" Tags: {', '.join(lion.get('tags', []))}")
# 3. List endangered
print("\n3. List of endangered species...")
endangered = db.list_species(endangered=True)
for species in endangered:
print(f" - {species['common_name']}: {species['population']}")
# 4. Update
print("\n4. Updating population...")
db.update_species("Lion", population=125)
lion = db.get_species_by_name("Lion")
print(f" New population: {lion['population']}")
# 5. Add observations (embedded in document!)
print("\n5. Adding observations...")
db.add_observation("Lion", "2024-01-15", "Serengeti", 12, "Pride with 2 cubs")
db.add_observation("Lion", "2024-01-20", "Masai Mara", 8, "Male coalition")
# 6. Fetch observations
print("\n6. Lion observations...")
observations = db.get_observations("Lion")
for obs in observations:
print(f" - {obs['date']}: {obs['count']}x @ {obs['location']}")
# 7. Statistics
print("\n7. Database statistics...")
stats = db.get_statistics()
print(f" Total species: {stats['total_species']}")
print(f" Endangered: {stats['endangered_count']}")
print(" By habitat:")
for habitat_stat in stats['by_habitat']:
print(f" - {habitat_stat['_id']}: {habitat_stat['count']} species")
db.close()
print("\nDemonstration complete")When to use MongoDB vs SQL?
Use MongoDB when:
- Flexible schema (data varies between records)
- Nested structures (JSON-like data)
- Rapid prototyping (no schema migrations)
- Horizontal scaling
- Non-relational data
Use SQL when:
- Rigid schema (structured data)
- Complex relationships (many JOINs)
- ACID transactions are critical
- Reporting and analytics
- Traditional business applications
Often: Both are used together - SQL for transactional data, MongoDB for logs, cache, sessions!
Summary - Module 4 Complete!
Congratulations! You've completed Module 4: Data Flow!
In this module you learned:
Lesson 1: Data Formats
- JSON, CSV, XML, YAML
- Serialization and deserialization
- Conversion between formats
Lesson 2: HTTP and requests
- HTTP protocol (GET, POST, PUT, DELETE)
- Status codes (200, 404, 500)
- The requests library
- Error handling, timeouts, sessions
Lesson 3: REST API
- REST principles (resources, methods, stateless)
- RESTful endpoints
- Pagination, filtering, sorting
- API best practices
Lesson 4: Web scraping
- BeautifulSoup4 (find, select)
- Extracting data from HTML
- Selenium for JavaScript
- Scraping ethics
Lesson 5: SQL and SQLite
- Relational databases
- SQL: CREATE, INSERT, SELECT, UPDATE, DELETE
- JOIN, WHERE, ORDER BY
- Python sqlite3
Lesson 6: SQLAlchemy ORM
- Python models and classes
- CRUD with ORM
- Relationships (One-to-Many)
- Query API
Lesson 7: NoSQL MongoDB
- Document database
- PyMongo
- Flexible schema
- MongoDB vs SQL
Final Safari Analogy: Now you can collect data (scraping), transmit it (HTTP), store it (SQL/MongoDB), and share it (REST API) - a complete data lifecycle in a Safari application!
You're ready for Module 5 and further adventures with Darwin! See you soon!
Docs entries used in this lesson
Spotted a mistake in this lesson?
Check yourself
Answer the questions from this lesson. Pick an answer to see right away whether it is correct.
1. The main difference between NoSQL and SQL:
2. In MongoDB, data is stored as:
These are 2 of 4 questions for this lesson. Solve the rest in the game.
Hands-on tasks in the game
- Code editor
Insert the document animal = {'name': 'Tygrys', 'weight': 200} into the 'animals' collection
- Code editor
Find all documents where weight > 100
- Click in order
Click the elements in the correct order:
- Vertical ordering
Arrange the steps for building a REST API client:
- Code editor
Create a SafariAPIClient class with methods: get_animals(), get_animal_by_id()
- Code editor
Create a DatabaseManager class with methods: save_animal(), get_all_animals(), search()
- Code editor
Create a complete Safari Data Hub with a menu, API fetching, database saving, and searching
- Vertical ordering
Arrange the steps for working with SQLAlchemy:
- Vertical ordering
Arrange the steps of a full data lifecycle:
Tasks after this lesson
- QuestionNoSQL vs SQL
- QuestionMongoDB
- Coding taskMongoDB Insert
- Coding taskMongoDB Find
- Click in orderBuilding find_all()
- Put in orderREST API Pattern
- Coding taskProject - API Client
- Coding taskProject - Database Manager
- Coding taskProject - Safari Data Hub
- Put in orderSQLAlchemy Workflow
- Put in orderFull Data Cycle
- QuestionSummary - JSON
- QuestionSummary - HTTP
- Project to submitSubmit Safari Data Hub to Mentor
- VideoModule Summary