NoSQL - MongoDB in the jungle

Congratulations! This is the last lesson of Module 4. Darwin here for the last time in this module!

You've learned SQL - relational databases with tables, rows, columns, and rigid schemas. Now it's time for NoSQL - flexible document databases without rigid structure!

Safari Analogy: SQL is like a catalog with drawers and index cards - everything has its place. MongoDB is a field notebook - you write observations in any form, without a rigid format!

What is NoSQL?

NoSQL (Not Only SQL) is a family of databases that don't use the relational table model.

Types of NoSQL databases:

1. Document (MongoDB, CouchDB) - JSON-like documents
2. Key-value (Redis, DynamoDB) - simple key: value pairs
3. Columnar (Cassandra, HBase) - columns instead of rows
4. Graph (Neo4j, ArangoDB) - nodes and relationships

In this lesson: MongoDB - the most popular document database!

MongoDB - document database

MongoDB stores data as BSON documents (Binary JSON):

1{
2  "_id": ObjectId("507f1f77bcf86cd799439011"),
3  "scientific_name": "Panthera leo",
4  "common_name": "Lion",
5  "population": 120,
6  "habitat": "savanna",
7  "endangered": true,
8  "observations": [
9    {"date": "2024-01-15", "location": "Serengeti", "count": 12},
10    {"date": "2024-01-20", "location": "Masai Mara", "count": 8}
11  ],
12  "tags": ["carnivore", "big cat", "africa"]
13}

Characteristics:

• Flexible schema - each document can have different fields
• Nested structures - objects and arrays
• Horizontal scaling
• Fast for certain use cases

MongoDB vs SQL

PyMongo - MongoDB in Python

PyMongo is the official MongoDB driver for Python.

Installation

1pip install pymongo

Note: You need a running MongoDB server:

• Install MongoDB Community: https://www.mongodb.com/try/download/community
• Or use MongoDB Atlas (cloud, free tier): https://www.mongodb.com/cloud/atlas

Connecting to MongoDB

1from pymongo import MongoClient
2
3# Local connection
4client = MongoClient("mongodb://localhost:27017/")
5
6# Or MongoDB Atlas (cloud)
7# client = MongoClient("mongodb+srv://username:password@cluster.mongodb.net/")
8
9# Select database
10db = client["safari_database"]
11
12# Select collection (like a table in SQL)
13species_collection = db["species"]

CRUD in MongoDB

CREATE - adding documents

1from pymongo import MongoClient
2from datetime import datetime
3
4client = MongoClient("mongodb://localhost:27017/")
5db = client["safari_database"]
6species = db["species"]
7
8# Add a single document
9lion = {
10    "scientific_name": "Panthera leo",
11    "common_name": "Lion",
12    "population": 120,
13    "habitat": "savanna",
14    "endangered": True,
15    "created_at": datetime.utcnow(),
16    "observations": [
17        {"date": "2024-01-15", "location": "Serengeti", "count": 12},
18        {"date": "2024-01-20", "location": "Masai Mara", "count": 8}
19    ],
20    "tags": ["carnivore", "big cat", "africa"]
21}
22
23result = species.insert_one(lion)
24print(f"Added document ID: {result.inserted_id}")
25
26# Add multiple documents
27many_species = [
28    {
29        "scientific_name": "Loxodonta africana",
30        "common_name": "Elephant",
31        "population": 450,
32        "endangered": True
33    },
34    {
35        "scientific_name": "Gorilla gorilla",
36        "common_name": "Gorilla",
37        "population": 230,
38        "endangered": True
39    }
40]
41
42result = species.insert_many(many_species)
43print(f"Added {len(result.inserted_ids)} documents")

READ - retrieving documents

1# All documents
2all_species = species.find()
3for doc in all_species:
4    print(doc["common_name"], doc["population"])
5
6# Single document
7lion = species.find_one({"common_name": "Lion"})
8print(lion)
9
10# Filtering - WHERE
11endangered = species.find({"endangered": True})
12savanna = species.find({"habitat": "savanna"})
13
14# Multiple conditions (AND)
15results = species.find({
16    "endangered": True,
17    "population": {"$gt": 100}  # $gt = greater than (>)
18})
19
20# OR
21results = species.find({
22    "$or": [
23        {"habitat": "savanna"},
24        {"habitat": "forest"}
25    ]
26})
27
28# Comparison operators
29# $gt - greater than (>)
30# $gte - greater than or equal (>=)
31# $lt - less than (<)
32# $lte - less than or equal (<=)
33# $ne - not equal (!=)
34# $in - in list
35
36large_population = species.find({"population": {"$gte": 200}})
37specific_habitats = species.find({"habitat": {"$in": ["savanna", "forest"]}})
38
39# Sorting
40sorted_species = species.find().sort("population", -1)  # -1 = descending
41
42# Limit
43top_5 = species.find().limit(5)
44
45# Projection - select only certain fields
46names_only = species.find({}, {"common_name": 1, "population": 1, "_id": 0})
47
48# Count
49count = species.count_documents({})
50endangered_count = species.count_documents({"endangered": True})

UPDATE - updating documents

1# Update a single document
2species.update_one(
3    {"common_name": "Lion"},  # Filter
4    {"$set": {"population": 125}}  # Update
5)
6
7# Update multiple documents
8species.update_many(
9    {"habitat": "savanna"},
10    {"$set": {"endangered": True}}
11)
12
13# Update operators:
14# $set - set value
15# $inc - increment
16# $push - add to array
17# $pull - remove from array
18
19# Examples
20species.update_one(
21    {"common_name": "Lion"},
22    {
23        "$inc": {"population": 5},  # Increase by 5
24        "$push": {  # Add observation
25            "observations": {
26                "date": "2024-01-25",
27                "location": "Ngorongoro",
28                "count": 10
29            }
30        }
31    }
32)

DELETE - deleting documents

1# Delete a single document
2species.delete_one({"common_name": "Test Species"})
3
4# Delete multiple documents
5species.delete_many({"population": 0})
6
7# Delete all (BE CAREFUL!)
8# species.delete_many({})

Safari example - complete MongoDB system

1from pymongo import MongoClient
2from datetime import datetime
3from typing import List, Dict, Optional
4
5class SafariMongoDB:
6    """Safari database management in MongoDB"""
7
8    def __init__(self, connection_string: str = "mongodb://localhost:27017/"):
9        self.client = MongoClient(connection_string)
10        self.db = self.client["safari_database"]
11        self.species = self.db["species"]
12
13    # === SPECIES ===
14
15    def create_species(self, scientific_name: str, common_name: str,
16                      population: int = 0, habitat: str = "",
17                      endangered: bool = False, tags: List[str] = None) -> str:
18        """Add a species"""
19        species_doc = {
20            "scientific_name": scientific_name,
21            "common_name": common_name,
22            "population": population,
23            "habitat": habitat,
24            "endangered": endangered,
25            "tags": tags or [],
26            "observations": [],
27            "created_at": datetime.utcnow(),
28            "updated_at": datetime.utcnow()
29        }
30
31        result = self.species.insert_one(species_doc)
32        return str(result.inserted_id)
33
34    def get_species(self, species_id: str) -> Optional[Dict]:
35        """Get species by ID"""
36        from bson import ObjectId
37        return self.species.find_one({"_id": ObjectId(species_id)})
38
39    def get_species_by_name(self, common_name: str) -> Optional[Dict]:
40        """Get species by name"""
41        return self.species.find_one({"common_name": common_name})
42
43    def list_species(self, endangered: Optional[bool] = None,
44                    habitat: Optional[str] = None,
45                    min_population: int = 0) -> List[Dict]:
46        """List species with filters"""
47        query = {"population": {"$gte": min_population}}
48
49        if endangered is not None:
50            query["endangered"] = endangered
51        if habitat:
52            query["habitat"] = habitat
53
54        return list(self.species.find(query).sort("common_name", 1))
55
56    def update_species(self, common_name: str, **kwargs) -> bool:
57        """Update a species"""
58        if not kwargs:
59            return False
60
61        kwargs["updated_at"] = datetime.utcnow()
62
63        result = self.species.update_one(
64            {"common_name": common_name},
65            {"$set": kwargs}
66        )
67        return result.modified_count > 0
68
69    def delete_species(self, common_name: str) -> bool:
70        """Delete a species"""
71        result = self.species.delete_one({"common_name": common_name})
72        return result.deleted_count > 0
73
74    # === OBSERVATIONS ===
75
76    def add_observation(self, common_name: str, observation_date: str,
77                       location: str, count: int, notes: str = "") -> bool:
78        """Add an observation to a species"""
79        observation = {
80            "date": observation_date,
81            "location": location,
82            "count": count,
83            "notes": notes,
84            "recorded_at": datetime.utcnow()
85        }
86
87        result = self.species.update_one(
88            {"common_name": common_name},
89            {"$push": {"observations": observation}}
90        )
91        return result.modified_count > 0
92
93    def get_observations(self, common_name: str) -> List[Dict]:
94        """Get observations for a species"""
95        species = self.species.find_one(
96            {"common_name": common_name},
97            {"observations": 1, "_id": 0}
98        )
99        return species.get("observations", []) if species else []
100
101    # === STATISTICS ===
102
103    def get_statistics(self) -> Dict:
104        """Database statistics"""
105        pipeline = [
106            {
107                "$group": {
108                    "_id": "$habitat",
109                    "count": {"$sum": 1},
110                    "total_population": {"$sum": "$population"}
111                }
112            },
113            {"$sort": {"count": -1}}
114        ]
115
116        by_habitat = list(self.species.aggregate(pipeline))
117
118        return {
119            "total_species": self.species.count_documents({}),
120            "endangered_count": self.species.count_documents({"endangered": True}),
121            "by_habitat": by_habitat
122        }
123
124    def close(self):
125        """Close connection"""
126        self.client.close()
127
128
129# === DEMONSTRATION ===
130
131print("=== SAFARI MONGODB SYSTEM ===\n")
132
133db = SafariMongoDB()
134
135# 1. Add species
136print("1. Adding species...")
137lion_id = db.create_species(
138    "Panthera leo", "Lion", 120, "savanna", True,
139    tags=["carnivore", "big cat", "africa"]
140)
141elephant_id = db.create_species("Loxodonta africana", "Elephant", 450, "savanna", True)
142gorilla_id = db.create_species("Gorilla gorilla", "Gorilla", 230, "forest", True)
143
144print(f"   Added 3 species")
145
146# 2. Fetch species
147print("\n2. Fetching species...")
148lion = db.get_species_by_name("Lion")
149print(f"   {lion['common_name']}: {lion['population']} individuals")
150print(f"   Tags: {', '.join(lion.get('tags', []))}")
151
152# 3. List endangered
153print("\n3. List of endangered species...")
154endangered = db.list_species(endangered=True)
155for species in endangered:
156    print(f"   - {species['common_name']}: {species['population']}")
157
158# 4. Update
159print("\n4. Updating population...")
160db.update_species("Lion", population=125)
161lion = db.get_species_by_name("Lion")
162print(f"   New population: {lion['population']}")
163
164# 5. Add observations (embedded in document!)
165print("\n5. Adding observations...")
166db.add_observation("Lion", "2024-01-15", "Serengeti", 12, "Pride with 2 cubs")
167db.add_observation("Lion", "2024-01-20", "Masai Mara", 8, "Male coalition")
168
169# 6. Fetch observations
170print("\n6. Lion observations...")
171observations = db.get_observations("Lion")
172for obs in observations:
173    print(f"   - {obs['date']}: {obs['count']}x @ {obs['location']}")
174
175# 7. Statistics
176print("\n7. Database statistics...")
177stats = db.get_statistics()
178print(f"   Total species: {stats['total_species']}")
179print(f"   Endangered: {stats['endangered_count']}")
180print("   By habitat:")
181for habitat_stat in stats['by_habitat']:
182    print(f"     - {habitat_stat['_id']}: {habitat_stat['count']} species")
183
184db.close()
185print("\nDemonstration complete")

When to use MongoDB vs SQL?

Use MongoDB when:

• Flexible schema (data varies between records)
• Nested structures (JSON-like data)
• Rapid prototyping (no schema migrations)
• Horizontal scaling
• Non-relational data

Use SQL when:

• Rigid schema (structured data)
• Complex relationships (many JOINs)
• ACID transactions are critical
• Reporting and analytics
• Traditional business applications

Often: Both are used together - SQL for transactional data, MongoDB for logs, cache, sessions!

Summary - Module 4 Complete!

Congratulations! You've completed Module 4: Data Flow!

In this module you learned:

Lesson 1: Data Formats

• JSON, CSV, XML, YAML
• Serialization and deserialization
• Conversion between formats

Lesson 2: HTTP and requests

• HTTP protocol (GET, POST, PUT, DELETE)
• Status codes (200, 404, 500)
• The requests library
• Error handling, timeouts, sessions

Lesson 3: REST API

• REST principles (resources, methods, stateless)
• RESTful endpoints
• Pagination, filtering, sorting
• API best practices

Lesson 4: Web scraping

• BeautifulSoup4 (find, select)
• Extracting data from HTML
• Selenium for JavaScript
• Scraping ethics

Lesson 5: SQL and SQLite

• Relational databases
• SQL: CREATE, INSERT, SELECT, UPDATE, DELETE
• JOIN, WHERE, ORDER BY
• Python sqlite3

Lesson 6: SQLAlchemy ORM

• Python models and classes
• CRUD with ORM
• Relationships (One-to-Many)
• Query API

Lesson 7: NoSQL MongoDB

• Document database
• PyMongo
• Flexible schema
• MongoDB vs SQL

Final Safari Analogy: Now you can collect data (scraping), transmit it (HTTP), store it (SQL/MongoDB), and share it (REST API) - a complete data lifecycle in a Safari application!

You're ready for Module 5 and further adventures with Darwin! See you soon!