Python course ยท Module 4: Data, APIs and Databases

NoSQL - MongoDB in the jungle

8 min read ยท
In this lesson7

Congratulations! This is the last lesson of Module 4. Darwin here for the last time in this module!

You've learned SQL - relational databases with tables, rows, columns, and rigid schemas. Now it's time for NoSQL - flexible document databases without rigid structure!

Safari Analogy: SQL is like a catalog with drawers and index cards - everything has its place. MongoDB is a field notebook - you write observations in any form, without a rigid format!

What is NoSQL?

NoSQL (Not Only SQL) is a family of databases that don't use the relational table model.

Types of NoSQL databases:

  1. Document (MongoDB, CouchDB) - JSON-like documents
  2. Key-value (Redis, DynamoDB) - simple key: value pairs
  3. Columnar (Cassandra, HBase) - columns instead of rows
  4. Graph (Neo4j, ArangoDB) - nodes and relationships

In this lesson: MongoDB - the most popular document database!

MongoDB - document database

MongoDB stores data as BSON documents (Binary JSON):

{
  "_id": ObjectId("507f1f77bcf86cd799439011"),
  "scientific_name": "Panthera leo",
  "common_name": "Lion",
  "population": 120,
  "habitat": "savanna",
  "endangered": true,
  "observations": [
    {"date": "2024-01-15", "location": "Serengeti", "count": 12},
    {"date": "2024-01-20", "location": "Masai Mara", "count": 8}
  ],
  "tags": ["carnivore", "big cat", "africa"]
}

Characteristics:

  • Flexible schema - each document can have different fields
  • Nested structures - objects and arrays
  • Horizontal scaling
  • Fast for certain use cases

MongoDB vs SQL

ConceptSQLMongoDB
DatabaseDatabaseDatabase
TableTableCollection
RowRowDocument
ColumnColumnField
Primary KeyPrimary Key_id (automatic)
JOINJOINEmbedded docs / $lookup
SchemaRigidFlexible

PyMongo - MongoDB in Python

PyMongo is the official MongoDB driver for Python.

Installation

pip install pymongo

Note: You need a running MongoDB server:

Connecting to MongoDB

from pymongo import MongoClient

# Local connection
client = MongoClient("mongodb://localhost:27017/")

# Or MongoDB Atlas (cloud)
# client = MongoClient("mongodb+srv://username:password@cluster.mongodb.net/")

# Select database
db = client["safari_database"]

# Select collection (like a table in SQL)
species_collection = db["species"]

CRUD in MongoDB

CREATE - adding documents

from pymongo import MongoClient
from datetime import datetime

client = MongoClient("mongodb://localhost:27017/")
db = client["safari_database"]
species = db["species"]

# Add a single document
lion = {
    "scientific_name": "Panthera leo",
    "common_name": "Lion",
    "population": 120,
    "habitat": "savanna",
    "endangered": True,
    "created_at": datetime.utcnow(),
    "observations": [
        {"date": "2024-01-15", "location": "Serengeti", "count": 12},
        {"date": "2024-01-20", "location": "Masai Mara", "count": 8}
    ],
    "tags": ["carnivore", "big cat", "africa"]
}

result = species.insert_one(lion)
print(f"Added document ID: {result.inserted_id}")

# Add multiple documents
many_species = [
    {
        "scientific_name": "Loxodonta africana",
        "common_name": "Elephant",
        "population": 450,
        "endangered": True
    },
    {
        "scientific_name": "Gorilla gorilla",
        "common_name": "Gorilla",
        "population": 230,
        "endangered": True
    }
]

result = species.insert_many(many_species)
print(f"Added {len(result.inserted_ids)} documents")

READ - retrieving documents

# All documents
all_species = species.find()
for doc in all_species:
    print(doc["common_name"], doc["population"])

# Single document
lion = species.find_one({"common_name": "Lion"})
print(lion)

# Filtering - WHERE
endangered = species.find({"endangered": True})
savanna = species.find({"habitat": "savanna"})

# Multiple conditions (AND)
results = species.find({
    "endangered": True,
    "population": {"$gt": 100}  # $gt = greater than (>)
})

# OR
results = species.find({
    "$or": [
        {"habitat": "savanna"},
        {"habitat": "forest"}
    ]
})

# Comparison operators
# $gt - greater than (>)
# $gte - greater than or equal (>=)
# $lt - less than (<)
# $lte - less than or equal (<=)
# $ne - not equal (!=)
# $in - in list

large_population = species.find({"population": {"$gte": 200}})
specific_habitats = species.find({"habitat": {"$in": ["savanna", "forest"]}})

# Sorting
sorted_species = species.find().sort("population", -1)  # -1 = descending

# Limit
top_5 = species.find().limit(5)

# Projection - select only certain fields
names_only = species.find({}, {"common_name": 1, "population": 1, "_id": 0})

# Count
count = species.count_documents({})
endangered_count = species.count_documents({"endangered": True})

UPDATE - updating documents

# Update a single document
species.update_one(
    {"common_name": "Lion"},  # Filter
    {"$set": {"population": 125}}  # Update
)

# Update multiple documents
species.update_many(
    {"habitat": "savanna"},
    {"$set": {"endangered": True}}
)

# Update operators:
# $set - set value
# $inc - increment
# $push - add to array
# $pull - remove from array

# Examples
species.update_one(
    {"common_name": "Lion"},
    {
        "$inc": {"population": 5},  # Increase by 5
        "$push": {  # Add observation
            "observations": {
                "date": "2024-01-25",
                "location": "Ngorongoro",
                "count": 10
            }
        }
    }
)

DELETE - deleting documents

# Delete a single document
species.delete_one({"common_name": "Test Species"})

# Delete multiple documents
species.delete_many({"population": 0})

# Delete all (BE CAREFUL!)
# species.delete_many({})

Safari example - complete MongoDB system

from pymongo import MongoClient
from datetime import datetime
from typing import List, Dict, Optional

class SafariMongoDB:
    """Safari database management in MongoDB"""

    def __init__(self, connection_string: str = "mongodb://localhost:27017/"):
        self.client = MongoClient(connection_string)
        self.db = self.client["safari_database"]
        self.species = self.db["species"]

    # === SPECIES ===

    def create_species(self, scientific_name: str, common_name: str,
                      population: int = 0, habitat: str = "",
                      endangered: bool = False, tags: List[str] = None) -> str:
        """Add a species"""
        species_doc = {
            "scientific_name": scientific_name,
            "common_name": common_name,
            "population": population,
            "habitat": habitat,
            "endangered": endangered,
            "tags": tags or [],
            "observations": [],
            "created_at": datetime.utcnow(),
            "updated_at": datetime.utcnow()
        }

        result = self.species.insert_one(species_doc)
        return str(result.inserted_id)

    def get_species(self, species_id: str) -> Optional[Dict]:
        """Get species by ID"""
        from bson import ObjectId
        return self.species.find_one({"_id": ObjectId(species_id)})

    def get_species_by_name(self, common_name: str) -> Optional[Dict]:
        """Get species by name"""
        return self.species.find_one({"common_name": common_name})

    def list_species(self, endangered: Optional[bool] = None,
                    habitat: Optional[str] = None,
                    min_population: int = 0) -> List[Dict]:
        """List species with filters"""
        query = {"population": {"$gte": min_population}}

        if endangered is not None:
            query["endangered"] = endangered
        if habitat:
            query["habitat"] = habitat

        return list(self.species.find(query).sort("common_name", 1))

    def update_species(self, common_name: str, **kwargs) -> bool:
        """Update a species"""
        if not kwargs:
            return False

        kwargs["updated_at"] = datetime.utcnow()

        result = self.species.update_one(
            {"common_name": common_name},
            {"$set": kwargs}
        )
        return result.modified_count > 0

    def delete_species(self, common_name: str) -> bool:
        """Delete a species"""
        result = self.species.delete_one({"common_name": common_name})
        return result.deleted_count > 0

    # === OBSERVATIONS ===

    def add_observation(self, common_name: str, observation_date: str,
                       location: str, count: int, notes: str = "") -> bool:
        """Add an observation to a species"""
        observation = {
            "date": observation_date,
            "location": location,
            "count": count,
            "notes": notes,
            "recorded_at": datetime.utcnow()
        }

        result = self.species.update_one(
            {"common_name": common_name},
            {"$push": {"observations": observation}}
        )
        return result.modified_count > 0

    def get_observations(self, common_name: str) -> List[Dict]:
        """Get observations for a species"""
        species = self.species.find_one(
            {"common_name": common_name},
            {"observations": 1, "_id": 0}
        )
        return species.get("observations", []) if species else []

    # === STATISTICS ===

    def get_statistics(self) -> Dict:
        """Database statistics"""
        pipeline = [
            {
                "$group": {
                    "_id": "$habitat",
                    "count": {"$sum": 1},
                    "total_population": {"$sum": "$population"}
                }
            },
            {"$sort": {"count": -1}}
        ]

        by_habitat = list(self.species.aggregate(pipeline))

        return {
            "total_species": self.species.count_documents({}),
            "endangered_count": self.species.count_documents({"endangered": True}),
            "by_habitat": by_habitat
        }

    def close(self):
        """Close connection"""
        self.client.close()


# === DEMONSTRATION ===

print("=== SAFARI MONGODB SYSTEM ===\n")

db = SafariMongoDB()

# 1. Add species
print("1. Adding species...")
lion_id = db.create_species(
    "Panthera leo", "Lion", 120, "savanna", True,
    tags=["carnivore", "big cat", "africa"]
)
elephant_id = db.create_species("Loxodonta africana", "Elephant", 450, "savanna", True)
gorilla_id = db.create_species("Gorilla gorilla", "Gorilla", 230, "forest", True)

print(f"   Added 3 species")

# 2. Fetch species
print("\n2. Fetching species...")
lion = db.get_species_by_name("Lion")
print(f"   {lion['common_name']}: {lion['population']} individuals")
print(f"   Tags: {', '.join(lion.get('tags', []))}")

# 3. List endangered
print("\n3. List of endangered species...")
endangered = db.list_species(endangered=True)
for species in endangered:
    print(f"   - {species['common_name']}: {species['population']}")

# 4. Update
print("\n4. Updating population...")
db.update_species("Lion", population=125)
lion = db.get_species_by_name("Lion")
print(f"   New population: {lion['population']}")

# 5. Add observations (embedded in document!)
print("\n5. Adding observations...")
db.add_observation("Lion", "2024-01-15", "Serengeti", 12, "Pride with 2 cubs")
db.add_observation("Lion", "2024-01-20", "Masai Mara", 8, "Male coalition")

# 6. Fetch observations
print("\n6. Lion observations...")
observations = db.get_observations("Lion")
for obs in observations:
    print(f"   - {obs['date']}: {obs['count']}x @ {obs['location']}")

# 7. Statistics
print("\n7. Database statistics...")
stats = db.get_statistics()
print(f"   Total species: {stats['total_species']}")
print(f"   Endangered: {stats['endangered_count']}")
print("   By habitat:")
for habitat_stat in stats['by_habitat']:
    print(f"     - {habitat_stat['_id']}: {habitat_stat['count']} species")

db.close()
print("\nDemonstration complete")

When to use MongoDB vs SQL?

Use MongoDB when:

  • Flexible schema (data varies between records)
  • Nested structures (JSON-like data)
  • Rapid prototyping (no schema migrations)
  • Horizontal scaling
  • Non-relational data

Use SQL when:

  • Rigid schema (structured data)
  • Complex relationships (many JOINs)
  • ACID transactions are critical
  • Reporting and analytics
  • Traditional business applications

Often: Both are used together - SQL for transactional data, MongoDB for logs, cache, sessions!

Summary - Module 4 Complete!

Congratulations! You've completed Module 4: Data Flow!

In this module you learned:

Lesson 1: Data Formats

  • JSON, CSV, XML, YAML
  • Serialization and deserialization
  • Conversion between formats

Lesson 2: HTTP and requests

  • HTTP protocol (GET, POST, PUT, DELETE)
  • Status codes (200, 404, 500)
  • The requests library
  • Error handling, timeouts, sessions

Lesson 3: REST API

  • REST principles (resources, methods, stateless)
  • RESTful endpoints
  • Pagination, filtering, sorting
  • API best practices

Lesson 4: Web scraping

  • BeautifulSoup4 (find, select)
  • Extracting data from HTML
  • Selenium for JavaScript
  • Scraping ethics

Lesson 5: SQL and SQLite

  • Relational databases
  • SQL: CREATE, INSERT, SELECT, UPDATE, DELETE
  • JOIN, WHERE, ORDER BY
  • Python sqlite3

Lesson 6: SQLAlchemy ORM

  • Python models and classes
  • CRUD with ORM
  • Relationships (One-to-Many)
  • Query API

Lesson 7: NoSQL MongoDB

  • Document database
  • PyMongo
  • Flexible schema
  • MongoDB vs SQL

Final Safari Analogy: Now you can collect data (scraping), transmit it (HTTP), store it (SQL/MongoDB), and share it (REST API) - a complete data lifecycle in a Safari application!

You're ready for Module 5 and further adventures with Darwin! See you soon!

Docs entries used in this lesson

Spotted a mistake in this lesson?

Check yourself

Answer the questions from this lesson. Pick an answer to see right away whether it is correct.

  1. 1. The main difference between NoSQL and SQL:

  2. 2. In MongoDB, data is stored as:

These are 2 of 4 questions for this lesson. Solve the rest in the game.

Hands-on tasks in the game

  • Code editor

    Insert the document animal = {'name': 'Tygrys', 'weight': 200} into the 'animals' collection

  • Code editor

    Find all documents where weight > 100

  • Click in order

    Click the elements in the correct order:

  • Vertical ordering

    Arrange the steps for building a REST API client:

  • Code editor

    Create a SafariAPIClient class with methods: get_animals(), get_animal_by_id()

  • Code editor

    Create a DatabaseManager class with methods: save_animal(), get_all_animals(), search()

  • Code editor

    Create a complete Safari Data Hub with a menu, API fetching, database saving, and searching

  • Vertical ordering

    Arrange the steps for working with SQLAlchemy:

  • Vertical ordering

    Arrange the steps of a full data lifecycle:

  1. QuestionNoSQL vs SQL
  2. QuestionMongoDB
  3. Coding taskMongoDB Insert
  4. Coding taskMongoDB Find
  5. Click in orderBuilding find_all()
  6. Put in orderREST API Pattern
  7. Coding taskProject - API Client
  8. Coding taskProject - Database Manager
  9. Coding taskProject - Safari Data Hub
  10. Put in orderSQLAlchemy Workflow
  11. Put in orderFull Data Cycle
  12. QuestionSummary - JSON
  13. QuestionSummary - HTTP
  14. Project to submitSubmit Safari Data Hub to Mentor
  15. VideoModule Summary

Useful articles