NestJS course Β· Module 9: Deployment and Infrastructure
Backup and Recovery - rescuing tributes after a disaster
In this lesson4
Guardian of the tributes! One night a migration with a typo deletes a column, and in the morning it turns out the last working copy of the database is three weeks old, because the backup script had been failing for a month and nobody noticed. Even the best forts can fall into ruins during a siege, which is why every wise legionary has a plan for rescuing his tributes.
Two numbers where everything starts
A recovery plan starts with two goals. RTO (Recovery Time Objective) is the maximum time to restore the service after a failure. RPO (Recovery Point Objective) is the acceptable data loss, measured in time: a backup once a day means you lose up to 24 hours of work. Want less - you need more frequent copies or continuous archiving of the change log.
Keep copies according to the 3-2-1 rule: three copies, on two different media, one of them outside the main location.
Database Backup Strategy
The PostgreSQL script makes a dump, checks it, sends it to S3 and cleans up old copies:
#!/bin/bash
# backup-database.sh
set -euo pipefail
# Without a database address the script fails instead of dumping a default database
: "${DATABASE_URL:?Set DATABASE_URL}"
BACKUP_DIR="${BACKUP_DIR:-/backups}"
DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_NAME="legionariusze_legion_backup_${DATE}.dump"
# Create backup (the custom format is already compressed)
echo "Creating database backup..."
pg_dump --format=custom --file="$BACKUP_DIR/$BACKUP_NAME" "$DATABASE_URL"
# Verify backup - pg_restore must be able to read the table of contents
pg_restore --list "$BACKUP_DIR/$BACKUP_NAME" > /dev/null
# Upload to S3
aws s3 cp "$BACKUP_DIR/$BACKUP_NAME" s3://legionariusze-legion-backups/
# Cleanup old backups (keep last 7 days)
find "$BACKUP_DIR" -name "*.dump" -mtime +7 -delete
echo "Backup completed: ${BACKUP_NAME}"set -euo pipefail aborts the script on the first error, an unset variable or a failure in a pipeline. The :? line is a fix: in the first version an empty DATABASE_URL made pg_dump silently connect to the default local database. The custom format is already compressed, so the separate gzip is gone, and pg_restore --list checks that the file can be read.
In MongoDB, which many NestJS applications run on, the same strategy looks like this:
#!/bin/bash
# backup-mongo.sh - the same strategy for MongoDB
set -euo pipefail
: "${MONGODB_URI:?Set MONGODB_URI}"
BACKUP_DIR="${BACKUP_DIR:-/backups/mongo}"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
ARCHIVE="$BACKUP_DIR/imperium_$TIMESTAMP.archive.gz"
mkdir -p "$BACKUP_DIR"
# Dump the database into a single compressed archive
mongodump --uri="$MONGODB_URI" --archive="$ARCHIVE" --gzip
# Verification: a trial restore without writing data
mongorestore --uri="$MONGODB_URI" --archive="$ARCHIVE" --gzip --dryRun
# Checksum for verifying the copy after transfer
sha256sum "$ARCHIVE" > "$ARCHIVE.sha256"
# Rotation: delete copies older than 7 days
find "$BACKUP_DIR" -name "imperium_*" -mtime +7 -delete
echo "Backup completed: $ARCHIVE"mongodump writes the database into a single compressed archive, mongorestore --dryRun tries to read it without writing any data, and the checksum lets you verify the copy after transfer. A dry run is not a test, though: a copy you have never restored into a separate database is only hope.
Disaster Recovery Plan
It is worth writing the recovery plan as code, so that nobody skips a step under stress:
// Recovery service
@Injectable()
export class RecoveryService {
async restoreFromBackup(backupFile: string) {
// 1. Download backup from S3
const backup = await this.downloadBackup(backupFile);
// 2. Create new database
await this.createRecoveryDatabase();
// 3. Restore data
await this.restoreData(backup);
// 4. Verify integrity
await this.verifyDataIntegrity();
// 5. Switch traffic
await this.switchTraffic();
}
}The service methods are a skeleton to fill in for your infrastructure. The order matters: you create the new database next to the damaged one, not on top of it, to preserve evidence for analysis, and you check integrity with record counts, a checksum and an application smoke test before switching traffic. The whole disaster recovery process goes like this:
- detect and assess the incident,
- activate the disaster recovery plan,
- restore services from backups,
- validate integrity and resume operations.
Infrastructure as code
Data is half of the recovery - the other half is infrastructure. Infrastructure as Code (IaC), e.g. Terraform or Pulumi, describes servers, networks and databases in files kept in a repository. It gives versioning, reproducibility and automation: after a disaster you recreate the environment with a single command, even in another region, instead of clicking through a console from memory.
The application image must be just as trustworthy. A secure Docker image build pipeline has these stages:
- an optimized Dockerfile with a multi-stage build,
- security and vulnerability scanning, e.g. Trivy,
- tagging the image and pushing it to a registry,
- image signing and attestation, e.g. with cosign.
The signature is created after the push, because it refers to a specific image digest in the registry; during recovery you deploy only images with a valid signature.
I recommend practising a full restore on a fresh environment once a month and measuring how long it takes - it is the only honest test of your RTO. In the project that closes the module you will combine backups with the whole deployment.
Remember: you can rebuild a fort from plans, but you cannot recover tributes without a copy that someone has actually checked.
Code for this lesson: scripts/backup.sh
#!/bin/bash
# Backup and Recovery - Saving the Tributes after a Disaster
# === CONFIGURATION ===
BACKUP_DIR="/backups/roman-empire"
DB_NAME="roman_empire"
DB_HOST="localhost"
DB_PORT="27017"
RETENTION_DAYS=7
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
BACKUP_PATH="$BACKUP_DIR/$TIMESTAMP"
echo "=== Roman Empire Database Backup ==="
echo "Timestamp: $TIMESTAMP"
echo "Database: $DB_NAME"
# === 1. MONGODUMP ===
mkdir -p "$BACKUP_PATH"
echo "Starting mongodump..."
mongodump \
--host "$DB_HOST" \
--port "$DB_PORT" \
--db "$DB_NAME" \
--out "$BACKUP_PATH"
# Check whether the backup succeeded
if [ $? -eq 0 ]; then
echo "Backup successful!"
else
echo "ERROR: Backup FAILED!"
exit 1
fi
# === 2. COMPRESSION ===
echo "Compressing backup..."
tar -czf "$BACKUP_PATH.tar.gz" -C "$BACKUP_DIR" "$TIMESTAMP"
rm -rf "$BACKUP_PATH"
echo "Compressed: $BACKUP_PATH.tar.gz"
# === 3. ROTATION (remove old backups) ===
echo "Removing backups older than $RETENTION_DAYS days..."
find "$BACKUP_DIR" -name "*.tar.gz" \
-mtime +$RETENTION_DAYS -delete
# === 4. RESTORE (recovery) ===
# mongorestore \
# --host "$DB_HOST" \
# --port "$DB_PORT" \
# --db "$DB_NAME" \
# --drop \
# "$BACKUP_PATH/$DB_NAME"
# === 5. CRON - automatic backups ===
# Add to crontab:
# 0 2 * * * /scripts/backup.sh >> /var/log/backup.log 2>&1
# (daily at 2:00 AM)
# === 6. BACKUP STRATEGY ===
# - Daily: full backup every 24h
# - Incremental: changes every 6h
# - Offsite: a copy on an external server
# - Test restore: test recovery every month!
echo "=== Backup Complete ==="
echo "File: $BACKUP_PATH.tar.gz"
echo "Size: $(du -h "$BACKUP_PATH.tar.gz" | cut -f1)"
Spotted a mistake in this lesson?
Check yourself
Answer the questions from this lesson. Pick an answer to see right away whether it is correct.
1. Infrastructure as Code (IaC) enables:
2. RTO (Recovery Time Objective) and RPO (Recovery Point Objective) define:
Hands-on tasks in the game
- Code editor
Write a backup script with mongodump, rotation, and integrity verification
- Vertical ordering
Arrange the disaster recovery process steps:
- Vertical ordering
Arrange the stages of a secure Docker image build pipeline: