NestJS course ยท Module 9: Deployment and Infrastructure
Blue-Green Deployment and Zero-Downtime - the art of seamless guard change
In this lesson6
Master of strategy! Senator Cicero says: "The Empire never sleeps, and the roads must always be open." Meanwhile every deployment means half a minute of 502 errors, because the old process has already gone out and the new one has not yet risen. The Roman legions changed the guard without leaving the walls undefended - applications must also update without interruptions.
Blue-Green Deployment is a technique in which we keep two identical production environments (blue and green). At any given moment only one serves traffic, and the other receives the new version. The switch happens instantly, with no downtime.
The Blue-Green Deployment concept
Imagine two city gates. Blue is open and serves travellers, while in green the guards install new fortifications. When green is ready, traffic moves over to it. The configuration of both slots is described by two interfaces:
// src/deployment/blue-green.config.ts
interface DeploymentSlot {
name: 'blue' | 'green';
port: number;
version: string;
status: 'active' | 'standby' | 'deploying';
healthCheckUrl: string;
startedAt: Date;
}
interface BlueGreenConfig {
activeSlot: 'blue' | 'green';
slots: {
blue: DeploymentSlot;
green: DeploymentSlot;
};
loadBalancerUrl: string;
healthCheckInterval: number; // ms
healthCheckRetries: number;
rollbackTimeout: number; // ms
}
const deploymentConfig: BlueGreenConfig = {
activeSlot: 'blue',
slots: {
blue: {
name: 'blue',
port: 3001,
version: '2.3.0',
status: 'active',
healthCheckUrl: 'http://blue.imperium.internal:3001/health',
startedAt: new Date(),
},
green: {
name: 'green',
port: 3002,
version: '2.4.0',
status: 'standby',
healthCheckUrl: 'http://green.imperium.internal:3002/health',
startedAt: new Date(),
},
},
loadBalancerUrl: 'http://lb.imperium.internal',
healthCheckInterval: 5000,
healthCheckRetries: 3,
rollbackTimeout: 30000,
};activeSlot points to the gate with traffic, and the switch is a change of target in the load balancer. A rollback means switching back, because the old version is still running. The price is double resources, and long-lived connections such as WebSockets must be moved deliberately.
Rolling Updates
A rolling update gradually replaces old instances with new ones, like centurions changing the guard, so some instances always serve traffic:
// src/deployment/rolling-update.service.ts
import { Injectable, Logger } from '@nestjs/common';
interface Instance {
id: string;
version: string;
status: 'running' | 'updating' | 'ready' | 'failed';
port: number;
}
@Injectable()
export class RollingUpdateService {
private readonly logger = new Logger(RollingUpdateService.name);
async performRollingUpdate(
instances: Instance[],
newVersion: string,
maxUnavailable: number = 1,
): Promise<void> {
this.logger.log(
`Starting rolling update to version ${newVersion}`
);
this.logger.log(
`Instances: ${instances.length}, max unavailable: ${maxUnavailable}`
);
// Update instances in batches - the whole batch at once
for (let i = 0; i < instances.length; i += maxUnavailable) {
const batch = instances.slice(i, i + maxUnavailable);
const results = await Promise.all(
batch.map((instance) => this.updateInstance(instance, newVersion)),
);
// A failed instance stops the whole rollout
if (results.includes(false)) {
throw new Error(`Rolling update to version ${newVersion} halted`);
}
}
}Promise.all updates the whole batch at once. The first version had a loop with await inside the batch, so despite the maxUnavailable parameter it always took down one instance at a time. A failed instance now stops the whole rollout instead of breaking the next ones - in a test with six instances the last two were left untouched.
Every instance goes through five steps in a fixed order:
private async updateInstance(instance: Instance, newVersion: string): Promise<boolean> {
instance.status = 'updating';
this.logger.log(`Updating instance ${instance.id}...`);
// 1. Remove the instance from the load balancer
await this.removeFromLoadBalancer(instance);
// 2. Wait for current requests to finish
await this.drainConnections(instance);
// 3. Update to the new version
await this.deployNewVersion(instance, newVersion);
// 4. Check the health check
const healthy = await this.waitForHealthy(instance);
if (healthy) {
instance.status = 'running';
instance.version = newVersion;
// 5. Add the instance back to the load balancer
await this.addToLoadBalancer(instance);
this.logger.log(`Instance ${instance.id} updated`);
return true;
}
instance.status = 'failed';
this.logger.error(
`Instance ${instance.id} - rollback!`
);
await this.rollback(instance);
return false;
}The instance leaves the load balancer, finishes ongoing requests, gets the new version, passes the health check and only then returns to the pool.
The helper methods are a skeleton to connect to your infrastructure:
private async removeFromLoadBalancer(instance: Instance) {
// Remove the instance from the load balancer pool
}
private async drainConnections(instance: Instance) {
// Wait for active connections to finish (graceful)
}
private async deployNewVersion(
instance: Instance, version: string
) {
// Deploy the new version to the instance
}
private async waitForHealthy(instance: Instance): Promise<boolean> {
const maxRetries = 10;
for (let i = 0; i < maxRetries; i++) {
try {
// Check the /health/ready endpoint of the new version
const response = await fetch(`http://localhost:${instance.port}/health/ready`, {
signal: AbortSignal.timeout(2000),
});
if (response.ok) return true;
} catch {
// The instance is not responding yet
}
await new Promise(r => setTimeout(r, 2000)); // pause after every attempt
}
return false;
}
private async addToLoadBalancer(instance: Instance) {
// Add the instance back to the pool
}
private async rollback(instance: Instance) {
// Restore the previous version and add the instance back to the pool
}
}waitForHealthy() polls /health/ready with a timeout and waits after every attempt. The first version waited only after an exception, so ten failed responses flew by in a fraction of a second.
Health Check-based Deployment
Health checks are the Empire's scouts - they check whether a new fortification is ready before we open the gates:
// src/health/deployment-health.controller.ts
import { Controller, Get } from '@nestjs/common';
import {
HealthCheck, HealthCheckService,
MongooseHealthIndicator, MemoryHealthIndicator,
DiskHealthIndicator,
} from '@nestjs/terminus';
@Controller('health')
export class DeploymentHealthController {
constructor(
private health: HealthCheckService,
private mongoose: MongooseHealthIndicator,
private memory: MemoryHealthIndicator,
private disk: DiskHealthIndicator,
) {}
// Liveness - is the application alive?
@Get('live')
@HealthCheck()
checkLiveness() {
return this.health.check([
() => this.memory.checkHeap('memory_heap', 200 * 1024 * 1024),
]);
}
// Readiness - is it ready for traffic?
@Get('ready')
@HealthCheck()
checkReadiness() {
return this.health.check([
() => this.mongoose.pingCheck('mongodb'),
() => this.memory.checkHeap('memory_heap', 200 * 1024 * 1024),
() => this.disk.checkStorage('disk', {
thresholdPercent: 0.9, path: '/',
}),
]);
}
// Startup - did the application start correctly?
@Get('startup')
@HealthCheck()
checkStartup() {
return this.health.check([
() => this.mongoose.pingCheck('mongodb'),
]);
}
}Liveness says whether the process is alive, readiness whether it can take traffic, and startup whether the application started correctly. TerminusModule.forRoot({ gracefulShutdownTimeoutMs: 10000 }) completes zero-downtime: after SIGTERM readiness returns 503 with the status shutting_down, while the application keeps serving traffic for another 10 seconds. In the test ordinary endpoints answered 200 during that time, so Kubernetes has time to take the pod out of the pool.
Database migrations - zero-downtime
Migrations are the hardest part of a zero-downtime deployment, because old and new code coexist with the same database. The rule: a migration must be backward compatible. ALTER TABLE legions RENAME COLUMN name TO legion_name causes downtime, because the old code looks for name and crashes. A safe field rename takes three deployments. The first adds the new field:
// Migration 1 (deployment 1): add the new legion_name field
import { Db } from 'mongodb';
export async function up(db: Db) {
await db.collection('legions').updateMany(
{},
[{ $set: { legion_name: '$name' } }]
);
// Old code still reads 'name' - it works!
}The pipeline update copies the value of name into legion_name in every MongoDB document, and the old code notices nothing.
The second deployment changes the code:
// Code v2 (deployment 2): write to both fields
async updateLegion(id: string, name: string) {
await this.model.updateOne(
{ _id: id },
{ $set: { name, legion_name: name } }
);
}
// Read from the new field
async getLegion(id: string) {
const doc = await this.model.findById(id);
return doc?.legion_name ?? doc?.name; // Fallback
}Code v2 writes to both fields and reads the new one, falling back to the old one for documents the migration did not cover.
The third cleans up:
// Migration 3 (after v2 is fully deployed): remove the old field
import { Db } from 'mongodb';
export async function up(db: Db) {
// First fill in documents written by old code during the rollout
await db.collection('legions').updateMany(
{ legion_name: { $exists: false } },
[{ $set: { legion_name: '$name' } }]
);
await db.collection('legions').updateMany(
{},
{ $unset: { name: '' } }
);
}First we fill in documents written by the old code after the first migration. The first version removed name straight away, and in the test a document added during the rollout lost its name.
Feature Flags
Feature flags are the emperor's secret orders - they turn features on and off without deploying new code:
// src/features/feature-flag.service.ts
import { Injectable } from '@nestjs/common';
interface FeatureFlag {
name: string;
enabled: boolean;
rolloutPercentage: number; // 0-100
allowedUsers: string[];
}
@Injectable()
export class FeatureFlagService {
private flags: Map<string, FeatureFlag> = new Map();
setFlag(flag: FeatureFlag) {
this.flags.set(flag.name, flag);
}
isEnabled(flagName: string, userId?: string): boolean {
const flag = this.flags.get(flagName);
if (!flag) return false;
// Flag completely disabled
if (!flag.enabled) return false;
// User on the allow list (e.g. testers)
if (userId && flag.allowedUsers.includes(userId)) {
return true;
}
// Gradual rollout (percentage rollout)
if (flag.rolloutPercentage < 100) {
// Flag name in the hash - every flag picks different users
const hash = this.hashUserId(`${flagName}:${userId || 'anonymous'}`);
return (hash % 100) < flag.rolloutPercentage;
}
return true;
}
private hashUserId(userId: string): number {
let hash = 0;
for (let i = 0; i < userId.length; i++) {
hash = ((hash << 5) - hash) + userId.charCodeAt(i);
hash |= 0;
}
return Math.abs(hash);
}
}I added setFlag(), because nothing filled the map before. The flag name in the hash makes every flag pick different users: without it, in the test the same 20% of users ended up in every rollout. In production flags are kept in a database or in a service such as Unleash.
The controller asks the service on every request:
// Usage in a controller
@Controller('legions')
class LegionsController {
constructor(private features: FeatureFlagService) {}
@Get()
async getLegions(@Req() req) {
if (this.features.isEnabled('new-ranking-system', req.user?.id)) {
return this.getNewRanking();
}
return this.getOldRanking();
}
}I left out the ranking methods, and req.user is set by the JWT guard.
PM2 Cluster Mode
PM2 in cluster mode is like deploying many garrisons - it uses all CPU cores, restarts processes and reloads them without downtime:
// ecosystem.config.cjs - PM2 configuration (.cjs, because a NestJS 12 project is an ES module)
module.exports = {
apps: [{
name: 'imperium-api',
script: 'dist/main.js',
instances: 'max', // Use all CPU cores
exec_mode: 'cluster', // Cluster mode
autorestart: true,
watch: false,
max_memory_restart: '500M',
// Zero-downtime reload
wait_ready: true, // Wait for process.send('ready')
listen_timeout: 10000, // Max time to become ready
kill_timeout: 5000, // Time for graceful shutdown
env_production: {
NODE_ENV: 'production',
PORT: 3000,
},
}],
};The file has the .cjs extension, because module.exports does not work in a project with "type": "module". wait_ready makes PM2 wait for the readiness signal (3 s by default, here listen_timeout is 10 s), and kill_timeout replaces the default 1.6 s for shutting down.
The readiness signal is sent by main.ts:
// In main.ts - signalling readiness
async function bootstrap() {
const app = await NestFactory.create(AppModule);
app.enableShutdownHooks(); // PM2 stops the process with SIGINT
await app.listen(3000);
// Tell PM2 the application is ready
if (process.send) {
process.send('ready');
}
}PM2 stops processes with SIGINT, which enableShutdownHooks() turns into an orderly retreat. process.send exists only when the process has an IPC channel to PM2.
PM2 commands for zero-downtime:
# Start from the configuration file (a .cjs file must be named explicitly)
pm2 start ecosystem.config.cjs --env production
# Reload without downtime (graceful)
pm2 reload imperium-api
# Instance status
pm2 status
# Real-time monitoring
pm2 monitIn cluster mode pm2 reload restarts processes one by one, so one of them always serves traffic.
Blue-green, rolling updates and feature flags are powerful tools in the arsenal of every architect of the Empire. I recommend starting with rolling updates and good probes and keeping blue-green for systems where a rollback must be instant.
Remember: the guard is changed so that the wall never stands empty, not even for a moment.
Code for this lesson: src/blue-green-deployment.ts
// Blue-Green Deployment and Zero-Downtime
// Deployment strategies without downtime
// 1. Blue-Green configuration
interface DeploymentSlot {
name: 'blue' | 'green';
port: number;
version: string;
status: 'active' | 'standby' | 'deploying';
}
const slots: Record<string, DeploymentSlot> = {
blue: {
name: 'blue',
port: 3001,
version: '2.3.0',
status: 'active',
},
green: {
name: 'green',
port: 3002,
version: '2.4.0',
status: 'standby',
},
};
// 2. Health Check Controller
// @Controller('health')
// class HealthController {
// @Get('live')
// liveness() { return { status: 'ok' }; }
//
// @Get('ready')
// readiness() {
// // TODO: Check MongoDB, memory, disk
// }
// }
// 3. Feature Flag Service
class FeatureFlagService {
private flags = new Map<string, { enabled: boolean; rollout: number }>();
isEnabled(flag: string, userId?: string): boolean {
const f = this.flags.get(flag);
if (!f || !f.enabled) return false;
if (f.rollout < 100 && userId) {
// TODO: Implement a canary release
// Use hash userId % 100 < rollout
}
return true;
}
}
// 4. PM2 Cluster Config
// ecosystem.config.js:
// {
// name: 'imperium-api',
// script: 'dist/main.js',
// instances: 'max',
// exec_mode: 'cluster',
// wait_ready: true,
// listen_timeout: 10000,
// }
// TODO: Implement a full rolling update service
// TODO: Add a database migration strategy (3 steps)
// TODO: Configure PM2 with graceful shutdown
Docs entries used in this lesson
Spotted a mistake in this lesson?
Check yourself
Answer the questions from this lesson. Pick an answer to see right away whether it is correct.
1. Blue-Green Deployment consists of:
Hands-on tasks in the game
- Code editor
Create a FeatureFlagService with an isEnabled method supporting rolloutPercentage and blue-green deployment configuration
- Vertical ordering
Arrange the rolling update steps for a single instance:
- Code editor
Create a .github/workflows/deploy.yml file with jobs: test, build, deploy triggered on push to main
- Click in order
Arrange the correct order of steps in a complete CI/CD pipeline:
- Vertical ordering
Arrange the correct health check decorator in a NestJS controller:
- Code editor
Create a configuration with Joi schema validating DATABASE_URL, PORT, NODE_ENV, and JWT_SECRET
- Click in order
Arrange security middleware in the recommended order of adding to main.ts:
Tasks after this lesson
- QuestionThe Blue-Green Deployment Concept
- Coding taskPractice: Blue-Green and Feature Flags
- Put in orderRolling Update Stages
- Coding taskPractice: Complete Deployment Pipeline
- Click in orderDeployment Pipeline Steps
- Put in orderHealth Check in Production Configuration
- Coding taskPractice: Environment Validation
- Click in orderNestJS Middleware Hardening Order
- Project to submitProject: The Empire's Production Infrastructure
- VideoModule Summary