Graceful Shutdown and Recovery - An Orderly Retreat of the Legions

Legion commander! Even the mighty Roman Empire had to know how to withdraw its troops in an orderly fashion. When Caesar ordered a retreat, every centurion knew exactly what to do - secure equipment, evacuate the wounded, burn bridges behind them. In NestJS, graceful shutdown is the same art - closing the application in a way that leaves no chaos behind.

What Is Graceful Shutdown?

When a NestJS application receives a shutdown signal (e.g., SIGTERM during deployment), it must:

• Complete active requests - do not abandon legionaries in the middle of battle
• Close database connections - secure the Empire archives
• Clear timers and intervals - recall the scouts
• Disconnect WebSocket clients - recall the messengers
• Save application state - write the battle report

Lifecycle Hooks in NestJS

NestJS offers special lifecycle hooks for handling shutdown. They are called in a specific order:

1// graceful-shutdown.service.ts
2import {
3  Injectable,
4  OnModuleDestroy,
5  BeforeApplicationShutdown,
6  OnApplicationShutdown,
7  Logger
8} from '@nestjs/common';
9
10@Injectable()
11export class LegionShutdownService
12  implements OnModuleDestroy, BeforeApplicationShutdown, OnApplicationShutdown
13{
14  private readonly logger = new Logger(LegionShutdownService.name);
15  private activeRequests = 0;
16
17  // 1. OnModuleDestroy - first signal to retreat
18  // Called when the module is destroyed
19  async onModuleDestroy() {
20    this.logger.warn('OnModuleDestroy: Initiating legion retreat procedure!');
21    // Close module-specific connections
22  }
23
24  // 2. BeforeApplicationShutdown - preparing for shutdown
25  // Receives signal (SIGTERM, SIGINT) as argument
26  async beforeApplicationShutdown(signal?: string) {
27    this.logger.warn(
28      `BeforeApplicationShutdown: Signal ${signal} - waiting for operations to complete`
29    );
30
31    // Wait until all active requests are completed
32    while (this.activeRequests > 0) {
33      this.logger.log(`Remaining ${this.activeRequests} active operations...`);
34      await new Promise(resolve => setTimeout(resolve, 1000));
35    }
36  }
37
38  // 3. OnApplicationShutdown - final shutdown
39  async onApplicationShutdown(signal?: string) {
40    this.logger.warn(`OnApplicationShutdown: Signal ${signal} - closing the gates of the Empire!`);
41    // Final resource cleanup
42  }
43}

Activating Shutdown Hooks

By default, NestJS does not listen for system signals. You must activate them in

main.ts

1// main.ts
2import { NestFactory } from '@nestjs/core';
3import { AppModule } from './app.module';
4import { Logger } from '@nestjs/common';
5
6async function bootstrap() {
7  const app = await NestFactory.create(AppModule);
8  const logger = new Logger('Bootstrap');
9
10  // Activate listening for SIGTERM and SIGINT signals
11  app.enableShutdownHooks();
12
13  // SIGTERM - sent by Docker/Kubernetes when stopping
14  // SIGINT  - sent by Ctrl+C in terminal
15
16  const port = process.env.PORT || 3000;
17  await app.listen(port);
18  logger.log(`Empire listening on port ${port}`);
19}
20
21bootstrap();

Safely Closing Database Connections

One of the most important aspects is safely disconnecting from the database:

1// database-shutdown.service.ts
2import { Injectable, OnModuleDestroy, Logger } from '@nestjs/common';
3import { InjectConnection } from '@nestjs/mongoose';
4import { Connection } from 'mongoose';
5
6@Injectable()
7export class DatabaseShutdownService implements OnModuleDestroy {
8  private readonly logger = new Logger(DatabaseShutdownService.name);
9
10  constructor(
11    @InjectConnection() private readonly connection: Connection,
12  ) {}
13
14  async onModuleDestroy() {
15    this.logger.warn('Closing connection to the Empire database...');
16
17    try {
18      // Mongoose - close connection gracefully
19      await this.connection.close();
20      this.logger.log('Empire archives secured - connection closed.');
21    } catch (error) {
22      this.logger.error('Error closing database:', error.message);
23    }
24  }
25}

Cleaning Up Timers and Intervals

Unclosed timers can block the Node.js process from closing:

1// scheduler-cleanup.service.ts
2import { Injectable, OnModuleDestroy, Logger } from '@nestjs/common';
3
4@Injectable()
5export class SchedulerCleanupService implements OnModuleDestroy {
6  private readonly logger = new Logger(SchedulerCleanupService.name);
7  private intervals: NodeJS.Timeout[] = [];
8  private timeouts: NodeJS.Timeout[] = [];
9
10  // Register intervals when creating
11  registerInterval(callback: () => void, ms: number): NodeJS.Timeout {
12    const interval = setInterval(callback, ms);
13    this.intervals.push(interval);
14    return interval;
15  }
16
17  registerTimeout(callback: () => void, ms: number): NodeJS.Timeout {
18    const timeout = setTimeout(callback, ms);
19    this.timeouts.push(timeout);
20    return timeout;
21  }
22
23  async onModuleDestroy() {
24    this.logger.warn('Recalling scouts - cleaning up timers and intervals...');
25
26    this.intervals.forEach(interval => clearInterval(interval));
27    this.logger.log(`Cleared ${this.intervals.length} intervals`);
28
29    this.timeouts.forEach(timeout => clearTimeout(timeout));
30    this.logger.log(`Wyczyszczono ${this.timeouts.length} timeouts`);
31
32    this.intervals = [];
33    this.timeouts = [];
34  }
35}

Health Check Recovery Pattern

When the application experiences problems, we need a recovery mechanism - an automatic return to health:

1// health-recovery.service.ts
2import { Injectable, Logger } from '@nestjs/common';
3
4@Injectable()
5export class HealthRecoveryService {
6  private readonly logger = new Logger(HealthRecoveryService.name);
7  private isHealthy = true;
8  private failureCount = 0;
9  private readonly MAX_FAILURES = 3;
10
11  reportFailure(component: string) {
12    this.failureCount++;
13    this.logger.warn(
14      `Component failure ${component}! Counter: ${this.failureCount}/${this.MAX_FAILURES}`
15    );
16
17    if (this.failureCount >= this.MAX_FAILURES) {
18      this.isHealthy = false;
19      this.logger.error('Empire in critical state! Starting recovery procedure...');
20      this.startRecovery();
21    }
22  }
23
24  private async startRecovery() {
25    this.logger.warn('Recovery procedure: attempting to restore services...');
26
27    try {
28      await this.reconnectDatabase();
29      await this.clearCache();
30      this.failureCount = 0;
31      this.isHealthy = true;
32      this.logger.log('Recovery completed successfully!');
33    } catch (error) {
34      this.logger.error('Recovery failed:', error.message);
35    }
36  }
37
38  private async reconnectDatabase() {
39    this.logger.log('Reconnecting to archives...');
40    await new Promise(resolve => setTimeout(resolve, 1000));
41  }
42
43  private async clearCache() {
44    this.logger.log('Clearing cache compartments...');
45    await new Promise(resolve => setTimeout(resolve, 500));
46  }
47
48  getHealthStatus() {
49    return {
50      healthy: this.isHealthy,
51      failureCount: this.failureCount,
52      maxFailures: this.MAX_FAILURES,
53    };
54  }
55}

Circuit Breaker Pattern - Protecting the Walls of the Empire

Circuit Breaker is a pattern that protects the application from cascading failures - like closing city gates when the enemy attacks:

1// circuit-breaker.service.ts
2import { Injectable, Logger } from '@nestjs/common';
3
4enum CircuitState {
5  CLOSED = 'CLOSED',       // Everything works - gates open
6  OPEN = 'OPEN',           // Failure - gates closed
7  HALF_OPEN = 'HALF_OPEN'  // Testing - gates ajar
8}
9
10@Injectable()
11export class CircuitBreakerService {
12  private readonly logger = new Logger(CircuitBreakerService.name);
13  private state = CircuitState.CLOSED;
14  private failureCount = 0;
15  private lastFailureTime = 0;
16  private readonly FAILURE_THRESHOLD = 5;
17  private readonly RECOVERY_TIMEOUT = 30000; // 30 sekund
18
19  async execute<T>(operation: () => Promise<T>): Promise<T> {
20    if (this.state === CircuitState.OPEN) {
21      if (Date.now() - this.lastFailureTime > this.RECOVERY_TIMEOUT) {
22        this.state = CircuitState.HALF_OPEN;
23        this.logger.warn('Circuit HALF_OPEN - testing one messenger...');
24      } else {
25        throw new Error('Circuit OPEN - Empire gates closed!');
26      }
27    }
28
29    try {
30      const result = await operation();
31
32      if (this.state === CircuitState.HALF_OPEN) {
33        this.state = CircuitState.CLOSED;
34        this.failureCount = 0;
35        this.logger.log('Circuit CLOSED - gates open again!');
36      }
37
38      return result;
39    } catch (error) {
40      this.failureCount++;
41      this.lastFailureTime = Date.now();
42
43      if (this.failureCount >= this.FAILURE_THRESHOLD) {
44        this.state = CircuitState.OPEN;
45        this.logger.error(
46          `Circuit OPEN po ${this.failureCount} failures - closing the gates!`
47        );
48      }
49
50      throw error;
51    }
52  }
53
54  getState() {
55    return {
56      state: this.state,
57      failureCount: this.failureCount,
58      threshold: this.FAILURE_THRESHOLD,
59    };
60  }
61}

Handling SIGTERM and SIGINT Signals

It is worth knowing when and why we receive these signals:

• SIGTERM - Docker/Kubernetes sends when stopping a container (you have ~30s to shut down)
• SIGINT - Ctrl+C in terminal (immediate shutdown)

1// main.ts - advanced signal handling
2process.on('SIGTERM', () => {
3  console.log('Received SIGTERM - starting graceful shutdown...');
4});
5
6process.on('SIGINT', () => {
7  console.log('Received SIGINT - immediate shutdown...');
8});
9
10// Catch unhandled exceptions
11process.on('uncaughtException', (error) => {
12  console.error('Unhandled exception:', error);
13  process.exit(1);
14});
15
16process.on('unhandledRejection', (reason) => {
17  console.error('Unhandled promise rejection:', reason);
18});

Graceful shutdown and recovery are the foundations of a reliable Empire. As Seneca said: "Luck favors the prepared" - an application that can safely shut down and recover from failure is like a legion ready for any eventuality.