Graceful Shutdown and Recovery - An Orderly Retreat of the Legions

You deploy a new version. Kubernetes sends a termination signal, the process disappears in half a second, and three payments half-way through being saved disappear with it. Even the mighty Roman Empire had to know how to withdraw its troops in good order: when Caesar ordered a retreat, every centurion knew how to secure the equipment and evacuate the wounded. In NestJS graceful shutdown is the same art - closing the application without leaving chaos behind.

What is Graceful Shutdown?

When an application receives a termination signal (e.g. SIGTERM during a deployment), it must:

• finish active requests - do not abandon legionaries mid-battle,
• close database connections - secure the Empire's archives,
• clear timers and intervals - recall the scouts,
• disconnect WebSocket clients - recall the messengers,
• save the application state - write a report from the battlefield.

Lifecycle Hooks in NestJS

NestJS calls three hooks in a fixed order. onModuleDestroy() runs after the signal, while the HTTP server still serves traffic. beforeApplicationShutdown(signal) receives the signal name; after it NestJS closes the HTTP server and waits for ongoing requests. onApplicationShutdown(signal) comes last, when the server has gone quiet:

1// graceful-shutdown.service.ts
2import {
3  Injectable,
4  OnModuleDestroy,
5  BeforeApplicationShutdown,
6  OnApplicationShutdown,
7  Logger
8} from '@nestjs/common';
9
10@Injectable()
11export class LegionShutdownService
12  implements OnModuleDestroy, BeforeApplicationShutdown, OnApplicationShutdown
13{
14  private readonly logger = new Logger(LegionShutdownService.name);
15  private activeOperations = 0;
16
17  // Background operations (imports, queue jobs) report their start and end
18  operationStarted() { this.activeOperations++; }
19  operationFinished() { this.activeOperations--; }
20
21  // 1. OnModuleDestroy - the first signal to retreat
22  // The HTTP server still accepts requests
23  async onModuleDestroy() {
24    this.logger.warn('OnModuleDestroy: Starting the legions retreat procedure!');
25    // Stop what requests no longer need (timers, subscriptions)
26  }
27
28  // 2. BeforeApplicationShutdown - preparing to shut down
29  // Receives the signal (SIGTERM, SIGINT) as an argument
30  async beforeApplicationShutdown(signal?: string) {
31    this.logger.warn(
32      `BeforeApplicationShutdown: Signal ${signal} - waiting for operations to finish`
33    );
34
35    // Wait for background operations, but for 8 seconds at most
36    const deadline = Date.now() + 8000;
37    while (this.activeOperations > 0 && Date.now() < deadline) {
38      this.logger.log(`${this.activeOperations} active operations left...`);
39      await new Promise(resolve => setTimeout(resolve, 1000));
40    }
41  }
42
43  // 3. OnApplicationShutdown - the final shutdown
44  // The HTTP server is already closed
45  async onApplicationShutdown(signal?: string) {
46    this.logger.warn(`OnApplicationShutdown: Signal ${signal} - closing the gates of the Empire!`);
47    // Final resource cleanup
48  }
49}

NestJS waits for HTTP requests by itself, so the counter only guards background work. The 8-second limit is not arbitrary: docker stop sends SIGKILL after 10 seconds, and that one cannot be handled.

Enabling Shutdown Hooks

By default NestJS does not listen to system signals, so without one line in main.ts none of these hooks runs after SIGTERM:

1// main.ts
2import { NestFactory } from '@nestjs/core';
3import { AppModule } from './app.module';
4import { Logger } from '@nestjs/common';
5
6async function bootstrap() {
7  const app = await NestFactory.create(AppModule);
8  const logger = new Logger('Bootstrap');
9
10  // Enable listening for the SIGTERM and SIGINT signals
11  app.enableShutdownHooks();
12
13  // SIGTERM - sent by Docker/Kubernetes when stopping
14  // SIGINT  - sent by Ctrl+C in the terminal
15
16  const port = process.env.PORT || 3000;
17  await app.listen(port);
18  logger.log(`The Empire is listening on port ${port}`);
19}
20
21bootstrap();

We call enableShutdownHooks() before listen(). On Windows SIGTERM does not work at all, but SIGINT does.

Closing database connections safely

We close the database only in onApplicationShutdown, because before that the HTTP server still serves requests that use it:

1// database-shutdown.service.ts
2import { Injectable, OnApplicationShutdown, Logger } from '@nestjs/common';
3import { InjectConnection } from '@nestjs/mongoose';
4import { Connection } from 'mongoose';
5
6@Injectable()
7export class DatabaseShutdownService implements OnApplicationShutdown {
8  private readonly logger = new Logger(DatabaseShutdownService.name);
9
10  constructor(
11    @InjectConnection() private readonly connection: Connection,
12  ) {}
13
14  async onApplicationShutdown() {
15    this.logger.warn('Closing the connection to the Empire database...');
16
17    try {
18      // Mongoose - close the connection gracefully
19      await this.connection.close();
20      this.logger.log('Empire archive secured - connection closed.');
21    } catch (error) {
22      this.logger.error('Error while closing the database:', error.message);
23    }
24  }
25}

The first version of this lesson closed the connection in onModuleDestroy - too early, because requests being served at that moment lost their database. MongooseModule and TypeOrmModule close their own connections, precisely in onApplicationShutdown; you write a service like this one for connections you create by hand.

Clearing timers and intervals

Timers left running can keep the Node.js process from exiting, and cleaning them up is a good job for onModuleDestroy:

1// scheduler-cleanup.service.ts
2import { Injectable, OnModuleDestroy, Logger } from '@nestjs/common';
3
4@Injectable()
5export class SchedulerCleanupService implements OnModuleDestroy {
6  private readonly logger = new Logger(SchedulerCleanupService.name);
7  private intervals: NodeJS.Timeout[] = [];
8  private timeouts: NodeJS.Timeout[] = [];
9
10  // Register intervals when creating them
11  registerInterval(callback: () => void, ms: number): NodeJS.Timeout {
12    const interval = setInterval(callback, ms);
13    this.intervals.push(interval);
14    return interval;
15  }
16
17  registerTimeout(callback: () => void, ms: number): NodeJS.Timeout {
18    const timeout = setTimeout(callback, ms);
19    this.timeouts.push(timeout);
20    return timeout;
21  }
22
23  async onModuleDestroy() {
24    this.logger.warn('Recalling the scouts - clearing timers and intervals...');
25
26    this.intervals.forEach(interval => clearInterval(interval));
27    this.logger.log(`Cleared ${this.intervals.length} intervals`);
28
29    this.timeouts.forEach(timeout => clearTimeout(timeout));
30    this.logger.log(`Cleared ${this.timeouts.length} timeouts`);
31
32    this.intervals = [];
33    this.timeouts = [];
34  }
35}

The service remembers every timer it created and clears them all at once during the retreat.

Health Check Recovery Pattern

When the application is in trouble, it needs a recovery mechanism - an automatic return to health:

1// health-recovery.service.ts
2import { Injectable, Logger } from '@nestjs/common';
3
4@Injectable()
5export class HealthRecoveryService {
6  private readonly logger = new Logger(HealthRecoveryService.name);
7  private isHealthy = true;
8  private failureCount = 0;
9  private readonly MAX_FAILURES = 3;
10
11  reportFailure(component: string) {
12    this.failureCount++;
13    this.logger.warn(
14      `Component ${component} failed! Counter: ${this.failureCount}/${this.MAX_FAILURES}`
15    );
16
17    if (this.failureCount >= this.MAX_FAILURES) {
18      this.isHealthy = false;
19      this.logger.error('The Empire is in a critical state! Starting the recovery procedure...');
20      this.startRecovery();
21    }
22  }
23
24  private async startRecovery() {
25    this.logger.warn('Recovery procedure: trying to restore services...');
26
27    try {
28      await this.reconnectDatabase();
29      await this.clearCache();
30      this.failureCount = 0;
31      this.isHealthy = true;
32      this.logger.log('Recovery completed successfully!');
33    } catch (error) {
34      this.logger.error('Recovery failed:', error.message);
35    }
36  }
37
38  private async reconnectDatabase() {
39    this.logger.log('Reconnecting to the archive...');
40    await new Promise(resolve => setTimeout(resolve, 1000));
41  }
42
43  private async clearCache() {
44    this.logger.log('Clearing the cache stores...');
45    await new Promise(resolve => setTimeout(resolve, 500));
46  }
47
48  getHealthStatus() {
49    return {
50      healthy: this.isHealthy,
51      failureCount: this.failureCount,
52      maxFailures: this.MAX_FAILURES,
53    };
54  }
55}

After three failures the service marks itself as unhealthy and tries to repair itself. startRecovery() runs in the background, without await, so whoever reports the failure does not wait.

Circuit Breaker Pattern - protecting the Empire's walls

A Circuit Breaker protects the application from cascading failures - like closing the city gates when the enemy attacks:

1// circuit-breaker.service.ts
2import { Injectable, Logger } from '@nestjs/common';
3
4enum CircuitState {
5  CLOSED = 'CLOSED',       // Everything works - gates open
6  OPEN = 'OPEN',           // Failure - gates closed
7  HALF_OPEN = 'HALF_OPEN'  // Testing - gates ajar
8}
9
10@Injectable()
11export class CircuitBreakerService {
12  private readonly logger = new Logger(CircuitBreakerService.name);
13  private state = CircuitState.CLOSED;
14  private failureCount = 0;
15  private lastFailureTime = 0;
16  private readonly FAILURE_THRESHOLD = 5;
17  private readonly RECOVERY_TIMEOUT = 30000; // 30 seconds
18
19  async execute<T>(operation: () => Promise<T>): Promise<T> {
20    if (this.state === CircuitState.OPEN) {
21      if (Date.now() - this.lastFailureTime > this.RECOVERY_TIMEOUT) {
22        this.state = CircuitState.HALF_OPEN;
23        this.logger.warn('Circuit HALF_OPEN - sending a messenger on trial...');
24      } else {
25        throw new Error('Circuit OPEN - the gates of the Empire are closed!');
26      }
27    }
28
29    try {
30      const result = await operation();
31
32      if (this.state === CircuitState.HALF_OPEN) {
33        this.state = CircuitState.CLOSED;
34        this.logger.log('Circuit CLOSED - the gates are open again!');
35      }
36      this.failureCount = 0; // we only count failures in a row
37
38      return result;
39    } catch (error) {
40      this.failureCount++;
41      this.lastFailureTime = Date.now();
42
43      if (this.failureCount >= this.FAILURE_THRESHOLD) {
44        this.state = CircuitState.OPEN;
45        this.logger.error(
46          `Circuit OPEN after ${this.failureCount} failures - closing the gates!`
47        );
48      }
49
50      throw error;
51    }
52  }
53
54  getState() {
55    return {
56      state: this.state,
57      failureCount: this.failureCount,
58      threshold: this.FAILURE_THRESHOLD,
59    };
60  }
61}

After five failures in a row the gates close and requests fail immediately instead of waiting for a dead service. After 30 seconds a trial call decides whether we return to CLOSED. Before, the counter did not reset after a success, so five failures spread over a week also opened the circuit.

Handling SIGTERM and SIGINT

It is worth knowing where signals come from:

• SIGTERM - sent by docker stop (SIGKILL after 10 s) and by Kubernetes (after 30 s by default),
• SIGINT - Ctrl+C in the terminal; with enableShutdownHooks() it also runs the full retreat.

1// main.ts - your own reactions to signals, next to enableShutdownHooks()
2process.once('SIGTERM', () => {
3  console.log('Received SIGTERM - starting graceful shutdown...');
4});
5
6process.once('SIGINT', () => {
7  console.log('Received SIGINT - starting graceful shutdown...');
8});
9
10// Catching unhandled exceptions
11process.on('uncaughtException', (error) => {
12  console.error('Unhandled exception:', error);
13  process.exit(1);
14});
15
16process.on('unhandledRejection', (reason) => {
17  console.error('Unhandled promise rejection:', reason);
18  process.kill(process.pid, 'SIGTERM'); // start the retreat instead of pretending nothing happened
19});

once matters: a permanent SIGTERM listener removes the default exit of Node.js, and after cleaning up NestJS re-sends the signal to end the process - with an on listener the application can hang. Merely listening to unhandledRejection also disables the default crash, so we start the retreat on purpose.

Test the retreat before it comes

Recovery you have not tested is only hope. In Jest tests describe groups cases, it describes one, and expect(result).toEqual(expected) compares the result with the expected value. beforeEach runs before every test, beforeAll once before all tests in a block. Since NestJS 12 new projects get Vitest with the same API, except that instead of jest.fn() you write vi.fn().

We will test a service that throws an HttpException with a response object and a status code when a legion is missing:

1// legions.service.ts
2@Injectable()
3export class LegionsService {
4  constructor(
5    @InjectRepository(Legion) private readonly repo: Repository<Legion>,
6  ) {}
7
8  async findOne(id: number) {
9    const legion = await this.repo.findOne({ where: { id } });
10    if (!legion) {
11      throw new HttpException(
12        { message: 'Legion not found', error: 'Not Found' },
13        HttpStatus.NOT_FOUND);
14    }
15    return legion;
16  }
17}

In a unit test we replace the real repository with a stand-in: getRepositoryToken(Legion) is the token under which @InjectRepository looks for the repository, and jest.fn() creates a function whose result we set ourselves:

1// legions.service.spec.ts
2describe('LegionsService', () => {
3  let service: LegionsService;
4  let repo: { findOne: jest.Mock; save: jest.Mock };
5
6  beforeEach(async () => {
7    repo = { findOne: jest.fn(), save: jest.fn() };
8
9    const moduleRef = await Test.createTestingModule({
10      providers: [
11        LegionsService,
12        { provide: getRepositoryToken(Legion), useValue: repo },
13      ],
14    }).compile();
15
16    service = moduleRef.get(LegionsService);
17  });
18
19  it('returns the legion with the given id', async () => {
20    // Arrange
21    const legion = { id: 9, name: 'Legio IX Hispana' };
22    repo.findOne.mockResolvedValue(legion);
23
24    // Act
25    const result = await service.findOne(9);
26
27    // Assert
28    expect(result).toEqual(legion);
29  });
30
31  it('throws HttpException when the legion does not exist', async () => {
32    repo.findOne.mockResolvedValue(null);
33
34    await expect(service.findOne(99)).rejects.toThrow(HttpException);
35  });
36});

The comments show the AAA pattern: Arrange prepares data and mocks, Act runs the operation, Assert checks the result. beforeEach builds a fresh module before every test, so mocks do not carry state between cases.

An end-to-end test starts the whole application and sends a real request through supertest:

1// test/health.e2e-spec.ts
2import request from 'supertest';
3import { Test } from '@nestjs/testing';
4import { INestApplication } from '@nestjs/common';
5import { AppModule } from '../src/app.module';
6
7describe('Health (e2e)', () => {
8  let app: INestApplication;
9
10  beforeAll(async () => {
11    const moduleFixture = await Test.createTestingModule({
12      imports: [AppModule],
13    }).compile();
14
15    app = moduleFixture.createNestApplication();
16    await app.init();
17  });
18
19  it('GET /health responds with 200', () => {
20    return request(app.getHttpServer()).get('/health').expect(200);
21  });
22
23  afterAll(async () => {
24    await app.close();
25  });
26});

We build the application once, in beforeAll, because it is expensive. app.close() in afterAll runs the same shutdown hooks, so the test also checks the retreat.

We test the exception filter from the first lesson of the module without a server, passing a mock ArgumentsHost:

1// legion-http-exception.filter.spec.ts
2it('should catch NotFoundException', () => {
3  const filter = new LegionHttpExceptionFilter();
4  const mockRes = { status: jest.fn().mockReturnThis(), json: jest.fn() };
5  const mockReq = { url: '/legions/99' };
6  const mockHost = {
7    switchToHttp: () => ({
8      getResponse: () => mockRes,
9      getRequest: () => mockReq,
10    }),
11  } as unknown as ArgumentsHost;
12
13  filter.catch(new NotFoundException(), mockHost);
14
15  expect(mockRes.status).toHaveBeenCalledWith(404);
16});

mockReturnThis() allows the response.status(404).json(...) chain, and toHaveBeenCalledWith(404) checks the argument the mock was called with.

There is one more catch: a filter registered with app.useGlobalFilters() in main.ts does not work in an e2e test, because the test does not run main.ts. Registering it in a module with the APP_FILTER token works everywhere and allows dependency injection:

1// app.module.ts - a global filter with Dependency Injection
2@Module({
3  providers: [
4    { provide: APP_FILTER, useClass: LegionHttpExceptionFilter },
5  ],
6})
7export class AppModule {}

You get a code coverage report with npm run test -- --coverage or the ready-made test:cov script.

I recommend a simple habit: give every shutdown hook and every filter at least one test, because failures are rare and never happen when you have time to check by hand. In the project that closes the module you will combine filters, logs, health checks and tests into one monitoring system.

Remember: fortune favours the prepared - an application that knows how to shut down and get back up after a failure is like a legion ready for anything.

1// Graceful Shutdown and Recovery - A Dignified Retreat of the Legions
2import { Injectable, OnModuleDestroy, BeforeApplicationShutdown, Logger } from '@nestjs/common';
3
4// ===========================================
5// 1. Graceful Shutdown Service
6// ===========================================
7
8@Injectable()
9export class LegionShutdownService
10  implements OnModuleDestroy, BeforeApplicationShutdown
11{
12  private readonly logger = new Logger(LegionShutdownService.name);
13  private activeRequests = 0;
14  private intervals: any[] = [];
15
16  // Track active requests
17  incrementRequests() { this.activeRequests++; }
18  decrementRequests() { this.activeRequests--; }
19
20  // Register intervals for cleanup
21  registerInterval(interval: any) {
22    this.intervals.push(interval);
23  }
24
25  // OnModuleDestroy - first step of the retreat
26  async onModuleDestroy() {
27    this.logger.warn('OnModuleDestroy: Starting the retreat procedure!');
28
29    // Clear the intervals
30    this.intervals.forEach(i => clearInterval(i));
31    this.logger.log('Cleared ' + this.intervals.length + ' intervals');
32    this.intervals = [];
33  }
34
35  // BeforeApplicationShutdown - wait for active operations
36  async beforeApplicationShutdown(signal?: string) {
37    this.logger.warn('BeforeApplicationShutdown: Signal ' + signal);
38
39    // Wait for active requests to complete
40    let waitCount = 0;
41    while (this.activeRequests > 0 && waitCount < 10) {
42      this.logger.log('Active operations: ' + this.activeRequests);
43      await new Promise(r => setTimeout(r, 1000));
44      waitCount++;
45    }
46
47    this.logger.log('All operations finished - the gates are closed');
48  }
49}
50
51// ===========================================
52// 2. Circuit Breaker Pattern
53// ===========================================
54
55enum CircuitState {
56  CLOSED = 'CLOSED',
57  OPEN = 'OPEN',
58  HALF_OPEN = 'HALF_OPEN'
59}
60
61class CircuitBreaker {
62  private state = CircuitState.CLOSED;
63  private failures = 0;
64  private lastFailure = 0;
65  private readonly threshold = 5;
66  private readonly timeout = 30000;
67
68  async execute<T>(operation: () => Promise<T>): Promise<T> {
69    if (this.state === CircuitState.OPEN) {
70      if (Date.now() - this.lastFailure > this.timeout) {
71        this.state = CircuitState.HALF_OPEN;
72      } else {
73        throw new Error('Circuit OPEN - the gates are closed!');
74      }
75    }
76
77    try {
78      const result = await operation();
79      if (this.state === CircuitState.HALF_OPEN) {
80        this.state = CircuitState.CLOSED;
81        this.failures = 0;
82      }
83      return result;
84    } catch (error) {
85      this.failures++;
86      this.lastFailure = Date.now();
87      if (this.failures >= this.threshold) {
88        this.state = CircuitState.OPEN;
89      }
90      throw error;
91    }
92  }
93
94  getState() {
95    return { state: this.state, failures: this.failures };
96  }
97}
98
99// ===========================================
100// 3. Demonstration
101// ===========================================
102
103const service = new LegionShutdownService();
104
105// Simulation of active requests
106service.incrementRequests();
107service.incrementRequests();
108console.log('=== Graceful Shutdown ===');
109console.log('Active tasks before shutdown: 2');
110
111service.decrementRequests();
112service.decrementRequests();
113console.log('Active tasks after completion: 0');
114console.log('');
115
116// Circuit Breaker
117const breaker = new CircuitBreaker();
118console.log('=== Circuit Breaker ===');
119console.log('State:', breaker.getState().state);
120console.log('');
121console.log('Hooks: OnModuleDestroy -> BeforeApplicationShutdown -> OnApplicationShutdown');
122console.log('Signals: SIGTERM (Docker), SIGINT (Ctrl+C)');
123console.log('app.enableShutdownHooks() - activation in main.ts');
124