We use cookies to enhance your experience on the site
Privacy Policy
OK
CodeWorlds

Production Best Practices - the rules of a true centurion

On production the code does not change. What changes is who is watching: nobody. The application runs at night, on Sundays and on holidays, and all you know about it are the signals it sends about itself. This lesson is about organising them.

A centurion did not inspect each legionary in person. He had three sources of knowledge: numerical reports on the state of the cohorts, watch records of individual incidents, and couriers' dispatches showing where an order had got stuck on the way. The same division holds to this day.

The three pillars of observability

The three pillars of observability are metrics, logs and traces. Not frontend, backend and database, for those are layers of an application. Not CPU, memory and disk, for those are resources - one kind of metric among many. And not development, testing and production, for those are environments.

Each pillar answers a different question, so none replaces the others:

  • Metrics tell you how much and how often - requests per second, response time, memory use. They are aggregated and cheap to store, so you keep them for months and build your alerts on them.
  • Logs tell you what exactly happened in one particular case - with the error's text and the request's identifier. They are expensive, because there are so many, but only they answer the question "why this request".
  • Traces tell you where the time went - they show one request's journey across all the services. Without them, with three services along the way, you know only that the answer took two seconds; you do not know which of them consumed it.

A metric tells you that something is wrong. A log tells you what broke. A trace tells you where.

The health check

An orchestrator - Kubernetes, Docker Swarm, or a plain load balancer - must know whether traffic may be sent to this instance. It asks through a health endpoint:

1@Controller('health')
2export class HealthController {
3  constructor(
4    private health: HealthCheckService,
5    private db: MongooseHealthIndicator,
6    private memory: MemoryHealthIndicator,
7  ) {}
8
9  @Get()
10  @HealthCheck()
11  check() {
12    return this.health.check([
13      () => this.db.pingCheck('database'),
14      () => this.memory.checkHeap('memory_heap', 300 * 1024 * 1024),
15    ]);
16  }
17}

The 

@nestjs/terminus
package supplies the parts ready-made: 
HealthCheckService
 gathers the results and assembles the response, 
MongooseHealthIndicator
 checks the database connection, and 
MemoryHealthIndicator
 watches heap use. There are more indicators - for Redis, for the disk, for any external HTTP service.

A health check should check the availability of the database, Redis and dependent services. Not whether the source code is up to date - that is not its role. Not the number of logged-in users - that is a business metric, not a health signal. And certainly not only whether the HTTP server responds.

That last distinction is the heart of it. An application whose database has died still answers HTTP - the process is alive, the port is open, 

200 OK
comes back without delay. A health check testing only that tells the orchestrator "all is well" about an instance that cannot serve a single real request. Check the dependencies without which the application can do nothing anyway.

Configuring monitoring

A monitoring system is set up in four steps, in this order:

  1. Install the monitoring agent - it is what gathers data from the machine and the application.
  2. Configure metrics collection - decide what you measure and how often.
  3. Set up alerting rules - fix the thresholds beyond which somebody is to be notified.
  4. Create visualization dashboards - charts to look at once you know something is happening.

The order is often reversed, and that is the commonest mistake in setting monitoring up. Dashboards come last, because until metrics are flowing you do not know what you would draw on them. Alerts come before them, because an alert arrives by itself whereas a dashboard has to be looked at - and at three in the morning nobody is looking.

Alert escalation levels

Not every threshold crossing means the same thing. Alerts fall into four levels:

  1. Warning (80% capacity) - a notification. Nothing is broken yet, but you are nearing the limit; there is time to react calmly.
  2. Critical (95% capacity) - an urgent response. The margin is nearly gone; failure is a matter of hours or minutes.
  3. Emergency (above 100%) - immediate intervention. The limit has been passed and users can already see it.
  4. Post-incident - analysis and lessons learned. Once the situation is contained you establish the cause and what to change so it does not return.

Note that the first two thresholds lie below a hundred per cent. That is deliberate: an alert at 100% is no longer a warning but a notification of failure. Sensible monitoring gives time to react, not a commentary on the fire.

The fourth level is often skipped, and it is the only one that changes anything for the future. Without a post-incident analysis the same alert will ring again next month.

Graceful shutdown

The last rule concerns switching off. When a new version is deployed the old instance receives a 

SIGTERM
signal - and what it does over the next few seconds decides whether anyone sees an error:

1async function bootstrap() {
2  const app = await NestFactory.create(AppModule);
3
4  app.enableShutdownHooks();
5
6  await app.listen(3000);
7}

enableShutdownHooks()
makes NestJS intercept the signal and, before closing, call the 
onModuleDestroy
and 
beforeApplicationShutdown
methods in the modules. The application thus has time to finish the requests in flight, close database connections and deregister from the service registry.

Without it a deployment cuts off the requests being served at that moment. At ten deployments a day that is ten bursts of errors nobody connects with the deployment - because in the logs they look like random dropped connections.

Summary

The centurion reads reports rather than inspecting every legionary, @name:

  • the three pillars of observability are metrics, logs and traces - not frontend/backend/database, not CPU/memory/disk, not environments,
  • a metric tells you that something is wrong, a log what broke, a trace where the time went,
  • a health check checks the availability of the database, Redis and dependent services - not the code's freshness, not the user count, and not merely whether HTTP responds,
  • an application with a dead database still returns 
    200
    - which is why checking HTTP alone is worthless,
  • @nestjs/terminus
    : 
    HealthCheckService
    assembles the result, 
    MongooseHealthIndicator
    tests the database, 
    MemoryHealthIndicator
    the heap,
  • configuring monitoring: install the agent → configure metrics collection → set up alerting rules → create dashboards,
  • escalation levels: Warning (80%) → Critical (95%) → Emergency (>100%) → Post-incident,
  • thresholds below 100% leave time to react; an alert at 100% is already a report of failure,
  • enableShutdownHooks()
    lets in-flight requests finish when an instance is shut down.

That closes our survey of production practice. For now remember: an application in production is only as good as the signals it sends about itself - because nobody is going to guess what is happening to it.

Go to CodeWorlds→