NestJS course ยท Module 6: Error Handling and Monitoring

Monitoring and Alerting - early warning systems

6 min read ยท
In this lesson6

The application works. No exception has been thrown, the logs are silent, the health check shines green. And yet users write that "the site drags". You check - responses arrive after eight hundred milliseconds instead of a hundred. Since when? Nobody knows. Logs record events, and this is a trend: something that grew over weeks and is invisible in any single entry.

Rome set signal towers on its borders. They did not report single events - they measured traffic: how many riders passed, how long the crossing took, how many sentries failed to return. Only from those numbers could you see something going wrong before a gate fell. That is what metrics are.

Four types of metric

The standard for collecting metrics is Prometheus - a system that collects and stores numbers describing your application. It offers four kinds of measurement, from simplest to most elaborate:

  1. Counter - only goes up. Requests served, errors seen. It never decreases; on restart it begins at zero.
  2. Gauge - can rise and fall. Active connections, memory in use, queue length. It takes the current value, like a needle on a dial.
  3. Histogram - the distribution of values across buckets. Instead of one number it remembers how many measurements fell into 0-100 ms, how many into 100-500 ms, and so on.
  4. Summary - like a histogram, but with percentiles computed on the client side, that is inside your application rather than in Prometheus.

For HTTP request duration the right choice is a Histogram. A Counter would say only how many requests there were, a Gauge how long the last one took. A Histogram shows the shape: that nine out of ten requests finish under 200 ms while every tenth exceeds a second. That shape reveals a problem an average would hide.

Defining a metric

You create a metric once, describing it with three fields:

import { Counter } from 'prom-client';

@Injectable()
export class PrometheusService {
  private readonly httpRequestCounter = new Counter({
    name: 'http_requests_total',
    help: 'Total HTTP requests',
    labelNames: ['method', 'route', 'status'],
  });

  recordRequest(method: string, route: string, status: number) {
    this.httpRequestCounter.inc({ method, route, status: String(status) });
  }
}

The order of the fields is conventional but always the same: name is the metric's identifier, help a human-readable description, labelNames the list of dimensions you will be able to slice the data by.

The labels are the interesting part. Thanks to them one counter answers many questions: how many POST requests there were, how many hit /legions, how many ended with a 500. Without labels you would need a separate counter for every combination.

Beware of one trap: a label with many possible values multiplies the number of data series. Putting a user id in a label creates as many series as you have users - and will bring Prometheus down. Labels must have a finite, small set of values.

System metrics for free

Before writing your own metrics, it is worth switching on the built-in ones:

import { collectDefaultMetrics, register } from 'prom-client';

collectDefaultMetrics();

collectDefaultMetrics() collects the default system metrics - processor and memory usage and the Node.js event loop lag. That last one is especially valuable: a growing event loop lag means something is blocking the thread and the application is falling behind, though no endpoint has failed yet.

Note what this function does not do: it sends nothing, draws no charts and resets no counters. It only starts collecting.

The /metrics endpoint

Prometheus does not accept data pushed by an application - it comes for it itself. Your job is to expose it at an agreed address:

@Controller()
export class MetricsController {
  @Get('/metrics')
  async getMetrics(@Res() res: Response) {
    res.set('Content-Type', register.contentType);
    res.send(await register.metrics());
  }
}

register is the registry of every defined metric, and register.metrics() returns them in the text format Prometheus understands. The Content-Type header must say plain text, not JSON - hence register.contentType, which sets the right value for you.

This model is called scraping: every dozen seconds or so Prometheus queries that address and records what it found. The application need not know who is watching it, or whether anyone is.

Five deployment stages

The whole road from nothing to a chart looks like this:

  1. Install the prom-client package.
  2. Define the metrics - Counter, Histogram, as many as needed.
  3. Create the /metrics endpoint.
  4. Configure the Prometheus scraper so it knows where and how often to ask.
  5. Visualise the metrics in Grafana - and only here do charts and alerts appear.

The division of labour between the last two often gets blurred. Prometheus collects and stores, Grafana draws and alerts. That separation lets you replace one without the other - and means the application knows neither.

Summary

The signal towers stand and the traffic is measured:

  • logs record events, metrics show trends - the latter is invisible in any single entry,
  • Prometheus collects and stores metrics; it does not log errors and repairs nothing,
  • four types from simplest: Counter (only rises), Gauge (rises and falls), Histogram (distribution across buckets), Summary (percentiles computed client-side),
  • for HTTP request duration the right type is a Histogram - it shows the shape an average would hide,
  • three fields define a metric: name, help, labelNames,
  • labels let you slice the data but must have a small set of values - a user id as a label will bring Prometheus down,
  • collectDefaultMetrics() collects system metrics - CPU, memory, event loop lag; it sends nothing,
  • Prometheus comes for the data itself (scraping), so you expose a /metrics endpoint with Content-Type set to plain text via register.contentType,
  • five stages: install prom-client, define metrics, the /metrics endpoint, configure the scraper, visualise in Grafana,
  • Prometheus collects, Grafana draws and alerts.

In the next lesson we will drop from trends down to a single bug - you will meet debugging techniques for when you know something is wrong but not where. For now remember: a log says what happened once; a metric says what keeps happening - and it is the one that warns you before a gate falls.

Code for this lesson: src/monitoring/monitoring-system.ts
// Monitoring and Alerting - Early Warning Systems
// Metrics, alerts and a dashboard for the Imperium
import { Injectable, Logger } from '@nestjs/common';

// ===========================================
// 1. Metrics Service (Prometheus style)
// ===========================================

@Injectable()
export class MetricsService {
  private logger = new Logger('Metrics');

  // Counters - only grow
  private counters = new Map<string, number>();

  // Histograms - distribution of values
  private histograms = new Map<string, number[]>();

  // Gauge - instant value (can grow and shrink)
  private gauges = new Map<string, number>();

  // Increment the counter
  incrementCounter(name: string, value: number = 1) {
    const current = this.counters.get(name) || 0;
    this.counters.set(name, current + value);
  }

  // Save histogram value
  observeHistogram(name: string, value: number) {
    if (!this.histograms.has(name)) {
      this.histograms.set(name, []);
    }
    this.histograms.get(name).push(value);
  }

  // Set gauge
  setGauge(name: string, value: number) {
    this.gauges.set(name, value);
  }

  // Get all metrics
  getMetrics() {
    const result: Record<string, any> = {};

    // Counters
    this.counters.forEach((v, k) => {
      result[k + '_total'] = v;
    });

    // Histograms - average and percentiles
    this.histograms.forEach((values, k) => {
      const sorted = [...values].sort((a, b) => a - b);
      const sum = values.reduce((a, b) => a + b, 0);
      result[k + '_avg'] = Math.round(sum / values.length);
      result[k + '_p95'] = sorted[Math.floor(sorted.length * 0.95)];
      result[k + '_count'] = values.length;
    });

    // Gauges
    this.gauges.forEach((v, k) => {
      result[k] = v;
    });

    return result;
  }
}

// ===========================================
// 2. Alert Service
// ===========================================

@Injectable()
export class AlertService {
  private logger = new Logger('Alerting');
  private alerts: Array<{
    name: string;
    severity: string;
    message: string;
    timestamp: Date;
  }> = [];

  // Alert threshold definitions
  private thresholds = {
    error_rate: { warn: 0.01, critical: 0.05 },
    response_time_ms: { warn: 500, critical: 2000 },
    memory_percent: { warn: 80, critical: 95 },
  };

  checkThreshold(metric: string, value: number) {
    const threshold = this.thresholds[metric];
    if (!threshold) return;

    if (value >= threshold.critical) {
      this.fireAlert(metric, 'CRITICAL', metric + ' = ' + value);
    } else if (value >= threshold.warn) {
      this.fireAlert(metric, 'WARNING', metric + ' = ' + value);
    }
  }

  private fireAlert(name: string, severity: string, message: string) {
    const alert = { name, severity, message, timestamp: new Date() };
    this.alerts.push(alert);
    this.logger.warn('ALERT [' + severity + ']: ' + message);
  }

  getActiveAlerts() {
    return this.alerts.slice(-20);
  }
}

console.log('=== Monitoring & Alerting ===');
console.log('Counter - counts events (requests, errors)');
console.log('Histogram - distribution of response times');
console.log('Gauge - current value (memory, connections)');
console.log('Alert thresholds: warn -> critical');

Spotted a mistake in this lesson?

Check yourself

Answer the questions from this lesson. Pick an answer to see right away whether it is correct.

  1. 1. What is Prometheus used for in the context of monitoring NestJS applications?

  2. 2. Which Prometheus metric type is best suited for measuring HTTP request duration?

These are 2 of 3 questions for this lesson. Solve the rest in the game.

Hands-on tasks in the game

  • Code editor

    Complete the PrometheusService with an httpRequestCounter field (Counter with labelNames: ['method', 'route', 'status_code']) and httpRequestDuration (Histogram with buckets: [0.1, 0.5, 1, 2, 5, 10])

  • Vertical ordering

    Order Prometheus metric types from simplest to most complex

  • Click in order

    Arrange the elements of a Counter definition with labelNames in the correct order

  • Code editor

    Complete the MetricsController with @Get('/metrics') that sets Content-Type to 'text/plain' and returns register.metrics() from prom-client

  • Vertical ordering

    Order the steps for implementing Prometheus monitoring in a NestJS application from installation to visualization

  1. QuestionPrometheus in NestJS
  2. QuestionPrometheus Metric Types
  3. Coding taskPrometheusService with Metrics
  4. Put in orderPrometheus Metrics - Types
  5. QuestioncollectDefaultMetrics
  6. Click in orderLabels in Prometheus Metrics
  7. Coding task/metrics Endpoint for Prometheus
  8. Put in orderNestJS Monitoring Setup Steps

Useful articles