Observability — Metrics with Prometheus — Eyes of the Empire

The Roman Empire maintained an extensive network of speculatores — scouts and spies who gathered information about the state of provinces, enemy army movements, and public sentiment. In modern applications, the role of speculatores is played by Prometheus — a system for collecting and analyzing metrics that allows us to see exactly what is happening in our application in real time.

What are Metrics?

Metrics are numerical measurements of the application's state over time. Unlike logs (text events), metrics are numerical series that are easy to aggregate and visualize:

1// Logs vs Metrics
2const comparison = {
3  logs: {
4    type: 'Text events',
5    example: '[INFO] Legionary recruited: leg_42',
6    usage: 'Debugging, auditing',
7    cost: 'High storage under heavy traffic',
8  },
9  metrics: {
10    type: 'Numerical time series',
11    example: 'http_requests_total{method="GET", status="200"} 1547',
12    usage: 'Monitoring, alerting, dashboards',
13    cost: 'Low — just numbers',
14  },
15};

Prometheus Metric Types

Prometheus defines four basic metric types:

1// Four Prometheus metric types
2interface PrometheusMetrics {
3  counter: {
4    description: 'Only increases (e.g., number of requests)';
5    example: 'http_requests_total';
6    analogy: 'Count of fallen in battles — only increases';
7  };
8  gauge: {
9    description: 'Increases and decreases (e.g., active connections)';
10    example: 'active_connections';
11    analogy: 'Number of legionaries in camp — changes';
12  };
13  histogram: {
14    description: 'Distribution of values in buckets (e.g., response times)';
15    example: 'http_request_duration_seconds';
16    analogy: 'Dispatch transit time — how many took 0-1s, 1-5s, 5-10s';
17  };
18  summary: {
19    description: 'Similar to histogram, but with quantiles (p50, p95, p99)';
20    example: 'request_duration_summary';
21    analogy: 'Median supply delivery time';
22  };
23}

Prometheus Integration with NestJS

We use the

@willsoto/nestjs-prometheus

1yarn add @willsoto/nestjs-prometheus prom-client

Module Configuration

1// metrics/metrics.module.ts
2import { Module } from '@nestjs/common';
3import {
4  PrometheusModule,
5  makeCounterProvider,
6  makeGaugeProvider,
7  makeHistogramProvider,
8} from '@willsoto/nestjs-prometheus';
9
10@Module({
11  imports: [
12    PrometheusModule.register({
13      path: '/metrics',        // Endpoint for Prometheus
14      defaultMetrics: {
15        enabled: true,         // Node.js metrics (CPU, memory, GC)
16      },
17    }),
18  ],
19  providers: [
20    // Counter — number of HTTP requests
21    makeCounterProvider({
22      name: 'http_requests_total',
23      help: 'Total number of HTTP requests',
24      labelNames: ['method', 'route', 'status'],
25    }),
26    // Gauge — active connections
27    makeGaugeProvider({
28      name: 'active_connections',
29      help: 'Number of active connections',
30    }),
31    // Histogram — response time
32    makeHistogramProvider({
33      name: 'http_request_duration_seconds',
34      help: 'HTTP response time in seconds',
35      labelNames: ['method', 'route', 'status'],
36      buckets: [0.01, 0.05, 0.1, 0.5, 1, 2, 5],
37    }),
38  ],
39  exports: [PrometheusModule],
40})
41export class MetricsModule {}

Using Metrics in a Service

1// legion/legion.service.ts
2import { Injectable } from '@nestjs/common';
3import { InjectMetric } from '@willsoto/nestjs-prometheus';
4import { Counter, Histogram } from 'prom-client';
5
6@Injectable()
7export class LegionService {
8  constructor(
9    @InjectMetric('http_requests_total')
10    private requestsCounter: Counter<string>,
11
12    @InjectMetric('http_request_duration_seconds')
13    private requestDuration: Histogram<string>,
14  ) {}
15
16  async recruitLegionary(dto: CreateLegionaryDto) {
17    const timer = this.requestDuration.startTimer({
18      method: 'POST',
19      route: '/legiones',
20    });
21
22    try {
23      const result = await this.legionRepository.create(dto);
24      this.requestsCounter.inc({
25        method: 'POST',
26        route: '/legiones',
27        status: '201',
28      });
29      return result;
30    } finally {
31      timer({ status: '201' });  // Record the duration
32    }
33  }
34}

The /metrics Endpoint

After configuration, the

/metrics

1# HELP http_requests_total Total number of HTTP requests
2# TYPE http_requests_total counter
3http_requests_total{method="GET",route="/legiones",status="200"} 1547
4http_requests_total{method="POST",route="/legiones",status="201"} 342
5
6# HELP http_request_duration_seconds HTTP response time
7# TYPE http_request_duration_seconds histogram
8http_request_duration_seconds_bucket{le="0.01"} 890
9http_request_duration_seconds_bucket{le="0.05"} 1200
10http_request_duration_seconds_bucket{le="0.1"} 1400

The RED Method — Key to Monitoring

RED is a proven method for monitoring services:

1const redMethod = {
2  rate: {
3    description: 'Number of requests per second',
4    metric: 'rate(http_requests_total[5m])',
5    alert: 'When it drops below normal — possible problem',
6  },
7  errors: {
8    description: 'Percentage of requests with errors',
9    metric: 'rate(http_requests_total{status=~"5.."}[5m])',
10    alert: 'When > 1% — something is wrong',
11  },
12  duration: {
13    description: 'Response time (p95, p99)',
14    metric: 'histogram_quantile(0.95, http_request_duration_seconds)',
15    alert: 'When p95 > 500ms — performance degradation',
16  },
17};

Grafana — Metrics Visualization

Prometheus collects metrics, but Grafana visualizes them as dashboards:

1# docker-compose.yml — Prometheus + Grafana
2services:
3  prometheus:
4    image: prom/prometheus:latest
5    volumes:
6      - ./prometheus.yml:/etc/prometheus/prometheus.yml
7    ports:
8      - "9090:9090"
9
10  grafana:
11    image: grafana/grafana:latest
12    ports:
13      - "3001:3000"
14    environment:
15      - GF_SECURITY_ADMIN_PASSWORD=imperium

Prometheus Configuration

1# prometheus.yml
2scrape_configs:
3  - job_name: 'roman-api'
4    scrape_interval: 15s
5    static_configs:
6      - targets: ['api:4000']
7    metrics_path: '/metrics'

Alert Rules

1# alert.rules.yml
2groups:
3  - name: roman-api-alerts
4    rules:
5      - alert: HighErrorRate
6        expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.01
7        for: 5m
8        labels:
9          severity: critical
10        annotations:
11          summary: "High error rate in the API"

Prometheus and Grafana are the eyes of our Empire — they see every movement, measure every operation, and raise alarms when something is wrong. Without metrics we are blind to performance problems.