Distributed Tracing with OpenTelemetry — Courier Routes of the Empire

The Roman Empire had an extensive cursus publicus system — a state courier service. Each dispatch had an identification marker and was tracked from sender to receiver, through all intermediate stations. In microservice applications, distributed tracing plays the same role — tracking requests through multiple services.

The Problem: A Lost Request

In a monolithic system, it is easy to trace the flow of a request — everything is in one process. But in a microservice architecture, a request passes through many services:

1// A request passes through multiple services
2const requestFlow = {
3  step1: 'User -> API Gateway',
4  step2: 'API Gateway -> Auth Service',
5  step3: 'Auth Service -> User Service (verification)',
6  step4: 'API Gateway -> Legion Service (business logic)',
7  step5: 'Legion Service -> MongoDB (write)',
8  step6: 'Legion Service -> Redis (cache)',
9  step7: 'Legion Service -> Notification Service (notification)',
10
11  problem: 'Which request is slow? Where is the bottleneck?',
12  solution: 'Distributed Tracing — trace the request through ALL services',
13};

What is Distributed Tracing?

Distributed tracing is a technique for tracking a request through multiple services. Each request receives a unique identifier (trace ID) that is propagated to every service:

1// Key concepts
2interface TracingConcepts {
3  trace: 'Full journey of a request (from client to response)';
4  span: 'Single operation within a trace (e.g., DB query)';
5  traceId: 'Unique ID for the entire request (like a dispatch number)';
6  spanId: 'Unique ID of a single operation';
7  parentSpanId: 'ID of the parent operation (links spans into a tree)';
8  context: 'Propagation — passing traceId between services';
9}
10
11// Example span tree
12const traceExample = {
13  traceId: 'abc-123-def-456',
14  rootSpan: {
15    name: 'POST /legiones',
16    duration: '250ms',
17    children: [
18      { name: 'auth.verify', duration: '15ms' },
19      {
20        name: 'legion.create',
21        duration: '180ms',
22        children: [
23          { name: 'mongodb.insert', duration: '45ms' },
24          { name: 'redis.set', duration: '5ms' },
25          { name: 'notification.send', duration: '120ms' },
26        ],
27      },
28    ],
29  },
30};

OpenTelemetry — The Observability Standard

OpenTelemetry (OTel) is an open standard for collecting telemetry data (traces, metrics, logs). It is supported by the CNCF and all major cloud providers:

1# Installing OpenTelemetry SDK for NestJS
2yarn add @opentelemetry/sdk-node
3yarn add @opentelemetry/auto-instrumentations-node
4yarn add @opentelemetry/exporter-trace-otlp-http
5yarn add @opentelemetry/resources
6yarn add @opentelemetry/semantic-conventions

Configuring OpenTelemetry in NestJS

1// tracing/tracing.ts — IMPORTANT: load BEFORE importing NestJS!
2import { NodeSDK } from '@opentelemetry/sdk-node';
3import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
4import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
5import { Resource } from '@opentelemetry/resources';
6import {
7  ATTR_SERVICE_NAME,
8  ATTR_SERVICE_VERSION,
9} from '@opentelemetry/semantic-conventions';
10
11const sdk = new NodeSDK({
12  resource: new Resource({
13    [ATTR_SERVICE_NAME]: 'roman-imperium-api',
14    [ATTR_SERVICE_VERSION]: '1.0.0',
15  }),
16  traceExporter: new OTLPTraceExporter({
17    url: process.env.OTEL_EXPORTER_URL || 'http://jaeger:4318/v1/traces',
18  }),
19  instrumentations: [
20    getNodeAutoInstrumentations({
21      '@opentelemetry/instrumentation-http': { enabled: true },
22      '@opentelemetry/instrumentation-express': { enabled: true },
23      '@opentelemetry/instrumentation-mongoose': { enabled: true },
24    }),
25  ],
26});
27
28sdk.start();
29console.log('OpenTelemetry initialized');
30
31// Graceful shutdown
32process.on('SIGTERM', () => {
33  sdk.shutdown().then(() => console.log('Tracing shut down'));
34});

Loading Before the Application

1// main.ts
2import './tracing/tracing';  // MUST be the first import!
3import { NestFactory } from '@nestjs/core';
4import { AppModule } from './app.module';
5
6async function bootstrap() {
7  const app = await NestFactory.create(AppModule);
8  await app.listen(4000);
9}
10bootstrap();

Manually Creating Spans

Auto-instrumentation handles HTTP and database calls, but we can create custom spans for business logic:

1import { Injectable } from '@nestjs/common';
2import { trace, SpanStatusCode } from '@opentelemetry/api';
3
4@Injectable()
5export class LegionService {
6  private tracer = trace.getTracer('legion-service');
7
8  async recruitLegionary(dto: CreateLegionaryDto) {
9    return this.tracer.startActiveSpan('legion.recruit', async (span) => {
10      try {
11        span.setAttribute('legionary.name', dto.name);
12        span.setAttribute('legionary.rank', dto.rank);
13
14        // Each operation is a separate span
15        const legionary = await this.tracer.startActiveSpan(
16          'mongodb.create',
17          async (dbSpan) => {
18            const result = await this.legionRepository.create(dto);
19            dbSpan.setAttribute('db.collection', 'legionaries');
20            dbSpan.end();
21            return result;
22          },
23        );
24
25        span.setStatus({ code: SpanStatusCode.OK });
26        return legionary;
27      } catch (error) {
28        span.setStatus({
29          code: SpanStatusCode.ERROR,
30          message: error.message,
31        });
32        span.recordException(error);
33        throw error;
34      } finally {
35        span.end();
36      }
37    });
38  }
39}

Context Propagation — Passing Context

The Trace ID is automatically passed between services via HTTP headers:

1// Propagation headers (W3C Trace Context)
2const traceHeaders = {
3  'traceparent': '00-abc123def456-span789-01',
4  // version-traceId-parentSpanId-flags
5};
6
7// Auto-instrumentation automatically adds these headers
8// You don't need to do this manually!

Jaeger — Trace Visualization

Jaeger is a popular tool for visualizing distributed traces:

1# docker-compose.yml — Jaeger
2services:
3  jaeger:
4    image: jaegertracing/all-in-one:latest
5    environment:
6      - COLLECTOR_OTLP_ENABLED=true
7    ports:
8      - "16686:16686"    # Jaeger UI
9      - "4318:4318"      # OTLP HTTP

Correlating Logs with Traces

Linking logs with trace IDs enables full diagnostics:

1// Adding traceId to logs
2import { context, trace } from '@opentelemetry/api';
3
4function getTraceId(): string {
5  const span = trace.getSpan(context.active());
6  return span?.spanContext().traceId || 'no-trace';
7}
8
9// In the logger
10this.logger.log({
11  message: 'Legionary recruited',
12  traceId: getTraceId(),
13  legionaryId: 'leg_42',
14});
15
16// Now in Jaeger you see the trace, and in logs — the details

Distributed tracing is the cursus publicus system of our Empire — every request is tracked from beginning to end, through all services. Thanks to OpenTelemetry and Jaeger, we can see exactly where a request spends its time and where problems arise.