Scaling and Microservices - building a network of legions

Commander of the legions! A single fort with a single Node.js process serves the whole province, and when traffic grows, the CPU hits a hundred percent and the fort falls. Consul Caesar.js will teach you how to build a network of provinces out of one Roman fort. Scaling and microservices are techniques for building applications that grow together with traffic and business complexity.

Horizontal vs Vertical Scaling

Vertical scaling means a bigger machine: more cores and memory for the same server. Horizontal scaling means more instances behind a load balancer. Node.js runs JavaScript in a single thread, so to use all the cores of one machine, the cluster module starts several processes that share a port:

1// cluster.mjs - several Node.js processes on one machine
2import cluster from 'node:cluster';
3import { availableParallelism } from 'node:os';
4
5const numCPUs = availableParallelism();
6
7if (cluster.isPrimary) {
8  console.log(`Primary ${process.pid} is running`);
9
10  // Fork workers
11  for (let i = 0; i < numCPUs; i++) {
12    cluster.fork();
13  }
14
15  cluster.on('exit', (worker, code, signal) => {
16    console.log(`Worker ${worker.process.pid} died`);
17    cluster.fork(); // Restart worker
18  });
19} else {
20  // Workers can share any TCP port
21  import('./dist/main.js').then(() => {
22    console.log(`Worker ${process.pid} started`);
23  });
24}

cluster.isPrimary replaced isMaster, deprecated since Node.js 16, and availableParallelism() is the recommended way to count cores today. The primary process hands connections to the workers in turn: in a test six requests reached six different processes. Workers do not share memory, so sessions and caches must live outside the process, e.g. in Redis - the Node.js documentation explicitly warns against keeping them in in-memory objects.

In containers I recommend one process per container and scaling the number of replicas, not cluster inside - the orchestrator looks after restarts better than a fork() loop.

Microservices Architecture

Microservices split an application into small, independently deployed services, like legions with their own commanders. They have a price: every boundary is a network connection that can fail and a separate deployment to watch, so a small team is often better served by a well-structured monolith. In NestJS a client of another microservice is registered by ClientsModule:

1// app.module.ts - microservice clients
2import { Module } from '@nestjs/common';
3import { ClientsModule, Transport } from '@nestjs/microservices';
4
5@Module({
6  imports: [
7    ClientsModule.register([
8      { name: 'TRIBUTE_SERVICE', transport: Transport.TCP, options: { host: 'tributes', port: 4001 } },
9      { name: 'LEGION_SERVICE', transport: Transport.TCP, options: { host: 'legions', port: 4002 } },
10    ]),
11  ],
12  controllers: [UserController],
13  providers: [UserService],
14})
15export class AppModule {}

Every client has a name, a transport (TCP here) and an address. The name then serves as an injection token:

1// User microservice - an HTTP gateway that asks other microservices
2import { Controller, Get, Inject, Param } from '@nestjs/common';
3import { ClientProxy } from '@nestjs/microservices';
4import { firstValueFrom } from 'rxjs';
5
6@Controller('users')
7export class UserController {
8  constructor(
9    private userService: UserService,
10    @Inject('TRIBUTE_SERVICE') private tributeService: ClientProxy,
11    @Inject('LEGION_SERVICE') private legionService: ClientProxy,
12  ) {}
13
14  @Get(':id/tributes')
15  async getUserTributes(@Param('id') userId: string) {
16    const user = await this.userService.findById(userId);
17    const tributes = await firstValueFrom(
18      this.tributeService.send('get_user_tributes', { userId }),
19    );
20
21    return { user, tributes };
22  }
23}
24
25// Message patterns
26export const USER_PATTERNS = {
27  GET_USER: 'get_user',
28  CREATE_USER: 'create_user',
29  UPDATE_USER: 'update_user',
30};

ClientProxy.send() sends a message and waits for the answer, returning an Observable that RxJS's firstValueFrom() turns into a Promise. The first version used .toPromise(), deprecated in RxJS 7, and did not inject userService. The controller is not really a user microservice, but an HTTP gateway that asks other services.

On the other side, a controller with @MessagePattern answers:

1// tribute.controller.ts - on the tribute microservice side
2import { Controller } from '@nestjs/common';
3import { MessagePattern, Payload } from '@nestjs/microservices';
4
5@Controller()
6export class TributeController {
7  @MessagePattern('get_user_tributes')
8  getUserTributes(@Payload() data: { userId: string }) {
9    return [{ userId: data.userId, amount: 1000, province: 'Gallia' }];
10  }
11}

The get_user_tributes pattern must match on both sides. Such a service starts through NestFactory.createMicroservice() with the same transport. send() is the request-response model; notifications that expect no answer use emit() and @EventPattern.

Deploying without downtime

Many instances let you update the application without interruption. Blue-green deployment keeps two identical environments and switches traffic from the old one to the new one - it gives zero downtime and an instant rollback by switching back, at the cost of double infrastructure. A zero-downtime deployment goes like this:

1. prepare the new version in a parallel environment,
2. run health checks and smoke tests,
3. switch traffic to the new version,
4. shut down the old version after verification.

Canary deployment gradually routes traffic to the new version, starting with a small percentage of users:

1. deploy the new version to a small percentage of users,
2. monitor metrics and errors,
3. gradually increase the traffic,
4. do a full rollout or a rollback.

In Kubernetes probes decide whether an instance gets traffic:

1# deployment.yaml - excerpt: deployment strategy and health probes
2spec:
3  replicas: 4
4  strategy:
5    type: RollingUpdate
6    rollingUpdate:
7      maxSurge: 1        # at most 1 extra pod during the rollout
8      maxUnavailable: 0  # an old pod goes away only once a new one is ready
9  template:
10    spec:
11      containers:
12        - name: api
13          image: registry.imperium.rome/api:2.4.0
14          readinessProbe:        # should it receive traffic?
15            httpGet:
16              path: /health/ready
17              port: 3000
18            initialDelaySeconds: 5
19            periodSeconds: 10
20            timeoutSeconds: 2
21            successThreshold: 2  # two successful checks in a row
22            failureThreshold: 3  # three failures = the pod drops out of traffic
23          livenessProbe:         # is the process alive?
24            httpGet:
25              path: /health/live
26              port: 3000
27            periodSeconds: 10
28            failureThreshold: 3  # three failures = container restart

The readiness probe removes a pod from traffic when it is not ready, and liveness restarts a container that stopped responding. A successThreshold greater than 1 is allowed only for readiness; by default periodSeconds is 10 and failureThreshold is 3. maxUnavailable: 0 makes sure a rolling update never reduces the number of ready pods.

In the last lesson of the module we will build blue-green, rolling updates and feature flags in code.

Remember: the Empire's strength did not lie in one big fort, but in a network of legions that could replace one another -, build your application the same way.

1// Scaling and Microservices - Building a Legion of Cohorts
2
3// 1. Vertical vs Horizontal Scaling
4// Vertical: bigger server (more CPU/RAM)
5// Horizontal: more instances (load balancing)
6
7// 2. NestJS Microservices
8import { Controller } from '@nestjs/common';
9// import { MessagePattern, EventPattern } from '@nestjs/microservices';
10
11// Microservice - TributeService
12// @Controller()
13// class TributeController {
14//   // Request-Response pattern
15//   @MessagePattern({ cmd: 'get_tribute' })
16//   getTribute(data: { provinceId: string }) {
17//     return { province: data.provinceId, amount: 1000 };
18//   }
19//
20//   // Event pattern (fire and forget)
21//   @EventPattern('tribute_collected')
22//   handleTributeCollected(data: { province: string; amount: number }) {
23//     console.log(`Tribute collected from ${data.province}`);
24//   }
25// }
26
27// 3. Communication between services
28// TCP Transport (default):
29// const app = await NestFactory.createMicroservice(AppModule, {
30//   transport: Transport.TCP,
31//   options: { host: '0.0.0.0', port: 3001 },
32// });
33
34// Redis Transport:
35// const app = await NestFactory.createMicroservice(AppModule, {
36//   transport: Transport.REDIS,
37//   options: { host: 'localhost', port: 6379 },
38// });
39
40// 4. Docker Compose for microservices
41const dockerCompose = `
42services:
43  api-gateway:
44    build: ./gateway
45    ports: ['3000:3000']
46    depends_on: [tribute-service, legion-service]
47
48  tribute-service:
49    build: ./tribute-service
50    ports: ['3001:3001']
51
52  legion-service:
53    build: ./legion-service
54    ports: ['3002:3002']
55
56  redis:
57    image: redis:7-alpine
58    ports: ['6379:6379']
59
60  mongodb:
61    image: mongo:7
62    ports: ['27017:27017']
63    volumes: [mongo-data:/data/db]
64
65volumes:
66  mongo-data:
67`;
68
69// 5. Architecture patterns
70// - API Gateway: a single entry point
71// - Service Discovery: automatic discovery of services
72// - Circuit Breaker: protection against cascading failures
73// - Saga Pattern: distributed transactions
74// - CQRS: separating reads from writes
75