PROJECT: Complete Legion Monitoring System

Chief technology legate! Consul Caesar.js has entrusted you with the ultimate challenge - creating a complete monitoring system for the entire Roman legion. This will be a real test of all the skills acquired in the Error Handling & Monitoring module!

Project Specification

Your system must be a complete solution for monitoring a legion consisting of many Roman forts. Each fort is a separate NestJS application instance, and you are building the central monitoring system.

Functional Requirements

1. Multi-Legion Monitoring

• Central Dashboard - overview of all forts in the legion
• Legion Registration - automatic registration of new forts
• Health Checks - health monitoring of each fort
• Legion Analytics - legion-wide analytics

2. Real-time Error Tracking

• Error Aggregation - collecting errors from all forts
• Error Classification - categorizing errors by type and severity
• Error Correlation - linking similar errors
• Alert System - intelligent notifications

3. Performance Monitoring

• System Metrics - CPU, memory, disk, network
• Application Metrics - response time, throughput, error rate
• Business Metrics - custom metrics specific to the legion
• SLA Monitoring - tracking performance targets

4. Advanced Alerting

• Smart Alerts - machine learning-based alerts
• Escalation Policies - alert escalation rules
• Multi-channel Notifications - Slack, email, SMS, webhook
• Alert Correlation - grouping related alerts

Architektura System

1// Architecture Overview
2/*
3┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
4│ Legion Alpha │ │ Legion Beta │ │ Legion Gamma │
5│ (NestJS) │ │ (NestJS) │ │ (NestJS) │
6│ │ │ │ │ │
7│ - Error Agent │ │ - Error Agent │ │ - Error Agent │
8│ - Metrics Agent │ │ - Metrics Agent │ │ - Metrics Agent │
9│ - Health Check │ │ - Health Check │ │ - Health Check │
10└─────────┬───────┘ └─────────┬───────┘ └─────────┬───────┘
11 │ │ │
12 └──────────────────────┼──────────────────────┘
13 │
14 ┌─────────────┴─────────────┐
15 │ Legion Monitor Center │
16 │ (Main NestJS App) │
17 │ │
18 │ - Data Aggregation │
19 │ - Alert Management │
20 │ - Dashboard API │
21 │ - Analytics Engine │
22 │ - Notification System │
23 └─────────────┬─────────────┘
24 │
25 ┌─────────────┴─────────────┐
26 │ External Systems │
27 │ │
28 │ - Prometheus/Grafana │
29 │ - Elasticsearch/Kibana │
30 │ - Slack/Teams │
31 │ - PagerDuty │
32 └───────────────────────────┘
33*/

System Implementation

1. Legion Agent (for each fort)

1// agents/cohort-agent.service.ts
2import { Injectable, Logger, OnModuleInit } from '@nestjs/common';
3import { Cron } from '@nestjs/schedule';
4import { HttpService } from '@nestjs/axios';
5import { ConfigService } from '@nestjs/config';
6
7export interface LegionMetrics {
8 cohortId: string;
9 cohortName: string;
10 centurion: string;
11 timestamp: Date;
12 system: {
13 cpu: number;
14 memory: number;
15 disk: number;
16 network: {
17 bytesIn: number;
18 bytesOut: number;
19 };
20 };
21 application: {
22 uptime: number;
23 activeConnections: number;
24 requestsPerSecond: number;
25 errorRate: number;
26 responseTime: {
27 avg: number;
28 p95: number;
29 p99: number;
30 };
31 };
32 business: {
33 tributesFound: number;
34 legionActivity: number;
35 missionsCompleted: number;
36 battlesWon: number;
37 };
38 health: {
39 overall: 'healthy' | 'warning' | 'critical';
40 services: Record<string, 'up' | 'down' | 'degraded'>;
41 };
42}
43
44@Injectable()
45export class LegionAgentService implements OnModuleInit {
46 private readonly logger = new Logger(LegionAgentService.name);
47 private readonly cohortId: string;
48 private readonly legionCenterUrl: string;
49 private isRegistered = false;
50
51 constructor(
52 private readonly httpService: HttpService,
53 private readonly configService: ConfigService,
54 ) {
55 this.cohortId = this.configService.get('COHORT_ID') || this.generateLegionId();
56 this.legionCenterUrl = this.configService.get('LEGION_CENTER_URL');
57 }
58
59 async onModuleInit() {
60 await this.registerWithLegion();
61 }
62
63 private async registerWithLegion() {
64 try {
65 const registrationData = {
66 cohortId: this.cohortId,
67 cohortName: this.configService.get('COHORT_NAME', 'Unknown Vessel'),
68 centurion: this.configService.get('CENTURION_NAME', 'Anonymous Centurion'),
69 version: process.env.npm_package_version || '1.0.0',
70 startTime: new Date(),
71 location: {
72 latitude: parseFloat(this.configService.get('COHORT_LAT', '0')),
73 longitude: parseFloat(this.configService.get('COHORT_LNG', '0')),
74 },
75 };
76
77 await this.httpService.post(
78 `${this.legionCenterUrl}/legion/cohorts/register`,
79 registrationData
80 ).toPromise();
81
82 this.isRegistered = true;
83 this.logger.log(` Legion ${this.cohortId} registered with legion center`);
84 } catch (error) {
85 this.logger.error('Failed to register with legion center:', error.message);
86 }
87 }
88
89 @Cron('0 * * * * *') // Every minute
90 async sendMetricsToLegion() {
91 if (!this.isRegistered) {
92 await this.registerWithLegion();
93 return;
94 }
95
96 try {
97 const metrics = await this.collectLegionMetrics();
98 
99 await this.httpService.post(
100 `${this.legionCenterUrl}/legion/metrics`,
101 metrics
102 ).toPromise();
103
104 this.logger.debug(` Metrics sent to legion center`);
105 } catch (error) {
106 this.logger.error('Failed to send metrics to legion center:', error.message);
107 }
108 }
109
110 async collectLegionMetrics(): Promise<LegionMetrics> {
111 const memUsage = process.memoryUsage();
112 const uptime = process.uptime();
113
114 return {
115 cohortId: this.cohortId,
116 cohortName: this.configService.get('COHORT_NAME', 'Unknown Vessel'),
117 centurion: this.configService.get('CENTURION_NAME', 'Anonymous Centurion'),
118 timestamp: new Date(),
119 
120 system: {
121 cpu: await this.getCpuUsage(),
122 memory: Math.round((memUsage.heapUsed / memUsage.heapTotal) * 100),
123 disk: await this.getDiskUsage(),
124 network: await this.getNetworkStats(),
125 },
126 
127 application: {
128 uptime,
129 activeConnections: await this.getActiveConnections(),
130 requestsPerSecond: await this.getRequestsPerSecond(),
131 errorRate: await this.getErrorRate(),
132 responseTime: await this.getResponseTimeStats(),
133 },
134 
135 business: {
136 tributesFound: await this.getTributesFound(),
137 legionActivity: await this.getLegionActivity(),
138 missionsCompleted: await this.getMissionsCompleted(),
139 battlesWon: await this.getBattlesWon(),
140 },
141 
142 health: await this.getHealthStatus(),
143 };
144 }
145
146 private async getCpuUsage(): Promise<number> {
147 return new Promise((resolve) => {
148 const startUsage = process.cpuUsage();
149 setTimeout(() => {
150 const endUsage = process.cpuUsage(startUsage);
151 const totalUsage = endUsage.user + endUsage.system;
152 const percentage = (totalUsage / 1000000) * 100;
153 resolve(Math.min(100, percentage));
154 }, 100);
155 });
156 }
157
158 private async getDiskUsage(): Promise<number> {
159 // Simplified disk usage - in real implementation use fs.statvfs
160 return Math.random() * 100;
161 }
162
163 private async getNetworkStats() {
164 // Simplified network stats - in real implementation read from /proc/net/dev
165 return {
166 bytesIn: Math.floor(Math.random() * 1000000),
167 bytesOut: Math.floor(Math.random() * 1000000),
168 };
169 }
170
171 private async getActiveConnections(): Promise<number> {
172 // Implementation depends on your server setup
173 return Math.floor(Math.random() * 100);
174 }
175
176 private async getRequestsPerSecond(): Promise<number> {
177 // Implementation using metrics collected by interceptors
178 return Math.random() * 1000;
179 }
180
181 private async getErrorRate(): Promise<number> {
182 // Implementation using error tracking
183 return Math.random() * 10;
184 }
185
186 private async getResponseTimeStats() {
187 return {
188 avg: 50 + Math.random() * 200,
189 p95: 100 + Math.random() * 400,
190 p99: 200 + Math.random() * 800,
191 };
192 }
193
194 private async getTributesFound(): Promise<number> {
195 // Business metric - implement based on your tribute service
196 return Math.floor(Math.random() * 50);
197 }
198
199 private async getLegionActivity(): Promise<number> {
200 // Business metric - implement based on your legion service
201 return Math.floor(Math.random() * 100);
202 }
203
204 private async getMissionsCompleted(): Promise<number> {
205 // Business metric - implement based on your mission service
206 return Math.floor(Math.random() * 20);
207 }
208
209 private async getBattlesWon(): Promise<number> {
210 // Business metric - implement based on your battle service
211 return Math.floor(Math.random() * 10);
212 }
213
214 private async getHealthStatus() {
215 // Implement comprehensive health checks
216 const services = {
217 database: Math.random() > 0.1 ? 'up' : 'down',
218 redis: Math.random() > 0.05 ? 'up' : 'down',
219 api: Math.random() > 0.02 ? 'up' : 'degraded',
220 auth: Math.random() > 0.01 ? 'up' : 'down',
221 };
222
223 const downServices = Object.values(services).filter(s => s === 'down').length;
224 const degradedServices = Object.values(services).filter(s => s === 'degraded').length;
225
226 let overall: 'healthy' | 'warning' | 'critical' = 'healthy';
227 if (downServices > 0) {
228 overall = 'critical';
229 } else if (degradedServices > 0) {
230 overall = 'warning';
231 }
232
233 return { overall, services };
234 }
235
236 private generateLegionId(): string {
237 return `cohort-${Math.random().toString(36).substr(2, 9)}`;
238 }
239
240 async sendErrorToLegion(error: any, context?: any) {
241 if (!this.isRegistered) return;
242
243 try {
244 const errorData = {
245 cohortId: this.cohortId,
246 timestamp: new Date(),
247 error: {
248 message: error.message,
249 stack: error.stack,
250 type: error.constructor.name,
251 code: error.code,
252 },
253 context: {
254 url: context?.url,
255 method: context?.method,
256 user: context?.user?.id,
257 requestId: context?.requestId,
258 ...context,
259 },
260 severity: this.classifyErrorSeverity(error),
261 };
262
263 await this.httpService.post(
264 `${this.legionCenterUrl}/legion/errors`,
265 errorData
266 ).toPromise();
267
268 this.logger.debug(' Error sent to legion center');
269 } catch (err) {
270 this.logger.error('Failed to send error to legion center:', err.message);
271 }
272 }
273
274 private classifyErrorSeverity(error: any): 'low' | 'medium' | 'high' | 'critical' {
275 if (error.name?.includes('Database') || error.code === 'ECONNREFUSED') {
276 return 'critical';
277 }
278 if (error.status >= 500) {
279 return 'high';
280 }
281 if (error.status >= 400) {
282 return 'medium';
283 }
284 return 'low';
285 }
286}

2. Legion Command - Central Monitoring Hub

1// legion-center/legion-monitor.service.ts
2import { Injectable, Logger } from '@nestjs/common';
3import { EventEmitter2 } from '@nestjs/event-emitter';
4import { Cron } from '@nestjs/schedule';
5
6export interface Legion {
7 cohorts: Map<string, LegionInfo>;
8 totalLegions: number;
9 healthyLegions: number;
10 warningLegions: number;
11 criticalLegions: number;
12 lastUpdate: Date;
13}
14
15export interface LegionInfo {
16 cohortId: string;
17 cohortName: string;
18 centurion: string;
19 lastSeen: Date;
20 metrics: LegionMetrics;
21 status: 'healthy' | 'warning' | 'critical' | 'offline';
22 alerts: Alert[];
23 errors: ErrorEvent[];
24}
25
26@Injectable()
27export class LegionMonitorService {
28 private readonly logger = new Logger(LegionMonitorService.name);
29 private legion: Legion = {
30 cohorts: new Map(),
31 totalLegions: 0,
32 healthyLegions: 0,
33 warningLegions: 0,
34 criticalLegions: 0,
35 lastUpdate: new Date(),
36 };
37
38 constructor(private readonly eventEmitter: EventEmitter2) {}
39
40 registerLegion(registrationData: any) {
41 const cohortInfo: LegionInfo = {
42 cohortId: registrationData.cohortId,
43 cohortName: registrationData.cohortName,
44 centurion: registrationData.centurion,
45 lastSeen: new Date(),
46 metrics: null,
47 status: 'healthy',
48 alerts: [],
49 errors: [],
50 };
51
52 this.legion.cohorts.set(registrationData.cohortId, cohortInfo);
53 this.updateLegionStats();
54 
55 this.logger.log(` Legion registered: ${cohortInfo.cohortName} (${cohortInfo.cohortId})`);
56 
57 this.eventEmitter.emit('cohort.registered', cohortInfo);
58 }
59
60 updateLegionMetrics(cohortId: string, metrics: LegionMetrics) {
61 const cohort = this.legion.cohorts.get(cohortId);
62 if (!cohort) {
63 this.logger.warn(`Received metrics from unregistered cohort: ${cohortId}`);
64 return;
65 }
66
67 cohort.metrics = metrics;
68 cohort.lastSeen = new Date();
69 cohort.status = this.determineLegionStatus(cohort);
70 
71 this.updateLegionStats();
72 this.analyzeMetricsForAlerts(cohort);
73 
74 this.eventEmitter.emit('cohort.metrics.updated', { cohortId, metrics });
75 }
76
77 recordLegionError(cohortId: string, errorData: any) {
78 const cohort = this.legion.cohorts.get(cohortId);
79 if (!cohort) {
80 this.logger.warn(`Received error from unregistered cohort: ${cohortId}`);
81 return;
82 }
83
84 const errorEvent: ErrorEvent = {
85 id: `error-${Date.now()}-${Math.random()}`,
86 timestamp: new Date(),
87 cohortId,
88 ...errorData,
89 };
90
91 cohort.errors.push(errorEvent);
92 
93 // Keep only last 100 errors per cohort
94 if (cohort.errors.length > 100) {
95 cohort.errors = cohort.errors.slice(-100);
96 }
97
98 this.processError(errorEvent);
99 this.eventEmitter.emit('cohort.error', errorEvent);
100 }
101
102 private determineLegionStatus(cohort: LegionInfo): 'healthy' | 'warning' | 'critical' | 'offline' {
103 if (!cohort.metrics) return 'offline';
104
105 const timeSinceLastSeen = Date.now() - cohort.lastSeen.getTime();
106 if (timeSinceLastSeen > 300000) { // 5 minutes
107 return 'offline';
108 }
109
110 if (cohort.metrics.health.overall === 'critical') {
111 return 'critical';
112 }
113
114 const { cpu, memory } = cohort.metrics.system;
115 const { errorRate } = cohort.metrics.application;
116
117 if (cpu > 90 || memory > 90 || errorRate > 10) {
118 return 'critical';
119 }
120
121 if (cpu > 70 || memory > 70 || errorRate > 5 || cohort.metrics.health.overall === 'warning') {
122 return 'warning';
123 }
124
125 return 'healthy';
126 }
127
128 private updateLegionStats() {
129 this.legion.totalLegions = this.legion.cohorts.size;
130 this.legion.healthyLegions = 0;
131 this.legion.warningLegions = 0;
132 this.legion.criticalLegions = 0;
133
134 for (const cohort of this.legion.cohorts.values()) {
135 switch (cohort.status) {
136 case 'healthy':
137 this.legion.healthyLegions++;
138 break;
139 case 'warning':
140 this.legion.warningLegions++;
141 break;
142 case 'critical':
143 this.legion.criticalLegions++;
144 break;
145 }
146 }
147
148 this.legion.lastUpdate = new Date();
149 }
150
151 private analyzeMetricsForAlerts(cohort: LegionInfo) {
152 const alerts: Alert[] = [];
153
154 // CPU Alert
155 if (cohort.metrics.system.cpu > 90) {
156 alerts.push(this.createAlert(
157 'high-cpu',
158 'critical',
159 `High CPU usage on ${cohort.cohortName}`,
160 `CPU usage is ${cohort.metrics.system.cpu.toFixed(1)}%`,
161 cohort.cohortId
162 ));
163 }
164
165 // Memory Alert
166 if (cohort.metrics.system.memory > 90) {
167 alerts.push(this.createAlert(
168 'high-memory',
169 'critical',
170 `High memory usage on ${cohort.cohortName}`,
171 `Memory usage is ${cohort.metrics.system.memory.toFixed(1)}%`,
172 cohort.cohortId
173 ));
174 }
175
176 // Error Rate Alert
177 if (cohort.metrics.application.errorRate > 10) {
178 alerts.push(this.createAlert(
179 'high-error-rate',
180 'high',
181 `High error rate on ${cohort.cohortName}`,
182 `Error rate is ${cohort.metrics.application.errorRate.toFixed(2)}%`,
183 cohort.cohortId
184 ));
185 }
186
187 // Response Time Alert
188 if (cohort.metrics.application.responseTime.p99 > 5000) {
189 alerts.push(this.createAlert(
190 'slow-response',
191 'medium',
192 `Slow response times on ${cohort.cohortName}`,
193 `P99 response time is ${cohort.metrics.application.responseTime.p99.toFixed(0)}ms`,
194 cohort.cohortId
195 ));
196 }
197
198 // Business Metrics Alerts
199 if (cohort.metrics.business.tributesFound === 0) {
200 alerts.push(this.createAlert(
201 'no-tributes',
202 'low',
203 `No tributes found on ${cohort.cohortName}`,
204 'Legion legion might need motivation!',
205 cohort.cohortId
206 ));
207 }
208
209 // Add new alerts and trigger them
210 for (const alert of alerts) {
211 if (!cohort.alerts.some(a => a.type === alert.type && a.severity === alert.severity)) {
212 cohort.alerts.push(alert);
213 this.eventEmitter.emit('alert.triggered', alert);
214 }
215 }
216 }
217
218 @Cron('0 */5 * * * *') // Every 5 minutes
219 checkLegionHealth() {
220 const offlineLegions = [];
221 const now = Date.now();
222
223 for (const [cohortId, cohort] of this.legion.cohorts) {
224 const timeSinceLastSeen = now - cohort.lastSeen.getTime();
225 
226 if (timeSinceLastSeen > 300000) { // 5 minutes offline
227 cohort.status = 'offline';
228 offlineLegions.push(cohort);
229 }
230 }
231
232 if (offlineLegions.length > 0) {
233 for (const cohort of offlineLegions) {
234 const alert = this.createAlert(
235 'cohort-offline',
236 'critical',
237 `Legion ${cohort.cohortName} is offline`,
238 `No communication for ${Math.round((now - cohort.lastSeen.getTime()) / 60000)} minutes`,
239 cohort.cohortId
240 );
241 
242 this.eventEmitter.emit('alert.triggered', alert);
243 }
244 }
245
246 this.updateLegionStats();
247 }
248
249 getLegionOverview(): Legion {
250 return this.legion;
251 }
252
253 getLegionDetails(cohortId: string): LegionInfo | null {
254 return this.legion.cohorts.get(cohortId) || null;
255 }
256
257 getLegionMetrics() {
258 const cohorts = Array.from(this.legion.cohorts.values());
259 
260 return {
261 legion: this.legion,
262 aggregatedMetrics: {
263 totalCpu: cohorts.reduce((sum, cohort) => sum + (cohort.metrics?.system.cpu || 0), 0),
264 totalMemory: cohorts.reduce((sum, cohort) => sum + (cohort.metrics?.system.memory || 0), 0),
265 totalErrors: cohorts.reduce((sum, cohort) => sum + cohort.errors.length, 0),
266 totalTributes: cohorts.reduce((sum, cohort) => sum + (cohort.metrics?.business.tributesFound || 0), 0),
267 averageResponseTime: cohorts.reduce((sum, cohort) => sum + (cohort.metrics?.application.responseTime.avg || 0), 0) / cohorts.length,
268 },
269 topPerformers: this.getTopPerformingLegions(),
270 worstPerformers: this.getWorstPerformingLegions(),
271 };
272 }
273
274 private getTopPerformingLegions(): LegionInfo[] {
275 return Array.from(this.legion.cohorts.values())
276 .filter(cohort => cohort.metrics && cohort.status === 'healthy')
277 .sort((a, b) => b.metrics.business.tributesFound - a.metrics.business.tributesFound)
278 .slice(0, 5);
279 }
280
281 private getWorstPerformingLegions(): LegionInfo[] {
282 return Array.from(this.legion.cohorts.values())
283 .filter(cohort => cohort.metrics && cohort.status !== 'healthy')
284 .sort((a, b) => b.metrics.application.errorRate - a.metrics.application.errorRate)
285 .slice(0, 5);
286 }
287
288 private createAlert(type: string, severity: string, title: string, description: string, cohortId: string): Alert {
289 return {
290 id: `alert-${Date.now()}-${Math.random()}`,
291 type,
292 severity,
293 title,
294 description,
295 cohortId,
296 timestamp: new Date(),
297 resolved: false,
298 };
299 }
300
301 private processError(errorEvent: ErrorEvent) {
302 // Implement error processing logic
303 // - Error correlation
304 // - Pattern detection
305 // - Automatic alert generation
306 
307 this.logger.debug(`Processing error from ${errorEvent.cohortId}: ${errorEvent.error.message}`);
308 }
309}

Tasks to Complete

Part 1: Basic System (4-6 hours)

1. Legion Agent Implementation

• Implement LegionAgentService with real metrics
• Add error tracking interceptor
• Create health check endpoints
• Add automatic registration with Legion Command

2. Legion Command Core

• Implement LegionMonitorService
• Create API endpoints for registration and metrics
• Add basic dashboard endpoint
• Implement real-time WebSocket communication

Part 2: Advanced Features (6-8 hours)

3. Intelligent Alerting

• Machine learning for problem prediction
• Alert correlation and deduplication
• Escalation policies
• Multi-channel notifications (Slack, email, SMS)

4. Analytics Engine

• Trend analysis i forecasting
• Anomaly detection
• Performance baselines
• Business metrics correlation

Part 3: Enterprise Features (8-10 hours)

5. Advanced Monitoring

• Custom metrics framework
• Distributed tracing
• Log aggregation
• SLA monitoring

6. Integration & Deployment

• Prometheus/Grafana integration
• Docker containerization
• Kubernetes deployment
• CI/CD pipeline

Part 4: Testing & Documentation (4-6 hours)

7. Comprehensive Testing

• Unit tests for all components
• Integration tests for legion communication
• Load testing for scalability
• E2E testing of the complete workflow

8. Documentation & Deployment Guide

• API documentation (Swagger)
• Deployment guide
• Monitoring runbooks
• Troubleshooting guide

Evaluation Criteria

Excellency Criteria (90-100%)

✓ Complete implementation of all required functionalities ✓ Performance - system handles >100 forts simultaneously ✓ Scalability - easy addition of new metric and alert types ✓ Reliability - system runs stably for 24h without interruptions ✓ Security - proper authentication and authorization ✓ Code Quality - clean code, proper patterns, comprehensive tests ✓ Documentation - complete documentation and deployment guides

Advanced Features (Bonus Points)

✓ Machine Learning - predictive alerting and anomaly detection ✓ Auto-scaling - automatic resource adjustment based on metrics ✓ Multi-region - support for geographically distributed legions ✓ Mobile App - React Native app for legion monitoring ✓ Voice Alerts - integration with Amazon Alexa for voice notifications

Good luck, legate! Your legion is counting on you - create a monitoring system that will be a legend across the provinces of technology!

private async getSystemMetrics(): Promise<SystemMetrics> { const memUsage = process.memoryUsage(); const cpuUsage = process.cpuUsage();

return { cpu: { usage: this.getCpuUsagePercent(cpuUsage), loadAverage: require('os').loadavg(), }, memory: { heapUsed: Math.round(memUsage.heapUsed / 1024 / 1024), heapTotal: Math.round(memUsage.heapTotal / 1024 / 1024), rss: Math.round(memUsage.rss / 1024 / 1024), external: Math.round(memUsage.external / 1024 / 1024), usagePercent: (memUsage.heapUsed / memUsage.heapTotal) * 100, }, gc: { lastDuration: 0, // Would need gc-stats module totalDuration: 0, collections: 0, }, eventLoop: { delay: await this.measureEventLoopDelay(), utilization: await this.measureEventLoopUtilization(), }, }; }

private async getApplicationMetrics(): Promise<ApplicationMetrics> { const httpStats = this.getPerformanceStats('http.request.duration', 60000); const dbStats = this.getPerformanceStats('database.query.duration', 60000);

return { http: { requestsPerSecond: this.calculateRate('http.requests', 60), averageResponseTime: httpStats?.average || 0, p95ResponseTime: httpStats?.p95 || 0, p99ResponseTime: httpStats?.p99 || 0, errorRate: this.calculateErrorRate(), activeConnections: this.gauges.get('http.connections') || 0, }, database: { activeConnections: this.gauges.get('db.connections') || 0, queryTime: { average: dbStats?.average || 0, p95: dbStats?.p95 || 0, p99: dbStats?.p99 || 0, }, queriesPerSecond: this.calculateRate('db.queries', 60), slowQueries: this.counters.get('db.slow_queries') || 0, }, cache: { hitRate: this.calculateCacheHitRate(), missRate: this.calculateCacheMissRate(), evictions: this.counters.get('cache.evictions') || 0, memoryUsage: this.gauges.get('cache.memory') || 0, }, }; }

private async getBusinessMetrics(): Promise<BusinessMetrics> { return { tributeOperations: { transfersPerMinute: this.calculateRate('tribute.transfers', 60), averageTransferTime: this.getPerformanceStats('tribute.transfer.duration')?.average || 0, failedTransfers: this.counters.get('tribute.transfers.failed') || 0, }, legionActivity: { activeUsers: this.gauges.get('legion.active_users') || 0, operationsPerUser: this.calculateOperationsPerUser(), sessionDuration: this.getPerformanceStats('user.session.duration')?.average || 0, }, cohortSystems: { healthScore: this.calculateOverallHealthScore(), maintenanceAlerts: this.counters.get('maintenance.alerts') || 0, systemUptime: process.uptime(), }, }; }

private getCustomMetrics(): Record<string, number> { const custom: Record<string, number> = {};

for (const [name, value] of this.gauges.entries()) { if (name.startsWith('custom.')) { custom[name.replace('custom.', '')] = value; } }

return custom; }

// Performance analysis methods analyzePerformanceTrends(hours: number = 24): any { const cutoff = new Date(Date.now() - hours * 60 * 60 * 1000); const recentSnapshots = this.snapshots.filter(s => s.timestamp >= cutoff);

if (recentSnapshots.length < 2) { return { message: 'Insufficient data for trend analysis' }; }

const trends = { responseTime: this.analyzeTrend(recentSnapshots.map(s => s.application.http.averageResponseTime)), memoryUsage: this.analyzeTrend(recentSnapshots.map(s => s.system.memory.usagePercent)), errorRate: this.analyzeTrend(recentSnapshots.map(s => s.application.http.errorRate)), throughput: this.analyzeTrend(recentSnapshots.map(s => s.application.http.requestsPerSecond)), };

return { period:

${hours} hours

, dataPoints: recentSnapshots.length, trends, recommendations: this.generateRecommendations(trends), }; }

detectPerformanceAnomalies(): any[] { const anomalies = []; const recentSnapshots = this.snapshots.slice(-100); // Last 100 snapshots

if (recentSnapshots.length < 10) return anomalies;

// Check for memory usage spikes const memoryValues = recentSnapshots.map(s => s.system.memory.usagePercent); const memoryAnomaly = this.detectSpike(memoryValues, 2.0); // 2 standard deviations if (memoryAnomaly) { anomalies.push({ type: 'memory_spike', severity: 'HIGH', description: 'Unusual memory usage spike detected', value: memoryAnomaly.value, threshold: memoryAnomaly.threshold, timestamp: new Date(), }); }

// Check for response time anomalies const responseTimeValues = recentSnapshots.map(s => s.application.http.averageResponseTime); const responseTimeAnomaly = this.detectSpike(responseTimeValues, 2.5); if (responseTimeAnomaly) { anomalies.push({ type: 'response_time_spike', severity: 'MEDIUM', description: 'Response time significantly higher than normal', value: responseTimeAnomaly.value, threshold: responseTimeAnomaly.threshold, timestamp: new Date(), }); }

return anomalies; }

// Utility methods private recordMetric(metric: PerformanceMetric) { const metricName = metric.name; if (!this.metrics.has(metricName)) { this.metrics.set(metricName, []); }

const metricArray = this.metrics.get(metricName)!; metricArray.push(metric);

// Keep only recent metrics (last hour) const hourAgo = new Date(Date.now() - 3600000); const filtered = metricArray.filter(m => m.timestamp >= hourAgo); this.metrics.set(metricName, filtered);

// Emit event for real-time monitoring this.eventEmitter.emit('metric.recorded', metric);

// Check thresholds if (metric.threshold) { if (metric.value >= metric.threshold.critical) { this.eventEmitter.emit('metric.critical', metric); } else if (metric.value >= metric.threshold.warning) { this.eventEmitter.emit('metric.warning', metric); } } }

private getPercentile(sortedValues: number[], percentile: number): number { const index = Math.ceil((percentile / 100) * sortedValues.length) - 1; return sortedValues[Math.max(0, index)]; }

private calculateStandardDeviation(values: number[]): number { const mean = values.reduce((sum, val) => sum + val, 0) / values.length; const squaredDiffs = values.map(val => Math.pow(val - mean, 2)); const avgSquaredDiff = squaredDiffs.reduce((sum, val) => sum + val, 0) / values.length; return Math.sqrt(avgSquaredDiff); }

private calculateRate(counterName: string, windowSeconds: number): number { const now = Date.now(); const windowStart = now - (windowSeconds * 1000);

const metrics = this.metrics.get(

${counterName}.count

) || []; const recentMetrics = metrics.filter(m => m.timestamp.getTime() >= windowStart);

if (recentMetrics.length === 0) return 0;

const totalCount = recentMetrics[recentMetrics.length - 1]?.value || 0; const initialCount = recentMetrics[0]?.value || 0;

return (totalCount - initialCount) / windowSeconds; }

private calculateErrorRate(): number { const totalRequests = this.counters.get('http.requests') || 0; const errorRequests = this.counters.get('http.requests.errors') || 0;

if (totalRequests === 0) return 0; return (errorRequests / totalRequests) * 100; }

private calculateCacheHitRate(): number { const hits = this.counters.get('cache.hits') || 0; const total = hits + (this.counters.get('cache.misses') || 0);

if (total === 0) return 0; return (hits / total) * 100; }

private calculateCacheMissRate(): number { return 100 - this.calculateCacheHitRate(); }

private calculateOperationsPerUser(): number { const totalOps = this.counters.get('user.operations') || 0; const activeUsers = this.gauges.get('legion.active_users') || 1;

return totalOps / activeUsers; }

private calculateOverallHealthScore(): number { const metrics = [ this.gauges.get('system.cpu.health') || 100, this.gauges.get('system.memory.health') || 100, this.gauges.get('database.health') || 100, this.gauges.get('cache.health') || 100, ];

return metrics.reduce((sum, val) => sum + val, 0) / metrics.length; }

private analyzeTrend(values: number[]): any { if (values.length < 2) return { trend: 'insufficient_data' };

const recent = values.slice(-Math.min(10, values.length)); const earlier = values.slice(0, Math.min(10, values.length));

const recentAvg = recent.reduce((sum, val) => sum + val, 0) / recent.length; const earlierAvg = earlier.reduce((sum, val) => sum + val, 0) / earlier.length;

const change = ((recentAvg - earlierAvg) / earlierAvg) * 100;

let trend = 'stable'; if (change > 10) trend = 'increasing'; else if (change < -10) trend = 'decreasing';

return { trend, change: Math.round(change * 100) / 100, recentAverage: Math.round(recentAvg * 100) / 100, earlierAverage: Math.round(earlierAvg * 100) / 100, }; }

private detectSpike(values: number[], threshold: number): any | null { if (values.length < 5) return null;

const recent = values.slice(-5); const historical = values.slice(0, -5);

const historicalMean = historical.reduce((sum, val) => sum + val, 0) / historical.length; const historicalStdDev = this.calculateStandardDeviation(historical);

const recentMax = Math.max(...recent); const spikeThreshold = historicalMean + (threshold * historicalStdDev);

if (recentMax > spikeThreshold) { return { value: recentMax, threshold: spikeThreshold, severity: recentMax > (historicalMean + 3 * historicalStdDev) ? 'HIGH' : 'MEDIUM', }; }

return null; }

private generateRecommendations(trends: any): string[] { const recommendations = [];

if (trends.responseTime?.trend === 'increasing') { recommendations.push('Response time is increasing - consider optimizing slow endpoints or scaling resources'); }

if (trends.memoryUsage?.trend === 'increasing') { recommendations.push('Memory usage is growing - check for memory leaks or consider increasing memory allocation'); }

if (trends.errorRate?.trend === 'increasing') { recommendations.push('Error rate is rising - investigate error logs and fix underlying issues'); }

if (trends.throughput?.trend === 'decreasing') { recommendations.push('Throughput is declining - check for bottlenecks in application or database'); }

return recommendations.length > 0 ? recommendations : ['Performance metrics are within normal ranges']; }

private async measureEventLoopDelay(): Promise<number> { return new Promise((resolve) => { const start = process.hrtime.bigint(); setImmediate(() => { const delay = Number(process.hrtime.bigint() - start) / 1000000; resolve(delay); }); }); }

private async measureEventLoopUtilization(): Promise<number> { // Simplified calculation - in real implementation would use perf_hooks return Math.random() * 100; // 0-100% }

private getCpuUsagePercent(cpuUsage: NodeJS.CpuUsage): number { // Simplified calculation return (cpuUsage.user + cpuUsage.system) / 1000000 * 100; }

private startPerformanceMonitoring() { // Take snapshots every 30 seconds setInterval(async () => { await this.getCurrentSnapshot(); }, 30000);

// Clean up old data every hour setInterval(() => { this.cleanupOldData(); }, 3600000); }

private cleanupOldData() { const cutoff = new Date(Date.now() - 24 * 60 * 60 * 1000); // 24 hours

for (const [name, metrics] of this.metrics.entries()) { const filtered = metrics.filter(m => m.timestamp >= cutoff); this.metrics.set(name, filtered); }

// Keep only recent snapshots this.snapshots = this.snapshots.filter(s => s.timestamp >= cutoff); } }

1
2Performance Metrics are your precise sextant on the sea of data - they allow you to navigate through performance issues and keep the fort on course for optimal efficiency!
3
4Remember: you cannot manage what you do not measure, and a good centurion knows the performance of every element of his fort!
5
6~~~sandpack/exercise_5_92