Google Cloud Multi-Region Live Traffic Visualizer

REGION: us-central1 (US) REGION: europe-west1 (EU) Browser Client App GCLB Anycast Balancer Cloud Run (US) hr-vacation-app us-central1 Cloud Run (EU) hr-vacation-app-eu europe-west1 AlloyDB Primary write-db.hr-vacation.internal READ-WRITE AlloyDB Replica read-db-eu.hr-vacation.internal READ-ONLY
DATABASE TRANSACTION LOGS
0 logs

Subsystem Workflow State Queue (/workflows)

Req ID Employee Sector Dates Requested Days Status Next Step Actions

Primary AlloyDB Resilient Console Inspector

Module 1 - Phase 2

Diagnosing & Remediating Multi-Region Cloud Infrastructure Outages via Agentic AI Tooling

Challenge Scenario

You are a Platform Cloud Engineer consulting for the Enterprise Architecture division at Cymbal Group. Following your successful migration of the Vacation Request Subsystem from a single-region footprint into a highly available, multi-region architecture, the Cymbal Group development team pushed a new code release (v2.0).

This new release was intended to integrate with the newly provisioned AlloyDB database cluster and dynamically route database read/write requests. However, immediately after deployment, the environment became unstable. During simulated peak quarterly review cycles, users are experiencing massive latency, and the backend Cloud Run service is throwing intermittent 500 Internal Server Errors.

You have been paged to resolve this outage. Because this is a restricted production environment, you do not have manual access to the Google Cloud Console. All diagnosis and remediation must be driven through your agentic IDEs, Google Agent Skills, and the Model Context Protocol (MCP) to read logs, diagnose the code logic or infrastructure misconfiguration, and deploy a fix.

Incident at a Glance

  • Customer: Cymbal Group — Enterprise Architecture division.
  • Workload: Multi-region Vacation Request Subsystem (primary us-central1, secondary europe-west1, fronted by a Global External Application Load Balancer).
  • Symptom 1 (Latency): European client requests experience high cross-region network propagation delay due to transatlantic routing hop.
  • Symptom 2 (Outage): Backend Cloud Run service throws intermittent HTTP 500 database pool exhaustion errors under sustained load.

High-Level Task List

  1. Investigate and Triage the Outage: Query Cloud Logging to identify why the European frontend is taking excessively long to respond and isolate the stack traces generating the HTTP 500 errors.
  2. Resolve the Infrastructure Fault (Regional Routing): Discover the incorrect database host configuration in the European frontend and update the environment variable to route queries locally.
  3. Resolve the Application Fault (Connection Leak): Refactor the backend Node.js connection logic to prevent creating a new database connection pool inside request handlers, resolving the connection pool leak.
  4. Deploy the Hotfix: Rebuild and push the updated image and deploy the new Cloud Run revisions.
  5. Validate System Recovery: Run load verification to confirm European response latency is minimized and error rates maintain 0% under peak loads.