Goal, Problem & Solution
Goal
- Enable proactive incident response by detecting cloud service degradation 30-60 minutes ahead of official GCP and AWS status pages, establishing the CCoE as the authoritative early-warning system for Japan operations.
- Reduce mean time to detect (MTTD) for cloud incidents from 30-60 minutes to under 5 minutes, minimizing downstream impact on business-critical workloads.
- Deliver unified multi-cloud observability across GCP and AWS Japan regions, covering 9 critical services that underpin Rakuten's cloud infrastructure.
Problem
- Official cloud status dashboards (GCP and AWS) lag 30-60 minutes behind real incidents and only surface issues exceeding internal thresholds.
- Regional outages in Japan often go undetected until end-users report them, causing cascading failures, SLA violations, and unplanned firefighting.
- No centralized, cross-cloud visibility — teams rely on separate per-provider dashboards with no correlation of incidents across GCP and AWS.
Solution
- Three-layer detection per cloud provider: official health APIs, synthetic API probes, and network-level checks — catching incidents at every level of the stack.
- Unified Prometheus backbone with automated alert routing through Alertmanager to Slack, providing a single pane of glass across both cloud providers.
- Fully containerized and reproducible via Docker Compose — deployable in minutes, scalable to additional cloud providers and regions.
Architecture Overview
Interval: 60s
Interval: 300s
Interval: 60s
Interval: 300s
GCP + AWS endpoints
Time-series DB (30-day retention)
Evaluates alert rules every 30s
Labels: service, region, severity
Metric prefixes: gcp_* / aws_*
Routes by severity
#ccoe-warnings
Historical trend analysis
Component Details
GCP Exporters
GCP Health Exporter (Port 8000) GCP
Purpose: Detect active GCP incidents affecting Japan regions
Data Sources:
- Primary: Public incidents.json feed (no authentication required)
- Fallback: GCP Personalized Service Health REST API (Application Default Credentials)
Workflow: Both sources are polled every 60s, deduplicated by incident ID (PSH takes priority), filtered to asia-northeast1 and asia-northeast2, then exposed as Prometheus metrics.
Key Metrics: gcp_incidents_active, gcp_incident_state, gcp_incident_duration_seconds
GCP Probe Exporter (Port 8001) GCP
Purpose: Detect service degradation by making real authenticated API calls
Probes (every 300s):
- Compute Engine: List instances in {region}-a zone
- BigQuery: Run
SELECT 1 AS health_checkquery - GKE: List clusters in region
- Pub/Sub: List topics in project
- Cloud Storage: List buckets in project
Requires: GCP credentials mounted from ~/.config/gcloud
Key Metrics: gcp_probe_success, gcp_probe_latency_ms (histogram), gcp_probe_errors_total
AWS Exporters
AWS Health Exporter (Port 8002) AWS
Purpose: Detect active AWS incidents affecting Japan regions
Data Sources:
- Primary: AWS Health API via boto3 (requires Business/Enterprise support plan)
- Fallback: AWS Service Health Dashboard RSS feeds (public, no auth needed)
Workflow: Both sources are polled every 60s, deduplicated by event ID (Health API takes priority), filtered to ap-northeast-1 and ap-northeast-3, then exposed as Prometheus metrics.
Key Metrics: aws_incidents_active, aws_incident_state, aws_incident_duration_seconds
AWS Probe Exporter (Port 8003) AWS
Purpose: Detect service degradation by making real authenticated API calls
Probes (every 300s, per region):
- EC2: Describe instances in region
- EKS: List clusters in region
- S3: List buckets (global)
- RDS: Describe DB instances in region
Requires: AWS credentials mounted from ~/.aws
Key Metrics: aws_probe_success, aws_probe_latency_ms (histogram), aws_probe_errors_total
Shared Infrastructure
Blackbox Exporter (Port 9115) Shared
Purpose: Network-level reachability checks for GCP and AWS API endpoints (no authentication)
Probe Types: HTTP (accepts 2xx/3xx/4xx as reachable), TCP (port 443 TLS handshake)
GCP Targets: Compute/GKE endpoints for asia-northeast1 and asia-northeast2, plus global BigQuery/Pub/Sub/Storage
AWS Targets: EC2/EKS/RDS endpoints for ap-northeast-1 and ap-northeast-3, plus global S3
Key Metrics: probe_success, probe_duration_seconds
Prometheus (Port 9090) Shared
Purpose: Scrapes all exporters (GCP + AWS), stores time-series data (30-day retention), evaluates alert rules every 30s
Alert Rules: 6 groups — GCP health, GCP probes, AWS health, AWS probes, blackbox, and system alerts
Alertmanager (Port 9093) Shared
Purpose: Receives fired alerts from Prometheus, deduplicates and groups them, routes to Slack channels
Routing: Critical alerts (incidents, probe failures) and warning alerts (latency, resource usage) are routed separately regardless of cloud provider
Grafana (Port 3000) Shared
Purpose: Dashboards and historical analysis. Prometheus is auto-provisioned as a datasource.
Data Flow
Collect
Exporters run on their own intervals: Health Exporters poll cloud status APIs (60s), Probe Exporters make synthetic API calls (300s), Blackbox Exporter probes endpoints (60s), Node Exporter reads VM stats (60s). GCP and AWS exporters run independently.
Scrape & Store
Prometheus scrapes each exporter's /metrics endpoint, stores the time-series data with labels (service, region, severity), and retains it for 30 days. GCP metrics use the gcp_* prefix, AWS metrics use aws_*.
Evaluate & Alert
Every 30 seconds, Prometheus evaluates alert rules for both providers (e.g., gcp_probe_success == 0, aws_probe_success == 0). Fired alerts go to Alertmanager, which deduplicates, groups by service/region, and sends to Slack.
Visualize
Grafana queries Prometheus for real-time dashboards and historical trend analysis across all monitored services, regions, and cloud providers.
Probe Exporter vs Blackbox Exporter
| Aspect | GCP Probe Exporter | AWS Probe Exporter | Blackbox Exporter |
|---|---|---|---|
| What It Tests | GCP service functionality | AWS service functionality | Network connectivity & endpoint reachability |
| Test Method | Authenticated GCP API calls | Authenticated AWS API calls | Unauthenticated HTTP/TCP probes |
| Authentication | GCP ADC (~/.config/gcloud) | AWS credentials (~/.aws) | None needed |
| Services | Compute, BigQuery, GKE, Pub/Sub, Storage | EC2, EKS, S3, RDS | All (via endpoint URLs) |
| What It Catches | Service degradation, slow responses, API errors | Service degradation, slow responses, API errors | Network outages, DNS failures, endpoint down |
| Interval | 300 seconds | 300 seconds | 60 seconds |
Services & Ports
| Service | Port | Container Name | Cloud | Image |
|---|---|---|---|---|
| GCP Health Exporter | 8000 | gcp-health-exporter | GCP | Custom (Dockerfile.exporter) |
| GCP Probe Exporter | 8001 | gcp-probe-exporter | GCP | Custom (Dockerfile.probe) |
| AWS Health Exporter | 8002 | aws-health-exporter | AWS | Custom (Dockerfile.aws_exporter) |
| AWS Probe Exporter | 8003 | aws-probe-exporter | AWS | Custom (Dockerfile.aws_probe) |
| Blackbox Exporter | 9115 | blackbox-exporter | Shared | prom/blackbox-exporter |
| Prometheus | 9090 | prometheus | Shared | prom/prometheus |
| Alertmanager | 9093 | alertmanager | Shared | prom/alertmanager |
| Grafana | 3000 | grafana | Shared | grafana/grafana |
| Node Exporter | 9100 | node-exporter | Shared | prom/node-exporter |
Separation Strategy
| Concern | GCP | AWS | Shared |
|---|---|---|---|
| Health Exporter | gcp_health_exporter.py | aws_health_exporter.py | - |
| Probe Exporter | gcp_probe_exporter.py | aws_probe_exporter.py | - |
| Dependencies | requirements.txt | aws_requirements.txt | - |
| Dockerfiles | Dockerfile.exporter, Dockerfile.probe | Dockerfile.aws_exporter, Dockerfile.aws_probe | - |
| Credentials | ~/.config/gcloud | ~/.aws | - |
| Metric Prefix | gcp_* | aws_* | probe_* (blackbox) |
| Alert Groups | gcp_health_alerts, gcp_probe_alerts | aws_health_alerts, aws_probe_alerts | blackbox_alerts, system_alerts |
| Config | - | - | prometheus.yml, alert_rules.yml, alertmanager.yml, docker-compose.yml |
How Alerts Reach Notification Channel
The platform watches Google Cloud and AWS continuously. When either cloud reports a problem, the system decides how much it matters to us and posts it to one of two Slack channels. The rule is simple: if it can hurt our Japan workloads, it is a Critical alert. If it is happening somewhere else, or the cloud provider is not sure it affects us, it is a Warning — logged for awareness, not for a wake-up call.
The Decision Rules at a Glance
| Where the signal comes from | Condition | Fires after |
|---|---|---|
| 🚨 Critical #ccoe-critical-alerts Our Japan workloads are affected right now — someone should look at this | ||
| GCPPersonalized Service Health | Google confirms the incident is Impacted or Related to our project and it affects Japan | 2 min |
| GCPPublic status page | Outage affects asia-northeast1 (Tokyo), asia-northeast2 (Osaka), or a global location | 2 min |
| AWSAccount health dashboard | AWS reports an open event on our account in ap-northeast-1 (Tokyo) or ap-northeast-3 (Osaka) | 2 min |
| AWSPublic status page | Outage affects ap-northeast-1 (Tokyo) or ap-northeast-3 (Osaka) — caught even if AWS has not linked it to our account | 2 min |
| GCPAWSCloud health feeds | More than 2 separate incidents at once — the signature of a widespread outage. Each incident counts once, however many regions it spans | 1 min |
| GCPAWSSynthetic API probes | A real API call we make ourselves (list VMs, SELECT 1, list buckets…) fails — we see the problem even if the provider has not published it yet |
2 min |
| GCPAWSSynthetic API probes | 4 or more of the last 6 test calls failed for the same service and region — a service that keeps failing and recovering, which a single-failure check can miss | 1 min |
| GCPAWSBlackbox probes | A cloud API endpoint stops responding to a plain connectivity check — catches network and DNS problems, not just service faults | 2 min |
| ⚠️ Warning #ccoe-warnings Worth knowing about — not our region, or not confirmed to touch us | ||
| GCPPersonalized Service Health | Google marks it Partially related, Unknown, or Not impacted | 2 min |
| GCPPersonalized Service Health | Google marks it Impacted or Related, but every affected location is outside Japan — real, just not ours to act on | 2 min |
| GCPPublic status page | Outage affects any other non-Japan location | 2 min |
| AWSAccount health dashboard | AWS reports an open event on our account in any non-Japan region | 2 min |
| AWSPublic status page | Outage affects any other non-Japan region — real, just not ours to act on | 2 min |
| GCPAWSCloud health feeds | An incident is still ongoing after 30 minutes — a nudge that it has not closed | 30 min |
| GCPAWSSynthetic API probes | Calls still succeed but are unusually slow (95% of calls over 5 seconds) — often the early sign of degradation | 3 min |
| GCPAWSBlackbox probes | An endpoint answers but takes more than 10 seconds | 2 min |
| Monitoring host | CPU, memory or disk on the monitoring VM goes above 80% — protects the monitor itself | 5 min |
One incident produces one alert, no matter how many regions it spans — every affected location is listed inside the single message. Japan above means Tokyo, Osaka, or a global / multi‑region location that includes Japan.
Each cloud is watched from two angles: the account view, which only shows what the provider has tied to our account, and the public status page, which covers every region and service. The public feed is what lets us see trouble outside Japan. An incident is considered over once the provider marks it resolved — or, if they never do, once it has gone quiet for a day.