Ensuring Solution and Operations Excellence

Metrics and Dashboards

Cloud Monitoring collects metrics from all GCP services automatically. You can also send custom metrics via the Monitoring API or OpenTelemetry.

# Install Ops Agent on a Compute Engine VM
curl -sSO https://dl.google.com/cloudagents/add-google-cloud-ops-agent-repo.sh
sudo bash add-google-cloud-ops-agent-repo.sh --also-install

# Write a custom metric using gcloud
gcloud monitoring metrics-descriptors create \
    custom.googleapis.com/myapp/queue_depth \
    --type=GAUGE \
    --value-type=INT64 \
    --description="Number of items in the processing queue"

# Create a dashboard from a JSON definition
gcloud monitoring dashboards create \
    --config-from-file=dashboard.json

Alerting Policies

Alerting policies define conditions, notification channels, and documentation for automated incident detection.

# Create an alerting policy for high CPU
gcloud monitoring policies create \
    --display-name="High CPU Alert" \
    --condition-display-name="CPU > 80% for 5 min" \
    --condition-filter='metric.type="compute.googleapis.com/instance/cpu/utilization"
        AND resource.type="gce_instance"' \
    --condition-threshold-value=0.8 \
    --condition-threshold-comparison=COMPARISON_GT \
    --condition-threshold-duration=300s \
    --notification-channels=projects/my-project/notificationChannels/12345 \
    --documentation="Investigate high CPU. Check for runaway processes or traffic spike."

Uptime Checks

Uptime checks probe endpoints from multiple global locations. They detect outages before users report them. Uptime checks can verify HTTP response codes, response body content, and SSL certificate expiration.

Log Types

Log Sinks and Routing

Log sinks export logs to external destinations for long-term storage, analysis, or SIEM integration.

# Create a log sink to BigQuery for audit logs
gcloud logging sinks create audit-to-bq \
    bigquery.googleapis.com/projects/my-project/datasets/audit_logs \
    --log-filter='logName:"cloudaudit.googleapis.com"' \
    --use-partitioned-tables

# Create a log sink to Cloud Storage for long-term archival
gcloud logging sinks create all-logs-archive \
    storage.googleapis.com/my-project-log-archive \
    --log-filter='severity >= WARNING'

# Create a log-based metric for error counting
gcloud logging metrics create api-errors \
    --description="Count of API 5xx errors" \
    --log-filter='resource.type="cloud_run_revision"
        AND httpRequest.status >= 500'

SLIs, SLOs, and SLAs

Error Budgets

An error budget is the amount of unreliability your SLO allows. If your SLO is 99.9% availability, your error budget is 0.1% (about 43 minutes per month of allowed downtime).

• Budget remaining — Continue deploying new features, take calculated risks.
• Budget exhausted — Freeze deployments, focus on reliability improvements, reduce change velocity.
• Budget policy — Define what happens at 50%, 75%, 100% budget consumption. Automate alerts.

# Create an SLO in Cloud Monitoring
gcloud monitoring slo create \
    --service=my-cloud-run-service \
    --display-name="Availability SLO" \
    --goal=0.999 \
    --rolling-period=30d \
    --request-based-sli-good-total-ratio-threshold-performance \
    --good-service-filter='metric.type="run.googleapis.com/request_count"
        AND metric.labels.response_code_class="2xx"' \
    --total-service-filter='metric.type="run.googleapis.com/request_count"'

Chaos Engineering

Chaos engineering is the practice of intentionally injecting failures into a system to test its resilience. The PCA exam tests your understanding of when and how to apply chaos engineering.

• Start small — Begin with non-production environments, graduate to production.
• Have a rollback plan — Every chaos experiment must be reversible.
• Monitor continuously — Watch SLIs during experiments to detect cascading failures.
• Automate over time — Move from manual game days to automated chaos frameworks (Chaos Monkey, Litmus).

Load Testing

Load testing validates that your architecture handles expected and peak traffic. On GCP, common approaches include:

Release Strategies on GCP

Incident Response

A structured incident response process is essential for operations excellence:

1. Detect — Alerting policies, uptime checks, SLO burn-rate alerts trigger notifications.
2. Triage — Determine severity (P1-P4), assign incident commander, open communication channel.
3. Mitigate — Rollback deployment, scale resources, enable failover. Focus on restoring service, not root cause.
4. Resolve — Fix the underlying issue. Deploy fix through normal CI/CD pipeline.
5. Post-mortem — Blameless analysis of what happened, what went well, what to improve. Document action items.

Cloud Trace collects latency data from applications to help identify performance bottlenecks. It integrates with Cloud Run, GKE, App Engine, and custom applications via OpenTelemetry.

• Automatic Tracing — Cloud Run, App Engine, and Cloud Functions automatically report traces.
• Custom Instrumentation — Use OpenTelemetry SDK to add spans for custom code paths.
• Trace Analysis — View request waterfall diagrams, identify slow spans, correlate with logs.
• Latency Distribution — Analyze p50, p95, p99 latency across services and time periods.

# Python — OpenTelemetry tracing with Cloud Trace exporter
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.exporter.cloud_trace import CloudTraceSpanExporter
from opentelemetry.sdk.trace.export import BatchSpanProcessor

# Set up Cloud Trace exporter
trace.set_tracer_provider(TracerProvider())
tracer_provider = trace.get_tracer_provider()
cloud_trace_exporter = CloudTraceSpanExporter(project_id="my-project")
tracer_provider.add_span_processor(BatchSpanProcessor(cloud_trace_exporter))

# Instrument your code
tracer = trace.get_tracer("my-service")

with tracer.start_as_current_span("process-request") as span:
    span.set_attribute("user.id", user_id)
    result = process_data(data)  # This call will be traced
    span.set_attribute("result.count", len(result))