Overview
Cloud Logging is a fully managed service that allows you to store, search, analyze, monitor, and alert on log data and events from Google Cloud and Amazon Web Services (AWS). It is a core component of the Google Cloud Operations Suite (formerly Stackdriver).
The “Security Camera” Analogy
Imagine a massive shopping mall. Cloud Logging is the central security office. Every entrance, cash register, and hallway has a camera (Resource Logs). Some cameras are always on (Admin Activity), while others only record if you pay for the extra storage (Data Access). If a shoplifter is spotted, the security team doesn’t just watch the screen; they can send the footage to a long-term vault (Cloud Storage), an analytics team to study patterns (BigQuery), or trigger an immediate alarm (Pub/Sub to Cloud Functions).
Detail Elaboration: The Log Lifecycle
Logs in GCP flow through a specific pipeline. Understanding this flow is critical for the ACE exam:
- Ingestion: Logs are generated by resources (GCE, GKE, App Engine) and sent to the Cloud Logging API.
- Log Router: The “Traffic Cop” of logging. It checks every log entry against Sinks to decide where the log should go.
- Storage: By default, logs are stored in
_Defaultor_Requiredlog buckets. - Retention: Logs aren’t kept forever by default (usually 30 days for standard logs).
Core Concepts & Best Practices
- Reliability: Centralizing logs ensures that even if a VM instance is deleted, its logs persist for troubleshooting.
- Security: Cloud Audit Logs (Admin Activity, Data Access, System Event, Policy Denied) provide the “Who did what, where, and when.”
- Cost Optimization: Use Exclusion Filters to drop high-volume, low-value logs (like routine health checks) before they are ingested to save money.
Comparison: Log Export Destinations
| Destination | Best For… | Retention Period | Analysis Tool |
|---|---|---|---|
| Cloud Storage | Compliance & Archiving (Cheap) | Years (Configurable) | GCS Fuse / Downloads |
| BigQuery | Big Data Analytics / SQL | Indefinite | Standard SQL |
| Pub/Sub | Real-time Streaming / Integration | Up to 7 days | Dataflow / Functions |
| Log Bucket | Standard Troubleshooting | 30 – 3650 Days | Log Explorer |
Scenario-Based Decision Matrix
IF the requirement is to keep logs for 7 years for regulatory audits… THEN use a Sink to Cloud Storage.
IF you need to trigger a Cloud Function whenever a Firewall rule is changed… THEN use a Sink to Pub/Sub.
IF you need to perform complex SQL joins across logs from multiple projects… THEN use a Sink to BigQuery.
IF you want to see how many 404 errors occur per minute… THEN create a Log-based Metric.
Exam Tips: Golden Nuggets
- Audit Logs: Admin Activity logs are ALWAYS on and free. Data Access logs (except BigQuery) are OFF by default because they generate massive volume.
- Log Sinks: Sinks are project-level by default, but can be “Aggregated Sinks” at the Folder or Organization level.
- The Agent: For “OS-level” logs (like Syslog or Windows Event Logs) on Compute Engine, you MUST install the Ops Agent (or the legacy Logging Agent).
- Retention: You cannot change the retention of the
_Requiredbucket (400 days). You CAN change it for the_Defaultbucket.
Cloud Logging Architecture Flow
Key Components
- Log Explorer: Interface for searching/querying logs.
- Log Buckets: Storage containers within Cloud Logging.
- Metrics: Convert log patterns into charts/alerts.
Common Pitfalls
- Forgetting to install Ops Agent for memory/disk logs.
- Not excluding high-volume logs (unexpected costs).
- Searching for Data Access logs when they aren’t enabled.
Architecture Patterns
- Centralized Security: Use an Aggregated Sink to send all Org logs to one project.
- SIEM Integration: Sink logs to Pub/Sub to feed Splunk or QRadar.