Exploring Data Transformation with Google Cloud

Data by itself has little value. It becomes valuable when it is collected, stored, processed, analyzed, and activated. Google Cloud provides managed services for every stage of this value chain, allowing organizations to build complete data platforms without managing infrastructure.

Stages of the Data Value Chain

Data Types

BigQuery is Google Cloud's fully managed, serverless, petabyte-scale data warehouse. It enables super-fast SQL queries using the processing power of Google's infrastructure. BigQuery is one of the most heavily tested services on the CDL exam.

Architecture: Separation of Storage and Compute

BigQuery's key architectural innovation is the separation of storage and compute. Data is stored in Google's distributed storage system (Colossus), while compute resources for queries are provisioned from a shared pool (Dremel engine). This means you can scale storage and compute independently.

  [Data Sources]
       |
  [ Ingest: Streaming / Batch / Transfer ]
       |
  +---------+      +-----------+
  | Colossus |      |  Dremel   |
  | (Storage)|      | (Compute) |
  +---------+      +-----------+
       |                 |
       +--------+--------+
                |
      [ BigQuery SQL Engine ]
                |
    +-----------+-----------+
    |           |           |
  [Looker]  [Data Studio] [BQML]
        

Key BigQuery Features

• Serverless — No infrastructure to manage. No clusters to provision or tune.
• Standard SQL — ANSI-compliant SQL (no proprietary query language).
• Built-in ML — BigQuery ML lets you create and execute ML models using SQL.
• Streaming ingestion — Insert rows in real-time for near-instant analysis.
• Geospatial analytics — Built-in GIS functions for location-based queries.
• Column-level security — Fine-grained access control on individual columns.
• BI Engine — In-memory analysis layer for sub-second dashboard queries.

BigQuery Pricing Models

# Query a public dataset (free tier)
bq query --use_legacy_sql=false \
  'SELECT name, SUM(number) as total
   FROM `bigquery-public-data.usa_names.usa_1910_current`
   WHERE year > 2000
   GROUP BY name
   ORDER BY total DESC
   LIMIT 10'

# Create a dataset and table
bq mk --dataset my_project:my_dataset
bq load --autodetect \
  my_dataset.sales_data \
  gs://my-bucket/sales.csv

Cloud Storage is Google Cloud's object storage service for storing any amount of unstructured data. It offers 11 nines (99.999999999%) durability, global accessibility, and four storage classes to optimize cost based on access frequency.

Storage Classes

Object Lifecycle Management

Lifecycle rules automatically transition or delete objects based on age, creation date, storage class, or other conditions. This is a key cost optimization feature for the exam.

# Set lifecycle rule: move to Coldline after 90 days, delete after 365 days
# lifecycle.json
{
  "lifecycle": {
    "rule": [
      {
        "action": {"type": "SetStorageClass", "storageClass": "COLDLINE"},
        "condition": {"age": 90}
      },
      {
        "action": {"type": "Delete"},
        "condition": {"age": 365}
      }
    ]
  }
}

# Apply lifecycle policy
gsutil lifecycle set lifecycle.json gs://my-bucket

Key Cloud Storage Concepts

Google Cloud offers multiple managed database services, each optimized for different workloads. The CDL exam tests your ability to match the right database to a given scenario.

Pub/Sub (Publish/Subscribe) is Google Cloud's fully managed messaging service for building event-driven architectures. It decouples event producers from consumers, enabling asynchronous communication between microservices, data pipelines, and applications.

How Pub/Sub Works

  [Publisher 1]  [Publisher 2]  [Publisher 3]
       |              |              |
       +------+-------+------+------+
              |              |
         [ Topic A ]    [ Topic B ]
              |              |
       +------+------+      |
       |             |       |
  [Subscription 1] [Sub 2] [Sub 3]
       |             |       |
  [Subscriber A] [Sub B] [Sub C]
  (push/pull)   (push)   (pull)
        

Key Pub/Sub Features

• At-least-once delivery — Every message is delivered at least once (may see duplicates).
• Push and pull — Pull: subscribers poll for messages. Push: Pub/Sub sends HTTP POST to an endpoint.
• Message retention — Up to 31 days (configurable). Replay old messages for recovery.
• Ordering — Optional message ordering within an ordering key for FIFO guarantees.
• Dead letter topics — Route undeliverable messages to a separate topic for debugging.
• Serverless & auto-scaling — Handles millions of messages per second with no capacity planning.

# Create a topic and subscription
gcloud pubsub topics create my-events
gcloud pubsub subscriptions create my-sub \
  --topic=my-events \
  --ack-deadline=60

# Publish a message
gcloud pubsub topics publish my-events \
  --message="Hello from CDL study guide"

# Pull messages
gcloud pubsub subscriptions pull my-sub --auto-ack --limit=5

Google Cloud offers two main data processing services. Dataflow is the serverless option based on Apache Beam. Dataproc is the managed Hadoop/Spark option for teams with existing Spark/Hadoop code.

Cloud Data Fusion

Cloud Data Fusion is a fully managed, visual ETL/ELT service. It provides a drag-and-drop interface for building data pipelines without writing code. Under the hood, it uses Dataproc for execution. Best for teams that prefer visual pipeline design or lack coding expertise.

Looker is Google Cloud's enterprise business intelligence (BI) platform. It provides a governed semantic layer (LookML) that ensures consistent metric definitions across an organization, plus dashboards, reports, and embedded analytics capabilities.

Google Cloud BI Tools

Service Selection Decision Tree

• "Store and analyze petabytes of structured data with SQL" → BigQuery
• "Store images, videos, backups, or any unstructured data" → Cloud Storage
• "Build a real-time event-driven pipeline" → Pub/Sub + Dataflow + BigQuery
• "Migrate existing Hadoop/Spark jobs" → Dataproc
• "Need a relational database for a web app" → Cloud SQL
• "Need global transactions with relational consistency" → Cloud Spanner
• "Build dashboards for business users" → Looker / Looker Studio
• "Real-time NoSQL for mobile/web apps" → Firestore
• "Massive throughput NoSQL for IoT/time-series" → Bigtable

Common Exam Scenarios