MLOps: DevOps for Machine Learning

MLOps (Machine Learning Operations) is the practice of applying DevOps principles — continuous integration, continuous delivery, monitoring, and automation — to machine learning systems. While traditional software delivers deterministic code, ML systems deliver trained models that depend on data, code, and configuration simultaneously.

Why ML is Different from Traditional Software

Google identifies three critical differences in ML systems: (1) ML development is experimental — you track not just code but experiments, hyperparameters, and metrics. (2) Testing is more complex — you must test data and model quality, not just code. (3) Deployment is not just about code — you deploy a complete ML pipeline that can automatically retrain and serve updated models.

Google defines three levels of MLOps maturity. Most organizations start at Level 0 and progressively automate. The exam frequently tests your ability to assess which level an organization is at and recommend the next steps to advance.

Level 0: Manual Process

At Level 0, data scientists work in Jupyter notebooks, manually train models, and hand off trained artifacts to engineers for deployment. There is no automation, no pipeline, and no continuous training.

Level 1: ML Pipeline Automation

At Level 1, the unit of deployment is not a model but an ML pipeline. The pipeline automates data ingestion, validation, transformation, training, evaluation, and model deployment. Continuous training is enabled — the pipeline can be triggered by new data, a schedule, or performance degradation.

# Level 1 pipeline pattern (conceptual)
# Deploy the PIPELINE, not the model

pipeline_steps = [
    "data_ingestion",      # Pull from BigQuery / GCS
    "data_validation",      # Schema + distribution checks (TFDV)
    "feature_engineering",   # Transform + Feature Store integration
    "model_training",        # Train with tracked hyperparameters
    "model_evaluation",      # Compare against baseline metrics
    "model_validation",      # Gate: only deploy if metrics pass threshold
    "model_deployment",      # Push to Vertex AI Endpoint
    "monitoring_setup",      # Configure drift detection + alerts
]

Level 2: CI/CD Pipeline Automation

Level 2 adds CI/CD for the ML pipeline itself. Changes to pipeline code, model architecture, or hyperparameters go through automated testing, building, and deployment — just like application code in traditional DevOps. This is full MLOps maturity.

Vertex AI provides a unified MLOps platform with integrated components that work together. Understanding which component to use for each task is critical for the exam.

Vertex AI Experiments provides a structured way to track, compare, and reproduce ML experiments. Every experiment run captures parameters, metrics, and artifacts — making it possible to answer "which configuration produced the best model?" at any time.

Tracking Metrics and Parameters

An experiment groups related runs. Each run captures a single training attempt with specific hyperparameters, metrics, and artifacts.

from google.cloud import aiplatform

# Initialize Vertex AI
aiplatform.init(
    project="my-project",
    location="us-central1",
    experiment="fraud-detection-v2",
)

# Start a run within the experiment
with aiplatform.start_run("xgboost-run-001") as run:
    # Log hyperparameters
    run.log_params({
        "learning_rate": 0.01,
        "max_depth": 6,
        "n_estimators": 500,
        "subsample": 0.8,
    })

    # Train model...
    # model = train_xgboost(params)

    # Log metrics
    run.log_metrics({
        "auc_roc": 0.945,
        "precision": 0.89,
        "recall": 0.82,
        "f1_score": 0.854,
    })

    # Log time-series metrics (e.g., per epoch)
    for epoch in range(10):
        run.log_time_series_metrics({
            "train_loss": 0.5 - epoch * 0.04,
            "val_loss": 0.55 - epoch * 0.03,
        })

Comparing Runs and Hyperparameter Tuning

Experiments lets you compare runs as a DataFrame, making it easy to identify the best-performing configuration. This integrates with Vertex AI Vizier for automated hyperparameter tuning.

# Get experiment results as a DataFrame
experiment_df = aiplatform.get_experiment_df("fraud-detection-v2")
print(experiment_df[[
    "run_name",
    "param.learning_rate",
    "param.max_depth",
    "metric.auc_roc",
    "metric.f1_score",
]].sort_values("metric.auc_roc", ascending=False))

# Find the best run
best_run = experiment_df.loc[experiment_df["metric.auc_roc"].idxmax()]
print(f"Best run: {best_run['run_name']}, AUC: {best_run['metric.auc_roc']}")

The Vertex AI Model Registry is a centralized repository for managing trained models across their lifecycle. It tracks versions, manages deployments, and maintains lineage back to the training data and pipeline that produced each model.

Model Versioning and Lineage

Each model in the registry can have multiple versions. Versions are immutable snapshots that capture the model artifact, framework, container image, and metadata. Lineage connects each version to the experiment run, training pipeline, and dataset that produced it.

from google.cloud import aiplatform

# Upload a model to the registry
model = aiplatform.Model.upload(
    display_name="fraud-detector",
    artifact_uri="gs://my-bucket/models/fraud-v2/",
    serving_container_image_uri="us-docker.pkg.dev/vertex-ai/prediction/sklearn-cpu.1-3:latest",
    labels={"team": "fraud", "env": "production"},
    version_aliases=["champion", "v2"],
    version_description="XGBoost model trained on Q4 2025 data, AUC=0.945",
)
print(f"Model resource: {model.resource_name}")
print(f"Version ID: {model.version_id}")

# List all versions
versions = aiplatform.Model.list(
    filter='display_name="fraud-detector"'
)
for v in versions:
    print(f"  Version {v.version_id}: {v.version_aliases} - {v.version_description}")

Deployment Management

The registry manages which model versions are deployed and where. You can use version aliases (like "champion" and "challenger") to implement blue-green or canary deployment patterns.

# Deploy from registry to an endpoint
endpoint = aiplatform.Endpoint.create(display_name="fraud-endpoint")

# Deploy champion model with 90% traffic
endpoint.deploy(
    model=model,
    deployed_model_display_name="champion-v2",
    traffic_percentage=90,
    machine_type="n1-standard-4",
    min_replica_count=1,
    max_replica_count=5,
)

# Deploy challenger model with 10% traffic (A/B test)
challenger = aiplatform.Model("projects/my-project/locations/us-central1/models/fraud-detector@challenger")
endpoint.deploy(
    model=challenger,
    deployed_model_display_name="challenger-v3",
    traffic_percentage=10,
    machine_type="n1-standard-4",
)

The Metadata Store captures the full lineage of your ML workflows — from raw data through feature engineering, training, evaluation, to deployed models. It answers questions like: "Which dataset was used to train this model?", "What pipeline produced this artifact?", and "Which models were trained on data from this source?"

Artifacts and Executions

Metadata is organized into three core concepts:

• Artifacts — datasets, models, metrics, schemas. Anything that is produced or consumed.
• Executions — pipeline steps, training jobs, preprocessing runs. Actions that produce or consume artifacts.
• Events — relationships between artifacts and executions (input/output edges in the lineage graph).

from google.cloud import aiplatform

# Log a dataset artifact
with aiplatform.start_run("training-run-042") as run:
    # Log input artifact (dataset)
    run.log_params({"dataset": "gs://bucket/data/fraud-2025q4.csv"})

    # After training, log the model artifact
    run.log_metrics({"auc": 0.945})

# Query lineage: which artifacts were produced by this execution?
experiment_df = aiplatform.get_experiment_df("fraud-detection-v2")
print(experiment_df)

Lineage Tracking

Lineage tracking allows you to trace the complete history of any artifact — forward (what used this dataset?) or backward (what produced this model?). This is essential for debugging (why did model performance degrade?), compliance (which data trained this model?), and reproducibility (can I retrain this exact model?).

Models degrade over time because the world changes — user behavior shifts, fraud patterns evolve, and market conditions fluctuate. Continuous training (CT) automatically retrains models to keep them current. This is a hallmark of Level 1+ MLOps maturity.

Continuous Training Triggers

# Continuous training trigger pattern
from google.cloud import aiplatform
from google.cloud import scheduler_v1

# Option 1: Schedule-based trigger (weekly retraining)
# Cloud Scheduler job triggers the pipeline via Pub/Sub
pipeline_job = aiplatform.PipelineJob(
    display_name="weekly-fraud-retrain",
    template_path="gs://my-bucket/pipelines/fraud-pipeline.yaml",
    pipeline_root="gs://my-bucket/pipeline-runs/",
    parameter_values={
        "training_data_uri": "bq://project.dataset.fraud_features",
        "threshold_auc": 0.92,
    },
)

# Create a recurring schedule
schedule = pipeline_job.create_schedule(
    display_name="weekly-fraud-retrain-schedule",
    cron="0 2 * * 0",  # Every Sunday at 2 AM
)

# Option 2: Drift-based trigger (conceptual)
# Model Monitoring detects drift -> sends alert -> Cloud Function triggers pipeline
# def handle_drift_alert(event, context):
#     if event["alertType"] == "feature_drift":
#         pipeline_job.submit()

CI/CD for ML extends traditional CI/CD to handle the unique challenges of ML systems. You must test not just code, but also data quality, model quality, and the full pipeline.

Testing Strategy for ML Systems

Deployment Gates

Deployment gates are automated checks that must pass before a model can be promoted to production. They prevent deploying models that are worse than the current production model.

# Deployment gate pattern in a Vertex AI Pipeline
def evaluate_and_gate(
    model_metrics: dict,
    baseline_metrics: dict,
    thresholds: dict,
) -> bool:
    """
    Deployment gate: model must beat baseline on all thresholds.
    Returns True if model should be deployed.
    """
    gates_passed = True

    # Gate 1: Absolute minimum performance
    if model_metrics["auc"] < thresholds["min_auc"]:
        print(f"FAIL: AUC {model_metrics['auc']:.3f} below minimum {thresholds['min_auc']}")
        gates_passed = False

    # Gate 2: Must beat or match current champion
    if model_metrics["auc"] < baseline_metrics["auc"] - thresholds["regression_tolerance"]:
        print(f"FAIL: AUC regressed vs champion")
        gates_passed = False

    # Gate 3: Latency requirement
    if model_metrics["p99_latency_ms"] > thresholds["max_latency_ms"]:
        print(f"FAIL: Latency {model_metrics['p99_latency_ms']}ms exceeds limit")
        gates_passed = False

    return gates_passed

Vertex AI Pipelines is the orchestration engine for MLOps. It runs ML workflows as directed acyclic graphs (DAGs) using the Kubeflow Pipelines SDK v2 or TFX. Pipelines are covered in depth in Course 15 — here we focus on how they fit into the MLOps architecture.

# Minimal Vertex AI Pipeline skeleton
from kfp import dsl
from google_cloud_pipeline_components.v1 import (
    bigquery as bq_components,
    custom_job as custom_components,
    endpoint as endpoint_components,
)

@dsl.pipeline(name="fraud-training-pipeline")
def fraud_pipeline(project: str, location: str):
    # Step 1: Extract training data from BigQuery
    data_op = bq_components.BigqueryQueryJobOp(
        query="SELECT * FROM fraud.features WHERE date > '2025-01-01'",
        project=project,
    )

    # Step 2: Train model (depends on data_op)
    train_op = custom_components.CustomTrainingJobOp(
        display_name="fraud-xgboost",
        script_path="train.py",
    ).after(data_op)

    # Step 3: Deploy (depends on train_op)
    deploy_op = endpoint_components.ModelDeployOp(
        model=train_op.outputs["model"],
    ).after(train_op)

This section maps to Section 5: Automating and orchestrating ML pipelines on the GCP Machine Learning Engineer exam. Here are the key patterns to recognize.

MLOps Maturity Assessment

Choosing the Right Vertex AI Component

Key Exam Takeaways

• MLOps is about deploying pipelines, not models. The pipeline is the deployable unit.
• Continuous training requires triggers (schedule, drift, performance) and validation gates.
• Data testing is as important as code testing in ML CI/CD.
• Metadata Store provides lineage; Experiments provides comparison.
• Model Registry manages versions and aliases for champion/challenger patterns.
• Level 2 maturity adds CI/CD for the pipeline code itself, not just for the model.