Vertex AI Feature Store

The Vertex AI Feature Store is a centralized, managed service for organizing, storing, and serving ML features. It solves three critical problems: (1) training-serving skew — features computed differently at training vs prediction time, (2) feature duplication — teams rebuilding the same features independently, and (3) point-in-time correctness — using future data that would not have been available at prediction time.

BigQuery-Backed Design (New Architecture)

The current Vertex AI Feature Store uses BigQuery as the offline store and a managed online serving layer for low-latency lookups. Features are defined as views or tables in BigQuery, and the Feature Store manages synchronization between offline and online stores.

Core Concepts

Feature Store provides two serving modes, each optimized for different access patterns. Understanding when to use each is critical for the exam.

Latency Requirements and Architecture

When to Use Each Mode

Create Feature Groups and Features

Feature groups are backed by BigQuery tables. You create a feature group by pointing to an existing BigQuery table, then register individual columns as features with metadata.

from google.cloud import aiplatform
from vertexai.resources.preview import feature_store

# Initialize
aiplatform.init(project="my-project", location="us-central1")

# Create a Feature Group backed by a BigQuery table
user_feature_group = feature_store.FeatureGroup.create(
    name="user_features",
    source=feature_store.utils.FeatureGroupBigQuerySource(
        uri="bq://my-project.features.user_features_table",
        entity_id_columns=["user_id"],
    ),
    description="User-level features for fraud detection",
)

# Register individual features with metadata
user_feature_group.create_feature(
    name="avg_transaction_amount_30d",
    description="Average transaction amount over last 30 days",
)
user_feature_group.create_feature(
    name="transaction_count_7d",
    description="Number of transactions in last 7 days",
)
user_feature_group.create_feature(
    name="account_age_days",
    description="Days since account creation",
)
user_feature_group.create_feature(
    name="is_verified",
    description="Whether user has completed identity verification",
)

Ingest Feature Values

Feature values come from batch ingestion (BigQuery tables updated by ETL pipelines) or streaming ingestion (real-time updates via the API). The BigQuery-backed architecture means batch ingestion is simply updating the underlying BigQuery table.

# Batch ingestion: update the BigQuery source table
# The Feature Store reads from BigQuery, so updating the table = updating features

# Option 1: BigQuery SQL (most common for batch)
# CREATE OR REPLACE TABLE features.user_features_table AS
# SELECT
#   user_id,
#   AVG(amount) OVER (PARTITION BY user_id ORDER BY timestamp
#     ROWS BETWEEN 30 PRECEDING AND CURRENT ROW) AS avg_transaction_amount_30d,
#   COUNT(*) OVER (PARTITION BY user_id ORDER BY timestamp
#     ROWS BETWEEN 7 PRECEDING AND CURRENT ROW) AS transaction_count_7d,
#   DATE_DIFF(CURRENT_DATE(), account_created, DAY) AS account_age_days,
#   is_verified,
#   CURRENT_TIMESTAMP() AS feature_timestamp
# FROM raw_data.transactions
# JOIN raw_data.users USING (user_id)

# Option 2: Streaming ingestion for real-time features
# (via the Feature Store write API)
from google.cloud.aiplatform_v1beta1 import FeatureOnlineStoreServiceClient

# client.write_feature_values(
#     feature_online_store="projects/.../featureOnlineStores/my-store",
#     feature_values=[{
#         "entity_id": "user_12345",
#         "feature_values": {"transaction_count_7d": 42, "is_verified": True}
#     }],
# )

Serve Features for Training (Offline)

For training, the Feature Store generates a training dataset by joining feature values with labels, respecting point-in-time correctness. This ensures each training example uses only features that were available at the time of the label.

# Create a Feature View for the fraud detection use case
fraud_feature_view = feature_store.FeatureView.create(
    name="fraud_detection_features",
    feature_registry_source=feature_store.utils.FeatureViewFeatureRegistrySource(
        feature_groups={
            user_feature_group.resource_name: feature_store.utils.FeatureViewFeatureRegistrySource.FeatureGroup(
                features=[
                    "avg_transaction_amount_30d",
                    "transaction_count_7d",
                    "account_age_days",
                    "is_verified",
                ],
            ),
        },
    ),
)

# Generate training data with point-in-time join
# Labels table has: entity_id, timestamp, label
training_data = fraud_feature_view.batch_serve(
    entity_df="bq://my-project.labels.fraud_labels",
    destination_uri="bq://my-project.training.fraud_training_data",
)
# Result: each row has features as-of the label timestamp

Serve Features for Prediction (Online)

For real-time predictions, the online store provides millisecond-latency lookups by entity ID. The prediction service fetches the latest feature values just before calling the model.

# Create an online store for low-latency serving
online_store = feature_store.FeatureOnlineStore.create(
    name="fraud-online-store",
    feature_online_store_type=feature_store.utils.FeatureOnlineStoreType.BIGTABLE,
    bigtable_auto_scaling=feature_store.utils.BigtableAutoScaling(
        min_node_count=1,
        max_node_count=3,
        cpu_utilization_target=70,
    ),
)

# Sync feature view to online store
sync_job = online_store.create_feature_view_sync(
    feature_view=fraud_feature_view,
)

# Fetch features at prediction time (single-digit ms)
feature_values = online_store.fetch_feature_values(
    feature_view="fraud_detection_features",
    entity_id="user_12345",
)
# Returns: {"avg_transaction_amount_30d": 450.0, "transaction_count_7d": 12, ...}

# Use these features in your prediction request
# prediction = endpoint.predict(instances=[feature_values])

Feature monitoring detects changes in feature distributions that could degrade model performance. It continuously compares current feature values against a baseline (typically training data distributions).

Drift Detection

Feature drift occurs when the statistical distribution of a feature changes over time. For example, if average transaction amounts increase due to inflation, a model trained on older data may underperform.

# Configure feature monitoring on a Feature View
# Monitoring checks feature distributions against a baseline

monitoring_config = feature_store.FeatureViewMonitoringConfig(
    snapshot_analysis=feature_store.SnapshotAnalysis(
        monitoring_interval_days=1,  # Check daily
    ),
    numerical_threshold_config=feature_store.NumericalThresholdConfig(
        value=0.3,  # Alert if Jensen-Shannon divergence > 0.3
    ),
    categorical_threshold_config=feature_store.CategoricalThresholdConfig(
        value=0.3,  # Alert if L-infinity distance > 0.3
    ),
)

# Alerts go to Cloud Monitoring, where you can configure
# notification channels (email, PagerDuty, Pub/Sub)

Staleness Alerts

Feature staleness occurs when feature values are not updated within the expected timeframe. If a daily feature pipeline fails, the Feature Store will serve yesterday's (or older) values, potentially degrading predictions. Staleness monitoring catches these failures.

Why It Matters

Point-in-time correctness is the most critical concept in feature engineering for ML. It means: when constructing training data, each example must use only features that were available at the time of the label. Using future data (data leakage) creates artificially high training metrics that do not generalize to production.

How Feature Store Handles It

The Feature Store's batch_serve method performs a point-in-time join: for each (entity_id, timestamp) pair in your labels table, it looks up the feature values that were current at or before that timestamp. This is also called an as-of join or temporal join.

# Point-in-time join: how it works
#
# Labels table:          Feature table (user_features):
# user_id | label_ts    | label     user_id | feature_ts  | txn_count_7d
# u001    | 2025-03-15  | fraud     u001    | 2025-03-10  | 5
# u001    | 2025-04-20  | legit     u001    | 2025-03-17  | 8
#                                   u001    | 2025-04-01  | 12
#                                   u001    | 2025-04-18  | 3
#
# Point-in-time result:
# user_id | label_ts    | label  | txn_count_7d
# u001    | 2025-03-15  | fraud  | 5     (used 03-10 value, NOT 03-17)
# u001    | 2025-04-20  | legit  | 3     (used 04-18 value, NOT future)
#
# Each row gets the feature value that was latest AT OR BEFORE label_ts

# In BigQuery SQL, this is an ASOF JOIN:
# SELECT l.*, f.txn_count_7d
# FROM labels l
# ASOF JOIN features f
#   ON l.user_id = f.user_id
#   AND f.feature_ts <= l.label_ts

Discovery and Reuse

Feature Store enables feature discovery — teams can browse existing features before building new ones. This eliminates duplicate work where multiple teams independently compute the same features (e.g., "user age" or "30-day transaction count") with slightly different definitions.

Governance

Feature governance includes access control (IAM policies on feature groups), lineage tracking (which models use which features), and documentation (descriptions and metadata on every feature). This ensures compliance, auditability, and organizational knowledge retention.

# List all available feature groups (discovery)
feature_groups = feature_store.FeatureGroup.list()

print(f"Available feature groups: {len(feature_groups)}")
for fg in feature_groups:
    features = fg.list_features()
    print(f"  {fg.name}: {len(features)} features - {fg.description}")
    for f in features:
        print(f"    - {f.name}: {f.description}")

Not every feature belongs in a Feature Store. Understanding when to use centralized features vs inline (computed at prediction time) is an important architectural decision.

Feature Store integrates with Vertex AI Pipelines at two key points in the ML lifecycle: (1) feature ingestion — pipeline steps that compute and write features, and (2) training data generation — pipeline steps that read features with point-in-time joins.

# Feature Store integration in a Vertex AI Pipeline
from kfp import dsl

@dsl.pipeline(name="fraud-pipeline-with-features")
def fraud_pipeline(project: str):
    # Step 1: Compute fresh features in BigQuery
    compute_features_op = bq_query_op(
        query="CALL features.compute_user_features()",
        project=project,
    )

    # Step 2: Generate training data from Feature Store
    # Point-in-time join: features + labels
    generate_training_data_op = feature_store_batch_serve_op(
        feature_view="fraud_detection_features",
        entity_source="bq://project.labels.fraud_labels",
        destination="bq://project.training.fraud_data",
    ).after(compute_features_op)

    # Step 3: Train model on feature-store-generated data
    train_op = custom_training_op(
        training_data=generate_training_data_op.outputs["training_data"],
    ).after(generate_training_data_op)

    # Step 4: Sync features to online store for serving
    sync_op = feature_view_sync_op(
        feature_view="fraud_detection_features",
        online_store="fraud-online-store",
    ).after(compute_features_op)

    # Step 5: Deploy model (uses online store at prediction time)
    deploy_op = model_deploy_op(
        model=train_op.outputs["model"],
    ).after(train_op, sync_op)  # Wait for both

Google has evolved the Feature Store architecture. The new architecture (BigQuery-backed) replaces the legacy architecture (Featurestore with EntityTypes). Understanding both is useful for the exam, as some older references still use legacy terminology.

This course maps to Section 2: Data Processing (feature engineering, data validation) and Section 4: ML Model Serving (online feature serving, training-serving consistency) on the GCP Machine Learning Engineer exam.

Key Exam Scenarios

Key Exam Takeaways

• Online serving = low-latency entity lookups for real-time predictions.
• Offline serving = batch feature retrieval with point-in-time joins for training.
• Point-in-time correctness prevents data leakage (using future data in training).
• Use Feature Store for shared, precomputed entity features. Use inline for request-level features.
• Feature monitoring detects drift (distribution changes) and staleness (ingestion failures).
• New architecture uses Feature Groups + Feature Views; legacy used Featurestore + EntityTypes.