Turning Raw Data into Powerful Features

Architecture

Vertex AI Feature Store is a centralized, managed repository for storing, organizing, and serving ML features. It solves several critical problems: feature reuse across teams, consistent feature values between training and serving, and point-in-time correctness for training data.

The architecture is organized around these concepts:

Featurestore: The top-level container (one per project/use-case). Contains all entity types and features.

Entity Type: A logical grouping of features about a specific entity (e.g., "user," "product," "transaction"). Each entity type has a unique ID column that identifies individual entities.

Feature: An individual measurable property within an entity type (e.g., "age," "total_purchases," "last_login_days_ago"). Features have a data type (INT64, FLOAT, STRING, BOOL, array types) and are versioned with timestamps.

Online serving provides low-latency feature lookups for real-time prediction. The Feature Store maintains an optimized online storage (backed by Bigtable) that returns feature values for a given entity ID in milliseconds. This is used when a prediction endpoint needs to fetch the latest features for a user at inference time.

Offline serving provides batch access to historical feature values for training. It performs point-in-time joins, ensuring that training examples use only features that were available at the time of each training label. This prevents data leakage from the future.

Operations: Create, Ingest, Serve, Share

The Feature Store workflow follows four steps:

1. Create: Define the featurestore, entity types, and features using the Vertex AI SDK or REST API. Specify feature names, types, and descriptions.

# Create Feature Store resources (requires GCP)
from google.cloud import aiplatform

aiplatform.init(project="my-project", location="us-central1")

# Create a featurestore
fs = aiplatform.Featurestore.create(
    featurestore_id="ecommerce_features",
    online_store_fixed_node_count=1
)

# Create an entity type
user_entity = fs.create_entity_type(
    entity_type_id="users",
    description="User-level features for churn prediction"
)

# Add features to the entity type
user_entity.batch_create_features(
    feature_configs={
        "age": {"value_type": "INT64"},
        "total_purchases": {"value_type": "DOUBLE"},
        "account_age_days": {"value_type": "INT64"},
        "region": {"value_type": "STRING"},
        "is_premium": {"value_type": "BOOL"},
    }
)

2. Ingest: Import feature values from BigQuery, GCS (CSV/Avro), or streaming sources. Ingestion is timestamp-aware — each feature value has an associated timestamp for point-in-time lookups.

3. Serve: Read features online (by entity ID for real-time) or offline (batch export with point-in-time joins for training datasets).

4. Share: Features registered in the Feature Store are discoverable by other teams via the Feature Registry. This promotes reuse and consistency across the organization.

Numerical Features

Numerical features often need transformation before they are useful for ML models. The key techniques:

Scaling (Min-Max): Maps values to [0, 1] range. Formula: (x - min) / (max - min). Use when you need bounded values and the distribution is roughly uniform. Sensitive to outliers.

Standardization (Z-score): Centers at 0 with unit variance. Formula: (x - mean) / std. Use for algorithms that assume Gaussian distributions (linear regression, SVMs). Less sensitive to outliers than min-max.

Log Transform: Compresses right-skewed distributions. Apply log(x + 1) for features like revenue, counts, or prices that span orders of magnitude.

Bucketizing / Binning: Converts continuous values into categorical bins. Use when the relationship between the feature and target is non-linear, or when you want to reduce noise. In BigQuery: ML.BUCKETIZE(age, [18, 25, 35, 50, 65]).

Categorical Features

One-Hot Encoding: Creates a binary column for each category. Works well for low-cardinality features (<50 categories). In BQML, this is automatic for STRING columns. Drawback: high-cardinality features create very sparse, wide vectors.

Label Encoding: Maps categories to integers (A=0, B=1, C=2). Simple but introduces an artificial ordinal relationship. Only use for tree-based models that can handle arbitrary splits.

Feature Hashing (Hashing Trick): Hashes category strings into a fixed number of buckets. Handles high-cardinality features without storing a vocabulary. Collisions are possible but acceptable for large bucket counts. In BQML: ML.HASH_BUCKETS(column, num_buckets).

Embeddings: Learn dense vector representations for categories. Best for very high cardinality (user IDs, product IDs). Requires a neural network to learn the embedding during training. Captures semantic similarity between categories.

Text and Temporal Features

Text features: For short text (product names, categories), use TF-IDF or bag-of-words. For longer text (reviews, descriptions), use pre-trained embeddings (BERT, Universal Sentence Encoder). In the GCP context, tf.Transform provides tft.compute_and_apply_vocabulary() and tft.tfidf() for scalable text featurization.

Temporal features: Extract components (day of week, hour, month, quarter). Create cyclical encodings to capture periodicity: encode hour as sin(2*pi*hour/24) and cos(2*pi*hour/24). This ensures hour 23 is close to hour 0. Create lag features (value at t-1, t-7) for time-series. Create rolling aggregates (7-day average, 30-day max).

Avoiding Data Leakage

Data leakage occurs when information from the target variable or from the future leaks into the training features. This causes artificially high training performance that does not generalize to production. Common sources:

Target leakage: A feature that is directly derived from or highly correlated with the label. Example: using "cancellation_date" as a feature to predict churn — the cancellation date IS the churn event.

Temporal leakage: Using future data to predict the past. Example: using a user's total lifetime purchases to predict whether they will make their first purchase. Always ensure features only use data available at prediction time.

Pre-processing leakage: Computing statistics (mean, std) on the full dataset including the test set, then using those statistics for scaling. Always fit preprocessing on the training set only, then transform both train and test.

Training-Serving Skew

Training-serving skew occurs when the feature engineering code used during training differs from what runs during prediction. This is one of the most insidious bugs in ML systems because the model will silently produce degraded results.

Common causes: different programming languages (Python training vs Java serving), different library versions, manual copy-paste of transformation logic, or different data access patterns (offline batch vs online lookup).

Solutions on GCP:

• tf.Transform: Define preprocessing as a TensorFlow graph that is saved with the model and applied identically at training and serving time
• BQML TRANSFORM clause: Feature engineering is part of the model definition, automatically applied during ML.PREDICT
• Feature Store: Centralized source of truth for feature values, same store serves training and prediction
• Vertex AI Pipelines: Orchestrate identical preprocessing steps for training and batch prediction

What They Are and When to Use Them

A feature cross is a synthetic feature formed by combining two or more existing features. For categorical features, a cross creates a new feature whose values are the Cartesian product of the original features' values. For example, crossing city (3 values) with device (2 values) creates 6 new combined values like "NYC_mobile," "NYC_desktop," "SF_mobile," etc.

Feature crosses are powerful because they allow linear models to learn non-linear patterns. Without crosses, a linear model can only learn that "being in NYC" has one effect and "using mobile" has another effect, independently. With a cross, the model can learn that the combination "NYC + mobile" has a unique effect that is different from the sum of the individual effects.

When to use feature crosses:

• When the interaction between features matters more than individual features
• When using linear models (logistic regression, linear regression) that cannot learn interactions natively
• When combining location + time, user segment + product category, or similar interaction pairs
• When features are already high-cardinality (crosses multiply cardinality — use hashing to manage)
• When using tree-based models (they learn interactions automatically through splits)

Examples

-- Feature cross in BQML
CREATE MODEL `project.dataset.model`
TRANSFORM(
  -- Cross two categorical features
  ML.FEATURE_CROSS(STRUCT(city, device_type)) AS city_device,
  -- Cross with hashing to limit dimensionality
  ML.FEATURE_CROSS(STRUCT(
    ML.HASH_BUCKETS(city, 10),
    ML.HASH_BUCKETS(zipcode, 50)
  )) AS city_zip_cross,
  -- Cross bucketed numerical with categorical
  ML.FEATURE_CROSS(STRUCT(
    ML.BUCKETIZE(age, [18, 30, 45, 60]),
    membership_tier
  )) AS age_tier_cross,
  label
)
OPTIONS(model_type='LOGISTIC_REG', input_label_cols=['label'])
AS SELECT * FROM `project.dataset.features`;

The Analyze-and-Transform Pattern

tf.Transform is a library for preprocessing data at scale. Its key innovation is the analyze-and-transform pattern that separates full-pass computations (like computing the mean of a column across the entire dataset) from per-example transformations (like subtracting that mean from each value).

The central concept is the preprocessing_fn — a pure function that defines all feature transformations. Inside this function, you use two types of operations:

Analyzers (full-pass): Functions like tft.mean(), tft.min(), tft.max(), tft.vocabulary() that need to see the entire dataset to compute a result. These run once during training as a Beam pipeline over the full dataset.

Mappers (per-example): Functions like tft.scale_to_0_1(), tft.apply_vocabulary() that transform individual examples using the values computed by analyzers. These become part of the saved TensorFlow graph.

The key benefit: the preprocessing function is exported as a TensorFlow graph that is attached to the model. At serving time, the same graph is applied automatically, guaranteeing identical preprocessing. No Python code runs at serving time — it is all graph ops. This eliminates training-serving skew.

import tensorflow_transform as tft

def preprocessing_fn(inputs):
    """Define feature transformations."""
    outputs = {}

    # Numerical: scale to [0, 1] using full-dataset min/max
    outputs['age_scaled'] = tft.scale_to_0_1(inputs['age'])

    # Numerical: z-score normalize
    outputs['income_normalized'] = tft.scale_to_z_score(inputs['income'])

    # Numerical: bucketize into quantile bins
    outputs['age_bucket'] = tft.bucketize(inputs['age'], num_buckets=5)

    # Categorical: compute vocabulary and map to integers
    outputs['city_index'] = tft.compute_and_apply_vocabulary(inputs['city'])

    # Categorical: hash to fixed buckets (high cardinality)
    outputs['zipcode_hash'] = tft.hash_strings(inputs['zipcode'], hash_buckets=100)

    # Text: compute TF-IDF
    outputs['description_tfidf'] = tft.tfidf(
        tft.compute_and_apply_vocabulary(inputs['description']),
        vocab_size=10000
    )

    # Pass through the label
    outputs['label'] = inputs['label']

    return outputs

Dataflow Integration

In production, tf.Transform runs on Apache Beam / Dataflow. The workflow is:

1. Define the preprocessing_fn with analyzers and mappers.

2. Run tft_beam.AnalyzeAndTransformDataset on the training data. This executes analyzers as Beam operations (distributed across Dataflow workers) and produces: (a) the transformed training data, and (b) a transform function (saved TF graph with computed constants like mean, vocabulary).

3. Use tft_beam.TransformDataset to apply the saved transform to eval/test data (using the constants from step 2, not recomputing them).

4. Attach the transform function to the serving model so identical preprocessing runs at inference time.

Choosing the right preprocessing approach depends on scale, model framework, and operational requirements. Here is the decision framework:

Methods

Feature selection reduces dimensionality, speeds up training, improves generalization, and makes models more interpretable. Three families of methods:

Filter methods: Use statistical tests to score features independently of the model. Fast but ignore feature interactions. Examples: correlation with target, chi-squared test (categorical features + categorical target), mutual information (measures any dependency, not just linear), ANOVA F-test (numerical features + categorical target).

Wrapper methods: Evaluate feature subsets by training models. Expensive but considers interactions. Examples: forward selection (start empty, add best feature), backward elimination (start full, remove worst), recursive feature elimination (RFE — train, remove least important, repeat).

Embedded methods: Feature selection happens during model training. The model itself learns which features matter. Examples: L1 regularization (Lasso — drives unimportant feature weights to zero), tree-based feature importance (how much each feature reduces impurity across all splits).

Feature Importance

Tree-based importance: Random forests and gradient-boosted trees compute feature importance as the total reduction in impurity (Gini or entropy) across all trees. Available in sklearn as model.feature_importances_. In BQML, use ML.FEATURE_IMPORTANCE(MODEL model_name).

Permutation importance: Shuffle one feature at a time and measure how much model performance degrades. Works with any model. More robust than tree-based importance for correlated features.

Correlation analysis: Compute pairwise correlations between features. Highly correlated features (>0.9) are redundant — drop one. Use Pearson for linear relationships, Spearman for monotonic relationships.

# Feature importance with sklearn
from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_selection import mutual_info_classif

# Tree-based importance
rf = RandomForestClassifier(n_estimators=100, random_state=42)
rf.fit(X_train, y_train)
importances = pd.Series(rf.feature_importances_, index=feature_names)
print(importances.sort_values(ascending=False))

# Mutual information (filter method)
mi_scores = mutual_info_classif(X_train, y_train)
mi_series = pd.Series(mi_scores, index=feature_names)
print(mi_series.sort_values(ascending=False))

High-yield topics for exam Sections 2 and 3 relating to feature engineering.

How to Explain This in 2 Minutes

Likely Interview Questions

Model Answers

System Design Scenario

Common Mistakes

• Fitting preprocessing on the full dataset including test data — Computing normalization statistics (mean, std) on the entire dataset before splitting leaks test-set information into training. Always fit transformations on the training split only. tf.Transform analyzers should run on the training partition. This is the most common source of inflated evaluation metrics that collapse in production.
• Using feature crosses with tree-based models — Decision trees and gradient-boosted models (XGBoost, LightGBM) learn feature interactions naturally through splits. Adding explicit feature crosses to these models adds dimensionality without improving performance. Feature crosses are primarily valuable for linear and logistic regression models that cannot learn interactions on their own.
• Ignoring cyclical encoding for time features — Extracting hour_of_day as an integer (0-23) implies that hour 23 and hour 0 are maximally distant, when they are actually adjacent. The correct approach is cyclical encoding: sin(2π * hour/24) and cos(2π * hour/24), which preserves the circular relationship. The same applies to day-of-week and month-of-year features.