MLOps with Vertex AI: Model Evaluation

Model evaluation answers three critical questions: Is this model good enough for production? Is it better than what we have? Is it fair and safe? Evaluation happens at multiple stages of the ML lifecycle:

Classification is the most common ML task on the exam. You must know when to use each metric and the tradeoffs between them.

The Confusion Matrix

Every classification metric derives from the confusion matrix. For binary classification:

Core Classification Metrics

Accuracy is the simplest metric but misleading for imbalanced datasets. If 99% of emails are not spam, a model that always predicts "not spam" achieves 99% accuracy but catches zero spam.

Precision matters when false positives are costly. Example: spam filter (you don't want to send real emails to spam), medical diagnosis (false positive means unnecessary treatment).

Recall matters when false negatives are costly. Example: fraud detection (missing a fraud case is expensive), cancer screening (missing a cancer case is dangerous).

F1 Score is the harmonic mean of precision and recall. Use it when you need to balance both. It is the default metric for imbalanced classification problems.

ROC-AUC and PR-AUC

ROC-AUC (Receiver Operating Characteristic - Area Under Curve) plots True Positive Rate vs False Positive Rate at all classification thresholds. AUC = 1.0 is perfect, AUC = 0.5 is random.

PR-AUC (Precision-Recall AUC) plots Precision vs Recall. More informative than ROC-AUC for highly imbalanced datasets where the positive class is rare (fraud, disease).

Mean Absolute Error (MAE): Average absolute difference between predictions and actual values. Robust to outliers. Easy to interpret (same units as target).

Mean Squared Error (MSE): Average squared difference. Penalizes large errors more heavily than MAE. Units are squared (harder to interpret).

Root Mean Squared Error (RMSE): Square root of MSE. Same units as target, but still penalizes large errors. The most commonly used regression metric.

R-Squared (Coefficient of Determination): Proportion of variance explained by the model. R² = 1.0 is perfect, R² = 0 means the model is no better than predicting the mean. Can be negative if the model is worse than the mean.

Mean Absolute Percentage Error (MAPE): Error as a percentage. Easy for stakeholders to understand ("our predictions are off by 5%"). Undefined when actual value is zero.

Evaluating generative AI requires fundamentally different approaches than classical ML. There is no single ground truth for open-ended text generation, and traditional metrics like accuracy do not apply.

Pointwise Evaluation

Pointwise evaluation scores a single model response on a rubric. A human or LLM judge rates the response on criteria like relevance, accuracy, completeness, and coherence, typically on a 1-5 scale.

# Pointwise evaluation criteria
criteria = {
    "relevance":    "Does the response address the question?",
    "accuracy":     "Is the information factually correct?",
    "completeness": "Does it cover all important aspects?",
    "coherence":    "Is the text well-organized and logical?",
    "safety":       "Is the response free from harmful content?",
}
# Each criterion scored 1-5, aggregate for overall quality

Pairwise Evaluation

Pairwise evaluation compares two responses and determines which is better. This is more reliable than pointwise scoring because humans and LLMs are better at relative comparisons than absolute ratings. Used heavily in RLHF for collecting preference data.

Automated Metrics: BLEU, ROUGE, Perplexity

LLM-Based Evaluation

LLM-based evaluation uses a strong model (e.g., Gemini Pro) to judge outputs from any model. Key dimensions evaluated:

Vertex AI provides built-in evaluation tools for both classical ML and generative AI models.

AutoML Model Evaluation

When you train an AutoML model, Vertex AI automatically generates evaluation metrics on the test set. These include confusion matrix, ROC curve, precision-recall curve, and feature importance for classification; MAE, RMSE, R² for regression.

# Access AutoML model evaluation via SDK
from google.cloud import aiplatform

model = aiplatform.Model("projects/my-proj/locations/us-central1/models/12345")

# Get all evaluation slices
evaluations = model.list_model_evaluations()
for eval in evaluations:
    print(f"Evaluation: {eval.display_name}")
    print(f"  Metrics: {eval.metrics}")
    print(f"  Slices: {eval.model_evaluation_slices}")

Gen AI Evaluation Service

The Vertex AI Gen AI Evaluation Service provides automated evaluation for generative AI models. It supports:

• Pointwise metrics: Fluency, coherence, safety, groundedness
• Pairwise comparison: Compare outputs from two models
• Task-specific: Summarization quality, QA accuracy, text generation
• Custom rubrics: Define your own evaluation criteria
• Batch evaluation: Evaluate hundreds of examples in parallel

# Vertex AI Gen AI Evaluation
from vertexai.evaluation import EvalTask, PointwiseMetric

eval_task = EvalTask(
    dataset=eval_dataset,      # List of {prompt, reference, response}
    metrics=[
        "fluency",
        "coherence",
        "safety",
        "groundedness",
        PointwiseMetric(
            metric="custom_quality",
            metric_prompt_template="Rate quality 1-5: {response}",
        ),
    ],
)

result = eval_task.evaluate(
    model=GenerativeModel("gemini-2.0-flash"),
)
print(result.summary_metrics)
print(result.metrics_table)  # Per-example scores

The quality of your evaluation is only as good as your evaluation dataset. A golden dataset (or gold standard dataset) is a curated collection of inputs and expected outputs used as the ground truth for evaluation.

Creating Golden Datasets

• Representative: Cover all important use cases and edge cases
• Diverse: Include examples from all user segments, languages, topics
• Annotated: Human-verified expected outputs (not model-generated)
• Versioned: Track changes to evaluation data over time
• Size: 100-500 examples minimum for meaningful evaluation

Avoiding Contamination

Models degrade over time as data distributions shift. Continuous evaluation monitors deployed model performance by periodically comparing predictions against ground truth labels that arrive with a delay.

Vertex AI Continuous Evaluation

Vertex AI supports continuous evaluation for deployed models. It samples predictions from the serving endpoint, collects ground truth labels (when available), and computes metrics over time windows.

A model can have excellent aggregate metrics but perform poorly for specific subgroups. Slice-based evaluation breaks down metrics by demographic or business-relevant segments.

Slice-Based Evaluation

Instead of looking only at overall accuracy, compute metrics for each slice of your data:

# Slice-based evaluation example
slices = {
    "gender": ["male", "female", "non-binary"],
    "age_group": ["18-30", "31-50", "51+"],
    "region": ["US", "EU", "Asia"],
}

for slice_name, values in slices.items():
    for value in values:
        subset = test_data[test_data[slice_name] == value]
        metrics = compute_metrics(subset.predictions, subset.labels)
        print(f"{slice_name}={value}: F1={metrics['f1']:.3f}, Recall={metrics['recall']:.3f}")

Fairness Metrics

A/B testing compares two model versions in production by splitting live traffic and measuring real-world performance. Unlike offline evaluation, A/B testing captures user behavior and business metrics.

Setting Up A/B Tests on Vertex AI

Vertex AI endpoints support traffic splitting natively. Deploy two model versions to the same endpoint and configure the traffic split:

# Deploy two models with traffic split
endpoint = aiplatform.Endpoint("projects/.../endpoints/12345")

# Champion: current model (90% traffic)
endpoint.deploy(
    model=champion_model,
    deployed_model_display_name="champion-v2",
    traffic_percentage=90,
    machine_type="n1-standard-4",
)

# Challenger: new model (10% traffic)
endpoint.deploy(
    model=challenger_model,
    deployed_model_display_name="challenger-v3",
    traffic_percentage=10,
    machine_type="n1-standard-4",
)

Statistical Significance

Do not declare a winner until you reach statistical significance. Key concepts:

• Sample size: Calculate minimum sample size before starting the test
• p-value: Probability the observed difference is due to chance (p < 0.05 typical threshold)
• Effect size: How large is the practical difference between models?
• Duration: Run long enough to capture weekly/seasonal patterns
• Multiple comparisons: If testing many metrics, apply Bonferroni correction

Choosing the Right Metric

GenAI Evaluation Methods

• Know the difference between pointwise (absolute scoring) and pairwise (comparative) evaluation
• BLEU/ROUGE are fast but limited; use LLM-based evaluation for quality
• Vertex AI Gen AI Evaluation Service supports both built-in and custom metrics
• For RAG: evaluate faithfulness (grounded in context) and relevance (answers the question)

When to Retrain

• Continuous evaluation metrics drop below threshold
• Data drift or concept drift detected
• Significant new training data is available
• Business requirements or target distribution changes
• Scheduled periodic retraining (weekly/monthly depending on domain)

How to Explain This in 2 Minutes

Likely Interview Questions

Model Answers

Choosing the Right Metric: I start by understanding the business cost structure. If a false positive is expensive (flagging a legitimate transaction as fraud causes customer friction), I optimize for precision. If a false negative is dangerous (missing a cancer diagnosis), I optimize for recall. For balanced datasets, accuracy and ROC-AUC work well. For imbalanced datasets (99% negative), I use F1-score and PR-AUC because accuracy is misleading—a model predicting all negatives gets 99% accuracy but is useless. For regression, I use RMSE when large errors matter disproportionately and MAE when I want robustness to outliers.

GenAI Evaluation: BLEU and ROUGE measure n-gram overlap with reference text, which is useful for translation and summarization but misses semantic quality. For production GenAI, I use LLM-as-judge: a strong model (like Gemini Pro) scores outputs on dimensions like helpfulness, accuracy, and safety. Pointwise evaluation assigns absolute scores to individual responses; pairwise evaluation compares two model outputs to determine which is better. On Vertex AI, the Gen AI Evaluation Service automates this with configurable rubrics and supports both built-in and custom evaluation metrics.

A/B Testing Models: I calculate the minimum sample size based on the expected effect size and desired statistical power (typically 80%) before starting. Traffic is split randomly at the user level (not request level) to avoid contamination. I run the test for at least one full business cycle to capture temporal patterns. I never peek at results early because repeated significance testing inflates the false positive rate. When comparing multiple variants, I apply Bonferroni correction. The primary metric is a business KPI, not an ML metric—click-through rate or revenue per user, not just model accuracy.

System Design Scenario

Common Mistakes

• Using accuracy on imbalanced datasets — A fraud detection model that predicts “not fraud” for every transaction achieves 99.9% accuracy but catches zero fraud. Always use F1, PR-AUC, or domain-specific metrics when classes are imbalanced.
• Stopping A/B tests early based on promising results — The peeking problem inflates false positive rates. Pre-commit to a sample size and test duration. If you must check early, use sequential testing methods that control for multiple looks.
• Evaluating GenAI with only automated metrics — BLEU and ROUGE have low correlation with human judgment for open-ended generation. Use LLM-as-judge for scalable quality assessment and periodic human evaluation to calibrate the automated system.