MLOps for Generative AI

Traditional MLOps focuses on training pipelines: ingest data, engineer features, train a model, validate metrics, deploy an endpoint, and monitor for data drift. GenAI MLOps adds entirely new dimensions that classical pipelines never anticipated.

The GenAI lifecycle is not a simple train-deploy-monitor loop. It is an iterative process with multiple decision points and feedback loops:

Stage-by-Stage Decisions

Model Selection: Choose between Gemini models (Pro, Flash, Ultra), open-source models on Model Garden, or third-party models. Consider cost, latency, task complexity, and compliance requirements. Vertex AI Model Garden provides 150+ foundation models.

Prompt Engineering: Develop system instructions, few-shot examples, and output formatting. Use Vertex AI Studio for rapid experimentation. Version all prompts in source control.

Fine-Tuning: When prompting alone is insufficient. Vertex AI supports supervised fine-tuning (SFT), RLHF, and distillation. Decision: fine-tune only when prompt engineering + RAG cannot achieve required quality.

Evaluation: Automated metrics (BLEU, ROUGE), LLM-as-judge, human evaluation, and domain-specific rubrics. Vertex AI Gen AI Evaluation Service provides built-in evaluation pipelines.

Deployment: Direct API calls (Vertex AI endpoints), cached responses for common queries, distilled models for cost optimization. Use traffic splitting for A/B testing.

Monitoring: Output quality drift, safety violations, hallucination rate, cost tracking, and latency monitoring. Set up alerts for quality degradation.

Prompts in production systems are not ad-hoc strings. They are versioned artifacts that must be managed with the same discipline as code. A prompt management system tracks prompt templates, variables, model configurations, and performance metrics.

Prompt Versioning & Registries

A prompt registry is a centralized store for all production prompts. Each entry includes the prompt template, the model it was tested with, evaluation scores, and metadata. Think of it as a model registry but for prompts.

# Prompt versioning pattern
PROMPT_REGISTRY = {
    "summarize_v1": {
        "template": "Summarize the following document in {num_sentences} sentences:\n\n{document}",
        "model": "gemini-2.0-flash",
        "temperature": 0.3,
        "version": "1.0.0",
        "eval_score": 0.87,
        "created": "2025-01-15",
    },
    "summarize_v2": {
        "template": "You are a technical writer. Create a {num_sentences}-sentence summary...\n\n{document}",
        "model": "gemini-2.0-flash",
        "temperature": 0.2,
        "version": "2.0.0",
        "eval_score": 0.92,
        "created": "2025-02-20",
    }
}

A/B Testing Prompts

In production, you can route a percentage of traffic to different prompt versions. Vertex AI endpoints support traffic splitting natively. For prompt-level A/B testing:

• Define a control prompt (current production version)
• Define a treatment prompt (the new version)
• Route 90/10 traffic split using endpoint config or application-level routing
• Collect quality metrics: user feedback, automated eval scores, latency
• Run for statistically significant sample size
• Promote the winner, archive the loser with full metadata

Dataset Preparation

Fine-tuning Gemini models on Vertex AI requires training data in JSONL format. Each line contains an input-output pair. Quality matters far more than quantity — 100 high-quality examples often outperform 10,000 noisy ones.

# Training data format for supervised fine-tuning
# Each line in the JSONL file:
{
  "messages": [
    {"role": "system", "content": "You are a medical coding assistant."},
    {"role": "user", "content": "Patient presents with acute bronchitis..."},
    {"role": "model", "content": "ICD-10: J20.9 - Acute bronchitis, unspecified"}
  ]
}

Supervised Fine-Tuning on Vertex AI

Vertex AI provides a managed fine-tuning service. You upload training data to Cloud Storage, specify the base model and hyperparameters, and Vertex AI handles the infrastructure. The tuned model is deployed as a new endpoint.

from google.cloud import aiplatform
from vertexai.tuning import sft

# Initialize Vertex AI
aiplatform.init(project="my-project", location="us-central1")

# Launch supervised fine-tuning job
tuning_job = sft.train(
    source_model="gemini-2.0-flash-001",
    train_dataset="gs://my-bucket/train.jsonl",
    validation_dataset="gs://my-bucket/val.jsonl",
    epochs=3,
    adapter_size=4,          # LoRA rank
    learning_rate_multiplier=1.0,
    tuned_model_display_name="medical-coder-v1",
)

# Monitor tuning progress
print(tuning_job.state)        # PIPELINE_STATE_RUNNING
print(tuning_job.tuned_model)  # Endpoint resource name

RLHF Pipelines

Reinforcement Learning from Human Feedback (RLHF) adds a second fine-tuning stage where a reward model learns human preferences. On Vertex AI, RLHF tuning involves:

• Step 1: SFT on instruction-following data
• Step 2: Collect human preference data (pairwise comparisons)
• Step 3: Train a reward model on preference data
• Step 4: Use PPO/DPO to align the model with the reward model

Evaluating generative AI is fundamentally different from evaluating classifiers. There is no single ground truth for open-ended text generation. GenAI evaluation uses a combination of automated metrics, LLM-based judging, and human evaluation.

LLM-as-Judge: Pointwise & Pairwise

Pointwise evaluation scores a single response on a rubric (e.g., 1-5 scale for relevance). Pairwise evaluation compares two responses and selects the better one. Pairwise is more reliable because humans (and LLMs) are better at comparisons than absolute ratings.

# LLM-as-Judge: Pointwise evaluation
from vertexai.generative_models import GenerativeModel

judge = GenerativeModel("gemini-2.0-pro")

JUDGE_PROMPT = """Rate the following response on a scale of 1-5 for:
- Relevance: Does it answer the question?
- Accuracy: Is the information correct?
- Completeness: Does it cover all aspects?

Question: {question}
Response: {response}

Output JSON: {"relevance": X, "accuracy": X, "completeness": X, "reasoning": "..."}"""

result = judge.generate_content(
    JUDGE_PROMPT.format(question=q, response=r)
)

RAGAS for RAG Evaluation

RAGAS (Retrieval-Augmented Generation Assessment) provides four key metrics for evaluating RAG pipelines:

Foundation models add new governance challenges. You must track not just model weights, but the entire configuration stack: base model version, fine-tuning data, prompt templates, retrieval configuration, and safety filters.

What to Version

• Base model: e.g., gemini-2.0-flash-001 (the specific version tag)
• Prompt templates: System instructions, few-shot examples, output schemas
• Fine-tuning artifacts: Training data hash, hyperparameters, adapter weights
• RAG configuration: Embedding model, chunk size, overlap, retrieval top-k
• Safety settings: Content filter thresholds, blocked categories
• Generation config: Temperature, top-p, top-k, max output tokens

Use Vertex AI Model Registry to register and track model versions. Each deployment should be a tagged combination of all the above. Use Git tags or semantic versioning for prompt+config bundles.

GenAI deployment is more varied than traditional ML deployment. The right pattern depends on latency requirements, cost constraints, and quality needs.

# Context caching for repeated queries (Vertex AI)
from vertexai.generative_models import GenerativeModel
from vertexai import caching

# Create a cached content object for large context
cached_content = caching.CachedContent.create(
    model_name="gemini-2.0-flash-001",
    contents=[large_document],
    ttl=datetime.timedelta(hours=1),
    display_name="product-manual-cache",
)

# Use cached content for multiple queries (saves input tokens)
model = GenerativeModel.from_cached_content(cached_content)
response = model.generate_content("What is the return policy?")

GenAI monitoring extends far beyond traditional ML monitoring. You cannot simply track prediction drift on a single metric. GenAI monitoring requires a multi-dimensional approach.

Setting Up Cost Tracking

# Cost tracking pattern for GenAI API calls
import time

class GenAICostTracker:
    def __init__(self):
        self.total_input_tokens = 0
        self.total_output_tokens = 0
        self.total_requests = 0

    def log_request(self, response):
        usage = response.usage_metadata
        self.total_input_tokens += usage.prompt_token_count
        self.total_output_tokens += usage.candidates_token_count
        self.total_requests += 1

    def estimate_cost(self, input_price_per_1k=0.000125, output_price_per_1k=0.000375):
        input_cost = (self.total_input_tokens / 1000) * input_price_per_1k
        output_cost = (self.total_output_tokens / 1000) * output_price_per_1k
        return {"input_cost": input_cost, "output_cost": output_cost,
                "total_cost": input_cost + output_cost}

RAG (Retrieval-Augmented Generation) pipelines have their own operational concerns that go beyond simple model deployment. Managing a RAG system in production requires continuous maintenance of the chunking pipeline, embedding model, and vector index.

Chunking Pipeline Operations

Documents must be split into chunks for embedding and retrieval. The chunking strategy directly impacts retrieval quality. Operational concerns include:

• Chunk size tuning: 256-1024 tokens per chunk, with overlap of 10-20%
• Incremental updates: When documents change, re-chunk and re-embed only the affected sections
• Metadata enrichment: Attach source, date, section headers to each chunk for filtering
• Deduplication: Remove near-duplicate chunks to improve retrieval precision

Embedding Updates & Index Management

When you update the embedding model (e.g., from text-embedding-004 to a newer version), you must re-embed all documents. This is a major operational task:

# Vertex AI Vector Search index management
from google.cloud import aiplatform

# Create a new index for updated embeddings
index = aiplatform.MatchingEngineIndex.create_tree_ah_index(
    display_name="product-docs-v2",
    dimensions=768,
    approximate_neighbors_count=150,
    distance_measure_type="DOT_PRODUCT_DISTANCE",
    shard_size="SHARD_SIZE_SMALL",
)

# Deploy index to an endpoint for real-time queries
index_endpoint = aiplatform.MatchingEngineIndexEndpoint.create(
    display_name="product-docs-endpoint",
    public_endpoint_enabled=True,
)
index_endpoint.deploy_index(
    index=index, deployed_index_id="prod_v2",
)

GenAI-Specific Monitoring

• Know that GenAI monitoring includes: output quality, safety, cost, latency, and hallucination rate
• Understand that traditional data drift detection does not directly apply to free-text inputs
• LLM-as-judge is the scalable approach for automated quality monitoring
• Cost monitoring is unique to GenAI because of per-token pricing variability

Fine-Tuning vs RAG Decision Framework

Key Exam Signals

• If the question mentions "latest data" or "real-time information" → RAG
• If the question mentions "specific format" or "consistent style" → Fine-tuning
• If the question mentions "monitoring output quality" → Continuous evaluation + LLM-as-judge
• If the question mentions "cost optimization" → Context caching, model distillation, or routing to Flash
• If the question mentions "prompt management" → Version control, prompt registry, A/B testing

How to Explain This in 2 Minutes

Likely Interview Questions

Model Answers

GenAI MLOps vs Traditional: Traditional MLOps manages code, data, and model weights through CI/CD pipelines. GenAI MLOps adds prompts as versioned artifacts, requires evaluation beyond numeric metrics (using LLM-as-judge and human preference), manages adapter weights from fine-tuning separately from base models, and must handle RAG pipeline configurations (chunk size, embedding model, retrieval strategy) as additional deployable artifacts. The feedback loop also changes—instead of periodic retraining, you iterate on prompts, update retrieval corpora, or fine-tune adapters on a much faster cadence.

Prompt Management: I would implement a prompt registry where each prompt template is versioned with its system instructions, few-shot examples, and output parsing logic. Changes go through code review. In production, I use traffic splitting to A/B test prompt variants, measuring quality via automated LLM-as-judge scoring and business metrics. Rollback is instant because switching prompts doesn’t require model redeployment. On Vertex AI, this integrates with the Gen AI Evaluation Service for automated quality gating before promoting a prompt to production.

Fine-Tuning vs RAG Decision: If the user needs up-to-date factual information or domain-specific knowledge, I use RAG—it avoids retraining and lets me update the knowledge base in real time. If the requirement is consistent output style, specific formatting, or behavioral alignment, I use supervised fine-tuning. For complex production systems, I combine both: fine-tune the model for tone and format, then use RAG to ground responses in current data. The key signal is whether the gap is in what the model knows versus how it communicates.

System Design Scenario

Common Mistakes

• Treating prompts as configuration, not artifacts — Prompts in production need the same rigor as code: version control, testing, staged rollout, and rollback capability. Hardcoding prompts in application code makes iteration slow and error-prone.
• Relying solely on automated metrics for GenAI evaluation — BLEU and ROUGE measure surface overlap, not quality. Production GenAI systems need LLM-based evaluation for nuance and periodic human evaluation to calibrate the automated judges.
• Fine-tuning when RAG would suffice — Fine-tuning is expensive, creates model management overhead, and bakes knowledge into weights that become stale. Default to RAG for knowledge augmentation and reserve fine-tuning for behavioral changes that prompting alone cannot achieve.