Build Generative AI Applications on Google Cloud

Generative AI applications on Google Cloud follow a set of repeatable architectural patterns. Understanding these patterns is critical for both the exam and real-world implementation. Each pattern trades off complexity for capability.

Effective prompting is the foundation of every GenAI application. The way you structure your prompt directly impacts the quality, consistency, and safety of the model's output.

System Instructions

System instructions define the model's persona, constraints, and output format. They persist across all turns in a conversation. In Vertex AI, system instructions are set when initializing the GenerativeModel object and are separate from user messages.

# System instruction example
model = GenerativeModel(
    "gemini-1.5-pro",
    system_instruction="""You are a GCP architecture advisor.
    Always respond with specific GCP services.
    Format responses as bullet points.
    Never recommend non-GCP alternatives."""
)

Zero-Shot Prompting

No examples provided. The model relies on pre-trained knowledge. Best for straightforward tasks where the expected output format is obvious.

# Zero-shot: classify without examples
prompt = """Classify this support ticket as BILLING, TECHNICAL, or GENERAL.

Ticket: "My Cloud Function keeps timing out after 60 seconds."
Category:"""

Few-Shot Prompting

Provide 2-5 examples to demonstrate the desired input/output pattern. Dramatically improves consistency for classification, extraction, and formatting tasks.

# Few-shot: guide output format with examples
prompt = """Extract the GCP service from each sentence.

Sentence: "We store our data in BigQuery."
Service: BigQuery

Sentence: "The model is deployed on Vertex AI."
Service: Vertex AI

Sentence: "We use Cloud Run to host our API."
Service:"""

Chain-of-Thought (CoT)

Ask the model to "think step by step" before answering. Significantly improves accuracy on reasoning, math, and multi-step logic problems. Can be combined with few-shot examples that include reasoning traces.

Output Formatting

Specify the exact output structure you need: JSON, CSV, markdown tables, or structured text. Use explicit format instructions and delimiters to get parseable responses.

# Force JSON output
prompt = """Analyze this error log and return a JSON object with keys:
- "service": the GCP service involved
- "error_type": classification of the error
- "severity": HIGH, MEDIUM, or LOW
- "recommendation": one-sentence fix

Error log: Cloud Function 'processOrder' OOM killed at 256MB.

Return ONLY valid JSON, no explanation."""

Grounding

Grounding connects the model to external data sources so it generates responses based on facts rather than training data alone. On Vertex AI, grounding can use Google Search or your own data stores as sources.

Grounding is distinct from RAG in that it is a built-in platform feature — you configure it at the API call level without building a retrieval pipeline. The model automatically searches and cites sources.

Temperature Tuning

Temperature controls randomness. For factual/analytical tasks, use temperature=0.0 to 0.3. For creative tasks like marketing copy, use 0.7 to 1.0. Higher temperatures increase diversity but reduce reliability.

Safety Settings

Vertex AI provides configurable safety filters across four harm categories: hate speech, dangerous content, sexually explicit content, and harassment. Each category can be set to BLOCK_NONE, BLOCK_ONLY_HIGH, BLOCK_MEDIUM_AND_ABOVE, or BLOCK_LOW_AND_ABOVE. For enterprise applications, use at least medium-level filtering.

Why RAG?

Large language models have a knowledge cutoff and can hallucinate facts. RAG solves both problems by retrieving relevant documents from your own data sources and injecting them into the prompt context before generation.

RAG Architecture: The Four-Step Pipeline

Every RAG system follows the same fundamental pipeline, regardless of implementation:

Step 1 — Embed: Convert documents into dense vector representations using an embedding model (e.g., textembedding-gecko or text-embedding-004). Chunk documents into 500-1000 token segments with overlap.

Step 2 — Store: Persist embeddings in a vector database. On GCP, options include Vertex AI Vector Search (managed, high-scale), AlloyDB with pgvector, Cloud SQL with pgvector, or Spanner with vector capabilities.

Step 3 — Retrieve: At query time, embed the user's question and perform approximate nearest neighbor (ANN) search to find the top-k most relevant document chunks.

Step 4 — Generate: Inject the retrieved chunks into the prompt as context and send to Gemini for answer generation. The model synthesizes a response using only the provided context.

Vertex AI RAG Tools

Vertex AI Agent Builder (RAG mode) provides a low-code/no-code RAG pipeline. Upload documents to a data store, and Agent Builder handles chunking, embedding, indexing, and retrieval automatically. Best for teams without ML engineering resources.

Custom RAG with Vertex AI Vector Search gives you full control over every stage. You define the chunking strategy, choose the embedding model, configure the ANN index parameters (tree depth, leaf size), and build your own retrieval logic. Best for production systems requiring fine-tuned relevance.

Vertex AI Agent Builder (formerly Dialogflow CX + Enterprise Search) is Google's managed platform for building search, conversational, and generative AI applications without writing ML code.

Data Stores

Data stores are the knowledge base for your application. You can ingest data from: Cloud Storage (PDFs, HTML, text), BigQuery tables, websites (crawled), or the Vertex AI API for structured data. Agent Builder automatically processes, chunks, and indexes the content.

Search Apps

Build enterprise-grade search experiences over your data stores. Features include: natural language queries, auto-generated summaries with citations, faceted filtering, and relevance tuning. Deploy as a widget or API endpoint.

Conversation Agents

Create multi-turn conversational agents grounded in your data. Agents can: answer questions from data stores, follow conversation flows (intents, pages), call external tools via webhooks, and hand off to human agents. Supports text and voice channels.

Grounding is a Vertex AI feature that attaches real-time information sources to your generative model calls. Unlike RAG (where you build the retrieval pipeline), grounding is a parameter you set in the API call.

Google Search Grounding

When enabled, the model automatically searches Google to find relevant information before generating a response. The response includes inline citations and source URLs. Ideal for questions about current events, facts, or anything beyond the model's training cutoff.

# Grounding with Google Search
from vertexai.generative_models import GenerativeModel, Tool
from vertexai.preview.generative_models import grounding

model = GenerativeModel("gemini-1.5-pro")
tool = Tool.from_google_search_retrieval(
    grounding.GoogleSearchRetrieval()
)
response = model.generate_content(
    "What were Google Cloud's latest announcements at Cloud Next?",
    tools=[tool]
)
print(response.text)

Custom Data Grounding

Ground responses against your own Vertex AI Agent Builder data stores. The model searches your indexed documents and generates answers with citations pointing to your content.

# Grounding with custom data store
tool = Tool.from_retrieval(
    grounding.Retrieval(
        grounding.VertexAISearch(
            datastore="projects/my-project/locations/global/collections/default_collection/dataStores/my-store"
        )
    )
)
response = model.generate_content(
    "What is our company's vacation policy?",
    tools=[tool]
)

Function calling lets the model generate structured function invocations instead of (or alongside) text. You define tools with their parameters, and the model decides when and how to call them based on the user's request.

How It Works

(1) You define a set of Tool objects with function declarations (name, description, parameters as JSON schema). (2) The model receives the user query and your tool definitions. (3) If the model determines a tool call is needed, it returns a FunctionCall object with the function name and arguments. (4) Your application executes the function and sends the result back. (5) The model generates a final response using the function result.

from vertexai.generative_models import (
    GenerativeModel, Tool, FunctionDeclaration
)

# Define the tool
get_weather = FunctionDeclaration(
    name="get_current_weather",
    description="Get the current weather for a given location",
    parameters={
        "type": "object",
        "properties": {
            "location": {
                "type": "string",
                "description": "City and state, e.g. San Francisco, CA"
            }
        },
        "required": ["location"]
    }
)

tool = Tool(function_declarations=[get_weather])
model = GenerativeModel("gemini-1.5-pro", tools=[tool])

# The model returns a FunctionCall, not text
response = model.generate_content("What's the weather in Tokyo?")
function_call = response.candidates[0].content.parts[0].function_call
print(f"Function: {function_call.name}")
print(f"Args: {dict(function_call.args)}")

Automatic Function Calling

Vertex AI supports automatic function calling where the SDK executes your Python functions directly based on the model's function calls. Define your functions, wrap them as tools, and the SDK handles the round-trip automatically.

Gemini models are natively multimodal, accepting text, images, video, audio, and PDF inputs in a single prompt. This enables powerful applications that understand and reason across different content types.

Text + Image

Send images alongside text prompts for visual Q&A, image captioning, diagram interpretation, OCR, and visual classification. Images can be passed as base64-encoded bytes, GCS URIs, or file objects.

Text + Video

Gemini can process video files (up to 1 hour for Gemini 1.5 Pro) for summarization, content moderation, visual search, and scene understanding. Video is processed frame by frame with audio transcription.

Common Multimodal Patterns

Once your GenAI application is built, you need to serve it reliably. Google Cloud offers several options depending on scale, latency requirements, and operational complexity.

Cloud Run Deployment Pattern

Cloud Run is the most common choice for GenAI apps. Package your application as a container, deploy it, and Cloud Run handles scaling (including scale-to-zero). Set environment variables for project ID and location. Use a service account with Vertex AI permissions.

# Dockerfile for a GenAI Flask app on Cloud Run
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
ENV PORT=8080
CMD ["gunicorn", "--bind", "0.0.0.0:8080", "app:app"]

# Deploy to Cloud Run
gcloud run deploy genai-app \
  --source . \
  --region us-central1 \
  --service-account genai-sa@my-project.iam.gserviceaccount.com \
  --set-env-vars PROJECT_ID=my-project,LOCATION=us-central1 \
  --allow-unauthenticated

This section distills the most heavily tested topics for the MLE exam from this course.

RAG vs Fine-Tuning Decision

Agent Builder vs Custom Code

Grounding Strategy Selection

• Current events / factual questions → Google Search grounding
• Internal company data → Custom data store grounding or RAG
• Need full control over retrieval → Custom RAG pipeline
• Simple factuality improvement → API-level grounding parameter
• Creative tasks (poetry, brainstorming) → No grounding needed

How to Explain This in 2 Minutes

Likely Interview Questions

Model Answers

System Design Scenario

Common Mistakes

• Choosing fine-tuning when RAG is needed — Fine-tuning does not reliably teach a model new factual knowledge. If users ask about specific documents, policies, or data, RAG is the correct pattern. Fine-tuning is for behavior and style, not knowledge injection.
• Ignoring chunk size trade-offs — Chunks that are too small lose context and produce fragmented answers. Chunks that are too large dilute relevance and waste context window tokens. Always experiment with chunk sizes (256, 512, 1024 tokens) and measure retrieval quality on representative queries.
• Using Google Search grounding for internal data — Google Search grounding only accesses public web content. For proprietary or internal documents, you must use custom data store grounding or a RAG pipeline. Mixing these up is a common exam and interview pitfall.