Fundamentals of Generative AI

Artificial Intelligence (AI) is the broad field of computer science focused on creating systems that can perform tasks that typically require human intelligence. This encompasses reasoning, learning, perception, language understanding, and decision-making. AI systems range from simple rule-based engines to sophisticated deep learning models.

Machine Learning (ML) is a subset of AI where systems learn patterns from data rather than being explicitly programmed. Instead of writing rules, you provide examples and the algorithm discovers the rules. ML powers recommendations, fraud detection, language translation, and thousands of other applications.

Deep Learning (DL) is a subset of ML that uses neural networks with multiple layers (hence "deep") to learn hierarchical representations of data. Deep learning excels at unstructured data — images, text, audio, and video — and is the foundation for modern generative AI.

AI vs ML vs Deep Learning

NLP Fundamentals

Natural Language Processing (NLP) is the branch of AI focused on enabling computers to understand, interpret, and generate human language. Key NLP tasks include:

Generative AI refers to artificial intelligence systems that can create new content — text, images, audio, video, code, and structured data — rather than merely classifying or analyzing existing data. These models learn the underlying distribution of their training data and generate novel outputs that are statistically similar to, but not copies of, the training examples.

Generative AI represents a paradigm shift from traditional ML. Where traditional systems consume data and produce decisions (classify this email, predict this price), generative systems produce entirely new artifacts. A generative AI model can write a marketing email, generate a product image, compose music, or write working code.

Generative vs Discriminative Models

This is a fundamental distinction tested on the exam. A discriminative model learns the decision boundary between classes — it models P(Y|X), the probability of a label Y given input X. A generative model learns the full joint distribution P(X,Y) or P(X), enabling it to generate new data points.

Use cases for generative AI span every industry: content creation (marketing copy, articles), code generation and debugging, customer service automation, document summarization, data augmentation, drug discovery, creative design, and educational tutoring. The exam expects you to identify which scenarios are best suited for generative AI versus traditional ML approaches.

A foundation model is a large AI model pre-trained on broad, diverse data that can be adapted to a wide range of downstream tasks. The term was coined by Stanford's HAI in 2021 to describe models like BERT, GPT, and PaLM that serve as a "foundation" upon which many applications are built.

Foundation models exhibit emergent capabilities — abilities that appear only at sufficient scale that were not explicitly trained for. For example, large language models can perform arithmetic, translate between languages they were not specifically trained on, and reason about novel problems. These emergent properties make foundation models uniquely powerful.

Key Characteristics of Foundation Models

The Transformer Architecture

The Transformer is the neural network architecture behind virtually all modern LLMs. Introduced in the 2017 paper "Attention Is All You Need" by Vaswani et al., it replaced recurrent architectures (RNNs, LSTMs) with a self-attention mechanism that processes all tokens in parallel rather than sequentially.

Key components of the Transformer:

• Self-Attention — allows each token to attend to every other token, capturing long-range dependencies
• Multi-Head Attention — runs multiple attention operations in parallel to capture different relationship types
• Positional Encoding — injects position information since attention is order-agnostic
• Feed-Forward Networks — applied to each position independently after attention
• Layer Normalization — stabilizes training of deep networks

Large Language Models (LLMs) are transformer-based foundation models trained specifically on text data. They predict the next token given a sequence of preceding tokens (autoregressive generation). Modern LLMs have billions to trillions of parameters and are trained on datasets containing trillions of tokens from books, websites, code repositories, and other text sources.

Diffusion Models

Diffusion models are a class of generative models that create data by learning to reverse a noise-adding process. During training, the model learns to progressively denoise a corrupted input. During generation, it starts from pure random noise and iteratively refines it into a coherent output.

Diffusion models power image generation (Imagen), video generation (Veo), and audio synthesis. They produce higher-quality, more diverse outputs than earlier approaches like GANs (Generative Adversarial Networks) and VAEs (Variational Autoencoders), and are more stable to train.

Machine learning approaches are categorized by the type of learning signal available during training. Understanding these distinctions is critical for the exam, as questions may ask when to use each approach.

Supervised, Unsupervised, and Reinforcement Learning

The ML lifecycle describes the end-to-end process of developing, deploying, and maintaining ML models. For generative AI projects, the lifecycle includes additional considerations around prompt engineering, evaluation of generated content, and responsible AI guardrails.

Choosing the Right Model

Selecting the right model is a critical decision. The exam tests whether you can match scenarios to appropriate model choices. Key factors include:

• Task type — text generation, image creation, classification, or multimodal
• Latency requirements — real-time (use smaller/faster models like Gemini Flash) vs. batch
• Cost constraints — larger models cost more per request; fine-tuning adds upfront cost
• Data sensitivity — on-premises deployment needs (use open-weight Gemma) vs. managed API
• Quality needs — complex reasoning tasks need larger models (Gemini Pro/Ultra)
• Customization depth — prompt engineering (fast) vs. fine-tuning (slower but more tailored)

Data is the fuel for all AI systems. The exam covers data types, data quality principles, and the distinction between structured/unstructured and labeled/unlabeled data.

Structured vs Unstructured Data

Labeled vs Unlabeled Data

Labeled data has known correct outputs (a tagged image, a classified document). It is required for supervised learning and fine-tuning. Creating labeled data is expensive and time-consuming.

Unlabeled data has no associated labels. It is abundant and cheap. Unsupervised learning and self-supervised pre-training (used for LLMs) work with unlabeled data. LLMs are pre-trained on massive unlabeled text corpora using self-supervised objectives (predict the next token).

Data Quality

Data quality directly impacts model performance. The principle "garbage in, garbage out" applies even more strongly to generative AI, where poor data leads to inaccurate, biased, or hallucinated outputs.

The generative AI technology stack can be understood as a layered architecture. Each layer builds on the one below and serves different stakeholders. The exam expects you to understand where Google's offerings fit within this landscape.

Google offers a comprehensive family of foundation models, each optimized for different modalities and use cases. Understanding the distinctions between these models is heavily tested on the exam.

The Gemini Family

Gemini is Google's most capable multimodal AI model family. It natively handles text, images, audio, video, and code in a single model. Gemini comes in multiple size variants optimized for different use cases:

Gemma, Imagen, and Veo

Beyond Gemini, Google offers specialized foundation models for specific modalities:

# Quick comparison: Gemini vs Gemma via Vertex AI SDK
from vertexai.generative_models import GenerativeModel

# Gemini Pro - managed API, multimodal
gemini = GenerativeModel("gemini-1.5-pro")
response = gemini.generate_content("Explain quantum computing in 3 sentences.")

# Gemma - open model, deploy to your own endpoint
# Deploy Gemma from Model Garden, then call your custom endpoint
gemma = GenerativeModel("gemma-7b")  # from your deployed endpoint
response = gemma.generate_content("Explain quantum computing in 3 sentences.")