🖼️ Vision Transformers: From Pixels to Patches - Technical Deep Dive
← Previous: Why ViT? 🏠 Home Next: Patch Embeddings →

🖼️ Vision Transformers: From Pixels to Patches

Master the complete mathematical and architectural foundations of Vision Transformers. From the revolutionary patch embedding concept to multi-head self-attention in the visual domain, understand every component that makes ViTs work.

🎯 What You'll Master: Patch tokenization mathematics, 2D positional encoding, visual attention mechanics, architecture scaling, memory analysis, and the complete ViT forward pass with real model specifications.

🧩 Step-by-Step ViT Architecture Walkthrough

Step 1: The Input Image Problem

Traditional CNNs process images through hierarchical local filters. Each layer can only "see" a small patch at once. To understand the whole image, information must flow through many layers.

Vision Transformer Solution:
Treat an image as a sequence of patches, just like text is a sequence of words. This allows every patch to "attend" to every other patch from layer 1.

Step 2: Converting Images to Patches

The first step transforms a 2D image into a 1D sequence that transformers can process.

Input Image
224×224×3
Height × Width × RGB
→
Patch Division
14×14 grid
224÷16 = 14 patches per side
→
Flattened Patches
196 patches
Each 16²×3 = 768 pixels
Patch Creation Mathematics:

1. Divide image into non-overlapping squares: 16×16 pixels each
2. Number of patches: (H÷P) × (W÷P) = (224÷16)² = 196 patches
3. Flatten each patch: 16×16×3 = 768 pixel values per patch
4. Result: 196 vectors, each containing 768 numbers

Step 3: The "Vocabulary" Problem in Vision

Text transformers have a fixed vocabulary (50K words/subwords). Images exist in continuous pixel space - any combination of RGB values is possible.

Key Difference:
• Text: Discrete vocabulary → lookup embedding table
• Vision: Continuous pixels → learn vocabulary during training
Linear Patch Embedding:

E = x_patches × W_projection + b

Where:
• x_patches: 196×768 (flattened patches)
• W_projection: 768×768 (learnable weight matrix)
• E: 196×768 (embedded patches)

This creates ViT's "vocabulary" by learning which visual patterns map to which embeddings

Step 4: The Classification Token

We need a single representation for the entire image. Unlike text where we might use the last token, vision adds a special [CLS] token.

Patch Embeddings
196×768
Image content
+
[CLS] Token
1×768
Learnable aggregator
=
Token Sequence
197×768
Ready for attention
💡 Why [CLS] works: Through attention, the [CLS] token learns to gather relevant information from all patch tokens to make the final classification decision.

Step 5: 2D Positional Encoding

After flattening patches, the model loses spatial relationships. We must encode "where" each patch came from.

The Problem: Two identical car patches look the same after flattening, but one might be "top-left corner" and another "bottom-right corner". Position matters for understanding spatial relationships.
Positional Encoding Process:

1. Create 197 position IDs: [CLS]=0, patches 1-196
2. Each position gets learnable embedding: 768 dimensions
3. Add to content: Final_token = Patch_embedding + Position_embedding

Key insight: Positions are fixed, but embeddings are learned
• Patch (0,0) always gets Position ID 1
• But Position 1's embedding vector is learned during training

Step 6: Multi-Head Self-Attention in Vision

Now we have 197 tokens (1 [CLS] + 196 patches), each with content and position information. Attention lets every token look at every other token.

Attention Process:

For each attention head:
1. Q = tokens × W_query (what am I looking for?)
2. K = tokens × W_key (what do I contain?)
3. V = tokens × W_value (what information can I provide?)
4. Attention_weights = softmax(Q × K^T / √d_k)
5. Output = Attention_weights × V

Result: Every patch can attend to every other patch globally
Visual Attention Examples:
• A "wheel" patch attends to other "car" patches
• "Sky" patches attend to other "sky" patches
• [CLS] token attends to most discriminative patches for classification

Implementation Pseudocode

Here's how the complete ViT forward pass actually works in code:

# B (batch size) ignored for brevity; add as leading dim in practice

# 1) Patchify + embed
patches = extract_patches(x, P=16) # (196, 768) = (N, 16*16*3)
tokens = patches @ W_p + b_p # (196, 768)

# 2) Add CLS + positions
cls = cls_param # (768,)
X = concat(cls[None, :], tokens, dim=0) # (197, 768)
X = X + E_pos # (197, 768)

# 3) L encoder layers (L=12 for ViT-Base)
for l in range(L):
# Pre-norm + Multi-Head Self-Attention
U = layernorm1(X) # (197, 768)
Q = U @ W_q; K = U @ W_k; V = U @ W_v # each (197, 768)

# Reshape for multi-head processing
Q = reshape(Q, (197, 12, 64)) # 12 heads, 64 dims each
K = reshape(K, (197, 12, 64))
V = reshape(V, (197, 12, 64))

# Attention computation per head
A = softmax(Q @ K.transpose(-1,-2) / sqrt(64)) # (12, 197, 197)
O = A @ V # (12, 197, 64)

# Concatenate heads and project
O = concat_heads(O) @ W_o # → (197, 768)
X = X + O # residual connection

# Pre-norm + MLP
U = layernorm2(X) # (197, 768)
M = gelu(U @ W1 + b1) @ W2 + b2 # 768→3072→768
X = X + M # residual connection

# 4) Classification readout
y = X[0] @ W_head + b_head # Extract [CLS] → (1000,)

Step 7: Feed-Forward Networks (MLP)

After attention gathers information, MLPs process each token independently with non-linear transformations.

Input
197×768
Post-attention tokens
→
Expand
197×3072
4× hidden dimension
→
Contract
197×768
Back to original size
💡 MLP Purpose: While attention mixes information between tokens, MLPs transform the information within each token through learned non-linear functions (GELU activation).

Step 8: Layer Stacking and Information Flow

ViT-Base uses 12 transformer layers. Each layer refines the representations progressively.

Layer Progression:
• Early layers (1-4): Learn local patterns, edges, textures
• Middle layers (5-8): Object parts, spatial relationships
• Late layers (9-12): Global context, semantic understanding

Each layer: LayerNorm → Attention → Residual → LayerNorm → MLP → Residual

Step 9: Classification Head

After 12 transformer layers, extract the [CLS] token for final classification.

[CLS] Token
1×768
Aggregated image info
→
Layer Norm
1×768
Normalized features
→
Linear Head
1×1000
Class probabilities
🎯 Complete Flow Summary:
Image → Patches → Linear Embedding → +[CLS] → +Position → 12×(Attention+MLP) → [CLS] → Classification

🧩 The Core Innovation: Treating Images as Sequences

💡 The Fundamental Insight

Vision Transformers revolutionized computer vision with a deceptively simple idea: treat an image as a sequence of patches, just like text is a sequence of words.

ViT Core Transformation:

Image(H × W × C) → Patches(N × (P² × C)) → Tokens(N × D)

Where:
• H, W = Image height, width
• C = Number of channels (3 for RGB)
• P = Patch size (typically 16×16)
• N = Number of patches = (H×W)/(P²)
• D = Embedding dimension (768 for ViT-Base)

🔍 Interactive Patch Tokenization

🧩 Patch Grid Visualizer

🏗️ ViT Architecture: Complete Mathematical Breakdown

📊 Architecture Flow: From Pixels to Predictions

Input Image
H×W×3
Raw pixels
→
Patch Embedding
N×D
Linear projection
→
Position + Class
(N+1)×D
Learnable embeddings
→
Transformer
L layers
Self-attention + MLP
→
Classification
Class logits
MLP head

🧮 Step 1: Patch Embedding Mathematics

The first crucial step converts image patches into embeddings that transformers can process.

Patch Embedding Process:

1. Extract patches: xp ∈ ℝN×(P²×C)
2. Linear projection: E = xpWp + bp
3. Where Wp ∈ ℝ(P²×C)×D is learnable projection

Key insight: Each patch becomes a D-dimensional vector
🔢 Patch Embedding Calculator
384×384
16×16

📍 Step 2: Positional Encoding for 2D Images

Unlike text, images have 2D spatial structure. ViTs use learnable positional embeddings to encode spatial relationships.

2D Positional Encoding:

z₀ = [xclass; xp1E; xp2E; ...; xpNE] + Epos

Where:
• xclass ∈ ℝD is learnable [CLS] token
• Epos ∈ ℝ(N+1)×D is learnable position embedding
• Each patch gets unique position information
📍 Positional Encoding Visualizer
8×8

🎯 Step 3: Multi-Head Self-Attention for Vision

Self-attention in vision enables each patch to attend to all other patches, creating global receptive fields from layer 1.

Visual Self-Attention Mathematics:

Attention(Q,K,V) = softmax(QKT/√dk)V

Where for each head h:
• Qh = zl-1Wqh ∈ ℝ(N+1)×dh
• Kh = zl-1Wkh ∈ ℝ(N+1)×dh
• Vh = zl-1Wvh ∈ ℝ(N+1)×dh
• dh = D/H (head dimension)
🎯 Multi-Head Attention
Params: 3×D²
Complexity: O(N²)
Global receptive field
🧠 MLP Block
Params: 8×D²
Hidden: 4×D
GELU activation
📊 Layer Normalization
Params: 2×D
Pre-norm architecture
Stabilizes training
🔗 Residual Connections
Params: 0
Enables deep networks
Gradient flow

📏 ViT Model Variants: Scaling Analysis

🔢 Complete Model Specifications

⚖️ ViT Model Comparison

💾 Memory Scaling: The Quadratic Challenge

Vision Transformers face the same quadratic scaling challenge as text transformers, but with 2D images creating even larger sequence lengths.

⚠️ ViT Computational Complexity:

Self-Attention Complexity:
• Memory: O(N²) where N = (H×W)/P²
• For 384×384 image with 16×16 patches: N = 576
• Attention matrix: 576² = 331,776 elements per head
• With 12 heads: ~4M attention weights per layer

Why This Matters:
• Doubling image size → 4× more attention computation
• 1024×1024 images → 4,096 patches → 16M attention matrix
• Memory becomes the primary constraint for high-resolution images
💾 Memory Scaling Calculator
384×384
8

🔬 Advanced ViT Concepts

🎯 Attention Pattern Analysis

Understanding what ViTs "see" through attention patterns reveals how they process visual information differently from CNNs.

🔍 Attention Pattern Simulator
Layer 6/12
Head 3/12

📈 Performance vs Scale: Empirical Analysis

Real-world performance data shows how ViT variants perform across different scales and datasets.

Model Parameters ImageNet Top-1 Training Data Memory (FP16) Training Time
ViT-Base/16 86M 77.9% JFT-300M 1.2GB 3 days (TPUv3)
ViT-Large/16 307M 85.2% JFT-300M 4.1GB 7 days (TPUv3)
ViT-Huge/14 632M 88.5% JFT-300M 8.7GB 14 days (TPUv3)
ViT-Giant/14 1.8B 90.1% JFT-3B 22GB 30+ days
🎯 Key Insights:
• ViTs scale predictably with parameters and data
• Larger models need exponentially more training data
• Memory scales roughly linearly with parameters
• Training time scales super-linearly with model size
• Performance gains diminish at very large scales

🚀 Production Considerations

⚡ Inference Optimization Strategies

✂️ Patch Size Optimization
Larger patches → Fewer tokens
16×16 vs 32×32 trade-off
Resolution vs efficiency
🎯 Attention Optimization
Linear attention variants
Sparse attention patterns
Local attention windows
📱 Mobile-Friendly ViTs
MobileViT architectures
Quantization strategies
Knowledge distillation
🔄 Dynamic Resolution
Adaptive input sizing
Multi-scale inference
Efficiency-accuracy trade-offs

🎯 When to Choose ViT: Decision Framework

✅ Use ViT When:
• Large datasets available (10M+ images)
• Complex visual reasoning required
• Transfer learning from large pre-trained models
• Multimodal applications (vision + language)
• Research exploring attention patterns

⚠️ Consider Alternatives When:
• Limited training data (<1M images)
• Mobile/edge deployment constraints
• Real-time inference requirements
• Simple classification tasks
• Extremely high-resolution images (>1024px)
🎓 Next Steps: Now that you understand ViT fundamentals, you're ready to explore patch embeddings in detail, cross-modal attention in CLIP, and advanced architectures like Swin Transformers. The mathematical foundation you've built here applies to all vision transformer variants.