Video Generation with AI. Working with Diffusion Transformers and Multimodal Learning Joseph Enochs

Dostępność produktu

Produkt został zweryfikowany pod kątem ułatwień dostępu więcej przeczytasz w deklaracji dostępności.

Spis treści książki

• Preface
  • Who Should Use This Book
  • Conventions Used in This Book
  • Using Code Examples
  • OReilly Online Learning
  • How to Contact Us
  • Acknowledgments
• 1. Foundations of Video Generation
  • Hands-On: Your First AI Video
    • Environment Setup
    • Loading the Model
    • Generate Your First Video
    • Save and Display Your Achievement
  • Introduction to Generative Video Models
    • What Makes Video Unique in AI
      • Temporal consistency
      • Higher computational demands
      • Sequence of events
    • Key Use Cases for Generative Video
      • Entertainment and media
      • Advertising and marketing
      • Healthcare and medical training
      • Cultural preservation and accessibility
  • The Architectures Behind the Models
    • GANs, Diffusion Models, and Variational Autoencoders
      • Generative adversarial networks (GANs): The artist and the critic
      • Variational autoencoders (VAEs): The magical middle ground
      • Diffusion models: The digital restoration artists
    • Modern Generative Techniques for Video
      • The revolutionary DiT architecture
      • Architectural components of modern video models
    • Open Models and Baseline Tools
      • Leading open source models
      • Essential development stack
    • Evaluation and Early Experiments
  • Getting Started with AI Video Generation
    • Exercise 1: Implementing Basic Evaluation Functions
    • Exercise 2: How Inference Steps Impact Quality
    • Exercise 3: How Prompt Styles Influence Output
  • Learning from Your Experiments
    • Trade-Off Between Inference Steps and Quality
    • Subjectivity of Evaluation Metrics
    • Recommended Settings by Use Case
    • Evaluation Guide
  • Hands-On Exercise: Create and Evaluate Your Own Video
  • Summary
• 2. Understanding and Preparing Video Data for Model Training
  • Hands-On: Working with Video Datasets
    • Setting Up the Workspace
    • Loading the Dataset
    • Common Challenges in Handling Video Data
      • Scalability
      • Privacy and ethics
      • Compliance and trust
  • Working with Video Datasets
  • Types of Video Data
    • Raw Video Data
    • Hands-On: Analyzing Raw Video Properties
    • Compressed Video Data
    • Hands-On: Comparing Raw and Compressed Video
    • Annotated Video Data
    • Hands-On: Simulating Video Streaming
    • Choosing Data Based on Use Case
    • Sourcing and Collecting Video Data
      • Publicly available datasets
      • Custom video capture
      • Collaborative data sharing
    • Automated Filtering Pipelines
  • Tools for Video Handling
    • Hands-On: Screening Quality with OpenCV
    • Hands-On: Normalizing and Exporting with FFmpeg
    • Hands-On: Detecting Scenes with PySceneDetect
  • Video Processing Pipelines for AI
  • Summary
• 3. Implementing Video Generation Pipelines
  • Building the Foundation: Data Pipeline and Encoding
    • Latent Space Compression with VAE
    • Understanding 3D Convolutions for Video VAE
      • Architecture setup and progressive downsampling
      • Forward pass and reparameterization trick
      • Deterministic inference path
    • Video Tokenization: From Pixels to Patches
    • Patch Embedding and Positional Encoding
      • Module initialization and architecture
      • Sinusoidal position encoding (intuition)
      • Forward pass: From videos to token sequences
      • Building toward attention
  • Attention Mechanisms: Understanding Relationships in Space and Time
    • What Is Attention?
    • Spatial Attention: Coherence Within Frames
    • Temporal Attention: Continuity Across Frames
    • The Alternating Block Architecture
  • The Diffusion Process: From Noise to Video
    • Forward and Reverse Processes
    • Noise Prediction and Scheduling
    • Adaptive Conditioning with AdaLN
  • Conditioning and Control: From Text to Video
    • Text Encoding with T5
    • Cross-Attention Integration
    • Classifier-Free Guidance
  • Latte Model Component Integration
    • Timestep Embedding: Encoding the Diffusion Schedule
    • Text Encoder: From Prompts to Embeddings
  • Putting It All Together: The Generation Pipeline
  • Summary
• 4. Training Video Generation Models
  • Configuration-Driven Training
    • YAML Configuration Structure
    • Loading and Managing Configurations
  • Dataset Preparation and Loading
    • Video Dataset Architecture
      • Dataset initialization and metadata management
      • Intelligent clip indexing
      • Efficient frame loading with caching
      • Temporal-consistent augmentation
      • Dataset interface and batching
    • Multiresolution Training Strategy
    • Temporal Sampling and Frame Selection
  • Training Infrastructure and Optimization
    • Anatomy of a Training Loop
    • Understanding Each Training Step
    • Memory-Efficient Training Strategies
      • Part 1: Trainer initialization and setup
        • Lattes simpler training approach
        • TF32 optimization (Lattes performance secret)
      • Part 2: Distributed training configuration
      • Part 3: Optimizer and scheduler configuration
      • Part 4: Gradient accumulation and memory management
      • Part 5: Core training step implementation
      • Part 6: Loss computation and model updates
      • Part 7: Training loop and monitoring
    • Learning Rate Scheduling and Warmup
    • Gradient Management and Stability
  • Loss Functions and Metrics
    • Loss Design: From Simple to Complex
      • Simple MSE loss (what Latte actually uses)
      • When (and why) you might need more
  • Scaling and Distributed Training
    • Data Parallelism (Latte Approach)
    • Advanced Parallelism
    • Practical Considerations
  • Summary
• 5. Fine-Tuning for Specific Video Tasks
  • Fine-Tuning Methods
    • LoRA: Low-Rank Adaptation for Video Models
    • How LoRA Works in CogVideo
    • Supervised Fine-Tuning (SFT) with DeepSpeed
    • Choosing Between LoRA and SFT
    • Decision Guide: When to Use Each Method
  • Common Fine-Tuning Issues
    • Issue 1: RuntimeErrorTensor Size Mismatch
    • Issue 2: CUDA Out of Memory (OOM)
    • Issue 3: Training Not Converging
    • Issue 4: Mixed Precision Training Fails
    • Issue 5: LoRA Weights Not Loading
    • Domain Adaptation and Transfer Learning
      • Understanding domain shift
      • Unsupervised domain adaptation
      • Cross-domain video generation
  • Few-Shot and Zero-Shot Video Generation
    • Meta-Learning for Video Generation
    • Compositional Video Generation
    • Prompt Engineering and In-Context Learning
  • Evaluation and Benchmarking Fine-Tuned Models
    • Task-Specific Evaluation Metrics
    • Ablation Studies and Analysis
    • Transfer Learning Effectiveness
  • Practical Execution: Running Your First Fine-Tuning
    • Setting Up Your Environment
    • Preparing Your Dataset
    • LoRA Fine-Tuning: Quick Start
    • SFT with DeepSpeed: Maximum Quality
    • Multi-GPU Training: Scaling Up
    • Inference with Fine-Tuned Models
    • Production Examples
  • Summary
• 6. Vision-Language Models for Video Understanding
  • From Generation to Understanding
    • The Two Sides of Video AI
    • The Paradigm Shift
    • Introducing Qwen3-VL
  • VLM Architecture
    • The Three-Component Model
    • Vision Encoder: Seeing the Video
    • The Projector: Bridging Two Worlds
    • The LLM: Reasoning Over Multimodal Tokens
    • Token Sequence Construction
    • Putting It Together: A VideoQA Class
    • 3D Positional Encoding for Video
    • Rotary Position Embedding (RoPE)
    • Extending to 3D: Temporal, Height, Width
    • Qwen3-VLs Interleaved-MRoPE Implementation
    • Timestamps Versus Frame Indices
    • Extending VideoQA with Temporal Grounding
    • Video Understanding Pipeline
    • The Complete Processing Pipeline
    • Video Understanding Task Types
    • Inference Code Example
      • Part 1: Model loading
      • Part 2: Message construction
      • Part 3: Processing and generation
    • Handling Long Videos
    • Adapting to Your Domain: Dataset Preparation
    • Training with LoRA
      • Using fine-tuned models
      • Evaluation
  • Summary
• 7. Audiovisual Synchronization with Cross-Modal Fusion
  • The Missing Modality: Audio
    • Why Post Hoc Audio Fails
  • Audio Representation: From Waveforms to Mel-Spectrograms
  • Introducing LTX-2
  • Unified DiT Architecture
    • The LTXModel Design
    • The Modality Dataclass
    • Bidirectional Cross-Modal Attention
    • AdaLN: Adaptive Layer Normalization
  • Audio Pipeline Deep Dive
    • Mel-Spectrogram Representation
    • Audio VAE Encoder
    • Per-Channel Statistics Normalization
    • Audio VAE Decoder
    • HiFi-GAN Style Vocoder
    • AudioPatchifier
  • Two-Stage Inference Pipeline
    • Pipeline Initialization: ModelLedger
    • Stage 1: CFG-Guided Generation
      • Classifier-free guidance (CFG)
      • The Euler denoising loop
    • Spatial Upsampling
    • Stage 2: Distilled Refinement
    • Final Decoding
  • Hands-On Implementation
    • Environment Setup
    • Hardware Requirements
    • Required Model Downloads
    • Text-to-Video Generation
    • Command-Line Interface
    • Image-to-Video Generation
    • Speech Synthesis
    • Fine-Tuning with LoRA
      • Dataset preparation
      • Training configuration
      • Launch training
    • Using Trained LoRAs for Inference
    • Architectural Evolution: From Research to Production
  • Summary
• 8. From Models to Systems
  • The Gap Between a Model and a Product
    • What Production AI System Means in 2026
    • Three Reference Architectures
  • LTX Desktop: Eight Patterns
    • Pattern 1: The Two-Process Architecture
      • Electron owns lifecycle, Python owns work
      • Spawning the backend
      • Health monitoring and restart
      • Backend adoption: Surviving stale processes
    • Pattern 2: Backend as Source of Truth
      • Capability detection at startup
      • The local-versus-API decision
      • Exposing the decision via /api/runtime-policy
    • Pattern 3: The Typed IPC Bridge
      • contextBridge and the sandboxed renderer
      • One Zod schema, two sides
      • Adding a new IPC method
    • Pattern 4: Localhost as a Security Boundary
      • Per-session auth tokens
      • Constant-time token comparison
      • Content security policy in production
      • Path validation across platforms
    • Pattern 5: Model Lifecycle: Download, License, Storage
      • Storage roots across platforms
      • The first-run flow
      • Streaming download progress
      • License acceptance as state
    • Pattern 6: State as Discriminated Unions
      • Why nullable-field structs hurt
      • Tagged unions in Python
      • Exhaustive pattern matching in handlers
    • Pattern 7: Service Protocols and Fakes (No Mocks)
      • Side effects behind protocols
      • Real and fake implementations
      • Composing the ServiceBundle
    • Pattern 8: Lock-Aware Handlers
      • The @with_state_lock decorator
      • The never hold the lock doctrine
      • Background tasks and the TaskRunner
  • The Landscape: Three Architectures, One Problem
    • ComfyUI: Community Plug-in Mesh
      • Architecture philosophy
      • The node abstraction
      • The custom node extension model
      • Strengths
      • Trade-offs
    • InvokeAI: Service-Oriented Monolith
      • Architecture philosophy
      • The invocation abstraction
      • The services layer
      • Strengths
      • Trade-offs
      • Comparison matrix
  • Capstone: Building a Custom Extension Across Three Platforms
    • The Feature: Add LTX-2 Audio-to-Video as a New Capability
    • Implementation in LTX Desktop
    • Implementation in ComfyUI
    • Implementation in InvokeAI
    • What the Three Implementations Reveal
      • Architectural principles recap
      • Practical applications
      • Design lessons
  • Summary and Conclusion: The Video AI Landscape
• Index

pokaż cały spis treści

ukryj recenzje