Hands-On LLM Serving and Optimization. Hosting LLMs at Scale Chi Wang, Peiheng Hu

Dostępność produktu

Produkt nie został jeszcze oceniony pod kątem ułatwień dostępu lub nie podano żadnych informacji o ułatwieniach dostępu lub są one niewystarczające. Prawdopodobnie Wydawca/Dostawca jeszcze nie umożliwił dokonania walidacji produktu lub nie przekazał odpowiednich informacji na temat jego dostępności.

Spis treści książki

• Preface
  • Why LLM Serving and Optimization?
  • What This Book Aims to Do
  • Who Should Read This Book
  • What This Book Isnt
  • How This Book Is Organized
  • How to Use This Book
  • What Youll Need
  • Conventions Used in This Book
  • Using Code Examples
  • OReilly Online Learning
  • How to Contact Us
  • Acknowledgments
• 1. Introduction to Model Serving and Optimization
  • Anatomy of a Model
    • Model Architecture
    • Model Data
    • Model Execution Code
  • Model Lifecycle: From Training to Serving
  • What Is Model Serving?
  • Why Study Model Serving?
  • Why Optimize Model Serving (Especially for LLMs)?
    • Example: Using a Model Serving Framework (vLLM) to Improve LLM Throughput
  • Model Serving Paradigms
    • On-Device (Edge) Serving
      • On-device serving design
      • Preparing a model for on-device serving
      • When on-device serving is the right choice
    • Single-Model Service
      • Routing choices in single-model serving
      • Horizontal and vertical scaling
      • Why single-model service is the default starting point
    • Multi-Model Service
      • Routing and autoscaling in multi-model service
      • When multi-model service becomes necessary
    • Model Serving Platforms
  • Summary
• 2. Large Language Model Serving
  • Inside the Mind of a Transformer
    • LLM Evolution
    • The Autoregressive Nature of Transformers
    • Decoder-Only Transformer Architecture
      • Model architecture
      • Transformer (decoder) block
    • Capture Token Context by Calculating Attention
      • Attention calculation
      • Multi-head attention
  • Executing LLM Generation: A Step-by-Step Walkthrough
    • Run the Qwen Model
    • Model Prediction, Line by Line
    • Enable the KV Cache to Boost Performance
    • The Prefill and Decode Phases
    • Run the LLM with a Serving Framework
    • Serve the LLM (Qwen) with vLLM
    • Performance Comparison: vLLM Versus Hugging Face Transformers
  • LLM Streaming Serving Basics
  • LLM Batch Serving Basics
  • Summary
• 3. Model Serving System Design: A Deep Dive
  • Build an Online LLM Serving Service from Scratch
    • Design Goals
    • Service Architecture
    • Implement Single Generation Request Handling
    • Batching
    • Streaming with Batching
    • Batch Serving with vLLM
  • A General Design for Single-Model LLM Serving
    • Requirements for Single-Model Serving
    • General Design
  • Build a Multi-Model Serving Service from Scratch
    • Design Goals
    • Service Architecture
    • Core Implementation
    • Using NVIDIA Triton as a Model Server
  • Trade-offs in Multi-Model Serving Designs
    • Challenges
    • A Cost-Optimized Multi-Model Design
    • A Latency-Optimized Multi-Model Design
  • Summary
• 4. Model Serving Best Practices
  • Model Serving in an Agentic World
    • Defining Agents
    • A Sample Knowledge Agent
    • The Agents Design
    • The Agents Internal Workflow
    • Agent Autonomy
      • Defining actions
      • Planning with LLMs
    • Retrieval-Augmented Generation (RAG)
      • Why RAG?
      • Index-building workflow (offline process)
      • The query/retrieval workflow (online process)
    • Cache-Augmented Generation (CAG)
    • How Agents Use Model Serving
  • LLM Serving in Enterprise Systems: An Overview
    • Public API Layer
    • Resource Management Layer
    • Model Selection and Orchestration Layer
    • Distributed Serving Layer
    • Core Inference Layer
    • Model Optimization Layer
    • Model Layer
  • Building with an Open Source Stack
    • Implementing Public API
    • Implementing Model Selection
    • Implementing a Model Serving Endpoint
      • Single-model hosting on Ray
      • Multi-model hosting on Ray
  • Building with a Cloud Vendor
    • Option 1: Fully Managed Foundation-Model Serving
      • Limitations
      • When to use
    • Option 2: One-Click Foundation-Model Deployment
      • Limitations
      • When to use
    • Option 3: Bring Your Own Model
      • Limitations
      • When to use
    • Option 4: Bring Your Own Code
      • Limitations
      • When to use
    • Option 5: Bring Your Own Serving Image
      • Limitations
      • When to use
    • Option 6: Build Your Own Serving Infrastructure
    • Comparing the Options
  • Build or Buy? Understanding Strategies
    • Why Knowing How to Build HelpsEven If You Wont Build
    • Our Selection Strategy
  • Measuring Performance in LLM Serving
    • Latency Metrics
    • Throughput Metrics
    • Best Practices for Performance Measurement
      • Identify the trade-off between latency and throughput
      • Set acceptable latency goals based on specific use cases
      • Decompose E2E latency into TTFT and ITL
      • Simulate real-world traffic patterns as closely as possible
      • Maintain consistency in your experiments
      • Monitoring and tracking hardware utilization
      • Avoid manipulating or inflating performance metrics
      • Continuously monitor performance in production
      • Run test suites periodically
  • Summary
• 5. Challenges When Serving LLMs
  • Why Optimizing LLM Serving is Important
    • Customer Experience
    • Cost Efficiency
    • Scalability, Peak Load Handling, and Feasibility
  • The Role of Accelerator Chips in LLM Serving
    • Reading GPU specs
      • GPU compute attribute
      • GPU memory attributes
      • GPU interconnect attributes
        • Intra-node interconnects
        • Inter-node interconnects
      • GPU power consumption
    • Comparing the Specs of Popular GPUs
  • Bottlenecks in LLM Model Loading
    • The Model Loading Process
    • Estimating Model Size
    • Estimating KV Cache Size
  • Bottlenecks in LLM Model Execution
    • Boundaries of GPU Compute and Memory Bandwidth
    • Arithmetic Intensity in Matrix Multiplications
    • Applying Arithmetic Intensity Analysis to the LLM Prefill and Decode Phases
  • Other AI Accelerators and Trends
  • Summary
• 6. Essential LLM Optimization Techniques
  • Request Batching and Scheduling-Level Optimizations
    • Why Do We Need Batching in Real-Time Serving?
    • Dynamic Batching in Online Inference
    • Continuous Batching for LLM Online Inference
    • Continuous Batching with Chunked Prefill
  • Scaling Attention and Kernel Optimization
    • Scalable Attention Mechanisms
    • Kernel Fusion and Custom Attention Kernels
      • Kernel fusion
      • FlashAttention
      • PagedAttention
  • Model Compression
    • Quantization
      • Quantization error
      • Storing numbers
      • How does quantization help in model serving?
      • Weight-only versus weight-and-activation quantization strategies
      • Hands-on quantization
        • Find the quantized model
        • Quantize a model yourself
        • Running benchmarks
        • Performance analysis
      • Other quantization methods
        • KV cache quantization and attention quantization
        • GGUF quantization
      • Accuracy trade-offs
      • Quantization-aware training
    • Distillation
    • Pruning
  • Prefix Caching
    • RadixAttention
    • Use Cases
    • Best Practices
    • Scaling Prefix Cache
  • Summary
• 7. Advanced LLM Optimization Techniques
  • Speculative Decoding
    • Detailed Steps
    • Tuning and Usage
      • Choose an existing small model
      • Use self-drafting
      • N-gram
      • Tune the number of draft tokens
      • Limitations
    • Hands-on Speculative Decoding
      • Enable speculative decoding
      • Run benchmarks
      • Performance analysis
  • Multi-GPU and Multi-Node Inferencing
    • Data Parallelism
    • Tensor Parallelism and Pipeline Parallelism
    • Expert Parallelism
  • Prefill-Decode Disaggregation
    • Overall Architecture
    • KV Cache Transfer
    • When to Use
  • Advanced KV Caching
    • Long-Context Serving
    • Cost and Latency Calculations
    • Self-Hosting LLMs
      • KV cache offloading
      • KV cache compression
      • KV cache blending
    • Hands-on LMCache
      • Enable LMCache and CPU offloading
      • Run benchmarks
      • Performance analysis
  • Summary
• 8. LLM Serving Frameworks
  • Why We Need Specialized LLM Serving Frameworks
  • vLLM
    • vLLMs Architecture
      • LLMEngine and EngineCore
      • Scheduler
      • ModelExecutor, (GPU) Worker, and ModelRunner
    • Model Initialization Workflow (with Multi-Process Worker)
    • Generation-Request Execution Workflow
    • Scheduler Deep Dive
      • Request scheduling workflow
      • Applying model optimization techniques
    • vLLMs Layered Optimization Strategy
  • TensorRT-LLM
  • SGLang
  • Llama.cpp
  • Selecting the Right Framework
  • Summary
• 9. LLM Optimization in Practice
  • LLM Serving Optimization Plan
  • Optimize Qwen3-14B serving with vLLM
    • Step 1: Examine the GPU hardware
    • Step 2: Generate Benchmark Traffic
    • Step 3: Define Evaluation Metrics
    • Step 4: Set Up the Model Serving Server
    • Step 5: Benchmark the Qwen3 Model with vLLM
    • Step 6: Benchmark the Quantized Qwen3 Model with vLLM
    • Step 7: Apply Additional Optimization Techniques
    • Step 8: Benchmark the Qwen3 Model with Distributed Serving
      • Host model in a distributed setup
      • Distributed serving performance analysis
  • Common Optimization Trade-0ffs
  • Summary
• 10. Advancements in LLM Serving
  • Semantic Caching
  • Performance Profiling Strategies
  • Multimodal Serving
    • Multimodal Input Processing
    • Architectural and System Implications
  • Edge AI: Drivers and Enablers
    • Specialized Low-Power Hardware
    • Model Compression and Optimization
    • Heterogeneous Compute
    • Thermal-Aware Scheduling
    • EdgeCloud Hybrid Compute
  • Multi-LoRA Serving
  • Model Serving in Reinforcement Learning
    • LLM Serving in RL
    • Determinism in RL Serving
  • Summary
• Index

pokaż cały spis treści

ukryj recenzje