Evals for AI Engineers. Systematically Measuring and Improving AI Applications Shreya Shankar, Hamel Husain

Dostępność produktu

Produkt został zweryfikowany pod kątem ułatwień dostępu więcej przeczytasz w deklaracji dostępności.

Spis treści książki

• Preface
  • What This Book Covers
  • Who Should Read This Book
  • A Note on the Pace of Change
  • What This Book Doesnt Cover
  • How This Book Is Organized
  • Conventions Used in This Book
  • OReilly Online Learning
  • How to Contact Us
  • Acknowledgments
• I. Foundations
• 1. Introduction
  • What Is Evaluation?
  • Evaluation Throughout the Lifecycle of an LLM
    • Pretraining
    • Post-training
    • Application
  • Evaluation in Applications
  • The Three Gulfs of LLM Application Development
    • The Gulf of Comprehension (Between the Developer and Data)
    • The Gulf of Specification (Between the Developer and the LLM Agent)
    • The Gulf of Generalization (Between the Data and the LLM Agent)
  • Why LLM Agent Evaluation Is Challenging
  • The LLM Evaluation Lifecycle: Bridging the Gulfs with Evaluation
    • Analyze
    • Measure
    • Improve
  • Putting It Together
• 2. LLMs and Evaluation Basics
  • Components of LLM Agents
    • Single LLM Call
    • Conversations
    • Retrieval
    • Tools
    • Agent Loops
  • Fundamentals of Prompting
    • Prompts for Single LLM Calls
    • Retrieval Fundamentals
    • Tools and Agents
  • Evaluation Setups
    • Absolute Evaluation
    • Comparative Evaluation
  • Putting It Together
• II. The Analyze Phase
• 3. Error Analysis
  • Establishing Terminology
  • Creating a Starting Dataset of Traces
    • Defining Dimensions
    • Generating Tuples and Queries
  • Open Coding: Reading and Labeling Traces
  • Axial Coding: Structuring Failure Modes
  • Labeling Traces with Structured Failure Modes
  • Iteration and Refinement
  • Common Pitfalls
  • Putting It Together
• 4. Collaborative Evaluation Practices
  • When a Single Expert Is Enough
  • A Collaborative Annotation Workflow
  • Measuring Inter-Annotator Agreement
    • Percent Agreement: Simple but Misleading
    • Cohens Kappa: Adjusting for Chance
    • Interpreting Kappa Scores
    • Python Implementation
    • When to Use Other Metrics
      • More than two annotators
      • Ordinal ratings
      • Missing data
  • Facilitating Alignment Sessions
    • Preparing for the Session
    • Running the Session
    • Techniques for Improving the Rubric
    • What to Avoid
    • Escalation
    • Save Everything
  • Common Pitfalls
    • Skipping Independent Annotation
    • Starting with a Vague Rubric
    • Using Only Percent Agreement
    • Ignoring Temporal Drift
    • Synthetic-Annotator Collusion
    • Focusing on Past Labels Instead of Future Consistency
  • Putting It Together
• III. The Measure Phase
• 5. Implementing Automated Evaluators
  • Defining the Right Metrics (What to Measure)
  • Implementing Metrics (How to Measure)
  • Writing LLM-as-Judge Prompts
  • Data Splits for Designing and Validating LLM-as-Judge
  • Iterative Prompt Refinement for the LLM-as-Judge
    • When to Stop Refining
    • If Alignment Stalls
  • Estimating True Success Rates with Imperfect Judges
    • Step 1: Measure Judge Accuracy
    • Step 2: Observe Raw Success Rate
    • Step 3: Correct the Observed Success Rate
    • Step 4: Quantify Uncertainty with a Bootstrap
  • Python Code for Estimating Success Rates
  • Optional: Group-wise Metrics for Evaluating Multiple Outputs
  • Common Pitfalls
  • Putting It Together
• 6. Evaluating Multi-Turn Conversations
  • Evaluating at Different Levels
    • Session Level
    • Turn Level
    • Conversational Coherence and Memory
  • Practical Strategies for Multi-Turn Evaluation
    • Collecting Traces
    • Isolating Failures
    • Operationalizing Session-Level Success
    • User Simulators for Generating Test Conversations
    • Stratifying by Conversation Length
    • Perturbation Testing
  • Automated Evaluation of Multi-Turn Traces
  • Common Pitfalls
  • Putting It Together
• 7. Evaluating Retrieval-Augmented Generation
  • Overview
  • Synthetically Generating Query-Answer Pairs
    • Creating Harder Synthetic Queries
    • Filtering Synthetic Questions
  • Metrics for Retrieval Quality
    • Precision@k and Recall@k
    • Mean Reciprocal Rank
    • Normalized Discounted Cumulative Gain (NDCG@k)
  • Evaluating and Optimizing Chunking Strategies
    • Tuning Chunk Size and Overlap
    • Other Chunking Approaches
  • Evaluating Generation Quality
  • Common Pitfalls
  • Putting It Together
• 8. Evaluating Tool Use and Complex Agents
  • What Are Tools and Why Do LLMs Use Them?
  • The Spectrum of Agency
  • Evaluating Tool Calls
    • Stage 1: Tool Selection
    • Stage 2: Argument Generation
    • Stage 3: Tool Execution
    • Stage 4: Output Handling
    • Why Evaluate Each Stage Separately
  • Debugging Multi-Step Traces
    • Traceability
    • From Traces to Systemic Patterns
    • The Transition Failure Heatmap
    • Investigating Hotspots
    • Evolving the Heatmap over Time
  • Handling Complex Input Data
    • Images
    • Long Documents
    • PDF Documents
  • Adversarial and Security Evaluation
  • Common Pitfalls
  • Putting It Together
• IV. Production and Improvement
• 9. Continuous Integration and Deployment for LLM Agents
  • CI: Building a Safety Net Against Regressions
    • Common Mistakes with CI
    • Revisiting Prompts When Models Improve
  • CD and Online Monitoring
    • Observability
    • Running Automated Evaluators in Production
    • Guardrails
    • Judge Drift
  • The Continuous Improvement Flywheel
  • Common Pitfalls
  • Putting It Together
• 10. Interfaces for Human Review
  • The Case for Custom Review Interfaces
    • Essential Interface Elements
    • Features That Improve Review Speed and Quality
  • Case Study: EvalGen
  • Selecting Traces for Human Review
    • Random Sampling
    • Uncertainty Sampling
    • Failure-Driven Sampling
  • Navigating Trace Groups and Discovering Patterns
    • Clustering
    • Search and Similarity Tools
  • Integrating Human Review into Engineering Workflows
  • Example Walkthrough: Reviewing Real Estate Assistant Emails
    • Starting with a Spreadsheet
    • A Simple Custom UI
    • An Advanced Review Interface
  • Case Study: DocWrangler for Prompt Refinement
  • Common Pitfalls
  • Putting It Together
• 11. Data Analysis for Traces
  • EDA and Clustering to Surface Hypotheses
    • Clustering
    • Beyond Clustering
  • From Clusters to Confirmed Hypotheses
  • Semantic Search to Find Related Traces
  • Using AI Coding Assistants
  • Common Pitfalls
  • Putting It Together
• 12. Improving LLM Agents
  • Accuracy Optimization
    • Low Effort: Prompt Refinement
    • Medium Effort: Changing How the Agent Works
    • High Effort: Model-Level Changes
  • Cost Optimization
    • Match Model Size to Task Complexity
    • Cut Token Usage
    • Make Prompts Cache-Friendly
    • Batching
    • Distillation
    • Model Cascades
      • Approach 1: Logprobs for classification tasks
      • Approach 2: Verbalized confidence for open-ended tasks (use with caution)
      • Tuning the cascade threshold
    • Routing
  • Closing: The Lifecycle Continues
  • Choosing an Improvement Strategy
  • Next Steps
• Bibliography
• Index

pokaż cały spis treści

ukryj recenzje