Site Reliability Engineering. How Google Runs Production Systems. 2nd Edition Betsy Beyer, Chris Jones, Christof Leng

Dostępność produktu

Produkt został zweryfikowany pod kątem ułatwień dostępu więcej przeczytasz w deklaracji dostępności.

Spis treści książki

• Foreword
• Preface
  • Guidance for Specific Groups of Readers
    • Aspiring SREs: Context and Overviews
    • Practitioners: Technical Implementation and Operations
    • Leaders: Strategy, Culture, and Organization
  • Conventions Used in This Book
  • Using Code Examples
  • OReilly Online Learning
  • How to Contact Us
  • Acknowledgments
• I. SRE in Context
• 1. Introduction: Modeling What We Do and Why
  • SRE Is a Dedicated Voice for Reliability
  • A Software Engineering Approach to Operations
  • What Do SREs Do?
    • Sociotechnical Foundations
    • Situational Awareness
    • Continuous Preparedness
    • Infrastructure Design
    • Resilient Products
    • Emergent Reliability
    • AI and SRE
  • Conclusion
• 2. The Value of Reliability
  • Why SRE Is Right for Google
  • The Value of SRE
    • To Customers
    • To the CFO
    • To Other Internal Technical Teams
    • To Organizational Culture
    • To Business Growth
    • Within the SRE Team
  • The Cost of Unreliable Systems
    • Direct Costs
    • Opportunity Costs
    • Increased Toil
    • Reduced Velocity
    • Retrofitting
    • Penalties
    • Cost to Manage
    • System Atrophy over Time
  • Patterns of SRE for Different Organizations
    • Early-Stage and Smaller Organizations
      • Automated unit and integration test coverage
      • Automated continuous integration/continuous deployment pipeline
      • Observability
      • Incident response process
      • Design review
    • Growing and Mature Organizations
      • Strategic engineering decisions
      • Managing complexity
      • Incident response
      • Centrally maintained tools
  • What to Consider When Adopting SRE
  • Conclusion
• 3. Trends in Production
  • The Cloud Computing Era
    • Capacity Management 2.0
    • Managed Services and Open Standards
    • Shared State
  • Infrastructure Trends
  • The Evolution of Production Platforms
    • The Rise of the Self-Service Ecosystem (20162026)
    • The Trend Toward Intent-Based Coordination Systems
  • From Components to Context
    • The Mythical Single Pane of Glass
    • Distributed Tracing as a Critical Primitive
    • Quality over Quantity
    • Democratization of Access
  • Organizations Rethinking Reliability
  • Googles Production Environment
    • Platformization
    • Digital Sovereignty and Embedded Compliance
    • The AI Infrastructure Explosion
  • Conclusion
• II. Sociotechnical Foundations
• 4. The Cultural Context of SRE
  • The Case for Culture
    • Key Elements of a Reliability-Focused Culture
    • Fostering a Reliability-Focused Culture
      • Data-driven mindset
      • Collaboration and shared responsibility
        • Within a team
        • Within the broader SRE organization
        • Outside SRE
      • Learning and continuous improvement
      • Blamelessness and blame awareness
      • Psychological safety
      • Trust
      • Empowerment
  • DORAs Findings on Culture and Performance
    • The Empirical Link: Culture and Performance
    • SRE Practices, Reliability, and Practitioner Well-Being
    • Westrums Typology of Organizational Culture
    • Characteristics of a Generative Culture
  • How Learning Drives Culture
    • Benchmarking Culture Through Confidence
    • Tailoring Training to Different Audiences
  • Practical Considerations
    • The Brownfield Evolution: Cultivating Reliability in Existing Systems
      • Educate your stakeholders
      • Select your starting point
      • Assess maturity and discover baseline reliability
      • Have the crucial conversation
      • Define initial SLOs
      • Introduce blameless postmortems
      • Spread the word and iterate
    • The Greenfield Design: Engineering Reliability from Day Zero
      • Use SLOs as a design input
      • Engineer the system to meet the SLOs
      • Establish operational readiness from the start
    • Error Budgets: SLO-Driven Operations
    • Revised Processes: Moving Beyond Cadence and Promises
      • From cadenced deployments to SLO-gated deployments
      • Feature promises versus reliability guarantees
      • Shifting team focus and effort
  • Common SRE Implementation Challenges
    • Getting Executive Support
    • Managing Tooling
    • Managing Expectations
      • Expecting perfection
      • Setting arbitrary reliability targets
      • Thinking there is an easy button
      • Believing its one and done
  • Conclusion
• 5. SRE Hiring, Training, and Career Management
  • Hiring SREs
    • Attributes of a Successful SRE
    • The Hiring Process
    • Hiring New Graduates into SRE: A Case Study in Ireland
      • Identifying the challenges
      • Raising awareness and skill building
      • Strategies for forging new SREs from new graduates
      • Collaborating effectively with staffing teams
    • SRE Hiring Antipatterns
  • Training SREs
    • Setting SREs Up for Success as an Organization
      • The case for investment in training
      • Training options by org size
      • Identifying your target audiences and their training needs
      • Types of SRE training
        • Informal learning methods
        • Team and service-specific training
        • Project-based learning initiatives
        • Formal training programs
        • Orientation
        • Breakage scenarios
        • On-call curriculum
        • Continuous education
        • Manager training
    • Designing Effective Training Content
    • Training Program Operations
    • Learning as a Vehicle for Individual Growth
      • Navigating early challenges
      • Essential learning
      • Managing your individual learning journey
  • Career Management
    • The Keys to a Successful SRE Career
      • Breadth
      • Initiative
      • Teamwork
    • The Individual Contributors Career Path in SRE
    • Attrition and Retention
  • Conclusion
• 6. Organizational Structures for SRE
  • Organizational Structures That Support SRE
    • Centralization Versus Decentralization
      • Context and expertise
      • Ownership
      • Career path
      • Economies of scale
    • Hybrid Approaches
    • Making SRE Transformation Work
  • Spotifys Distributed Reliability Engagement Model
    • Reliability Ownership Without a Centralized SRE Organization
    • Incident Management as a Shared Capability
    • Scaling the Model
    • Trade-offs and Lessons
  • Team Design
    • Team Roles
    • On-Call Considerations
  • Collaboration
    • Product Development
    • SRE-Internal Collaboration
  • Maturity Matrices
    • Absent
    • Reactive
    • Proactive
    • Strategic
    • Visionary
  • Scorecards
    • Why Do They Fall Short?
    • Why Are Scorecards Hard to Get Rid Of?
    • Case Study: ProdExGoogles Production Excellence Program
    • Effective Scorecard Use
      • Case study: Large financial services firm
      • Case study: National retail chain
  • Conclusion
• 7. SRE Engagement Models
  • What Is an SRE Engagement Model?
    • Shared Understanding of Objectives
    • Effective Communication and Collaboration
    • Improved Understanding of Roles
  • The Maturity of SRE Engagements
    • The How: Intent
    • The What: Impact
    • The Why: Value
  • Scoping SRE Engagements
    • Kitchen Sink
    • Embedded
    • Product/Application
    • Infrastructure
    • Tools
    • Consulting
    • Blended
    • Scoping Considerations
  • SRE for Consumers Versus Enterprises
    • Engagement Commonalities
    • Engagement Differences
      • SLOs and error budgets
      • Incident management
      • Generic mitigations
      • System design
    • Organizing Teams
  • The Evolution of Google SREs Engagement Model
    • The Early Engagement Model (PRRs)
    • A Tiered Engagement Model
    • An Outcome-Focused Approach
  • Case Study: Outcomes-Focused Engagements in Persistent Disk
    • Old Model
    • Evolution
    • Results
  • Case Study: The Microsoft Azure Model
    • Brief History
    • Azure SREs Evolution
    • The Azure SRE Playbook
    • Scaling SRE in Azure
    • SRE Return on Investment
    • Takeaways
  • Disney Global SRE: Embedding Reliability Across the Magic
    • Embedded SRE: From Transactional to Relational
    • Aligning SRE with Business Goals
    • Centralized SRE: Scaling Reliability Without Losing Context
    • Engagements That Pivot, Not End
    • How Disney Organizes On-Call
    • What We Got Wrong: Leadership Alignment Must Be Ongoing
    • The Results
  • How SRE Collaborates with Its Partners
    • SRE and Product Management
    • Service Lifecycle Engagement
    • Beyond Production Readiness Reviews
  • Conclusion
• III. Situational Awareness and Continuous Preparedness
• 8. Service Level Objectives
  • The Basics of SLOs
    • External Versus Internal
    • Selecting SLIs and Thresholds
    • Recommendations for Beginners
    • Why Not 100%?
  • Pitfalls and Remedies in SLO Adoption
  • Critical User Journeys
  • Per-Customer SLOs
    • The Motivation for Per-Customer Reliability
    • How to Implement Per-Customer SLOs
      • Scope: Selecting workloads and SLIs
      • SLO thresholds and burn rate
    • Operationalizing Per-Customer SLOs
  • Implicit SLOs
  • Client-Side SLOs
    • Complex Instrumentation
    • High Noise Floor
    • Undefined Denominators
    • Trustworthy SLIs
  • SLO Tooling and Support Mechanisms
    • Automatic Alert Configurations on SLOs
    • SLO Taxonomy
  • Measuring SLO Failure Concentration
  • Weaknesses of the SLO Model
  • Additional Resources
  • Conclusion
• 9. Observability and Monitoring
  • From Monitoring to Observability: A Modern Approach
    • From Manual to Out-of-the-Box Dashboards
    • From Static Thresholds to Intelligent Alerting
    • From Investigation to Automatic Mitigation
    • Moving Forward
  • High-Availability Telemetry Versus Analytics Telemetry
    • High-Availability Telemetry
    • Analytics Telemetry
    • Design Considerations
      • High-availability monitoring systems
      • Analytics monitoring systems
    • Seamless User Experience
  • Ingestion Evolution
    • To Standardize or Not to Standardize
    • The Advent of OpenTelemetry
    • Case Study: Waze Telemetry over Time
    • Present Challenges and the Future
    • The Opportunity of eBPF with Observability
    • The Observability of AI Systems
  • Client-Side Telemetry
    • Why Client Monitoring
    • Implementation
    • Client Telemetry Architecture Overview
      • Device considerations
      • Privacy and security in client telemetry collection
      • Scaling
        • Covering different aspects of an application
        • Covering a user population
        • Scaling to the ecosystem
      • A final thought on client devices
  • Customer-Centric Monitoring
    • Measuring Experienced Reliability
    • Defense in Depth
      • Automated CUJ change supervision
      • CUJ-focused SLOs
      • Supporting infrastructure
  • How Customer-Centric Monitoring Improves the User Experience: A Google Maps Case Study
  • Observability DataValue Versus Cost
    • Observability Data Retention
    • Googles Strategic Approach to Data Retention
    • Horizontal Monitoring Data: A Centralized Approach
    • Adapting for the Future
  • Observability Data: Privacy, Compliance and Security Concerns
    • Compliance with Regulatory Requirements
    • Data Categorization
    • Compliance by Design
      • Data minimization
      • Data locale
      • Data wipeout
      • Access control
      • Encrypted data
    • E2E Observability Privacy and Security Considerations
  • Variation-Based Monitoring and Statistics
    • The Imperfect Lens of Monitoring
    • Understanding the Limits: Cardinality and Noise
    • Alerting with the Binomial Distribution
    • Composing SLOs
      • Reasoning about dependencies
      • Multiple fail-close dependencies
      • Increasing resilience with hedging
      • The important caveat: Correlation
    • Extending Confidence Intervals to Relative Metrics: Alerting on Latencies
  • Observability and AI
    • AI Approaches to Proactive Monitoring
    • AI for Toil Reduction
  • Conclusion
• 10. Incident Management and On-Call
  • Incidents, Outages, and Bugs
  • Foundations of On-Call
  • Structuring an On-Call Rotation
    • Optimal Paging Load
    • Hours of Availability
    • Response Time to Notifications
    • System Properties
    • Customer Requirements
    • Additional Implementation Considerations
    • Escalation Rotations
  • Incident Response Teams
    • Organization
      • Large organizations: Federating incident response
      • Small organizations: Bootstrapping the first IRT
    • Influence on Organizational Technical Policy
    • Potential Drawbacks
    • Alternatives
      • Consensus-driven response
      • Executives as incident commanders
  • The Lifecycle of an Incident
    • Detection
    • Triage
    • Investigation
      • Root cause analysis
      • OODA
      • Multiple hypotheses
    • Mitigation
    • Resolution
    • Learning
  • Collaboration and Communication
  • Training and Development
    • The Ghost in the Machine
      • Developing judgment
      • Staying calm in crisis
      • Adapting leadership skills
      • Cultivating systems thinking
  • The Role of AI in Incident Response
    • Detection and Triage
    • Outage Communication
    • Investigation
    • Hypothesis Generation
    • On-call Handoff
    • Autonomy in Production
    • A North Star for Agent Safety
  • Case Study: Incident Response and Management at Etsy
    • The Cultural Foundation
      • Blameless culture as bedrock
      • Etsys trust culture and psychological safety
    • Our Incident Response Evolution
    • Incident Response in Action
      • Detection and declaration
      • The LSE coordinator role
      • Cross-organizational collaboration
    • Our Postmortem Process
    • Lessons for Other Organizations
    • Takeaways
  • Conclusion
• 11. Learning from Incidents
  • Principles and Practices
    • Foundations
    • Operationalizing the Principles
    • Evolving the Learning from Incident Practices
  • Theoretical Research in the Field of Safety Science
    • The Swiss Cheese Model
    • Risk Management in a Dynamic Society
    • A Tale of Two Safeties
    • How Complex Systems Fail
    • Four Concepts for Resilience
    • Engineering a Safer World
  • Case Study: Paris Data Center Outage
    • Timeline
      • Tuesday, 25 April 2023
      • Wednesday, 26 April 2023
    • Interviews
    • Analysis
    • Action Items
    • Outcome
  • Conclusion
• 12. Safety Engineering for SRE with STPA and CAST
  • Limits of Traditional Causality Models
  • STAMP: A Modern Causality Model for Complex Systems
  • STPA and CAST: Analysis Methods Based on STAMP
    • STPA: Forward-Looking Hazard Analysis
    • CAST: Backward-Looking Accident Analysis
  • STPA: A Method for Anticipating Incidents
    • Step 1: Scope the Analysis
    • Step 2: Model the System
    • Step 3: Identify Unsafe Control Actions
    • Step 4: Build Causal Scenarios
  • Applying STPA to the Maps Data Updates Approval Process
    • The Google Maps Incident in Japan
    • Applying STPA
      • Step 1
      • Step 2
      • Step 3
      • Step 4
      • Results and return on investment
  • CAST: A New Method for Postmortems
    • The Case of the Misnamed Cities
    • Applying CAST
      • Step 1: Assemble basic information
      • Step 2: Model the control structure
      • Step 3: Analyze contributions of each controller
      • Step 4: Identify control structure flaws
      • Step 5: Create an improvement plan
  • Why Systems Theory and Safety Engineering Matters for SRE
    • Why STPA?
    • Why CAST?
    • Contrasting Traditional SRE Methods with STPA/CAST
  • Driving Adoption of STPA and CAST
    • Introducing STPA to Your Organization
    • Adopting CAST: Overcoming Tradition
      • Impact
      • Scalability
      • Efficiency
  • Conclusion
• IV. Infrastructure Design
• 13. Automation and Tooling
  • Levels of Automation
  • Scaling Automation
    • Example: Capacity Provisioning
    • Example: Incident Management
  • When to Automate and When Not To
    • The Pitfalls
    • A Framework for Decision-Making
      • Quantifiables
      • Nonquantifiables
    • Automation Examples
      • Automated hardware fault mitigation
      • Automated regional rollout system
      • Automating restarts for a memory leak
  • Incremental Automation
    • From Toil to Automation: A Practical Framework
    • Prioritization: The Automation Snowball
    • The Benefits of an Incremental Approach
    • Important Early Considerations
    • The Final Stretch and Fostering Collaboration
  • Coevolution of Distributed Systems and Automation at Scale
  • Toil
    • Why Toil Is a Problem
    • Measuring Toil and Defining Success
    • Measuring Operational Workload
    • What Success Looks Like
    • Effective Strategies for Addressing Toil
      • Identify issues early
      • Reward improvements
      • Recognize the real problem
      • Set realistic long-term goals
      • What doesnt work: Lessons and cautionary tales
  • When to Toil and When Not To
    • When to Toil
      • Toiling to mitigate
      • Toiling to learn
      • Toiling to apply human judgment
    • When Not to Toil
      • Toiling to close product gaps
      • Toiling that scales with the system
      • Toiling to limit production access
  • Human Factors in Automation
    • User Interface
    • Observability
    • Speed of Operations
    • Team Implications
  • Conclusion
• 14. Testing
  • Testing Basics
    • Models of Testing
    • Test Types Defined
      • Unit tests
      • Integration tests
      • System tests
        • Smoke tests
        • Performance tests
        • Regression tests
      • Resilience tests
      • Production tests
      • Other kinds of tests
    • Getting Started with Testing
      • Test infrastructure and best practices
      • Configuration basics
        • Configuration tests
        • Configuration drift
      • What to prioritize
      • Testing by exception
      • Monitoring pre-production
  • Advanced Testing
    • Canaries
      • Where canaries are particularly valuable
      • Canary analysis
    • Testing for Emergency Rollouts
    • Disaster Recovery and Resilience
      • Introduction to DiRT
      • Modeling service-level tests
  • Client-Focused Change Supervision
    • An Example Rollout
    • Implementation
      • The simple approach
      • Handling multiple hops
      • Attribution metrics
    • Using This Data
    • Challenges with This Approach
    • Benefits Realized by Google
  • Fault Injection Testing
    • Limits of Traditional Testing
    • Why Fault Injection?
    • Getting Started with Fault Injection Testing
    • Testing Infrastructure
    • Testing Dependencies
    • Modeling Large-Scale Outages
  • Formal Methods
    • Using Formal Methods
    • Two Classes of Formal Verifiers
      • Model checking
      • Theorem proving
    • Comparing the Two Approaches
      • Formal methods for testing
    • Use of Formal Methods at Google
      • Verifying behavior of network switches using P4-BAR
      • Formal verification in hardware design
      • Fiat Cryptography and BoringSSL
      • Analysis of C++ code via Tricorder, Clang-Tidy, and Clang Static Analyzer
    • Agentic Testing
      • Classic software authoring (level 1)
      • Agents generating code, data, content, etc. (level 2)
      • Agentic tool use (level 3)
  • Conclusion
• 15. Capacity Planning
  • Engineering Ourselves Out of Capacity Planning
  • Introduction to Core Concepts
    • Demand Forecasting
    • Quantify Uncertainty
    • Business-Driven Demand
    • Supply Planning
  • Designing for Resilience and Risk
    • Understanding Traffic Requirements
    • Traffic Management and Tagging
    • Capacity Isolation and Failure Domains
      • Good enough capacity isolation
      • Capacity isolation as a customer
        • Failure domains
        • Advanced failure domains
        • Putting it into practice
    • Overload Mitigation Strategies
      • Criticality and load shedding
      • Crumple zones
    • Planning for Outages and Incidents
  • Optimization and Resource Efficiency
    • Capacity Efficiency
    • Resource Pooling and Flex
      • The theory: Sum of peaks versus peak of sums
      • Flex: Googles resource pooling system
      • Scaling composite services
        • Autoscaling with Flex
        • The value of a unified abstraction
    • Reclaiming Stranded Capacity
  • Case Study: Throwing the Plan out the Window (Google Meet During COVID-19)
    • Capacity Estimation
    • Dependency Mapping
    • Bottleneck Mitigation
    • Graceful Degradation
    • Production Changes
    • Reevaluating Sustainability
    • Lessons Learned
  • Case Study: Scaling Google for the AI Age
    • Navigating Supply Constraints
    • Maintaining an End-to-End Perspective
    • Proactive Demand Shaping
    • Recovering Stranded Capacity
    • Increasing Elasticity via AI Agents
    • Lessons Learned
  • Capacity Planning Principles
  • Conclusion
• 16. Data Management
  • Legal and Ethical Considerations
    • Legal Concerns
      • Age of your users
      • Data retention
      • Data residency
      • Restrictions on usage of data
    • Ethical Concerns
  • Data Classification
    • Original Data Versus Derived Data
    • Identifiable Versus Nonidentifiable Data
    • High-Value Personal Data
  • The Hierarchy of Data Needs
    • Data Integrity
      • Data resiliency
      • Consistency checks
      • Checksums
      • Application-level verification
    • Data Security
      • Data access
      • Test data
    • Data Availability
    • Data Efficiency
      • Storage costs
      • Striping data across regions
  • What Failure Scenarios Do You Need to Consider?
    • Operator Error or Malicious Action
    • Bad Data Written by Applications
    • Data Destroyed or Corrupted by a Storage System
    • Data Unavailability or Loss Due to Hardware Failure
    • Data Unavailability or Loss Due to Region Failure
  • SLOs
    • Recovery Point Objective
    • Restore Time Objective
    • Restore Window Objective (RWO)
  • Replication
    • Considerations When Deciding Whether to Use Replication
    • Dealing with the Loss of a Replica
      • Single-leader topologies
      • Multileader topologies
  • Backups and Restores
    • Snapshots Versus Backups
    • Point-in-Time Recovery
    • Full, Incremental, or Differential?
      • Full backups
      • Incremental backups
      • Differential backups
    • Smart or Simple?
    • Restore Testing
  • Creating a Backup Plan
    • The Planning Framework
    • Keep Business Needs in Mind at All Times
    • Classify Your Data Sources
    • Set the Objectives for Each Bucket
    • Decide on the Backup Configuration for Each Bucket
    • Decide How You Will Perform Restore Testing
    • Write the Backup Plan
    • Implement the Backup Plan
  • Creating a Data Recovery Policy
  • Conclusion
• V. Resilient Products and Emergent Reliability
• 17. Designing for Resilience
  • The Importance of Resilience
    • Emerging Reliability
    • Higher and Lower Probability
    • User Expectations
  • Googles Principles for Reliability and Resilience
    • Enforcement Through Technology
    • The Role of Uniformity
  • Risk Management
    • The Role of Teams
    • The Role of Leadership
    • Probabilities
  • Change Management
    • Progressive Rollout
    • Change Concurrency
    • Change Supervision
    • Safe Mitigations
    • Risk Tolerance
    • Defense in Depth
  • Dependency Management
    • Representations and Controls
    • Resilience to Dependency Failures
    • Defense of Dependents
    • Managing Fallbacks
  • Failure Domains
    • Type of Failures and Domains
    • Aligning Failure Domains
    • Recovering Failure Domains
    • Preventing Loss
  • Overload Management
    • Avoiding Cascading Failures
    • Server-Side Controls
    • Client-Side Controls
    • Isolating Overload
    • Actionable Mitigations
  • Input Validation
    • Validation Outcomes
    • Types of Bad Input
    • Actionable Hardening
  • Data Resilience
    • Deglobalize Data Loss
    • Delete and Undelete Data Correctly
    • Independent Fallbacks for Critical Data
    • Recover Fast from Data Loss
  • Qualification
    • Environments and Shifting Left
    • Realistic Environments
    • Practical Validation
  • Emergency Mode
    • Incident Modes
    • For System Developers
    • For Incident Responders
    • For Leadership
    • Response CUJs
    • Training and Practicing
  • Outlook
    • Drivers and Costs
    • Taking a Snapshot
    • Making It Practical
    • Failures Without People
  • Conclusion
• 18. Platform Engineering in SRE
  • Platform Engineering
    • What Is a Platform?
    • Curating Solutions
    • Platform Policies
  • Platforms as Gardens
    • Complexity
    • Abstractions and Coupling
    • Control: Direct and Indirect
    • Choices and Cognitive Load
    • Incentives Rule Everything Around Us
    • Paths and Guardrails
    • Exceptions
    • Feedback
    • From Gardens to Ecosystems
  • Platforms as Ecosystems
    • Shifting Left and Shifting Down
    • Effectiveness
  • Frameworks
  • Seeing Like a State Machine
    • Legibility and Introspection
    • Remediation
    • Green Fields and Brown Fields
  • Conclusion
• VI. Outlook
• 19. SRE at the Tipping Point
  • The Software Development Lifecycle
    • The SDLC and Non-Functional Requirements
    • The SDLC Under Tension and Transition
    • The Social SDLC
  • Production Work in the Era of AI Dev
    • Rate of Change
    • Strategic Incoherence and Technical Debt
    • Versioning
    • Staging and Test Environments
    • Production Debugging
    • Incidents
    • Skill Erosion
    • Observability and Monitoring
    • Formal Methods
    • Leadership and the Interpretation of Value
  • Conclusion
• 20. How SRE Supports AI
  • Core ML Concepts
  • Delivering AI and ML Systems
    • Applications and Agents
    • Model Serving
    • Model Creation
  • How SRE Supports AI Workloads
    • Service Level Objectives in AI Systems
    • AI Reliability
    • Practical Observability and Capacity Challenges
    • Launching an AI Feature
  • How SRE Qualifies New Architectures and Models
    • Product Lifecycle Integration
      • Common testing strategy from EVT forward
      • Validation with first-party models
    • Continuous Hardware Qualification
    • What It Means for Accelerators to be SLO-Eligible
    • Qualification of Candidate Models
      • Performance and quality
      • Model lifecycle
    • AI Workflow Evaluation
  • Utilization, Efficiency, and Capacity Planning
    • Accelerator Utilization and the Loss Framework
      • Max utilization loss
      • Unrealized demand loss
      • Redundancy (N + 1) loss
      • Disaggregation imbalance loss
      • Load-balancing loss
      • Fragmentation loss
    • Centralized Fleet Management to Enable Agility
    • Accelerator Hardware Fungibility
    • Automating Capacity Planning
    • Scheduling Primitives
    • Job-Specific Optimizations
  • Energy Consumption and Sustainability
    • Shifting Landscape
    • Power Oversubscription
    • SLOs
    • Optimization Principles
    • Safety Principles
  • Security and Compliance
    • Threats and Challenges
    • Scaling Through the Platform
    • Enhanced Observability for AI Security
  • Conclusion
• 21. How AI Supports SRE
  • Complexity, Scale, and Risk
  • Structuring for Evolution
  • The Hierarchy of Production Health
  • Sociotechnical Foundations
    • The Role of Humans
    • The Safety and Reliability of AI Solutions
  • Situational Awareness
    • From Metrics to Meaning
    • Case Study: Detectr
    • AI Impact Assessment
  • Continuous Preparedness
    • Managing Incident Complexity
    • Mitigation Versus Root Cause Analysis
    • Learning from Failure
  • Infrastructure Design
    • Adaptive Change Management
    • Eliminating Toil
    • Verifying Resilience
  • Resilient Products
    • AI as an SRE Force Multiplier
    • AI Design and Launch Reviews
  • Emergent Reliability
    • The Path to Autonomy
    • Case Study: AI Operator
    • The Organizational Shift
  • Conclusion
• 22. SRE in Diverse Environments
  • SRE When Faced with Staffing and Organizational Constraints
    • The Solo SRE: Surviving and Scaling SRE in a Startup or as Part of a New, Small Team
      • Prioritization is key
      • The first 90 days
      • Leveraging AI for help
      • Playbooks
      • Scaling yourself
    • SRE Applied in Hybrid and Legacy Environments
    • Organizational, Process, and Access Constraints
      • Access and authority during incidents
      • Autonomy and authority to prevent incidents
    • Technical Constraints and Obstacles
      • Closed systems
      • Heterogeneity and silos
      • Humans in the loop
  • Real-World Examples
    • Case Study: Rapidly Scaling B2B SaaS Startup
      • The scenario
      • The adoption approach
      • The results
    • Case Study: Large Multinational Investment and Consumer Bank
      • The scenario
      • The adoption approach
      • The results
    • Case Study: Large-Scale Mortgage Lender
      • The scenario
      • The adoption approach
      • The results
    • SRE Managing Real-World Safety: An Example with Waymo
      • Traditional role of SRE on-call
      • SRE coordinating a large-scale crisis
      • Why SRE can manage a crisis and not just an outage
      • When SRE isand is nota good fit for crisis coordinator
  • SRE for Security and Privacy
    • IT Disruptions After Crowdstrike Software Update
    • Log4j Vulnerability
  • Conclusion
• 23. Future of SRE
  • What Wont Change
  • What Is Transforming
    • Shift Left Turns Into Shift Down
    • Risk as a Leading Indicator of Reliability
    • Evolving the Practice of SRE with AI
• A. Calculating the ROI of Training Investment
• B. SRE Team Charter Template
  • <Team Name> Charter
    • Mission
    • Scope
    • Services
    • Success
    • Values
• C. SRE Team Charter Team Example
  • gShoe SRE Charter
    • Mission
    • Scope
    • Services
    • Success
    • Values
• D. Example Production Meeting Minutes
• E. The Mathematics of Composing SLOs
  • Serial Dependencies
  • Parallel Dependencies (with Hedging)
• F. The Mathematics of Latency Alerting
• G. Modern Observability: Statistics and ML
  • A Statistical Approach for Error Rates
  • Handling Cyclical Metrics with Machine Learning
  • A Cost-Effective Application: Post-Alert Analysis
• Index

pokaż cały spis treści

ukryj recenzje