Building Event-Driven Microservices

Spis treści książki

• Preface
  • Conventions Used in This Book
  • OReilly Online Learning
  • How to Contact Us
  • Acknowledgments
• 1. Why Event-Driven Microservices
  • What Are Event-Driven Microservices?
  • Introduction to Domain-Driven Design and Bounded Contexts
    • Leveraging Domain Models and Bounded Contexts
    • Aligning Bounded Contexts with Business Requirements
  • Communication Structures
    • Business Communication Structures
    • Implementation Communication Structures
    • Data Communication Structures
    • Conways Law and Communication Structures
  • Communication Structures in Traditional Computing
    • Option 1: Make a New Service
    • Option 2: Add It to the Existing Service
    • Pros and Cons of Each Option
    • The Team Scenario, Continued
    • Conflicting Pressures
  • Event-Driven Communication Structures
    • Events Are the Basis of Communication
    • Event Streams Provide the Single Source of Truth
    • Consumers Perform Their Own Modeling and Querying
    • Data Communication Is Improved Across the Organization
    • Accessible Data Supports Business Communication Changes
  • Asynchronous Event-Driven Microservices
    • Example Team Using Event-Driven Microservices
  • Synchronous Microservices
    • Drawbacks of Synchronous Microservices
      • Point-to-point couplings
      • Dependent scaling
      • Service failure handling
      • API versioning and dependency management
      • Data access tied to the implementation
      • Distributed monoliths
      • Testing
    • Benefits of Synchronous Microservices
  • Summary
• 2. Event-Driven Microservice Fundamentals
  • Building Topologies
    • Microservice Topology
    • Business Topology
  • The Contents of an Event
  • The Structure of an Event
    • Unkeyed Event
    • Entity Event
    • Keyed Event
  • Materializing State from Entity Events
  • Event Data Definitions and Schemas
  • Microservice Single Writer Principle
  • Powering Microservices with the Event Broker
    • Event Storage and Serving
    • Additional Factors to Consider
      • Support tooling
      • Hosted services
      • Client libraries and processing frameworks
      • Community support
      • Long-term and tiered storage
  • Event Brokers Versus Message Brokers
    • Consuming from the Immutable Log
      • Consuming as an event stream
      • Consuming as a queue
    • Providing a Single Source of Truth
  • Managing Microservices at Scale
    • Putting Microservices into Containers
    • Putting Microservices into Virtual Machines
    • Managing Containers and Virtual Machines
  • Paying the Microservice Tax
  • Summary
• 3. Communication and Data Contracts
  • Event-Driven Data Contracts
    • Using Explicit Schemas as Contracts
    • Schema Definition Comments
    • Full-Featured Schema Evolution
    • Code Generator Support
    • Breaking Schema Changes
      • Accommodating breaking schema changes for entities
      • Accommodating breaking schema changes for events
  • Selecting an Event Format
  • Designing Events
    • Tell the Truth, the Whole Truth, and Nothing but the Truth
    • Use a Singular Event Definition per Stream
    • Use the Narrowest Data Types
    • Keep Events Single-Purpose
      • Example: Overloading event definitions
    • Minimize the Size of Events
    • Involve Prospective Consumers in the Event Design
    • Avoid Events as Semaphores or Signals
  • Summary
• 4. Integrating Event-Driven Architectures with Existing Systems
  • What Is Data Liberation?
    • Compromises for Data Liberation
    • Converting Liberated Data to Events
  • Data Liberation Patterns
  • Data Liberation Frameworks
  • Liberating Data by Query
    • Bulk Loading
    • Incremental Timestamp Loading
    • Autoincrementing ID Loading
    • Custom Querying
    • Incremental Updating
    • Benefits of Query-Based Updating
    • Drawbacks of Query-Based Updating
  • Liberating Data Using Change-Data Capture Logs
    • Benefits of Using Data Store Logs
    • Drawbacks of Using Data Base Logs
  • Liberating Data Using Outbox Tables
    • Performance Considerations
    • Isolating Internal Data Models
    • Ensuring Schema Compatibility
      • Benefits of event-production with outbox tables
      • Drawbacks of event production with outbox tables
    • Capturing Change-Data Using Triggers
      • Benefits of using triggers
      • Drawbacks of using triggers
  • Making Data Definition Changes to Data Sets Under Capture
    • Handling After-the-Fact Data Definition Changes for the Query and CDC Log Patterns
    • Handling Data Definition Changes for Change-Data Table Capture Patterns
  • Sinking Event Data to Data Stores
  • The Impacts of Sinking and Sourcing on a Business
  • Summary
• 5. Event-Driven Processing Basics
  • Composing Stateless Topologies
    • Transformations
    • Branching and Merging Streams
  • Repartitioning Event Streams
    • Example: Repartitioning an Event Stream
  • Copartitioning Event Streams
    • Example: Copartitioning an Event Stream
  • Assigning Partitions to a Consumer Instance
    • Assigning Partitions with the Partition Assignor
    • Assigning Copartitioned Partitions
    • Partition Assignment Strategies
      • Round-robin assignment
      • Static assignment
      • Custom assignment
  • Recovering from Stateless Processing Instance Failures
  • Summary
• 6. Deterministic Stream Processing
  • Determinism with Event-Driven Workflows
  • Timestamps
    • Synchronizing Distributed Timestamps
    • Processing with Timestamped Events
      • Example: Selecting order of events when processing multiple partitions
  • Event Scheduling and Deterministic Processing
    • Custom Event Schedulers
    • Processing Based on Event Time, Processing Time, and Ingestion Time
    • Timestamp Extraction by the Consumer
    • Request-Response Calls to External Systems
  • Watermarks
    • Watermarks in Parallel Processing
  • Stream Time
    • Stream Time in Parallel Processing
  • Out-of-Order and Late-Arriving Events
    • Late Events with Watermarks and Stream Time
    • Causes and Impacts of Out-of-Order Events
      • Sourcing from out-of-order data
      • Multiple producers to multiple partitions
    • Time-Sensitive Functions and Windowing
      • Tumbling windows
      • Sliding windows
      • Session windows
  • Handling Late Events
  • Reprocessing Versus Processing in Near-Real Time
  • Intermittent Failures and Late Events
  • Producer/Event Broker Connectivity Issues
  • Summary and Further Reading
• 7. Stateful Streaming
  • State Stores and Materializing State from an Event Stream
  • Recording State to a Changelog Event Stream
  • Materializing State to an Internal State Store
    • Materializing Global State
    • Advantages of Using Internal State
      • Scalability requirements are offloaded from the developer
      • High-performance disk-based options
      • Flexibility to use network-attached disk
    • Disadvantages of Using Internal State
      • Limited to using runtime-defined disk
      • Wasted disk space
    • Scaling and Recovery of Internal State
      • Using hot replicas
      • Restoring and scaling from changelogs
      • Restoring and scaling from input event streams
  • Materializing State to an External State Store
    • Advantages of External State
      • Full data locality
      • Technology
    • Drawbacks of External State
      • Management of multiple technologies
      • Performance loss due to network latency
      • Financial cost of external state store services
      • Full data locality
    • Scaling and Recovery with External State Stores
      • Using the source streams
      • Using changelogs
      • Using snapshots
  • Rebuilding Versus Migrating State Stores
    • Rebuilding
    • Migrating
  • Transactions and Effectively Once Processing
    • Example: Stock Accounting Service
    • Effectively Once Processing with Client-Broker Transactions
    • Effectively Once Processing Without Client-Broker Transactions
      • Generating duplicate events
      • Identifying duplicate events
      • Guarding against duplicates
      • Maintaining consistent state
  • Summary
• 8. Building Workflows with Microservices
  • The Choreography Pattern
    • A Simple Event-Driven Choreography Example
    • Creating and Modifying a Choreographed Workflow
    • Monitoring a Choreographed Workflow
  • The Orchestration Pattern
    • A Simple Event-Driven Orchestration Example
    • A Simple Direct-Call Orchestration Example
    • Comparing Direct-Call and Event-Driven Orchestration
    • Creating and Modifying an Orchestration Workflow
    • Monitoring the Orchestration Workflow
  • Distributed Transactions
    • Choreographed Transactions: The Saga Pattern
      • Choreography Example
    • Orchestrated Transactions
  • Compensation Workflows
  • Summary
• 9. Microservices Using Function-as-a-Service
  • Designing Function-Based Solutions as Microservices
    • Ensure Strict Membership to a Bounded Context
    • Commit Offsets Only After Processing Has Completed
      • When the function has completed its processing
      • When the function has first started
    • Less Is More
  • Choosing a FaaS Provider
  • Building Microservices Out of Functions
  • Cold Start and Warm Starts
  • Starting Functions with Triggers
    • Triggering Based on New Events: The Event-Stream Listener
    • Triggering Based on Consumer Group Lag
    • Triggering on a Schedule
    • Triggering Using Webhooks
    • Triggering on Resource Events
  • Performing Business Work with Functions
  • Maintaining State
  • Functions Calling Other Functions
    • Event-Driven Communication Pattern
    • Direct-Call Pattern
      • Choreography and asychronous function calling
      • Orchestration and synchronous function calling
        • Event streamtriggered processing
        • Queue-triggered event processing
  • Termination and Shutdown
  • Tuning Your Functions
    • Allocating Sufficient Resources
    • Batch Event-Processing Parameters
  • Scaling Your FaaS Solutions
  • Summary
• 10. Basic Producer and Consumer Microservices
  • Where Do BPCs Work Well?
    • Integration with Existing and Legacy Systems
      • Example: Sidecar pattern
    • Stateful Business Logic That Isnt Reliant Upon Event Order
      • Example: Book publishing
    • When the Data Layer Does Much of the Work
    • Independent Scaling of the Processing and Data Layer
      • Example: Perform aggregations on event data to build user engagement profiles
  • Hybrid BPC Applications with External Stream Processing
    • Example: Using an External Stream-Processing Framework to Join Event Streams
  • Summary
• 11. Heavyweight Framework Microservices
  • A Brief History of Heavyweight Frameworks
  • The Inner Workings of Heavyweight Frameworks
  • Benefits and Limitations
  • Cluster Setup Options and Execution Modes
    • Use a Hosted Service
    • Build Your Own Full Cluster
    • Create Clusters with CMS Integration
      • Deploying and running the cluster using the CMS
      • Specifying resources for a single job using the CMS
  • Application Submission Modes
    • Driver Mode
    • Cluster Mode
  • Handling State and Using Checkpoints
  • Scaling Applications and Handling Event Stream Partitions
    • Scaling an Application While It Is Running
    • Scaling an Application by Restarting It
    • Autoscaling Applications
  • Recovering from Failures
  • Multitenancy Considerations
  • Languages and Syntax
  • Choosing a Framework
  • Example: Session Windowing of Clicks and Views
  • Summary
• 12. Lightweight Framework Microservices
  • Benefits and Limitations
  • Lightweight Processing
  • Handling State and Using Changelogs
  • Scaling Applications and Recovering from Failures
    • Event Shuffling
    • State Assignment
    • State Replication and Hot Replicas
  • Choosing a Lightweight Framework
    • Apache Kafka Streams
    • Apache Samza: Embedded Mode
  • Languages and Syntax
  • Stream-Table-Table Join: Enrichment Pattern
  • Summary
• 13. Integrating Event-Driven and Request-Response Microservices
  • Handling External Events
    • Autonomously Generated Events
    • Reactively Generated Events
  • Handling Autonomously Generated Analytical Events
  • Integrating with Third-Party Request-Response APIs
  • Processing and Serving Stateful Data
    • Serving Real-Time Requests with Internal State Stores
    • Serving Real-Time Requests with External State Stores
      • Serving requests via the materializing event-driven microservice
      • Serving requests via a separate microservice
  • Handling Requests Within an Event-Driven Workflow
    • Processing Events for User Interfaces
      • Example: Newspaper publishing workflow (approval pattern)
      • Separating the editor and advertiser approval services
  • Micro-Frontends in Request-Response Applications
  • The Benefits of Microfrontends
    • Composition-Based Microservices
    • Easy Alignment to Business Requirements
  • Drawbacks of Microfrontends
    • Potentially Inconsistent UI Elements and Styling
    • Varying Microfrontend Performance
    • Example: Experience Search and Review Application
  • Summary
• 14. Supportive Tooling
  • Microservice-to-Team Assignment System
  • Event Stream Creation and Modification
  • Event Stream Metadata Tagging
  • Quotas
  • Schema Registry
  • Schema Creation and Modification Notifications
  • Offset Management
  • Permissions and Access Control Lists for Event Streams
  • State Management and Application Reset
  • Consumer Offset Lag Monitoring
  • Streamlined Microservice Creation Process
  • Container Management Controls
  • Cluster Creation and Management
    • Programmatic Bringup of Event Brokers
    • Programmatic Bringup of Compute Resources
    • Cross-Cluster Event Data Replication
    • Programmatic Bringup of Tooling
  • Dependency Tracking and Topology Visualization
    • Topology Example
  • Summary
• 15. Testing Event-Driven Microservices
  • General Testing Principles
  • Unit-Testing Topology Functions
    • Stateless Functions
    • Stateful Functions
  • Testing the Topology
  • Testing Schema Evolution and Compatibility
  • Integration Testing of Event-Driven Microservices
  • Local Integration Testing
    • Create a Temporary Environment Within the Runtime of Your Test Code
    • Create a Temporary Environment External to Your Test Code
    • Integrate Hosted Services Using Mocking and Simulator Options
    • Integrate Remote Services That Have No Local Options
  • Full Remote Integration Testing
    • Programmatically Create a Temporary Integration Testing Environment
      • Populating with events from production
      • Populating with events from a curated testing source
      • Creating mock events using schemas
    • Testing Using a Shared Environment
    • Testing Using the Production Environment
  • Choosing Your Full-Remote Integration Testing Strategy
  • Summary
• 16. Deploying Event-Driven Microservices
  • Principles of Microservice Deployment
  • Architectural Components of Microservice Deployment
    • Continuous Integration, Delivery, and Deployment Systems
    • Container Management Systems and Commodity Hardware
  • The Basic Full-Stop Deployment Pattern
  • The Rolling Update Pattern
  • The Breaking Schema Change Pattern
    • Eventual Migration via Two Event Streams
    • Synchronized Migration to the New Event Stream
  • The Blue-Green Deployment Pattern
  • Summary
• 17. Conclusion
  • Communication Layers
  • Business Domains and Bounded Contexts
  • Shareable Tools and Infrastructure
  • Schematized Events
  • Data Liberation and the Single Source of Truth
  • Microservices
  • Microservice Implementation Options
  • Testing
  • Deploying
  • Final Words
• Index

pokaż cały spis treści

ukryj recenzje