Building Medallion Architectures. Designing with Delta Lake and Spark Piethein Strengholt

(ebook) (audiobook) (audiobook)

Promocja Przejdź

Building Medallion Architectures. Designing with Delta Lake and Spark Piethein Strengholt - okladka książki

Building Medallion Architectures. Designing with Delta Lake and Spark Piethein Strengholt - audiobook MP3

Building Medallion Architectures. Designing with Delta Lake and Spark Piethein Strengholt - audiobook CD

Autor:: Piethein Strengholt
Wydawnictwo:: O'Reilly Media (Z chęcią przeczytam książkę w języku polskim)
Ocena:: Bądź pierwszym, który oceni tę książkę
Stron:: 396
Dostępne formaty:: ePub

Mobi

Ebook (160,65 zł najniższa cena z 30 dni)

~~199,00 zł~~ (-15%)
169,14 zł

Dodaj do koszyka lub Kup na prezent Kup 1-kliknięciem

(160,65 zł najniższa cena z 30 dni)

Przenieś na półkę

Do przechowalni

To deliver the insights that give them a competitive advantage, organizations increasingly turn to the proven Medallion architecture. Yet implementing a robust data architecture can be difficult, particularly when it comes to using the Medallion architecture's Bronze, Silver, and Gold layers—done wrong, it can hamper your ability to make data-driven decisions. This practical guide helps you build a Medallion architecture the right way with Azure Databricks and Microsoft Fabric.

Drawing on hands-on experience from the field, Piethein Strengholt demystifies common assumptions and complex problems you'll face when embarking on a new data architecture. Architects and engineers of all stripes will find answers to the most typical questions along with insights from real organizations about what's worked, what hasn't, and why.

You'll learn:

Learn how to build a Medallion architecture with Azure Databricks and Microsoft Fabric
Gain insights from three real case studies that illustrate practical field experience and lessons learned
Explore scaling considerations, including governance, security, generative AI, and more
Make informed decisions when designing or implementing new data architectures
Get proven patterns for success that align with broader organizational objectives

Wybrane bestsellery

O autorze książki

Piethein Strengholt jest głównym inspektorem danych w Microsoft Netherlands. Jest też aktywnym blogerem i regularnie komentuje najnowsze trendy w zarządzaniu danymi. Specjalizuje się w takich zagadnieniach jak siatka danych, nadzorowanie danych i strategie działania na dużą skalę.

Ebooka "Building Medallion Architectures. Designing with Delta Lake and Spark" przeczytasz na:

czytnikach Inkbook, Kindle, Pocketbook, Onyx Booxs i innych
systemach Windows, MacOS i innych

systemach Windows, Android, iOS, HarmonyOS
na dowolnych urządzeniach i aplikacjach obsługujących formaty: PDF, EPub, Mobi

Masz pytania? Zajrzyj do zakładki Pomoc »

Oceny i opinie klientów: Building Medallion Architectures. Designing with Delta Lake and Spark Piethein Strengholt

(0)

Szczegóły książki

ISBN Ebooka:: 978-10-981-7879-6, 9781098178796
Data wydania ebooka :: 2025-03-28 Data wydania ebooka często jest dniem wprowadzenia tytułu do sprzedaży i może nie być równoznaczna z datą wydania książki papierowej. Dodatkowe informacje możesz znaleźć w darmowym fragmencie. Jeśli masz wątpliwości skontaktuj się z nami sklep@helion.pl.
Język publikacji:: angielski
Rozmiar pliku ePub:: 12.1MB
Rozmiar pliku Mobi:: 41.8MB

Zgłoś erratę
Kategorie:
Big Data » Analiza danych

Dostępność produktu

Produkt nie został jeszcze oceniony pod kątem ułatwień dostępu lub nie podano żadnych informacji o ułatwieniach dostępu lub są one niewystarczające. Prawdopodobnie Wydawca/Dostawca jeszcze nie umożliwił dokonania walidacji produktu lub nie przekazał odpowiednich informacji na temat jego dostępności.

Spis treści książki

Foreword
Preface
- Who Should Read This Book
- Navigating This Book
- Conventions Used in This Book
- Using Code Examples
- OReilly Online Learning
- How to Contact Us
- Acknowledgments
I. Understanding the Medallion Framework
1. The Evolution of Data Architecture
- What Is a Medallion Architecture?
- A Brief History of Data Warehouse Architecture
  - OLTP Systems
  - Data Warehouses
  - The Staging Area
  - Inmon Methodology
  - Kimball Methodology
  - Key Takeaways from Traditional Data Warehouses
- A Brief History of Data Lakes
  - Hadoops Distributed File System
  - MapReduce
  - Apache Hive
    - External and internal tables
    - Hive Metastore
  - Spark Project
  - Moving Forward with Data Lakes
- A Brief History of Lakehouse Architecture
  - Founders of Spark
  - Emergence of Open Table Formats
  - The Rise of Lakehouse Architectures
  - Medallion Architecture and Its Practical Challenges
- Conclusion
2. Laying the Groundwork
- Foundational Preconditions
- Extra Landing Zones
- Raw Data
- Batch Processing
- Real-Time Data Processing
  - Spark Structured Streaming
  - Change Data Feed
  - Change Data Capture
  - Considerations and Learning Resources
- ETL and Orchestration Tools
- Managing Delta Tables
  - Z-Ordering
  - V-Ordering
  - Table Partitioning
  - Liquid Clustering
  - Compaction and Optimized Writes
  - DeltaLog
- Conclusion
3. Demystifying the Medallion Architecture
- The Three-Layered Design
- Bronze Layer
  - Processing Hierarchy
  - Processing Full Data Loads
  - Processing Incremental Data Loads
  - Data Historization Within the Bronze Layer
  - Schema Evolution and Management
  - MergeSchema and Schema Enforcement
  - Technical Validation Checks
  - Usage and Governance
  - The Bronze Layer in Practice
- Silver Layer
  - Cleaning Data Activities
  - Designing the Silver Layers Data Model
    - Conforming and renaming columns
    - Denormalization
    - Slowly changing dimensions
    - Surrogate keys
  - Harmonization with Other Sources
  - 3NF and Data Vault
  - Operational Querying and Machine Learning
  - Managing Overlapping Requirements
  - Automation Tasks
  - The Silver Layer in Practice
- Gold Layer
  - Star Schema
    - Loading the dimension tables
    - Loading the fact tables
    - Optimizing loads
  - Star Schema Design Nuances
  - Curated, Semantic, and Platinum Layers
  - One-Big-Table Design
  - Serving Layer
  - The Gold Layer in Practice
- Conclusion
II. Crafting the Medallion Layers
4. Building a Medallion Foundation with Microsoft Fabric
- Our Case Study: Oceanic Airlines
- Introducing Microsoft Fabric
  - Domains
  - Workspaces and Capacities
  - OneLake
  - Data Engineering with Spark
  - Data Warehousing with T-SQL
  - Other Fabric Workload Types
- Setting Up the Foundation
  - Setting up Capacities
  - Setting up Domains
  - Setting up Workspaces
  - Creating Lakehouses
  - Capacity Considerations
  - Domain Considerations
  - Workspace Considerations
  - Lakehouse Entities Considerations
  - Storage Account Considerations
- Conclusion
5. Construct the Bronze Layer
- Building the Data Pipeline
  - Deploying the AdventureWorks Sample Database
  - Set Up an Azure SQL Database Connection
  - Creating a New Data Pipeline
    - Building the ForEach loop
    - Configuring the CopyTable activity
  - Additional Considerations
- Implementation of Lakehouse Tables
  - Traverse Parquet Files to Managed Delta Tables
  - Using External Tables
  - Updating Tables with MERGE Operations
  - Spark Structured Streaming
    - Example with Azure Event Hubs
  - Using Change Data Capture
  - Navigating Data Handling Techniques
- Schema Management
  - Create Tables Without Defining Schemas
  - Define Schemas with the DataFrame API
  - SQL DDL Statements
  - YAML or JSON Configurations
  - Metadata-Driven Approach
  - Databricks Auto Loader
  - Third-Party Tools
  - Handling Schema Evolution
- Conclusion
6. Build the Silver Layer
- Quick Recap
- Implementation of a Metadata-Driven Approach
  - Implementation of the Metadata Store
  - Implementation of Dynamic Data Validations
  - Improvement Areas
- Data Cleansing
  - Implementation of Data Cleansing Tasks
  - Data Cleansing Considerations
  - Data Transformation Frameworks and Data Quality Tools
- Optimization of Query Performance with Denormalization
- Lightweight Enrichments
- Data Historization
- Optimization Jobs
- Orchestration with Apache AirFlow
- Final Recommendations
- Silver-Layer Data as a Product
- Conclusion
7. Streamline the Gold Layer
- Design of the Gold Layer
  - Transform Data Using a Star Schema
    - Creation of the Gold-layer tables
    - Creation of the dimensional table for address
    - Creation of the dimensional table for customer
    - Creation of the dimensional table for date
    - Creation of the dimensional table for product
    - Creation of the fact table for sales
  - Creation of the Semantic Model
  - Creation of the First Power BI Report
  - Creation of Task Flows
  - Enhancements for Gold-Layer Design
  - Microsoft Fabric in Practice
- Data Products
  - Introduction to data product guidelines
  - Types of data products
  - Data modeling guidance
  - Governance guidance
- Data Governance with Microsoft Purview
  - Microsoft Purview Design Considerations
    - Governance domains
    - Collections
    - Microsoft Purview data products
  - Guidance for Medallion Architectures
- Conclusion
III. Real-World Case Studies
8. Case Study: Data, Analytics and Business Strategy at AP Pension
- Medallion Architecture
- Other Considerations
- Final Recommendations
9. Case Study: Amadeus, a Tech Leader in the Travel Industry
- Medallion Architecture
- FinOps
- Data Models
- Data Contracts
- Data Governance
10. Case Study: Strategic Data Transformation at ZEISS
- Data Platform Evolution
- Medallion Architecture
- Data Products and Sharing
- Recommendations and Best Practices
IV. Scaling, Governance, and the Future of Medallion Architectures
11. Scaling the Medallion Architecture
- Decentralization of Data Management
  - Flexibility in Federation
  - Medallion Mesh
  - Number of Medallion Architectures
- Medallion Inner Architecture Variations
  - Separate Data Product Layers
  - Tailored Medallions Architectures
  - Adaptability of the Bronze Layer
  - Silver Layer Variations
  - Gold Layer Variations
  - Enterprise Data Models
  - Master Data Management
  - Reference Data Management
- Conclusion
12. Medallion Governance and Security
- Data Governance
  - Governance Within a Medallion Architecture
  - Unity Catalog
  - Medallion Architecture with Unity Catalog
- Data Contracts
  - Contracts Within a Catalog
  - Contracts Within a Metastore
  - Data Contracts Using YAML Files and GitOps
  - Other Data Contract Specifications
- Data Security and Access Management
- Conclusion
13. Future Medallion Architectures with Generative AI
- Unstructured Data Processing
  - Retrieval-Augmented Generation
  - Bronze Layer
  - Silver Layer
  - Gold Layer
- Integration of LLMs and Medallion Architectures
- Role of Agents
- Training and Fine-Tuning LLMs
- Future of Medallion Architectures
- Conclusion
Index