Data Lake, Warehouse & Lakehouse
The platform beneath ETRM analytics decides whether they succeed. This course teaches lake, warehouse, and lakehouse patterns, then designs a governed medallion architecture, bronze to silver to gold, that the whole trading estate can build on.
The pipeline at the heart of this course
Why architecture decides success
The same analytics ambition succeeds on one platform and stalls on another, and the difference is architecture. This course is about the platform beneath ETRM analytics: where data lands, how it is conformed, how it is served, and how all of that is governed. Get the architecture right and adding a new mart is a day's work; get it wrong and every new question is a fresh integration project.
You learn the three dominant patterns, data lake, data warehouse, and lakehouse, not as buzzwords but as engineering trade-offs. The lake offers cheap, flexible storage but weak guarantees; the warehouse offers strong guarantees but rigidity and cost; the lakehouse aims to combine the storage economics of the lake with the transactional guarantees of the warehouse, which is why it has become the default for platforms like this.
The course frames the platform choice as the decision that quietly determines everything downstream, because it does. A well-architected platform makes the next mart, the next feed, the next analytical question a routine addition; a poorly chosen one makes each of them a fresh integration project. You learn to reason about lake, warehouse, and lakehouse as trade-offs against the specific demands of trading data rather than as competing fashions.
Open table formats and ACID on the lake
The lakehouse is made possible by open table formats that bring database guarantees to files in object storage: ACID transactions, so a half-written batch never corrupts a table; schema enforcement, so bad data is rejected at write; and time travel, so any past version of a table can be queried. The course examines these formats and what they make possible for a trading platform that must be both flexible and trustworthy.
Time travel in particular matters for ETRM, because it lets the platform answer as-of questions, what did this mart contain at end-of-day, without bespoke history tables. It complements the bitemporal patterns from earlier courses and underpins the audit requirements the STP course formalizes.
Open table formats are the enabling technology the course spends real time on, because they are what make the lakehouse more than a slogan. ACID transactions, schema enforcement, and time travel on cheap object storage are exactly the guarantees a trading platform needs, and understanding how they work is what lets you use them well and avoid their pitfalls. The course connects these capabilities directly to the bitemporal and audit requirements the rest of the program established.
The medallion architecture
The organizing pattern is the medallion: bronze for raw ingested data, exactly as it arrived; silver for conformed, cleaned, canonical data; and gold for the business-ready marts that positions, P&L, and dashboards consume. Each layer has a clear contract, and data only moves upward through explicit transformations, so lineage is traceable and reprocessing is safe.
You design each layer for ETRM: bronze ingesting trade feeds, FpML, market data, and reference feeds; silver holding canonical trades, golden reference data, and bitemporal curves; gold serving positions, P&L, exposure, and dashboard marts. The layering is what keeps a growing platform comprehensible.
The medallion pattern is the organizing idea, and the course shows how it keeps a growing platform comprehensible. Bronze, silver, and gold each have a clear contract, data only moves upward through explicit transformations, and lineage is traceable end to end, so that as the platform grows it stays a system rather than becoming a swamp. You learn to design each layer specifically for ETRM, with the feeds, canonical models, and marts the platform actually needs.
Partitioning, formats, and performance
Under the architecture sit the physical choices that determine performance and cost: partitioning strategy, so queries scan only what they need; columnar formats and compression, so storage is cheap and scans are fast; and file layout, so small-file problems do not cripple the platform. The course is concrete about these because they are where lakehouse projects most often disappoint.
You also learn the compute side: separating storage from compute so each scales independently, choosing engines for batch and interactive workloads, and the FinOps discipline that keeps cost proportional to value, a theme that connects to the wider platform-cost concerns of the trading estate.
The physical choices under the architecture are where lakehouse projects most often disappoint, so the course is concrete about them. Partitioning, file layout, columnar formats, and the small-file problem determine whether the platform performs or crawls, and getting them wrong can erase the benefits of every good high-level decision. You learn these details because they are the difference between a lakehouse that works and one that merely exists on a diagram.
Governance and a reference architecture
A platform without governance becomes a swamp. The course integrates catalog, lineage, data quality, and access control as first-class layers spanning bronze to gold, so that every dataset is discoverable, its origins traceable, and its access controlled. This is what lets a trading platform be both open to analytics and safe for regulated data.
You leave with a reference ETRM lakehouse architecture you can defend end to end. The artifact below sketches the medallion layers and a silver-layer transformation with schema enforcement.
Governance is what the course insists must be built in rather than added later, because a platform without it becomes unusable at exactly the moment it becomes valuable. Catalog, lineage, quality, and access control spanning every layer are what let a trading platform be both open to analytics and safe for regulated data. You finish with a reference architecture that is not just performant but discoverable, traceable, and controlled, which is what production actually requires.
The medallion layers as contracts
The power of the medallion architecture is that each layer is a contract. Bronze promises only to preserve raw data faithfully, exactly as it arrived, so that nothing is ever lost and any later reprocessing can start from the true source. Silver promises conformed, cleaned, canonical data that downstream consumers can trust without knowing where it came from. Gold promises business-ready marts shaped for specific consumption, positions, P&L, exposure, dashboards.
Because each layer has a clear contract, data only ever moves upward through explicit, testable transformations, and the lineage from a gold number back to its bronze source is traceable. This is what keeps a growing platform comprehensible: a new requirement adds a gold mart or a silver model, but it does not tangle the whole system, because the layering constrains how data flows.
The course designs each layer specifically for ETRM, and shows how the medallion pattern interacts with the open table formats that give the lake database-like guarantees. The combination, clear layer contracts plus ACID and time travel underneath, is what makes a lakehouse suitable for regulated trading data rather than merely fashionable.
Performance, cost, and the physics of the platform
Beneath the architecture sits physics, and ignoring it is how lakehouse projects disappoint. Query performance is largely decided by how much data a query must scan, which is decided by partitioning and file layout. Get these right and a query touches only the relevant slice; get them wrong and every query scans everything, and the platform feels slow no matter how much compute you throw at it. The course is concrete about partitioning strategy, file sizing, and the small-file problem that quietly cripples many lakes.
Cost follows the same physics. Separating compute from storage lets each scale independently and lets you pay for compute only when you use it, but it also makes cost a function of how efficiently your queries scan and how well your data is laid out. The course introduces the FinOps discipline of keeping cost proportional to value, so that the platform stays affordable as it grows.
You leave able to reason about the platform not just as boxes on a diagram but as a physical system with performance and cost consequences, and to make the layout and format choices that keep it fast and economical at scale.
Where this course sits, and what it unlocks
Architecture is the eighth course, the platform beneath everything built so far. It takes the canonical trades, reference data, curves, positions, and P&L from the earlier courses and organizes them into a governed medallion lakehouse that the real-time, dashboard, and STP courses all build on. It is where the pieces stop being separate schemas and become one coherent platform.
The medallion layering and open table formats you learn here provide the ACID guarantees, time travel, and governance that regulated trading data demands, and they set up the event-sourcing and streaming courses that add the real-time and audit dimensions on top.
Common platform failures, and how to avoid them
The most common lakehouse failure is the swamp: a lake with no governance, where data lands but no one can find it, trust it, or trace its lineage, so it slowly becomes unusable. Catalog, lineage, quality, and access control, applied as first-class layers, are what keep a lake a lakehouse rather than a swamp.
A second failure is ignoring physical layout, poor partitioning and small files, so that every query scans everything and the platform feels slow no matter the compute. A third is uncontrolled cost, where separating compute and storage without FinOps discipline produces surprising bills. The course addresses each with concrete layout, format, and governance practices.
By avoiding these patterns you get a platform that stays fast, affordable, governed, and trustworthy as it grows, the foundation the rest of the bundle assembles on.
A worked example: a trade feed from bronze to gold
Follow a trade feed up the medallion layers. It lands in bronze exactly as it arrived, raw and unaltered, so nothing is lost and the true source is always available for reprocessing. A silver transformation conforms it into the canonical model, casting types with proper precision, applying schema enforcement so malformed rows are quarantined rather than silently dropped, and joining in governed reference data.
From silver, a gold transformation aggregates the conformed trades into a position mart, shaped for the dashboards and P&L that consume it. At each step the transformation is explicit and testable, and the lineage from the gold position back to the bronze source is traceable. The open table format underneath provides ACID guarantees, so a half-written batch never corrupts a table, and time travel, so any past version of any layer is queryable.
This one feed shows the medallion pattern working end to end, and why the layering matters: each layer's contract keeps the platform comprehensible and its data traceable as it grows.
Operational realities: governance, cost, and scale
Running a lakehouse in production is a governance and cost discipline as much as an engineering one. The course covers keeping the platform governed as it grows, catalog so datasets are discoverable, lineage so their origins are traceable, quality so they are trustworthy, and access control so regulated data is protected, because without these a growing lake becomes an unusable swamp.
Cost and performance are managed through physical layout and FinOps: partitioning and file sizing so queries scan only what they need, and cost monitoring so that the flexibility of separated compute and storage does not produce surprising bills. The course leaves you able to run the platform, not just design it, keeping it fast, affordable, and governed at scale.
Design trade-offs: lake, warehouse, or lakehouse
The platform choice is a genuine trade-off, not a fashion. A data warehouse gives strong guarantees, transactions, schema, fast structured queries, but at higher cost and with rigidity that makes new data types and exploratory work painful. A data lake gives cheap, flexible storage that accepts anything, but without guarantees it degrades into a swamp where nothing can be trusted. Each solves one problem by creating the other's.
The lakehouse exists to refuse that choice, bringing transactional guarantees, schema enforcement, and time travel to cheap object storage through open table formats. The course teaches why this has become the default for platforms like ETRM: the workload genuinely needs both the warehouse's trustworthiness, for regulated position and P&L data, and the lake's flexibility, for the variety of trade, market, and reference feeds. But it also teaches the lakehouse's own trade-offs, the operational maturity it demands and the small-file and layout pitfalls that can erase its performance, so the choice is made with eyes open.
Carrying the platform into the capstone
The lakehouse is where the STP flow lives at rest. Bronze receives the raw feeds the capture and ingestion stages produce, silver holds the canonical trades and conformed reference and curve data the flow operates on, and gold serves the positions, P&L, and exposures the control and dashboard stages consume. The medallion layering is what keeps the capstone's data comprehensible as it flows.
Time travel, which the lakehouse provides, is also what lets the capstone answer as-of questions about any stage without bespoke history tables, complementing the event-sourced backbone. So the platform is not a backdrop to the capstone; it is the substrate the whole flow reads from and writes to, and its governance layer is what makes the flow's data discoverable, traceable, and controlled end to end.
Performance and scale considerations
The lakehouse's performance is decided by physical choices that are easy to get wrong, so the course treats them as core material rather than tuning trivia. Partitioning by the dimensions queries filter on, controlling file sizes to avoid the small-file problem, and clustering data so scans read only what they need are what determine whether the platform performs or crawls. The course is concrete because these are where lakehouse projects most often disappoint in practice.
Separating storage from compute is the architectural lever that lets the platform scale each independently, and the course shows how to exploit it: sizing compute to the workload, isolating heavy analytical jobs from interactive ones, and keeping cost proportional to value through FinOps discipline. The aim is a platform that stays fast and affordable as both data and query load grow.
Testing and validating the platform
A data platform is testable at the layer boundaries the medallion pattern creates, and the course uses them. Schema enforcement between bronze and silver is a test in itself, quarantining data that does not conform, and the course shows how to assert that conformed silver data matches the canonical model and that gold marts reconcile to their silver sources. These layer contracts turn the architecture into something whose correctness can be checked rather than assumed.
The course also teaches validating the platform's guarantees directly: confirming that ACID transactions prevent partial writes from being visible, that time travel returns the expected historical version, and that access controls actually restrict what they claim to. You finish able to demonstrate that the platform delivers the guarantees the rest of the program relies on, which is what makes it safe to build on.
How it works, concretely
# Silver: conform raw bronze trades into the canonical model, with
# schema enforcement so bad rows are quarantined, not silently dropped.
from pyspark.sql import functions as F
bronze = spark.read.format("delta").load("s3://etrm/bronze/trades")
canonical = (bronze
.withColumn("trade_id", F.col("raw_id").cast("long"))
.withColumn("quantity", F.col("qty").cast("decimal(20,6)"))
.withColumn("price", F.col("px").cast("decimal(20,8)"))
.withColumn("trade_date", F.to_date("trade_dt"))
.filter(F.col("quantity").isNotNull() & F.col("price").isNotNull()))
# Rows that fail the contract go to a quarantine table for repair
quarantine = bronze.subtract(canonical.select(bronze.columns))
quarantine.write.format("delta").mode("append").save("s3://etrm/quarantine/trades")
(canonical.write
.format("delta")
.option("mergeSchema", "false") # enforce the silver schema, no drift
.mode("append")
.save("s3://etrm/silver/trades"))
# Delta gives ACID + time travel: any past version of silver is queryable.Chapters 141 to 160
Twenty sequential chapters. Within the full bundle these are chapters 141 through 160 of 240.
- 141Why platform architecture decides analytics successWhy the platform choice decides whether analytics scale or stall. The platform choice determines whether adding a mart is a day's work or a fresh project.
- 142Data lake fundamentals and object storageData-lake fundamentals and cheap flexible object storage. Object storage gives cheap, flexible capacity but, alone, weak guarantees.
- 143Data warehouse fundamentals and MPPData-warehouse fundamentals and massively parallel processing. The warehouse gives strong guarantees and fast structured queries at higher cost and rigidity.
- 144The lakehouse pattern and open table formatsThe lakehouse pattern and open table formats. The lakehouse aims to combine the lake's economics with the warehouse's guarantees.
- 145Medallion architecture: bronze, silver, goldOrganizing data as bronze, silver, and gold layers. The medallion layers give each stage a clear contract and keep lineage traceable.
- 146Batch vs streaming ingestion patternsChoosing batch or streaming ingestion per source. Batch and streaming ingestion suit different data, and the platform supports both.
- 147Schema-on-read vs schema-on-writeWeighing schema-on-read against schema-on-write. The schema-on-read versus schema-on-write choice shapes flexibility and safety.
- 148Partitioning, clustering, and file layoutPartitioning, clustering, and file layout for performance. Partitioning and file layout are where lakehouse performance is won or lost.
- 149Columnar formats: Parquet and ORCUsing columnar formats like Parquet and ORC. Columnar formats are what make large scans fast and storage cheap.
- 150Open table formats: Delta, Iceberg, HudiComparing Delta, Iceberg, and Hudi table formats. Open table formats are what bring database guarantees to files in object storage.
- 151ACID transactions on the lakeGetting ACID transactions on the data lake. ACID transactions are what stop a half-written batch from ever corrupting a table.
- 152Time travel and versioned dataUsing time travel and versioned data. Time travel lets any past version of a table be queried without bespoke history tables.
- 153The conformed layer and shared dimensionsBuilding the conformed layer of shared dimensions. The conformed layer is where shared dimensions make marts consistent with each other.
- 154Gold marts for position, P&L, and riskServing gold marts for position, P&L, and risk. Gold marts are the business-ready tables positions, P&L, and dashboards consume.
- 155Compute engines and separation of storageSeparating compute from storage so each scales. Separating storage from compute lets each scale independently and cost-effectively.
- 156Data governance and catalog integrationIntegrating catalog and governance across the platform. Catalog and lineage are what keep a growing platform discoverable and traceable.
- 157Cost, performance, and FinOps trade-offsManaging cost, performance, and FinOps trade-offs. FinOps discipline keeps platform cost proportional to the value it delivers.
- 158Security, access control, and row-level policyApplying security, access control, and row-level policy. Row-level policy is what makes regulated trading data safe to open up for analytics.
- 159Migration patterns from legacy warehousesMigrating from legacy warehouses safely. Migration patterns show how to move off legacy warehouses without a big-bang risk.
- 160A reference ETRM lakehouse architectureAssembling a reference ETRM lakehouse architecture. You finish with a reference ETRM lakehouse architecture you can defend end to end.
What you'll be able to do
- Choose between lake, warehouse, and lakehouse for a given need
- Design a governed medallion architecture with clear layer contracts
- Use open table formats for ACID and time travel on the lake
- Lay out data for performance and control cost
- Integrate catalog, lineage, and access control across the platform
Audience & prerequisites
Platform and data engineers choosing and building the lakehouse beneath ETRM analytics, and architects who must justify their platform choices.
Prove the module by building it
Design a medallion lakehouse for ETRM: bronze ingestion, silver conformed models, and gold marts, with schema enforcement and a governance layer.
This course project is one of twelve that culminate in the bundle's end-to-end capstone, a complete working ETRM data platform run as a straight-through pipeline.
Learn it your way
This course is included in the ETRM Data Engineering and Analytics bundle, available self-paced with PDF handbooks, slide decks, video explainers, and hands-on labs and a project, and deliverable as private corporate training.
Explore the full 12-course bundle and its end-to-end capstone project →
Frequently asked questions
What does the Data Lake, Warehouse & Lakehouse course cover?
The platform architecture beneath ETRM analytics: lake, warehouse, and lakehouse patterns, and when each is right. It runs to 20 sequential chapters (chapters 141 to 160 of the 240-chapter bundle).
Where does this sit in the learning path?
It is course 8 of 12 in the ETRM Data Engineering and Analytics sequence. It builds on the courses before it and prepares for those after it.
Is there a project?
Yes. Design a medallion lakehouse for ETRM: bronze ingestion, silver conformed models, and gold marts, with schema enforcement and a governance layer.
What technologies are involved?
The course works with Spark, Delta Lake, S3, Snowflake, Parquet, chosen to reflect how real ETRM data platforms are built.
Is it available self-paced?
Yes. Every course in the bundle is available self-paced with PDF handbooks, slide decks, video explainers, and hands-on labs and a project, with lifetime access.
Can my team take it as corporate training?
Yes. The whole bundle, or individual courses, can be delivered as private corporate training mapped to your stack and data. Use the contact form to scope it.
Take this course, or the whole platform
Enroll self-paced, or bring the ETRM Data Engineering bundle to your team.