Data Engineering & Pipelines Foundations
The foundation that ties the whole track together: how data moves through modern systems, ETL and ELT, orchestration, data modeling, and the role of the data engineer, before you specialize in any single platform.
Why start with the foundation
It is tempting to jump straight into a cloud or a warehouse, but the engineers who go furthest first understand the shape of the whole system. Knowing what a data engineer actually does, how data flows from source to insight, and where each platform fits turns the rest of this track from a list of tools into a coherent discipline. This program builds that understanding deliberately, so everything after it has a place to land.
The orientation is toward the long game. This track leads, for those who want it, all the way to Principal Engineer and Director of Cloud and Data Platforms, and that journey rests on foundations. An engineer who understands pipelines, modeling, and orchestration as concepts can learn any specific platform quickly, which is exactly the adaptability that senior and lead roles demand.
The payoff is leverage: an engineer who understands pipelines, modeling, and orchestration as concepts learns any specific platform quickly, which is exactly the adaptability that carries you from a first data job toward senior and lead roles.
Pipelines, ETL, and ELT
The heart of data engineering is moving and shaping data reliably, so the program covers ingestion patterns and change data capture, the shift from ETL to ELT, transformation and the central role of SQL, and the data-quality checks that keep a pipeline trustworthy. Idempotency and safe reprocessing are treated as first-class, because in real systems pipelines fail and must be rerun without corrupting data.
These concepts are platform-independent, which is the point. Whether you later build on AWS, Snowflake, or Spark, the same pipeline thinking applies, so learning it once, clearly, pays off across every platform in the track. This program teaches the ideas that every later program then makes concrete.
Because these pipeline ideas are platform-independent, the effort you invest here compounds across every later program, so the foundation is the single highest-return place to start rather than a hurdle before the real material.
Modeling, orchestration, and reliability
Good pipelines rest on good data models, so the program covers relational and dimensional modeling, the medallion (bronze, silver, gold) pattern that structures modern lakehouses, schema evolution and contracts, and partitioning and file formats. This modeling knowledge is what makes a data platform queryable and maintainable rather than a swamp.
Finally the program builds the operational backbone: orchestration and scheduling, DAGs and dependency management, idempotent and retryable tasks, and the monitoring and observability that keep pipelines healthy. The end-to-end project ties it together into a real, reliable pipeline, the reference the rest of the track builds on.
The end-to-end project you build becomes the reference the rest of the track extends, so you finish the first program already holding a complete, working pipeline rather than only concepts.
See the method, not just the topic
A representative worked example from the program, so you can see the level of concreteness the curriculum works at.
# An Airflow-style DAG: ingest -> transform (medallion) -> publish.
# Each task is idempotent, so a retry re-runs it safely.
with DAG("daily_sales", schedule="@daily", catchup=False) as dag:
ingest = PythonOperator(
task_id="ingest_bronze",
python_callable=load_raw, # overwrites today's partition
)
transform = PythonOperator(
task_id="build_silver",
python_callable=clean_and_conform, # deterministic from bronze
)
publish = PythonOperator(
task_id="build_gold",
python_callable=aggregate_marts, # rebuilds gold from silver
)
ingest >> transform >> publish # dependency, left to right
# Because each task rebuilds its own partition from the layer below,
# a failed run can be retried end to end with no double-counting.
# This idempotent, layered design is the core of reliable pipelines.The full syllabus
Four modules of five chapters each, sequenced so the material builds cumulatively. Each chapter carries a note on what it teaches.
Module 1How data moves
- 01What a data engineer does and why it mattersWhat the data-engineering role actually involves. You leave knowing what the job is before you specialize.
- 02Sources, sinks, and the shape of a pipelineThe anatomy of a pipeline, from source to sink. Every pipeline you build later has this shape.
- 03Batch versus streaming, and when to use eachChoosing between batch and streaming. The choice drives most of your architecture.
- 04The modern data stack, from a heightA high-level map of the modern data stack. The map turns a list of tools into a system.
- 05Where cloud, warehouses, and processing fitWhere clouds, warehouses, and engines each fit. You see how the whole track fits together.
Module 2ETL and ELT
- 06Extract, transform, load, and the ELT shiftThe move from ETL to ELT, and why it happened. ELT is why cloud warehouses changed the game.
- 07Ingestion patterns and change data captureIngestion patterns and change data capture. Capturing only what changed keeps pipelines efficient.
- 08Transformation and the role of SQLTransformation and the central role of SQL. SQL does most of the transformation work.
- 09Data quality, validation, and testingBuilding data quality into the pipeline. Untested pipelines quietly corrupt data.
- 10Idempotency and safe reprocessingIdempotency and safely reprocessing data. Idempotency is what lets you safely rerun a failed job.
Module 3Data modeling for pipelines
- 11Relational modeling and normalizationRelational modeling and normalization. Normalization keeps operational data clean.
- 12Dimensional modeling for analyticsDimensional modeling for analytics. Dimensional models make analytics fast.
- 13The medallion (bronze, silver, gold) patternThe bronze, silver, gold medallion pattern. The medallion pattern structures modern lakehouses.
- 14Schema evolution and contractsHandling schema change with contracts. Contracts keep changes from breaking downstream.
- 15Partitioning and file formatsPartitioning and choosing file formats. The right format and partitioning cut cost and time.
Module 4Orchestration and reliability
- 16Orchestration and schedulingOrchestrating and scheduling work. Orchestration turns scripts into a reliable system.
- 17DAGs and dependency managementManaging dependencies with DAGs. DAGs express what must run before what.
- 18Idempotent, retryable tasksMaking tasks idempotent and retryable. Retryable tasks survive the failures that will happen.
- 19Monitoring, alerting, and observabilityMonitoring and observing pipeline health. You cannot operate what you cannot observe.
- 20The end-to-end pipeline projectBuilding the end-to-end reference pipeline. The project is the reference the whole track builds on.
How the program is taught
The program is hands-on and project-driven: you work with the platform-independent foundation of data engineering through real labs rather than watching from a distance, and it builds toward a capstone you can keep and show. Every concept is applied, because platform skills are built by doing, not by reading about them.
It is structured so a motivated learner can start where they are and build steadily, with worked examples and code throughout. The through-line is always real, production-shaped work, so at every stage you are learning the platform the way practitioners actually use it.
Prerequisites and pace
No data background is required. If you are new to coding or SQL, pair it with the Programming and Data Analytics tracks; otherwise it is the ideal first step. The pace builds from foundations to a capstone, and the most effective approach is to complete each lab rather than skim it, since the labs accumulate into the project.
For someone working toward the senior end of this track, consistency matters more than speed: steady progress through the material, and through the capstone, is what builds durable capability and a portfolio that demonstrates it.
What makes this program different
It teaches the concepts, pipelines, modeling, orchestration, that every platform later makes concrete, so you learn the discipline before the tools. That focus is what turns knowledge of a platform into the ability to build and operate real systems on it.
The other distinction is the orientation toward the whole journey. Every program in this track is designed to fit with the others and to build toward the senior technical-leadership destination, so this one is taught as a step on that path rather than an isolated course.
What you will be able to do
- Understand how data moves through a modern data platform
- Design ETL and ELT pipelines with data quality built in
- Model data for both operational and analytical use
- Orchestrate reliable, idempotent, monitored pipelines
- See where every later platform fits in the whole
Who should take it
- Newcomers building a base before choosing a platform
- Analysts and developers moving into data engineering
- Career changers from a non-data background
- Anyone who wants the map before the details
Common questions
Do I need a data background to start here? No. This program is designed as the entry point, and it pairs with the sibling Programming and Data Analytics tracks for anyone who needs the absolute basics of coding and SQL first or in parallel.
Which platform should I learn after it? Any of them, and the program helps you choose by showing where each fits. Most people take SQL next, then one cloud, then a warehouse and a processing engine.
Where it leads, toward Principal
In the near term, this program opens Data Engineer and Analytics Engineer roles, and it is the base every more senior title in this track builds on. The capstone is concrete evidence of that capability, which matters more than any list of topics studied.
In the longer term, it is one rung on the ladder toward Principal Engineer and Director of Cloud and Data Platforms. That destination is reached through breadth across the whole stack plus the architecture, operations, and leadership judgment the senior programs emphasize, and this program contributes a genuine, in-demand piece of that breadth.
How it fits the journey
This program is the first program and the map for everything after it. It rests on the foundations before it and connects to the programs around it, so taking it in sequence builds cumulative command rather than isolated knowledge.
Because every program is also complete in itself, you can enter here if this is exactly the platform your goals require, and still get a whole, finished program. The sequence is a guide, not a gate, all the way up to the Principal and Director destination.
What you build and keep
Build an end-to-end data pipeline from scratch: ingest from a source, transform through a medallion (bronze, silver, gold) layout, model the result for analytics, orchestrate it as a scheduled, retryable, monitored job, and document it as the reference project the rest of the track builds on.
Format: Self-paced with hands-on labs; the recommended starting point and the map for the whole track.
Run this program for your team
Every program can be delivered as a private, tailored cohort for your organization, aligned to your systems, policies, and career frameworks.
Scope a corporate cohortFrequently asked questions
What is the Data Engineering & Pipelines Foundations program?
The foundation that ties the whole track together: how data moves through modern systems, ETL and ELT, orchestration, data modeling, and the role of the data engineer, before you specialize in any single platform.
Who is this program for?
It suits newcomers building a base before choosing a platform, along with others described on this page.
How is it delivered?
Self-paced with hands-on labs; the recommended starting point and the map for the whole track.
Is there a project or capstone?
Build an end-to-end data pipeline from scratch: ingest from a source, transform through a medallion (bronze, silver, gold) layout, model the result for analytics, orchestrate it as a scheduled, retryable, monitored job, and document it as the reference project the rest of the track builds on.
How does this fit the wider journey?
This is the first program and the map for the whole track. If you are from a non-data background, pair it with the Programming and Data Analytics tracks for coding and SQL. Everything after it, the clouds, warehouses, engines, and databases, slots into the picture this program builds.
Can my organization run this as a private cohort?
Yes. Every program can be delivered as a tailored corporate cohort. Contact us to scope it.