Programme 8 of 14 · Processing & streaming

Apache Spark - Large-Scale Processing Mastery

765 words3 min read

Master Apache Spark for batch processing, structured streaming, machine learning, and production deployments on YARN, Kubernetes, and managed cloud Spark, the engine behind large-scale data processing.

Apache SparkSpark SQLStructured StreamingMLlibKubernetes
Apache Spark - Large-Scale Processing Mastery: the syllabus at a glance1Spark core2Spark SQL andperformance3Streaming4ML andproductionProject
Curriculum · 20 chapters in 4 modules

The full syllabus

Four modules of five chapters each, sequenced so the material builds cumulatively. The brochure carries a note on what each chapter teaches.

M1 Spark core Ch 1-5 · 5 chapters
  1. The Spark execution model
  2. RDDs, DataFrames, and Datasets
  3. The Spark APIs in Scala, Python, and Java
  4. Transformations and actions
  5. Partitioning and shuffles
M2 Spark SQL and performance Ch 6-10 · 5 chapters
  1. Spark SQL and the DataFrame API
  2. The Catalyst optimizer
  3. Performance tuning and caching
  4. Handling skew and spills
  5. Joins and aggregations at scale
M3 Streaming Ch 11-15 · 5 chapters
  1. Structured Streaming
  2. Windowing and watermarks
  3. Kafka integration
  4. Stateful stream processing
  5. Exactly-once semantics
M4 ML and production Ch 16-20 · 5 chapters
  1. MLlib for machine learning
  2. MLflow and model serving
  3. Running Spark on YARN and Kubernetes
  4. Managed cloud Spark
  5. The Spark processing capstone
Where Apache Spark - Large-Scale Processing Mastery can leadThis programmeopens roles inData Engineer (Spark)Big Data EngineerStreaming / Platform EngineerData Architecttoward Principal Cloud & Data Platforms
FAQ

Frequently asked questions

What is the Apache Spark - Large-Scale Processing Mastery programme?

Master Apache Spark for batch processing, structured streaming, machine learning, and production deployments on YARN, Kubernetes, and managed cloud Spark, the engine behind large-scale data processing.

Who is this programme for?

It suits data engineers processing data at scale, along with others described on this page.

How is it delivered?

Self-paced with hands-on labs from Spark core to a production processing capstone.

Is there a project or capstone?

Build a large-scale Spark processing system: a tuned batch pipeline and a structured-streaming job reading from Kafka with exactly-once semantics, deployed on Kubernetes or managed cloud Spark, with performance analysis showing how you handled skew, shuffles, and caching.

How does this fit the wider journey?

Spark opens the processing-and-streaming stage and underpins Databricks. Its performance-tuning depth is a hallmark of senior data engineers, and the batch-and-streaming command it builds is central to large-scale platform work.

Can my organization run this as a private cohort?

Yes. Every programme can be delivered as a tailored corporate cohort. Contact us to scope it.

How it is delivered

Three ways to take Apache Spark Large-Scale Processing Mastery

Self-paced is a document-and-media programme with lifetime access - no live sessions. Cohort and enterprise add live instructor-led training. Mentorship is not offered in any tier.
Feature Self-paced Cohort Enterprise
Format Written chapters, video explainers and podcasts Everything in self-paced, plus scheduled live sessions Everything in cohort, delivered privately to your team
Live sessions None Scheduled, instructor-led Scheduled, instructor-led, private
Mentorship Not offered Not offered Not offered
Access Lifetime Lifetime Lifetime for every enrolled seat
Pace Entirely your own Guided schedule with a peer group Agreed with your desk
Tailoring Fixed curriculum Fixed curriculum Sequenced to your markets, systems and governance
Best for Individuals learning around a job Individuals who want structure and deadlines Desks building the same capability together
For individuals

Get the programme guide

The full chapter list, what each module covers, and how the tiers compare - sent to your inbox as a PDF.

For teams

Run this for my desk

Private delivery for your desk, sequenced to your markets and systems. Tell us the team and we will scope it.

Run this for my desk