Streaming & Real-Time Risk
Real-time changes what a desk can see. This course processes trades and market data in motion, maintaining live positions and firing limit and risk alerts within seconds, with exactly-once guarantees and correct handling of time.
The pipeline at the heart of this course
Why real-time changes the desk
When a desk can only see its risk at end-of-day, it is managing yesterday. Streaming changes that: by processing trades and market data as they happen, the platform can maintain live positions and surface breaches within seconds. This course builds that capability, and it depends on everything before it, the models, the events, the limits, now driven in motion.
The shift from batch to streaming is not just faster batch; it is a different computational model. Data is unbounded and arrives continuously and out of order, and the system must produce correct, timely results despite that. The course teaches this model rigorously, because naive streaming produces confident, wrong numbers.
The course frames streaming as a genuinely different computational model rather than faster batch, because treating it as the latter is how people produce confident, wrong numbers. Data is unbounded, arrives continuously and out of order, and must yield correct, timely results despite that, and the techniques that make this possible are specific and learnable. The course teaches them rigorously because the failure mode, plausible live numbers that are quietly incorrect, is both easy to reach and hard to detect.
The log abstraction and stream processing
Streaming rests on the log: an ordered, partitioned, replayable record of events, the same abstraction the event-sourcing course used, now consumed continuously. You learn producers and consumers, topics and partitions, and how partitioning enables parallelism while preserving the ordering that correctness needs.
Stateful stream processing is the core skill: maintaining local state, a running position, a current exposure, that updates with each event. The course covers how that state is kept, scaled, and made fault-tolerant, because a streaming position engine that loses its state on restart is worse than useless.
Stateful stream processing is the core skill the course builds toward, because a stream that cannot maintain state cannot maintain a position. Keeping a running position or exposure in local state, and keeping that state correct, scalable, and fault-tolerant, is what separates a real-time risk engine from a toy, and the course spends time on how that state is stored, scaled, and recovered after failure. A streaming position engine that loses its state on restart is worse than none.
Time, windows, and late data
The hardest part of streaming is time. Event time, when something actually happened, differs from processing time, when the system saw it, and events arrive late and out of order. Windowing, tumbling, sliding, session, groups events for aggregation, and watermarks tell the system when a window is safe to close despite stragglers.
The course is concrete about why this matters for trading: a position computed on processing time can be wrong when a trade arrives late; a risk aggregate closed too early can miss data. You learn to reason in event time and to handle late data explicitly, so that live numbers are both timely and correct.
Time is the hard part, and the course does not soften it. Event time versus processing time, windowing, and watermarks are the machinery that lets a stream produce correct results despite late and out-of-order data, and reasoning in event time is what keeps a live position from being wrong when a trade arrives late. The course makes these ideas concrete with trading examples, because they are the difference between a live view that can be trusted and one that cannot.
Joining streams and live risk
The signature pattern is joining streams: trades against market data, so that a market move immediately revalues open positions, and positions against limits, so that a breach fires the instant it occurs. This course builds that join, maintaining live positions in stream state and revaluing them as curves tick, then checking limits inline.
Exactly-once processing is essential here: a trade counted twice, or a limit alert fired on stale state, undermines trust in the whole system. The course covers the mechanisms, idempotency, transactional state, checkpointing, that deliver exactly-once guarantees under failure, building on the idempotency established in earlier courses.
Joining streams is the signature pattern the course builds toward, because it is what makes risk truly live. Trades joined against market data so a move immediately revalues positions, and positions joined against limits so a breach fires the instant it occurs, are what give a desk a live picture. Exactly-once processing is what makes this trustworthy under failure, and the course shows how idempotency and transactional state deliver it.
Reconciling streaming with batch truth
Real-time systems must still agree with the official, batch-computed books. The course closes by reconciling the streaming view with batch truth, so the live numbers on the desk and the end-of-day numbers of record never silently diverge, and by tuning latency, backpressure, and scaling for production. You leave with a real-time risk pipeline: ingest, join, compute, check, and serve.
The artifact below shows a stateful stream processor maintaining live positions and firing limit alerts.
Reconciliation with batch truth is the discipline the course ends on, because a fast view that disagrees with the official books is a liability, not an asset. The streaming numbers on the desk and the end-of-day numbers of record must never silently diverge, and the course shows how to reconcile them continuously. You finish with a real-time risk pipeline that is both fast and honest, which is the only kind worth running.
Event time, watermarks, and the correctness of live numbers
The subtlest source of wrong answers in streaming is time. A naive system processes events in the order it happens to receive them, but events arrive late and out of order, and a position or risk number computed on arrival order can simply be wrong. The course teaches the discipline of reasoning in event time, when things actually happened, rather than processing time, when the system saw them.
Windowing groups events for aggregation, and watermarks are the mechanism that tells the system when a window is safe to finalize despite the possibility of stragglers. Set the watermark too aggressively and you finalize before late data arrives, producing wrong results; too conservatively and you add latency. The course makes these trade-offs concrete for trading, where both correctness and timeliness matter, and shows how to handle genuinely late data explicitly rather than silently mis-handling it.
The payoff is live numbers that are both timely and correct, a position that reflects all the trades that belong in it even if some arrived late, a risk aggregate that is not quietly wrong because a window closed too soon. This correctness under the reality of imperfect timing is what separates a trustworthy real-time system from a fast but unreliable one.
Exactly-once, state, and reconciling with batch truth
Two things make streaming hard to get right in trading: exactly-once processing and stateful correctness under failure. A trade counted twice inflates a position; a limit alert fired on stale state erodes trust. The course covers the mechanisms that deliver exactly-once guarantees, idempotent processing, transactional state updates, and checkpointing, so that a consumer restart or a rebalance never double-counts a trade or double-fires an alert.
Stateful stream processing, maintaining a running position or exposure in local state, is the core technique, and its correctness depends on that state surviving failures. The course shows how checkpointing and fault-tolerant state stores keep the running state correct across restarts, building on the idempotency established in the event-sourcing course.
Finally, the course insists that the real-time view must still agree with the official, batch-computed books, and shows how to reconcile the two so that the live numbers on the desk and the end-of-day numbers of record never silently diverge. This reconciliation is the same discipline the position-and-P&L and STP courses demand, applied to the streaming layer.
Where this course sits, and what it unlocks
Streaming is the tenth course, adding real-time to everything the platform has built. It consumes the events and curves from earlier courses and drives live positions, real-time revaluation, and inline limit checks, feeding the dashboards course and forming a key stage of the STP capstone. It is where end-of-day becomes now.
The event-time reasoning and exactly-once guarantees you learn here are what make real-time risk trustworthy rather than merely fast, and they build directly on the idempotency established in the event-sourcing course.
Common streaming failures, and how to avoid them
The defining streaming failure is computing on processing time instead of event time, so that late or out-of-order events produce wrong positions and risk numbers that look authoritative. Reasoning in event time, with windows and watermarks, is what keeps live numbers correct under the reality of imperfect timing. A second failure is losing stateful correctness across restarts, where a consumer that loses its running state on failure produces wrong positions; checkpointing and fault-tolerant state prevent this.
A third failure is weak processing guarantees, where a rebalance or retry double-counts a trade or double-fires an alert, eroding trust; exactly-once semantics address this. And a fourth is letting the streaming view drift from batch truth, which reconciliation catches.
By avoiding these patterns you get real-time risk that is both timely and correct, and that agrees with the official books, which is the only kind a desk will actually rely on.
A worked example: a market move firing a limit alert
Trace a market move through the streaming pipeline. A price tick arrives on the market-data stream and is joined, in event time, against the live positions maintained in stream state. The affected positions are revalued immediately, and their updated exposure is checked against limits inline. One position's exposure now breaches its limit, and a breach alert is emitted downstream within seconds of the tick.
The worked example shows the stateful join, the revaluation, and the inline limit check as one continuous flow, and highlights the guarantees that make it trustworthy: event-time processing so a late tick is handled correctly, and exactly-once semantics so the alert fires once, not zero or twice. The desk sees the breach essentially as it happens, rather than at end-of-day, which is the entire point of the real-time layer.
This demonstrates the course's synthesis: streams joined statefully, in event time, with exactly-once guarantees, turn a market move into an actionable alert in seconds.
Operational realities: scaling, failure, and reconciliation
Operationally, a streaming system must scale with load, recover from failures without losing or double-counting, and agree with the batch books. The course covers partitioning and consumer rebalancing for scale, checkpointing and fault-tolerant state for recovery, and backpressure handling so a burst does not overwhelm the pipeline.
Reconciliation with batch truth is the final operational discipline: the live streaming view and the official end-of-day numbers must be reconciled so they never silently diverge, and any divergence must be caught and explained. This is the same tie-out principle from the position-and-P&L course, applied continuously to the real-time layer, and it is what lets a desk trust the live screen as much as the official books.
Design trade-offs: latency versus correctness under late data
Streaming forces a choice between answering quickly and answering completely. Close a window and emit a result as soon as processing time says the window is over, and the desk sees numbers fast but a late-arriving trade can make them wrong. Wait long enough to be sure every straggler has arrived, and the numbers are correct but no longer timely. In trading, both wrong-but-fast and right-but-late are failures.
The course teaches the event-time and watermark machinery that navigates this: reasoning in event time so results reflect when things actually happened, and using watermarks to decide when a window is safe to close, with explicit handling for the late data that arrives after. The design accepts a bounded, deliberate latency in exchange for correctness, rather than either racing ahead into wrong answers or waiting indefinitely for completeness. This is the hardest idea in streaming and the one that separates a live risk view that can be trusted from one that cannot.
Carrying streaming into the capstone
Streaming is what makes the capstone's real-time stages real-time: live positions maintained as trades arrive, revaluation as curves tick, and limit alerts that fire within seconds rather than at end-of-day. The exactly-once guarantees you build here are what let the capstone drive these live numbers without double-counting under failure or rebalance, which is essential when those numbers gate trading.
The reconciliation between streaming and batch, which the course insists on, is also a capstone concern: the fast, live view and the official, batch-computed books must never silently diverge, and the flow's credibility depends on their agreement. So streaming does not replace the batch truth in the capstone; it runs alongside it, faster, and reconciles to it, giving the desk speed without giving up the authority of the books of record.
Performance and scale considerations
Streaming performance is about sustained throughput and bounded latency under continuous load, which is a different discipline from batch, and the course teaches it as such. Partitioning enables parallelism, but the course shows how to partition so that related events stay ordered while unrelated ones scale out, and how backpressure and flow control keep the pipeline stable when input outpaces processing. The aim is a pipeline that stays within its latency budget even under bursts.
State is the scaling challenge specific to stateful streaming, because the running positions and exposures a stream maintains must be kept, scaled, and recovered without loss. The course covers how local state is stored and checkpointed, how consumers rebalance without dropping state, and how to keep state size bounded so the pipeline stays fast as the book grows. These are the concerns that separate a real-time engine that scales from one that seizes up.
Testing and validating streaming
Streaming correctness is hard to test precisely because of time, and the course teaches the techniques that make it tractable. Replaying a fixed stream of events with known timestamps and asserting the resulting windows and aggregations is how you test event-time logic deterministically, and deliberately injecting late and out-of-order events confirms the watermark handling behaves as intended. Exactly-once is tested by simulating failures and restarts and confirming nothing is double-counted.
The course also treats reconciliation against batch as a continuous validation: the streaming position must agree with the batch-computed position of record, and any divergence is a test failure the platform surfaces. This is the check that keeps the fast live view honest, so building it is part of building the pipeline. You finish able to demonstrate that your streaming numbers are both timely and correct, which is the whole point.
How it works, concretely
// Maintain a live net position per book, update on every trade,
// and emit an alert the instant a limit is breached.
KStream<String, Trade> trades = builder.stream("trades");
KTable<String, Double> livePosition = trades
.groupBy((k, t) -> t.bookId()) // partition by book
.aggregate(
() -> 0.0, // initial state
(book, t, pos) -> pos + t.signedQty(), // fold each trade
Materialized.as("live-position-store")); // fault-tolerant state
// Join live position against the limit table; emit breaches downstream
livePosition.toStream()
.join(limits, // KTable<book, Limit>
(pos, lim) -> new Check(pos, lim))
.filter((book, c) -> Math.abs(c.pos()) > c.limit()) // breach test
.mapValues(c -> new Alert(c, Instant.now()))
.to("limit-alerts");
// Exactly-once is enabled at the topology level so a rebalance or
// restart never double-counts a trade or double-fires an alert.Chapters 181 to 200
Twenty sequential chapters. Within the full bundle these are chapters 181 through 200 of 240.
- 181Why real-time changes what a desk can seeWhy real-time visibility changes what a desk can manage. Seeing risk live rather than at end-of-day is what changes how a desk can act.
- 182Streaming fundamentals and the log abstractionThe log abstraction that streaming rests on. The log abstraction is the same one event sourcing used, now consumed continuously.
- 183Producers, consumers, topics, and partitionsProducers, consumers, topics, and partitions. Partitioning is what enables parallelism while preserving the ordering correctness needs.
- 184Stream processing vs batch processingContrasting stream processing with batch. Stream processing produces continuous results over unbounded, out-of-order data.
- 185Stateful stream processing and local stateMaintaining local state in a stream processor. Local state is what lets a stream maintain a running position or exposure.
- 186Windowing: tumbling, sliding, and sessionWindowing events with tumbling, sliding, and session windows. Windowing groups unbounded events into the finite sets aggregation needs.
- 187Event time vs processing timeReasoning in event time versus processing time. Event time versus processing time is the distinction that makes live numbers correct.
- 188Watermarks and handling late dataUsing watermarks to handle late-arriving data. Watermarks tell the system when a window is safe to close despite stragglers.
- 189Joining trade and market-data streamsJoining trade and market-data streams. Joining trade and market streams is what makes a market move immediately revalue positions.
- 190Maintaining live positions in a streamMaintaining live positions inside a stream. Maintaining live positions in stream state is the signature real-time capability.
- 191Real-time revaluation on market movesRevaluing positions in real time as markets move. Real-time revaluation is what lets the desk see P&L move with the market.
- 192Streaming limit checks and breach alertsChecking limits and firing breach alerts inline. Streaming limit checks fire a breach alert the instant it occurs.
- 193Exactly-once processing guaranteesGuaranteeing exactly-once processing under failure. Exactly-once processing is what keeps a trade from being counted twice under failure.
- 194Backpressure and flow controlHandling backpressure and flow control. Backpressure and flow control keep the pipeline stable under load.
- 195Scaling and rebalancing consumersScaling and rebalancing stream consumers. Consumer scaling and rebalancing are what let throughput grow without losing correctness.
- 196Fault tolerance and checkpointingAchieving fault tolerance with checkpointing. Checkpointing is what lets a streaming job recover its state after a failure.
- 197Serving real-time data to dashboardsServing real-time data to dashboards. Serving real-time data to dashboards connects this course to the dashboards one.
- 198Latency budgets and performance tuningMeeting latency budgets and tuning performance. Latency budgets are what keep live numbers actually live under production load.
- 199Reconciling streaming with batch truthReconciling the streaming view with batch truth. Reconciling streaming with batch is what keeps the fast view honest against the books.
- 200A real-time risk pipeline end to endAssembling a real-time risk pipeline end to end. You finish with a real-time risk pipeline that maintains live positions and alerts.
What you'll be able to do
- Maintain live positions in a stateful stream processor
- Reason in event time and handle late data with watermarks
- Join trade and market streams to drive real-time risk
- Guarantee exactly-once processing under failure
- Reconcile the streaming view with batch truth
Audience & prerequisites
Engineers building real-time position and risk, and teams moving from end-of-day batch to live visibility without sacrificing correctness.
Prove the module by building it
Build a streaming pipeline that maintains live positions and fires limit and risk alerts within seconds of a trade or market move, with exactly-once guarantees.
This course project is one of twelve that culminate in the bundle's end-to-end capstone, a complete working ETRM data platform run as a straight-through pipeline.
Learn it your way
This course is included in the ETRM Data Engineering and Analytics bundle, available self-paced with PDF handbooks, slide decks, video explainers, and hands-on labs and a project, and deliverable as private corporate training.
Explore the full 12-course bundle and its end-to-end capstone project →
Frequently asked questions
What does the Streaming & Real-Time Risk course cover?
Processing trades and market data in motion to drive live position, exposure, and alerting. It runs to 20 sequential chapters (chapters 181 to 200 of the 240-chapter bundle).
Where does this sit in the learning path?
It is course 10 of 12 in the ETRM Data Engineering and Analytics sequence. It builds on the courses before it and prepares for those after it.
Is there a project?
Yes. Build a streaming pipeline that maintains live positions and fires limit and risk alerts within seconds of a trade or market move, with exactly-once guarantees.
What technologies are involved?
The course works with Kafka, Flink, Kafka Streams, Windowing, WebSockets, chosen to reflect how real ETRM data platforms are built.
Is it available self-paced?
Yes. Every course in the bundle is available self-paced with PDF handbooks, slide decks, video explainers, and hands-on labs and a project, with lifetime access.
Can my team take it as corporate training?
Yes. The whole bundle, or individual courses, can be delivered as private corporate training mapped to your stack and data. Use the contact form to scope it.
Take this course, or the whole platform
Enroll self-paced, or bring the ETRM Data Engineering bundle to your team.