Straight-Through Processing (Capstone)
This is the capstone. Straight-through processing wires every prior module into one automated, controlled, exception-managed flow from trade capture to settlement, and this course builds it end to end, the point where all twelve courses become a single working system.
The pipeline at the heart of this course
What STP means and why it is last
Straight-through processing is the goal every ETRM platform is ultimately built toward: a trade flows from capture to settlement automatically, touched by human hands only when something needs repair. It is the last course in this program by design, because you cannot build STP until you have built everything it connects. Every prior course, the model, reference data, the lifecycle, FpML, curves, position and P&L, credit and limits, the lakehouse, event sourcing, streaming, and dashboards, is a stage in the flow this course assembles.
This course is where the platform becomes a system. It is less about a new technique and more about integration, control, and operability: wiring the pieces into one pipeline, handling the exceptions that inevitably arise, and running the whole thing with the reconciliation and observability that a production trading platform demands.
The course is explicit that STP is the synthesis of the program rather than a new topic, and that framing shapes how it is taught. Every prior module is a stage in the flow assembled here, so the capstone is less about learning something new and more about integrating what you already built into one automated, controlled, exception-managed pipeline. The course spends its effort on the integration, control, and operability that turn a set of components into a system.
Designing the pipeline as connected stages
You design STP as a sequence of connected stages, each owning a clear responsibility and a clear contract with the next: capture records the trade, enrichment applies reference data, representation normalizes FpML, valuation strikes it against curves, aggregation produces position and P&L, control checks credit and limits, and settlement confirms and pays. The course shows how to compose these so that data flows straight through when all is well.
The event-sourced backbone from earlier is what makes the flow auditable and recoverable: each stage emits events, so the pipeline's entire history is reconstructable, and any stage can be safely reprocessed. Streaming and batch stages coexist, real-time where latency matters, batch where completeness does, and the course shows how to combine them coherently.
Designing the pipeline as connected stages with clear contracts is the structural idea, and the course makes it concrete. Each stage owns a responsibility and a contract with the next, so data flows straight through when all is well and peels off cleanly when it is not, and the event-sourced backbone makes the whole flow auditable and reprocessable. The course shows how to combine streaming and batch stages coherently rather than treating them as rival paradigms.
Exception management and the repair queue
No real flow is exception-free. The defining feature of a good STP system is not that nothing goes wrong but that exceptions are handled cleanly: a trade that fails validation, a missing curve, a limit breach peels off into a repair queue, is fixed or overridden with audit, and rejoins the flow, while everything else continues straight through. The course builds this exception-management layer explicitly.
The metrics that matter follow directly: the straight-through rate, the proportion of trades that flow with no manual touch, and the manual-touch count, which the desk works to drive down. These are the operational KPIs of an STP platform, and the course shows how to instrument and improve them.
Exception management is the feature that defines a good STP system, and the course treats the repair queue as central rather than peripheral. The measure of the system is not that nothing goes wrong but that exceptions are handled cleanly, fixed or overridden with audit and rejoined, while everything else flows on. The straight-through rate and manual-touch count that follow are the operational KPIs the course teaches you to instrument and improve.
Reconciliation, idempotency, and control
Every handoff in the flow is a place data can diverge, so reconciliation runs at each one: does what left this stage match what arrived at the next. Idempotency, established across the earlier courses, is what makes safe reprocessing possible, a trade reprocessed after a failure must not double-settle. The course brings these guarantees together into a controlled, auditable flow.
Governance and controls span the whole pipeline: who can override an exception, how changes are authorized, how the complete audit trail is maintained. This is where the platform meets the regulatory and control expectations that trading demands, and where the event-sourced foundation pays off fully.
Reconciliation, idempotency, and control are what make the automated flow safe, and the course brings them together deliberately. Every handoff is reconciled, every operation is idempotent so reprocessing is safe, and governance spans the whole pipeline so overrides and changes are authorized and recorded. This is where the platform meets the control and regulatory expectations that trading demands, and where the event-sourced foundation pays off fully.
Orchestration, observability, and the reference implementation
Finally the course covers running STP in production: orchestration that controls the flow, observability and tracing so that a trade can be followed through every stage, SLAs and failure handling including dead-letter design, and the settlement and downstream feeds that close the loop. You leave having assembled the complete STP reference implementation, the working synthesis of the entire program.
The artifact below shows the STP pipeline as orchestrated, reconciled, exception-handled stages, the shape of the capstone system.
Running STP in production is the final concern, and the course does not stop at design. Orchestration, observability and tracing, SLAs, dead-letter handling, and the settlement and downstream feeds that close the loop are what make the flow operable rather than merely designed. You finish having assembled the complete STP reference implementation, which is the working synthesis of everything the program set out to teach.
Exception management and the straight-through rate
The defining feature of a good straight-through-processing system is not that nothing ever goes wrong, but that when something does, it is handled cleanly without stopping everything else. This course builds the exception-management layer explicitly: a trade that fails validation, a missing curve, or a limit breach peels off into a repair queue, is fixed or overridden with a full audit record, and rejoins the flow, while every clean trade continues straight through untouched.
This design is what makes automation safe. Without a repair queue, a single bad trade either halts the pipeline or, worse, is forced through and corrupts downstream state. With one, the pipeline degrades gracefully: the straight-through rate, the proportion of trades that flow with no manual touch, is the headline metric the desk works to raise, and the manual-touch count is what it works to lower. The course shows how to instrument both and how to drive automation up over time by attacking the most common exception causes.
You learn to treat exceptions not as failures of the system but as a normal, expected part of its operation, designed for from the start. This is the operational maturity that distinguishes a real STP platform from a demo that only works on clean data.
Reconciliation, idempotency, and running STP in production
STP wires many stages together, and every handoff between stages is a place data can diverge, so the course runs reconciliation at each one: what left a stage must match what arrived at the next, and any break is caught immediately rather than discovered at settlement. This per-handoff reconciliation is what gives an automated flow the integrity that lets a desk trust it without watching every trade.
Idempotency, established across the earlier courses, is what makes safe reprocessing possible: a trade reprocessed after a failure must not double-settle, and a stage re-run must produce the same result. The course brings these guarantees together so that the pipeline can recover from failures by reprocessing, rather than by manual intervention, which is essential for a flow that runs continuously.
Finally, the course covers running STP as a production system: orchestration that controls the flow, observability and tracing so that any trade can be followed through every stage, SLAs and dead-letter handling for failures, and the settlement and downstream feeds that close the loop. Assembling all of this is the capstone of the capstone, the point where every skill in the bundle becomes one working, operable, auditable system.
Where this course sits: the capstone that assembles everything
Straight-through processing is deliberately the twelfth and final course, because it is not a new topic so much as the assembly of all the previous ones into a single working system. Every earlier course built a component, the model, the reference data, the lifecycle, FpML ingestion, curves, position and P&L, credit and limits, the lakehouse, event sourcing, streaming, dashboards, and STP wires them into one automated flow from capture to settlement.
This is why it comes last and why it doubles as the bundle's capstone. You cannot build straight-through processing until you have built everything it connects, and having built them, assembling them into an automated, controlled, observable flow is the natural culmination. The capstone project of the whole bundle is precisely this assembly.
Common STP failures, and how to avoid them
The first STP failure is having no exception path, so that a single bad trade either halts the pipeline or is forced through and corrupts downstream state. A repair queue, peeling exceptions off cleanly while everything else flows straight through, is what makes automation safe. A second failure is skipping per-handoff reconciliation, so that data diverges between stages and the break is discovered only at settlement; reconciling at every handoff catches it immediately.
A third failure is non-idempotent stages, so that reprocessing after a failure double-settles or double-counts; the idempotency built across the earlier courses is what makes safe recovery possible. A fourth is running STP with no observability, so that a stuck trade cannot be found; tracing and monitoring are essential for a continuous flow.
By avoiding these patterns you get an STP platform that is not a fragile demo but a real, operable, auditable system, one that degrades gracefully, recovers safely, and can prove exactly what happened to every trade. Building it is the moment every skill in the bundle becomes one capability.
A worked example: a clean trade and an exception, side by side
Follow two trades through the STP pipeline to see the design work. The first is clean: captured, enriched with reference data, valued against curves, aggregated into position and P&L, checked against limits, reconciled at each handoff, and settled, all automatically, with no human touch. It contributes to the straight-through rate the desk works to maximize.
The second hits an exception: its counterparty is missing a required piece of static data, so enrichment fails. Rather than halting the pipeline or forcing the trade through, the exception peels the trade off into the repair queue, where a steward supplies the missing data, and the trade rejoins the flow and completes. The clean trade was never affected. The worked example shows both paths through the orchestrated pipeline shown earlier, and highlights how the repair queue makes automation safe.
This side-by-side is the essence of STP: clean trades flow straight through, exceptions are handled cleanly without stopping everything, and both are reconciled and audited. It is the whole bundle, working as one system.
Operational realities: running STP as a living system
STP is not a batch job that finishes; it is a living system that runs continuously and must be operated as such. The course covers orchestration that keeps the flow moving, observability and tracing so any trade can be followed through every stage, SLAs so slow stages are noticed, and dead-letter handling so that genuinely unprocessable trades are isolated rather than blocking the flow.
The metrics that matter, the straight-through rate and the manual-touch count, are monitored continuously and drive ongoing improvement: the team attacks the most common exception causes to push automation up over time. Running STP well is a discipline of continuous measurement and improvement, and the course leaves you able not just to build the pipeline but to operate it as the reliable, auditable backbone of a trading platform.
Assembling all of this, the flow, the exceptions, the reconciliation, the observability, into one working system is the capstone of the entire bundle, the point at which every skill from the previous eleven courses becomes a single, operable capability.
Design trade-offs: automation versus control
Straight-through processing is often described as if more automation is always better, but the real design problem is the balance between automation and control. Automate aggressively with weak checks and the flow is fast but unsafe, propagating errors straight through to settlement before anyone notices. Gate every step with heavy manual controls and the flow is safe but no longer straight-through, defeating the purpose. The art of STP is maximizing automation while keeping the controls that make automation safe.
This capstone teaches that balance concretely. Inline validation, credit and limit checks, and per-handoff reconciliation are the controls that let the flow be trusted to run unattended, and the exception path, the repair queue, is what lets it handle the genuinely hard cases without either stopping the whole flow or forcing them through unchecked. The straight-through rate is the measure of how well the balance is struck: a high rate with strong controls is the goal, and a high rate achieved by weakening controls is a trap.
The capstone as the synthesis of the program
This course is where the entire program becomes one thing. Every prior module is a stage in the flow assembled here: the data model gives it a shared language, reference data enriches it, the lifecycle and FpML courses represent trades within it, curves and position and P&L value them, credit and limits control them, the lakehouse and event sourcing give it a platform and an audit backbone, streaming drives its real-time stages, and dashboards make it observable. STP is not a thirteenth topic; it is the integration of the other twelve.
That is why building it is the truest test of the program. Wiring twelve modules into one automated, reconciled, exception-managed flow from capture to settlement is only possible because each module was built to fit, on the same model, with the same temporal discipline, the same idempotency, and the same auditability. When the flow runs end to end on real-shaped trades and reconciles cleanly, you have not just finished a course; you have demonstrated a working ETRM data platform, which is the portfolio piece the whole program was built to produce.
Performance and scale considerations
An STP pipeline's performance is measured end to end, from capture to settlement, and the course treats the flow's throughput and latency as a whole rather than stage by stage. The slowest stage sets the pace, so the course shows how to identify and relieve bottlenecks, how to run independent stages in parallel, and how to combine fast streaming stages with heavier batch stages so neither holds up the flow unnecessarily. The goal is a pipeline that keeps up with the trading it serves.
Scale in STP is also about the exception path, because a flood of exceptions can overwhelm the repair queue and collapse the straight-through rate. The course shows how to keep the queue manageable by fixing the upstream causes of exceptions rather than just processing them, so that the pipeline's automation improves over time rather than degrading. Throughput and straight-through rate are treated together, because a fast pipeline that touches everything manually is not straight-through at all.
Testing and validating the STP pipeline
The capstone's defining test is an end-to-end run: feed real-shaped trades in at capture and confirm they emerge correctly settled, with the exceptions peeling off to the repair queue exactly as designed. The course treats this full-flow test as the proof that the twelve modules truly integrate, because a pipeline that works stage by stage but breaks at the seams is not a working platform. Running it on sample data is how you demonstrate the whole thing holds together.
The course also teaches validating the guarantees that make STP safe: per-handoff reconciliation catches divergence between stages, idempotency is tested by reprocessing trades and confirming nothing double-settles, and the straight-through-rate and manual-touch metrics are measured to confirm the automation is real. You finish having not just built the pipeline but demonstrated that it flows correctly, reconciles at every step, and recovers safely, which is what turns the capstone into a genuine portfolio piece.
How it works, concretely
# The straight-through pipeline: each stage is idempotent, reconciled,
# and routes exceptions to a repair queue instead of failing the flow.
from airflow.decorators import dag, task
@dag(schedule="@continuous", catchup=False)
def stp_pipeline():
@task
def capture(trade): return record_event("CAPTURED", trade)
@task
def enrich(t): # reference & master data validation
ok, t2 = apply_reference_data(t)
return t2 if ok else to_repair(t, reason="missing_static")
@task
def value(t): # curves -> MTM (point-in-time)
return revalue(t, as_of=business_time())
@task
def control(t): # inline credit & limit check
if breaches_limit(t): return to_repair(t, reason="limit_breach")
return t
@task
def reconcile(t): # every handoff is reconciled
assert ties_to_blotter(t), "STP break"
return t
@task
def settle(t): return record_event("SETTLED", t)
# straight-through when clean; exceptions peel off to the repair queue
settle(reconcile(control(value(enrich(capture(next_trade()))))))
stp_pipeline()
# Straight-through rate and manual-touch count are emitted as metrics
# so the desk can drive automation up over time.Chapters 221 to 240
Twenty sequential chapters. Within the full bundle these are chapters 221 through 240 of 240.
- 221What straight-through processing means and why it mattersWhat straight-through processing is and why it is the goal. The capstone is where the whole program becomes a single working system.
- 222The end-to-end flow from capture to settlementTracing the end-to-end flow from capture to settlement. The end-to-end flow is the object every prior course was quietly building toward.
- 223Designing the STP pipeline as connected stagesDesigning the pipeline as clearly connected stages. Designing the pipeline as connected stages with clear contracts is what makes it composable.
- 224Integrating trade capture and the lifecycle modelIntegrating trade capture and the lifecycle model. The lifecycle model from earlier is the spine the capture stage runs on.
- 225Wiring in reference and master data validationWiring in reference and master-data validation. Reference-data validation is the first place a trade can peel off to the repair queue.
- 226Feeding market data and curves into valuationFeeding market data and curves into valuation. Curve-fed valuation in the flow is what makes the pipeline's numbers reproducible.
- 227Computing position and P&L in the flowComputing position and P&L inside the flow. Position and P&L in the flow are the aggregation the control stage checks against.
- 228Embedding credit and limit checks inlineEmbedding credit and limit checks inline. Inline credit and limit checks are what let the flow enforce risk appetite automatically.
- 229The event-sourced backbone of STPUsing the event-sourced backbone for auditability. The event-sourced backbone is what makes the whole flow auditable and recoverable.
- 230Streaming versus batch stages in the pipelineCombining streaming and batch stages coherently. Combining streaming and batch stages coherently is a core capstone design decision.
- 231Exception management and the repair queueManaging exceptions with a repair queue. The repair queue is what lets exceptions be handled without stopping the whole flow.
- 232Straight-through rates and manual-touch metricsTracking straight-through rate and manual-touch metrics. Straight-through rate and manual-touch count are the operational KPIs the desk drives.
- 233Reconciliation at every handoffReconciling at every handoff in the flow. Per-handoff reconciliation is what catches divergence the moment it appears.
- 234Idempotency and safe reprocessingEnsuring idempotent, safe reprocessing. Idempotency is what lets a trade be reprocessed after a failure without double-settling.
- 235Orchestration and workflow controlOrchestrating and controlling the workflow. Orchestration is what controls the flow and sequences its stages reliably.
- 236Observability, tracing, and SLAsAdding observability, tracing, and SLAs. Tracing lets a single trade be followed through every stage of the pipeline.
- 237Failure handling and dead-letter designHandling failure with dead-letter design. Dead-letter design is how genuinely unprocessable items are handled safely.
- 238Settlement, confirmation, and downstream feedsClosing the loop with settlement and downstream feeds. Settlement and downstream feeds are what close the loop and confirm the trade.
- 239Governance, controls, and audit across STPGoverning controls and audit across the whole flow. Governance and controls across STP are what meet the audit expectations trading demands.
- 240The complete STP reference implementationDelivering the complete STP reference implementation. You finish having assembled the complete STP reference implementation end to end.
What you'll be able to do
- Wire all prior modules into one automated capture-to-settlement flow
- Manage exceptions with a repair queue and drive up the straight-through rate
- Reconcile at every handoff and reprocess idempotently
- Orchestrate, observe, and trace the pipeline in production
- Assemble a complete, auditable STP reference implementation
Audience & prerequisites
Senior engineers and architects assembling the whole platform into one automated flow. This capstone assumes the material of the earlier eleven courses.
Prove the module by building it
Assemble the full STP pipeline end to end, integrating all eleven prior modules into one automated flow from capture through settlement, with exception handling, reconciliation, and observability.
This course project is one of twelve that culminate in the bundle's end-to-end capstone, a complete working ETRM data platform run as a straight-through pipeline.
Learn it your way
This course is included in the ETRM Data Engineering and Analytics bundle, available self-paced with PDF handbooks, slide decks, video explainers, and hands-on labs and a project, and deliverable as private corporate training.
Explore the full 12-course bundle and its end-to-end capstone project →
Frequently asked questions
What does the Straight-Through Processing (Capstone) course cover?
The capstone: wiring every prior module into one automated, controlled, exception-managed flow from trade capture to settlement. It runs to 20 sequential chapters (chapters 221 to 240 of the 240-chapter bundle).
Where does this sit in the learning path?
It is course 12 of 12 in the ETRM Data Engineering and Analytics sequence. As the final capstone, it assembles everything from the earlier eleven courses into one straight-through flow.
Is there a project?
Yes. Assemble the full STP pipeline end to end, integrating all eleven prior modules into one automated flow from capture through settlement, with exception handling, reconciliation, and observability.
What technologies are involved?
The course works with Orchestration, Kafka, Airflow, Reconciliation, Observability, chosen to reflect how real ETRM data platforms are built.
Is it available self-paced?
Yes. Every course in the bundle is available self-paced with PDF handbooks, slide decks, video explainers, and hands-on labs and a project, with lifetime access.
Can my team take it as corporate training?
Yes. The whole bundle, or individual courses, can be delivered as private corporate training mapped to your stack and data. Use the contact form to scope it.
Take this course, or the whole platform
Enroll self-paced, or bring the ETRM Data Engineering bundle to your team.