Event Sourcing & Audit Trail Design
Trading systems need a perfect audit trail. This course records every change as an immutable event, so state can be rebuilt, audited, and replayed, and builds an event-sourced trade store with projections and full point-in-time reconstruction.
The pipeline at the heart of this course
Why trading needs a perfect audit trail
A trading system must be able to answer, at any time and for any regulator, exactly what it knew and did and when. State-oriented persistence, where a row is overwritten as it changes, cannot do this: the history is gone the moment it is updated. Event sourcing takes the opposite approach, recording every change as an immutable event and deriving current state from the sequence, so history is never lost.
This course builds an event-sourced core for ETRM. It is the natural conclusion of the lifecycle course: where that course modeled events, this one makes the event log the source of truth and treats current state as a projection. The payoff is a system that is auditable by construction and reconstructable to any past moment.
The course is honest that event sourcing is a deliberate choice with real costs, and it earns its place in trading specifically. The overhead of an append-only log, projections, and snapshots would be hard to justify in a domain that does not need perfect audit, but trading does, and the questions the pattern answers, exactly what happened, in what order, and what did we know when, are precisely the ones regulators and risk functions ask. The course makes the case for when the pattern is worth its weight.
Events, commands, and the append-only log
The building blocks are commands, which express intent, and events, which record what actually happened. Commands may be rejected; events are immutable facts. The event log is append-only, no event is ever changed or deleted, and this immutability is what gives the system its integrity. You learn to model trade events as immutable facts and to design the event store that holds them.
Designing good events is a craft: they must capture what changed with enough context to be meaningful forever, be versioned so the schema can evolve, and be ordered so that replay is deterministic. The course covers event schema design with formats like Avro and the ordering and partitioning that keep a distributed log consistent.
The distinction between commands and events is the conceptual foundation, and the course makes it concrete because it is what keeps the log trustworthy. Commands express intent and may be rejected; events record what actually happened and are immutable, and never confusing the two is what makes the log a reliable record rather than a mutable database in disguise. You learn to design events that carry enough context to remain meaningful forever.
Projections and CQRS
If events are the source of truth, the shapes the application reads from are projections built by replaying events. This is the essence of CQRS, separating the write side, which appends events, from the read side, which serves purpose-built projections: a current-position view, a P&L view, an audit view. Each projection is derived, disposable, and rebuildable from the log.
The course shows how projections are built and kept up to date, and why this separation is powerful: read models can be optimized independently, added without touching the write side, and rebuilt from scratch if they are ever wrong, because the events remain the truth. This directly feeds the dashboards and streaming courses.
Projections and CQRS are where the pattern pays off in flexibility, and the course shows how to exploit that. Because read models are derived and disposable, they can be optimized independently, added without touching the write side, and rebuilt from scratch if they are ever wrong, since the events remain the truth. This is a genuinely different way of thinking about reads, and the course spends time making it intuitive rather than just stating it.
Replay, snapshots, and point-in-time
Because state is derived from events, the system can rebuild any state by replaying the log, and can rebuild state as of any past moment by replaying up to that point. This is the reconstruction capability that audit and risk depend on. The course also addresses its cost: replaying millions of events is slow, so snapshots periodically capture state to bound how far back a replay must go.
Idempotency and exactly-once semantics get careful treatment, because replaying events that have side effects, or processing the same event twice, must not corrupt state. These properties, established here, are what make the streaming and STP courses safe.
Replay, snapshots, and point-in-time reconstruction are the capabilities that make event sourcing worth the effort, and the course treats their cost seriously. Replaying millions of events is slow, so snapshots bound how far back a replay must go, and the course shows how to design them so that current state stays fast while full history stays available. Idempotency and exactly-once semantics, established here, are what make the whole thing safe under failure.
Reconciliation and an event-sourced core
Finally the course connects event sourcing to the operational world through change data capture, turning changes in operational stores into events, and reconciliation, ensuring the event stream and any external state agree. Corrections are modeled as compensating events, never as edits to history, preserving the immutable log.
You leave with an event-sourced ETRM core: commands, an append-only log, projections, snapshots, and point-in-time reconstruction. The artifact below shows the event store and a projection rebuilt by replay.
Connecting the event-sourced core to the operational world is the last piece, and the course does not leave it abstract. Change data capture turns changes in operational stores into events, reconciliation ensures the event stream and any external state agree, and corrections are compensating events rather than edits, so history is never rewritten. You finish with an event-sourced ETRM core that is auditable by construction and reconstructable to any past moment.
Projections, CQRS, and disposable read models
The liberating idea in event sourcing is that read models are disposable. Because the event log is the source of truth, any view you read from, current positions, a P&L summary, an audit trail, is just a projection built by replaying events, and if it is ever wrong or its shape no longer serves you, you can throw it away and rebuild it from the log. Nothing is lost, because the truth was never in the projection.
This is the essence of CQRS, the separation of the write side that appends events from the read side that serves projections. It lets you optimize each independently: the write side for integrity and throughput, each read model for the specific queries it serves. It lets you add a new view without touching the write side, and it lets you fix a buggy projection by correcting its logic and replaying, rather than by risky in-place data surgery.
The course shows how to build and maintain projections, how to keep them current as events arrive, and how to rebuild them safely. This directly enables the dashboards and streaming courses, whose read models and live views are, in this architecture, just more projections of the same authoritative event log.
Replay, snapshots, and reconstruction as of any time
The signature capability of an event-sourced system is reconstruction: because state is derived from events, you can rebuild the state as of any past moment by replaying events up to that point. For a trading platform this is transformative, it means risk can reconstruct a book exactly as it stood before a market move, and audit can answer precisely what was known and when, from the same authoritative log.
The course is honest about the cost, replaying millions of events is slow, and addresses it with snapshots: periodically capturing an aggregate's state so that a replay need only start from the last snapshot rather than the beginning of time. This bounds reconstruction cost without compromising the log as the source of truth, since snapshots are an optimization, not an alternative truth.
Idempotency and exactly-once semantics get careful treatment here too, because replaying events that have side effects, or processing an event twice, must never corrupt state. These properties, established in this course, are what make the streaming and STP courses safe to build, since both depend on being able to reprocess without double-counting.
Where this course sits, and what it unlocks
Event sourcing is the ninth course, generalizing the event-over-state pattern from the lifecycle course into the platform's persistence strategy. It gives the platform a perfect audit trail and point-in-time reconstruction, which the dashboards course exploits for time-travel views and the STP capstone relies on for auditability and safe reprocessing.
The projections and idempotency you build here are what make the streaming course's read models and exactly-once guarantees possible, so this course is the backbone on which the real-time and automated layers rest.
Common event-sourcing failures, and how to avoid them
The subtle failure in event sourcing is treating projections as authoritative rather than disposable, so that fixing a projection means risky data surgery instead of a simple replay. Keeping the event log as the single source of truth, with every read model derived and rebuildable, avoids this entirely. A second failure is unbounded replay cost, letting event histories grow so large that reconstruction becomes impractical; snapshots bound this without compromising the log.
A third failure is editing history, correcting a mistake by mutating a past event, which destroys the immutability the whole approach depends on. Corrections must be new compensating events, never edits. The course is emphatic on this because a mutable event log is not an audit trail at all.
By avoiding these patterns you get an event-sourced core that delivers on its promise: an immutable, replayable history from which any state, as of any time, can be reconstructed with confidence.
A worked example: reconstructing a book before a crash
Suppose risk needs to know exactly how a book stood the moment before a market crash, to understand what the desk was exposed to. In a state-oriented system this is often impossible, because the state has since been overwritten. In the event-sourced core, it is a query: replay the events up to that timestamp, and the book's state as of that instant is reconstructed exactly.
The worked example shows the replay, folding trade-booked and amended events up to the chosen point-in-time, rebuilding the position that existed then. Because the events are immutable and the state is derived from them, the reconstruction is authoritative, not an approximation. Risk can now analyze precisely what the desk held before the crash, and audit can answer exactly what was known and when.
This is the capability that justifies event sourcing for trading: perfect historical reconstruction from an immutable log, which no mutable-state design can match.
Operational realities: snapshots, schema evolution, and rebuilds
Operationally, replaying long event histories is bounded with snapshots, periodically capturing state so a replay starts near the point of interest rather than at the beginning of time. The course covers when and how to snapshot without compromising the log as the source of truth, since snapshots are an optimization, never an alternative truth.
Schema evolution and projection rebuilds are the other operational realities. Event schemas evolve over time, and the course shows how to version them so old events remain interpretable. And because projections are disposable, a buggy read model is fixed by correcting its logic and replaying, not by risky data surgery, so operating an event-sourced system is, in this respect, safer than operating a mutable one.
Design trade-offs: event sourcing versus state, and its cost
Event sourcing is powerful but not free, and the course is honest about its cost. Storing every change as an immutable event gives perfect auditability and point-in-time reconstruction, but it makes the simplest query, what is the current state, into a replay, and it demands snapshots, projection management, and careful schema evolution that a plain updatable table never needs. For a domain without strong audit requirements, that overhead would be hard to justify.
Trading is precisely the domain where it is justified, because the questions event sourcing answers, exactly what happened, in what order, and what did we know when, are regulatory and operational necessities rather than nice-to-haves. The course teaches how to contain the cost: snapshots to bound replay, well-designed events to keep the log meaningful, and projections built only for the read shapes the application actually needs. The trade-off is deliberate, applied where its benefits are worth its weight.
Carrying event sourcing into the capstone
The event-sourced log is the backbone that makes the STP capstone auditable and recoverable. Each stage of the flow emits events, so the entire journey of a trade through capture, enrichment, valuation, control, and settlement is reconstructable, and any stage can be safely reprocessed by replaying from the log. Corrections are compensating events, never edits, so the capstone's history is never rewritten.
This is what lets the capstone meet the governance and reconstruction expectations that trading demands. When a regulator or a control function asks the platform to show exactly what it did and when, the answer is a replay of the log, and that capability, built here, is what turns a fast automated flow into a fast automated flow you can defend.
Performance and scale considerations
Event sourcing's main performance concern is replay cost, and the course addresses it as a central design problem rather than a footnote. Rebuilding state by folding millions of events is slow, so snapshots capture state periodically to bound how far back a replay must go, and the course shows how to choose snapshot frequency to balance replay speed against snapshot storage. The everyday read stays fast because it starts from a recent snapshot, not the beginning of time.
The append-only log also has to sustain high write throughput without losing ordering, so the course covers partitioning the log for parallelism while preserving the per-aggregate order that correctness needs. Projections are kept current incrementally rather than rebuilt from scratch each time, and the course shows how to scale them independently of the write side, which is one of the pattern's key advantages.
Testing and validating an event-sourced core
Event sourcing is unusually testable, and the course makes the most of it. Because state is a pure function of events, you can assert that a known event sequence produces a known state, that replaying events is deterministic, and that a projection rebuilt from scratch matches the one maintained incrementally. This last test is powerful, because it directly validates that the incremental read model has not drifted from the truth in the log.
The course also teaches testing point-in-time reconstruction and idempotency explicitly: rebuild state as of past moments and confirm it matches history, and apply events more than once to confirm the result is unchanged. These are exactly the properties audit and recovery depend on, so testing them directly is how you earn confidence in the core. You finish able to validate that your event-sourced system is auditable and recoverable in fact, not just in design.
How it works, concretely
-- The single source of truth: an append-only, immutable event log
CREATE TABLE event_store (
global_seq bigserial PRIMARY KEY, -- total order
aggregate_id bigint NOT NULL, -- e.g. trade_id
event_type text NOT NULL,
version int NOT NULL, -- per-aggregate order
payload jsonb NOT NULL,
occurred_at timestamptz NOT NULL,
UNIQUE (aggregate_id, version)
);
-- No UPDATE or DELETE is ever issued against this table.
-- A projection: current net position, rebuilt purely from events.
-- Replay all TRADE_BOOKED / TRADE_AMENDED events up to an optional as-of.
SELECT aggregate_id AS trade_id,
SUM((payload->>'signed_qty')::numeric) AS net_qty
FROM event_store
WHERE event_type IN ('TRADE_BOOKED','TRADE_AMENDED')
AND occurred_at <= COALESCE(:as_of, 'infinity'::timestamptz) -- point in time
GROUP BY aggregate_id;
-- Snapshots bound replay cost for hot aggregates
CREATE TABLE aggregate_snapshot (
aggregate_id bigint PRIMARY KEY,
version int NOT NULL, -- events up to here are folded in
state jsonb NOT NULL,
taken_at timestamptz NOT NULL
);Chapters 161 to 180
Twenty sequential chapters. Within the full bundle these are chapters 161 through 180 of 240.
- 161Why trading systems need a perfect audit trailWhy regulated trading needs a perfect, immutable audit trail. The audit trail is a regulatory necessity, and event sourcing provides it by construction.
- 162State-oriented vs event-oriented persistenceContrasting state-oriented with event-oriented persistence. State-oriented persistence destroys history the moment it overwrites a row.
- 163Events, commands, and the append-only logModeling commands, events, and the append-only log. The append-only log is what gives the system its integrity and reconstructability.
- 164The event store and its data modelDesigning the event store and its data model. The event store's design determines how efficiently events are written and replayed.
- 165Modeling trade events as immutable factsRecording trade events as immutable facts. Modeling events as immutable facts is what keeps the log trustworthy forever.
- 166Projections and building read modelsBuilding read models as projections of events. Projections are the read shapes the application actually queries, built from events.
- 167CQRS: separating writes from readsSeparating writes from reads with CQRS. CQRS lets read and write sides be optimized and scaled independently.
- 168Rebuilding state by replaying eventsRebuilding current state by replaying events. Rebuilding state by replay is what makes any past state recoverable.
- 169Point-in-time state and temporal queriesAnswering point-in-time queries from the log. Point-in-time reconstruction is the capability audit and risk most depend on.
- 170Idempotency and exactly-once semanticsGuaranteeing idempotency and exactly-once handling. Idempotency and exactly-once are what make replay and reprocessing safe.
- 171Event schemas and evolution with AvroEvolving event schemas safely with Avro. Avro schemas let events evolve without breaking existing consumers.
- 172Ordering, partitioning, and consistencyGetting ordering, partitioning, and consistency right. Ordering and partitioning are what keep a distributed log consistent.
- 173Snapshots to bound replay costUsing snapshots to bound replay cost. Snapshots bound replay cost so current state is fast to obtain for hot aggregates.
- 174The audit trail as a first-class productTreating the audit trail as a first-class product. Treating the audit trail as a product is what makes it genuinely useful, not incidental.
- 175Change data capture from operational storesCapturing change from operational stores. Change data capture turns changes in operational stores into events for the log.
- 176Reconciling event streams and stateReconciling event streams against state. Reconciling stream and state is what catches silent divergence early.
- 177Compensating events and correctionsCorrecting with compensating events, never edits. Compensating events correct mistakes without ever rewriting history.
- 178Regulatory reconstruction and lineageReconstructing history for regulators and lineage. Regulatory reconstruction is exactly what an event-sourced core makes straightforward.
- 179Testing and validating an event-sourced systemTesting and validating an event-sourced system. Testing an event-sourced system has its own techniques the course makes concrete.
- 180An event-sourced ETRM core end to endAssembling an event-sourced ETRM core end to end. You finish with an event-sourced ETRM core with projections, snapshots, and replay.
What you'll be able to do
- Model changes as immutable events in an append-only log
- Build read models as projections and apply CQRS
- Reconstruct any state as of any past moment by replay
- Bound replay cost with snapshots without compromising the log
- Guarantee idempotency so reprocessing never corrupts state
Audience & prerequisites
Engineers who need a perfect audit trail and the ability to reconstruct history, and anyone building the immutable core a trading platform can trust.
Prove the module by building it
Implement an event-sourced trade store: append-only events, projections for read models, and a full audit trail that can rebuild any state as of any time.
This course project is one of twelve that culminate in the bundle's end-to-end capstone, a complete working ETRM data platform run as a straight-through pipeline.
Learn it your way
This course is included in the ETRM Data Engineering and Analytics bundle, available self-paced with PDF handbooks, slide decks, video explainers, and hands-on labs and a project, and deliverable as private corporate training.
Explore the full 12-course bundle and its end-to-end capstone project →
Frequently asked questions
What does the Event Sourcing & Audit Trail Design course cover?
Recording every change as an immutable event, so state can be rebuilt, audited, and replayed with confidence. It runs to 20 sequential chapters (chapters 161 to 180 of the 240-chapter bundle).
Where does this sit in the learning path?
It is course 9 of 12 in the ETRM Data Engineering and Analytics sequence. It builds on the courses before it and prepares for those after it.
Is there a project?
Yes. Implement an event-sourced trade store: append-only events, projections for read models, and a full audit trail that can rebuild any state as of any time.
What technologies are involved?
The course works with Kafka, Event store, CQRS, Avro, Debezium, chosen to reflect how real ETRM data platforms are built.
Is it available self-paced?
Yes. Every course in the bundle is available self-paced with PDF handbooks, slide decks, video explainers, and hands-on labs and a project, with lifetime access.
Can my team take it as corporate training?
Yes. The whole bundle, or individual courses, can be delivered as private corporate training mapped to your stack and data. Use the contact form to scope it.
Take this course, or the whole platform
Enroll self-paced, or bring the ETRM Data Engineering bundle to your team.