Market Data, Curves & Time-Series
Market data is the fuel of valuation and risk. This course engineers the time-series and curve infrastructure, bootstrapping forward curves, storing them bitemporally, and serving point-in-time values that make every revaluation reproducible.
The pipeline at the heart of this course
Market data as the fuel
No position is worth anything until it is valued, and no valuation is possible without market data: prices, curves, volatilities. This course builds the infrastructure that turns raw quotes into the curves valuation consumes. It is unglamorous and absolutely central, because the quality and reproducibility of every risk and P&L number depends on it.
The defining challenge is time. Market data is observed continuously, revised, and consumed as of specific moments, and a revaluation done today for last Tuesday must use the curve as it was known then, not as it is known now. Getting this right is what separates a reproducible risk system from one that can never quite explain its own numbers.
The unglamorous truth the course keeps in view is that the curve service is where the reproducibility of the entire platform is decided. Every position's value, every P&L number, every risk measure traces back to the curves it was struck against, and if those curves cannot be retrieved exactly as they were, none of the downstream numbers can be reproduced. The bitemporal design is not a refinement bolted on for completeness; it is the property that makes the whole platform auditable.
Time-series modeling and bitemporality
You begin with the time-series data model: how observations are stored, indexed, and retrieved efficiently at scale. Then comes the crucial refinement, bitemporality, where every data point carries both the date it refers to and the date it was observed. This two-axis model is what lets the system answer both what is the curve for next month and what did we think the curve for next month was, last Tuesday.
The course is concrete about storage: columnar formats for scan efficiency, compression for volume, and indexing for point-in-time retrieval. These are not incidental; the difference between a curve service that answers in milliseconds and one that stalls a revaluation is in these choices.
The two-axis view of time is the idea the course spends the most effort making intuitive, because it is genuinely counterintuitive at first. A data point does not just have a value and a date; it has the date it refers to and the date it was observed, and revisions add new observations rather than overwriting old ones. Once this clicks, a whole class of reconciliation problems dissolves, because the question of what we knew when finally has a precise, queryable answer.
Bootstrapping forward curves
A forward curve is not observed directly; it is constructed from the prices of traded instruments through bootstrapping, solving for the curve that reprices the market. The course teaches the mechanics: which instruments anchor which parts of the curve, how interpolation fills the gaps, and how the choice of interpolation method changes the risk the curve implies.
Energy and commodity curves add seasonality, monthly and yearly shapes that a naive interpolation would smooth away wrongly. You learn to construct curves that respect that structure, and to represent volatility surfaces and correlation matrices with the same care, because options valuation depends on them.
Bootstrapping is where market data becomes a usable curve, and the course is concrete about the mechanics because the choices matter. Which instruments anchor which parts of the curve, how the gaps between them are interpolated, and how seasonality is preserved all change the risk the curve implies, and a curve that reprices the market but carries the wrong risk is a subtle trap. You learn to construct curves that are both consistent with observed prices and sensible in the risk they express.
Point-in-time retrieval and snapshots
The heart of a curve service is point-in-time retrieval: given a valuation date and an as-of date, return exactly the curve that was in effect. The course shows how bitemporal storage makes this a query rather than an archaeology project, and how end-of-day snapshots freeze the official curves that P&L and risk are struck against.
You also handle the messy reality: gaps where a quote is missing, spikes from bad ticks, and the validation that catches them before they poison a curve. A single bad tick that survives into a curve can misstate the risk of an entire book, so validation is not optional.
Point-in-time retrieval and end-of-day snapshots are what turn the store into a service the rest of the platform can rely on. The snapshot fixes the official curves that P&L and risk are struck against, and point-in-time retrieval lets any past valuation be reproduced against exactly the curve that was live then. The course shows how these two capabilities, together, serve both the trader who wants the latest view and the risk function that needs a frozen, reproducible one.
Serving curves at scale
Finally, the curve service must serve valuation at scale, thousands of revaluations striking against the same curves, with consistency and speed. The course covers caching, snapshot distribution, and the interface valuation consumes. You leave with a production curve pipeline: ingest, bootstrap, validate, snapshot, store bitemporally, and serve point-in-time.
The artifact below shows the bitemporal curve store and a point-in-time retrieval query, the query that makes every downstream valuation reproducible.
Serving curves at scale is the engineering that makes all of this usable in production, and the course does not skip it. Thousands of revaluations striking against the same curves demand fast, consistent retrieval, which drives the storage-format, caching, and interface choices. You finish with a pipeline that goes from raw quotes to validated, bitemporally stored, point-in-time-retrievable curves served fast enough for the valuation workloads that depend on them.
Bitemporality, in depth: the two axes of time
The single most important idea in this course is bitemporality, and it repays careful thought. There are two independent time axes for market data. One is the date the data is about, the tenor of a forward point, the date a fixing applies to. The other is the date on which the system observed or recorded that value. These are genuinely independent: today you might record a revised value for a fixing that applied last week.
Modeling both axes explicitly is what lets the system answer two very different questions without confusion. What is the curve for next month? uses the tenor axis. What did we think next month's curve was, as of last Tuesday's close? uses the observation axis. A system that collapses the two, storing only the latest value, can answer the first question but not the second, and the second is exactly what risk reconstruction and dispute resolution require.
The course shows how bitemporal storage turns these into simple queries rather than archaeology, and how it underpins reproducible valuation: revaluing last Tuesday's book uses last Tuesday's observation date and gets precisely the curves that were live then, no matter how they have been revised since. This reproducibility is a property regulators increasingly expect and desks increasingly depend on.
Construction, validation, and the danger of a single bad tick
Curve construction is where market data becomes a financial instrument's valuation, and small errors here have large consequences. A single bad tick, a fat-fingered quote, a stale price, a decimal error, that survives into a curve can misstate the value and risk of an entire book. The course therefore treats validation not as a preliminary chore but as a core part of construction.
You learn robust validation techniques: detecting spikes against a robust measure of dispersion so that a genuine move is not mistaken for an error and vice versa, handling gaps where a quote is simply missing, and reconciling constructed curves against independent checks. The aim is a curve you can stake a book on, with the confidence that comes from knowing every input was validated.
Interpolation gets similar rigor, because the choice of method is not neutral, it changes the risk the curve implies between observed points, and in energy and commodity markets it must respect seasonality that a naive method would smooth away. The course connects these choices to their downstream effect on position, P&L, and risk, so you understand not just how to build a curve but what your construction choices commit the desk to.
Where this course sits, and what it unlocks
Curves is the fifth course, supplying the valuation inputs that the position-and-P&L course consumes. Without reproducible curves, P&L cannot be reproducible, and without reproducible P&L, risk reconstruction and dispute resolution are impossible. The bitemporal curve store you build here is therefore a linchpin of the whole platform's reproducibility.
The course also connects backward to the modeling course, whose bitemporal patterns it applies to market data, and forward to the streaming course, which drives real-time revaluation off the same curves. Getting the curve infrastructure right unlocks trustworthy valuation everywhere it is used.
Common market-data failures, and how to avoid them
The most consequential market-data failure is the single bad tick that survives into a curve and misstates the value and risk of an entire book. Validation, detecting spikes robustly, handling gaps, reconciling against independent checks, is the defense, and the course treats it as a core part of construction rather than an afterthought.
A second failure is collapsing the two axes of time, storing only the latest value, which makes reproducible point-in-time valuation impossible and turns every dispute into archaeology. Bitemporal storage prevents this. A third is interpolation that ignores seasonality, smoothing away the monthly and yearly structure that energy and commodity curves must respect, which silently distorts risk.
By avoiding these patterns you get a curve service you can stake a book on: validated, reproducible, and faithful to the real shape of the market.
A worked example: revaluing a book as of last week
Consider the question risk asks constantly: what was this book worth at last Tuesday's close, and what was its risk. Answering it correctly requires the bitemporal curve store. The revaluation runs with a valuation date and an as-of observation date of last Tuesday, and the point-in-time query returns exactly the curves that were live then, not today's revised values.
The book's positions are revalued against those historical curves, producing the mark-to-market and risk as they genuinely stood, reproducibly, no matter how the curves have been revised since. This is the difference between a risk system that can defend its historical numbers and one that cannot, and it falls directly out of storing both axes of time. The worked example shows the query, the retrieved curve, and the revaluation, making the abstract idea of bitemporality concrete.
The same mechanism resolves disputes, when a counterparty questions a historical valuation, and satisfies regulators, who increasingly expect reproducible point-in-time numbers. One design choice, bitemporal storage, unlocks all of it.
Operational realities: snapshots, gaps, and serving load
Operationally, a curve service must produce official end-of-day snapshots reliably, because P&L and risk are struck against them, and must handle the daily mess of missing quotes and bad ticks without producing a broken curve. The course covers the snapshot process, the validation that guards it, and the fallbacks for gaps, so that the official curves are always complete and trustworthy.
Serving load is the other operational concern: thousands of revaluations striking against the same curves demand a service that answers fast and consistently, which the course addresses with caching, columnar storage, and snapshot distribution. The result is a curve service that is not only correct and reproducible but also fast enough and robust enough for production valuation at scale.
Design trade-offs: intraday freshness versus reproducibility
A curve service is pulled in two directions. Traders want the freshest possible curves, updated intraday as the market moves, so their views reflect the world now. Risk and P&L want frozen, official curves struck at end-of-day, so that yesterday's numbers can be reproduced exactly today and every time after. A service that serves only fresh curves cannot reproduce history; one that serves only frozen curves is blind intraday.
The bitemporal design this course teaches resolves the tension rather than choosing a side. Every curve point records both the tenor it refers to and the moment it was observed, so the same store serves the live intraday view, by asking for the latest observation, and the reproducible historical view, by asking for the observation as of a past moment. End-of-day snapshots then mark which observations are the official ones. This is why the bitemporal investment, which looks like overkill for a simple curve, is exactly what a trading platform needs.
Carrying curves into the capstone
In the STP flow, the curve service is what the valuation stage calls, and its point-in-time guarantee is what makes the capstone's valuations reproducible and auditable. When a trade is revalued in the flow, it is struck against the curve as of the business moment, and because that curve can be retrieved again identically, any later reconciliation or audit reproduces the exact number the flow produced.
The validation you build here also feeds the capstone's integrity. A bad tick that survives into a curve would misprice every position that touches it, and the flow would carry that error straight through to P&L. So the curve service is not just a data source for the capstone; it is one of the guarantees the capstone's reproducibility rests on.
Performance and scale considerations
Curve retrieval sits directly under valuation, so its performance sets a ceiling on how fast the whole platform can revalue, and the course treats this as a first-order concern. Bitemporal storage can be slow if queried naively, so the course teaches indexing on the observation and tenor axes, snapshotting official curves so end-of-day retrieval does not replay history, and columnar storage so that reading a whole curve is a single efficient scan.
At scale, thousands of revaluations strike against the same curves simultaneously, which makes caching and consistent snapshot distribution essential. The course shows how to serve a stable, agreed set of curves to a large valuation run without each consumer re-querying the store, and how to keep those served curves consistent so that two revaluations in the same run cannot disagree about the market.
Testing and validating curves
A curve is a construction, so it can be tested against the market it was built from: a well-bootstrapped curve reprices its anchor instruments, and the course uses that as a direct correctness check. If the curve does not return the prices of the instruments it was built from, something is wrong, and this test catches a whole class of construction errors before the curve ever reaches valuation.
The course also teaches testing the reproducibility that is the curve service's whole point: retrieve a past curve as of a past moment, and confirm it matches what was used then, exactly. Combined with the tick-validation tests that confirm bad observations are rejected, this gives a curve service whose reproducibility and quality are demonstrated rather than assumed, which is what every downstream number depends on.
How it works, concretely
-- Each curve point carries both axes of time
CREATE TABLE curve_point (
curve_id text NOT NULL, -- e.g. 'BRENT.FWD'
tenor date NOT NULL, -- the date the point refers to
value numeric(20,8) NOT NULL,
observed_at timestamptz NOT NULL, -- when we learned this value
observed_to timestamptz NOT NULL DEFAULT 'infinity',
PRIMARY KEY (curve_id, tenor, observed_at)
);
-- Point-in-time: "the BRENT forward curve as we knew it at :as_of"
SELECT tenor, value
FROM curve_point
WHERE curve_id = 'BRENT.FWD'
AND observed_at <= :as_of -- known by the as-of moment
AND observed_to > :as_of -- and not yet superseded
ORDER BY tenor;
-- Revaluing last Tuesday's book uses last Tuesday's :as_of and gets
-- exactly the curve that was live then, no matter how it was revised since.import numpy as np
def interpolate(tenors, values, target):
"""Log-linear interpolation, the workhorse for forward curves."""
t = np.array([d.toordinal() for d in tenors], float)
v = np.log(np.array(values, float))
x = target.toordinal()
return float(np.exp(np.interp(x, t, v))) # flat-extrapolate the ends
def validate_ticks(series, z=6.0):
"""Reject spikes: a tick more than z robust-sigmas from the median."""
a = np.array(series, float)
med = np.median(a)
mad = np.median(np.abs(a - med)) or 1e-9
return [x for x in series if abs(x - med) / (1.4826 * mad) <= z]Chapters 81 to 100
Twenty sequential chapters. Within the full bundle these are chapters 81 through 100 of 240.
- 081Market data as the fuel for valuation and riskWhy every valuation and risk number depends on clean market data. No position has a value until market data turns it into one, which is why this course is pivotal.
- 082Quotes, fixings, and observation dataModeling quotes, fixings, and raw observations. Quotes and fixings are the raw observations everything else is constructed from.
- 083Time-series data models and storageDesigning time-series storage that scales and retrieves fast. The time-series model determines how efficiently curves can be stored and retrieved at scale.
- 084Bitemporal market data: as-of and observation dateStoring market data bitemporally by as-of and observation date. The two time axes are what make a revaluation reproducible rather than approximate.
- 085Forward curves: concepts and constructionUnderstanding forward curves and how they are constructed. A forward curve is constructed, not observed, so its construction method matters.
- 086Bootstrapping a curve from instrumentsBootstrapping a curve from the prices of traded instruments. Bootstrapping solves for the curve that reprices the market instruments that anchor it.
- 087Interpolation methods and their trade-offsChoosing interpolation methods and understanding their risk impact. The interpolation choice quietly changes the risk the curve implies, so it is deliberate.
- 088Seasonality in energy and commodity curvesRespecting seasonality in energy and commodity curves. Energy and commodity seasonality must be respected or the curve misstates real structure.
- 089Volatility surfaces and their representationRepresenting volatility surfaces for options valuation. Volatility surfaces are what options valuation depends on, so their representation is careful.
- 090Correlation matrices and their data modelModeling correlation matrices and their storage. Correlation data underpins portfolio-level risk and must be modeled coherently.
- 091Interest-rate and discount curvesBuilding interest-rate and discount curves. Discount curves connect commodity valuation to the time value of money.
- 092FX curves and cross-currency mechanicsHandling FX curves and cross-currency mechanics. FX curves and cross-currency mechanics are what let a book be valued in one base currency.
- 093Curve versioning and point-in-time retrievalVersioning curves for exact point-in-time retrieval. Point-in-time retrieval is the query that makes every downstream valuation reproducible.
- 094Handling gaps, spikes, and bad ticksHandling gaps, spikes, and bad ticks safely. Handling gaps and spikes is what keeps a single bad observation from poisoning a curve.
- 095Market-data validation and quality controlValidating market data before it can poison a curve. Validation catches bad ticks before they can misstate the risk of an entire book.
- 096Snapshotting curves for end-of-daySnapshotting official curves at end of day. End-of-day snapshots freeze the official curves that P&L and risk are struck against.
- 097Intraday vs end-of-day market dataDistinguishing intraday from end-of-day market data. The intraday and end-of-day distinction is what serves both traders and risk from one store.
- 098Storage formats: columnar and compressionChoosing columnar formats and compression for scale. Columnar storage and compression are what make curve retrieval fast and cheap at scale.
- 099Serving curves to valuation at scaleServing curves to valuation at production scale. Serving curves to valuation at scale is where storage and interface design are tested.
- 100A production curve pipeline end to endAssembling ingest, bootstrap, validate, snapshot, and serve. You finish with a production curve pipeline from ingestion to point-in-time serving.
What you'll be able to do
- Store market data bitemporally for reproducible point-in-time valuation
- Bootstrap and interpolate forward curves that respect seasonality
- Validate market data so a single bad tick cannot poison a curve
- Snapshot official end-of-day curves for P&L and risk
- Serve curves to valuation at production scale
Audience & prerequisites
Data engineers building the market-data and curve infrastructure behind valuation and risk, and quants who need that infrastructure to be reproducible.
Prove the module by building it
Build a curve-construction pipeline: ingest quotes, bootstrap a forward curve, store it bitemporally, and expose point-in-time curve retrieval for revaluation.
This course project is one of twelve that culminate in the bundle's end-to-end capstone, a complete working ETRM data platform run as a straight-through pipeline.
Learn it your way
This course is included in the ETRM Data Engineering and Analytics bundle, available self-paced with PDF handbooks, slide decks, video explainers, and hands-on labs and a project, and deliverable as private corporate training.
Explore the full 12-course bundle and its end-to-end capstone project →
Frequently asked questions
What does the Market Data, Curves & Time-Series course cover?
Forward curves, volatility surfaces, and the time-series engineering that feeds every valuation and risk number. It runs to 20 sequential chapters (chapters 81 to 100 of the 240-chapter bundle).
Where does this sit in the learning path?
It is course 5 of 12 in the ETRM Data Engineering and Analytics sequence. It builds on the courses before it and prepares for those after it.
Is there a project?
Yes. Build a curve-construction pipeline: ingest quotes, bootstrap a forward curve, store it bitemporally, and expose point-in-time curve retrieval for revaluation.
What technologies are involved?
The course works with Time-series DB, NumPy, pandas, Interpolation, Arrow, chosen to reflect how real ETRM data platforms are built.
Is it available self-paced?
Yes. Every course in the bundle is available self-paced with PDF handbooks, slide decks, video explainers, and hands-on labs and a project, with lifetime access.
Can my team take it as corporate training?
Yes. The whole bundle, or individual courses, can be delivered as private corporate training mapped to your stack and data. Use the contact form to scope it.
Take this course, or the whole platform
Enroll self-paced, or bring the ETRM Data Engineering bundle to your team.