Skip to content
Corporate & Investment Banking My progress Français

Home / Capital Markets & Securities Services / Cross-functional / Module 28

28Market Data & Reference Data

The Data Layer Beneath Every CIB Business: From the vendor contract to the regulatory report, following one instrument through the machinery that decides what a bank believes to be true

The data layer that decides what the bank believes to be true.

Reading time
2 h 30
Sections
49
Recall questions
58
Quiz questions
15

Two threads run through this module

The business thread asks a deceptively simple question: what does the bank actually believe, and who decided? A bond has one ISIN but five prices, three of which disagree. A client group has one name but forty legal entities across nine jurisdictions. Somewhere in the bank a rule, a person or a machine resolves that ambiguity into a single number and a single hierarchy, and that resolution flows straight into P&L, capital, collateral calls and the accounts. Market data and reference data are not plumbing. They are the mechanism by which a bank forms a view of the world and is then held to it.

The technology thread asks how that belief is manufactured at scale, at speed, and with proof. A tier-one bank ingests hundreds of feeds, maintains millions of instruments, and answers to regulators who expect it to reconstruct any number on demand years later. That combination — high throughput, high accuracy, full traceability — is one of the harder engineering problems in the industry, and it is the problem most often underestimated by teams who have only worked on the trading applications sitting on top of it.

How to read this module

Read it in order. The five parts build on each other: Part 1 establishes what the data is and what the market for it looks like, Part 2 follows a single instrument through the full lifecycle, Part 3 maps the people, Part 4 opens the systems, and Part 5 examines what goes wrong and where the money and risk actually move. Every technical term is defined at first use. Nothing assumes prior knowledge of banking.

Coloured boxes carry specific weight. Key insight marks a point worth remembering after everything else fades. Going deeper offers detail for readers who want mechanism rather than summary. IT & data lens connects a business step to the systems that carry it. Watch out flags a common misconception or a genuine trap. Worked example applies the concept to the running case. In a nutshell closes each part. Check your understanding tests whether the part landed.

You will understand how a raw price becomes an authorised internal truth

The path from an exchange tick to an approved valuation mark involves at least six deliberate decisions, each of which can be defended or challenged. By the end of Part 2 you will be able to name each one, say who makes it, and explain what happens downstream when it is made badly.

You will be able to map every actor in the data supply chain

Exchanges, vendors, aggregators, standards bodies, regulators, internal stewards and internal consumers each want something different from the same data. Part 3 gives you the full cast, the commercial tension between them, and the reason the bank's data function sits uncomfortably in the middle of all of it.

You will understand where the number actually comes from

For a liquid share the answer is trivial. For an illiquid bond, a bespoke swap or a private credit position it is a judgement built from imperfect sources, subject to an independent price verification process and ultimately signed off by someone who can be held responsible. You will be able to explain that construction and its control.

You will be able to trace data front to back

From feed handler to golden copy to risk engine to regulatory submission, you will be able to follow one field through every system it touches, name the identifier that binds the chain together, and explain why an error at the front is never contained at the front.

You will recognise the risk, cost and transformation landscape

Market data is one of the largest discretionary technology line items in a bank, and one of the most concentrated supplier markets in financial services. You will understand the six risks the function carries, the metrics that separate a well-run data operation from a lucky one, and which of the current transformation trends genuinely move cost or risk.

1Data is the raw material of the bank, and it is bought, not found

Before following the process, you need to know what the material is. This part builds the concept of financial data from zero, sorts the market into its distinct product classes, sizes the industry, and explains why a business with no inventory and no balance sheet exposure nonetheless behaves like a commodity supply chain with a concentrated supplier base.

Financial data is a manufactured product, not a natural fact

A price is an event, and an event has to be captured by someone before it can be sold. When two parties agree to trade a share on a regulated exchange, the exchange records the price, the quantity and the timestamp. That record does not exist anywhere else. The exchange owns it, publishes it under rules set by its regulator, and charges for access to it. The same is true of a bond traded over the counter, where the record is created when the trade is reported to an approved publication arrangement, and of a corporate action, where the record originates with the issuer and travels through the custodian chain. In every case the data is the by-product of an economic event, captured by an institution that has a commercial interest in that capture.

This matters because it makes data a product with a supply chain rather than a public good. The bank does not observe the market directly. It buys the observation, in a specific format, under a specific licence, with specific restrictions on who inside the bank may see it and what they may do with it. Understanding market data means understanding that chain of ownership, because almost every constraint that shapes a bank's data architecture — the entitlement systems, the usage reporting, the caching rules, the redistribution controls — descends from a contract, not from a technical limitation.

Market data and reference data answer different questions about the same instrument

Market data is what the instrument is worth right now. Prices, quotes, volumes, order-book depth, yields, spreads, index levels, volatility surfaces. It changes constantly, sometimes millions of times a second in aggregate across the venues a bank consumes. Its defining property is that it is time-stamped and perishable: a price without a timestamp is meaningless, and a price from four hours ago may be worse than no price at all.

Reference data is what the instrument is. The issuer, the maturity date, the coupon, the day-count convention, the currency, the listing venue, the identifiers, the credit rating, the corporate structure of the legal entity that issued it. It changes rarely but changes consequentially, and its defining property is that it is descriptive rather than perishable. A bond's maturity date does not move; when it does move, because the terms were amended, every downstream calculation that used the old date is now wrong and no one will notice until something fails.

The two are useless apart. A price with no instrument attached is a number floating in space. An instrument with no price cannot be valued, risked or reported. The industry treats them as separate disciplines with separate teams, separate vendors and separate systems, largely for historical reasons, and the seam between them is where a disproportionate share of operational failures occurs. Throughout this module the two are treated as one supply chain with two kinds of cargo.

The bank consumes data in five distinct roles, and each role wants something different

The trading desk wants speed and completeness of the live picture. A market maker quoting a corporate bond needs the current state of the market with the lowest achievable latency, and cares far more about receiving every update in order than about whether the number is philosophically the right one. Milliseconds matter; auditability matters less.

Risk management wants completeness and consistency across the whole book. A value-at-risk calculation is only as good as its worst-covered position. A single instrument without a price forces a proxy, and a proxy is an assumption the risk committee has to accept. Risk does not need microsecond latency, but it needs every position priced from a consistent source on a consistent date.

Finance and product control want accuracy that will survive an audit. The valuation that reaches the income statement must be independently verified, documented and defensible to an external auditor and, for the fair-value hierarchy under IFRS 13, classified by how observable its inputs were. This is where the concept of an official mark is born.

Compliance and regulatory reporting want traceability. Every transaction report submitted under MiFIR or EMIR carries dozens of reference-data fields, and the regulator's question is never simply "is this right?" but "show me where it came from and prove it has not been altered." Lineage is the product here, not the number.

Client-facing and digital channels want presentability and licence cleanliness. The moment the bank shows a price to a client on a portal, that price has been redistributed, and redistribution is a separate and considerably more expensive licence category than internal use. The commercial constraint arrives before the technical one.

Where the data function sits inside the bank

Market data and reference data are not a business line, and that is the source of most of their organisational difficulty. Equity Capital Markets earns fees. A rates desk earns a spread. The data function earns nothing and spends heavily, which means it is a cost centre serving internal clients who each believe their requirement is the priority. It typically reports into Chief Operating Officer or Chief Data Officer structures, sits alongside Operations and Technology, and is measured on availability, quality and cost rather than revenue.

The historical pattern is fragmentation followed by painful centralisation. For most of the last three decades every desk bought its own data, built its own instrument database and negotiated its own vendor terms, because that was faster and the desk was paying. The resulting estate — dozens of overlapping instrument masters, duplicate vendor contracts, inconsistent identifiers — became untenable once regulators started asking consolidated questions after 2008. The Basel Committee's principles for risk data aggregation, published in 2013 as BCBS 239, made the fragmentation a supervisory finding rather than an inefficiency, and most large banks have spent the decade since consolidating towards a smaller number of authoritative sources. That programme is not finished at any major institution.

The centralisation argument is not purely about efficiency, and framing it that way is why so many data programmes stall. Consolidating instrument masters saves money, but the saving is rarely large enough to justify the disruption on its own. The real argument is that a bank with three instrument masters cannot answer a supervisory question in one voice, cannot net exposures reliably across desks, and cannot prove that the number in the regulatory report is the same number that drove the trading decision. The business case is about the ability to make a single, defensible statement — a capability that only becomes visible when it is demanded under pressure.

The market for financial data sorts into nine product classes

The industry sells nine broadly distinct classes of content, and they differ in latency, price, complexity and who inside the bank consumes them. Sorting them this way — rather than by vendor, which is how procurement usually sees it — makes the architectural consequences visible. The classes below run roughly from fastest and most technically demanding to slowest and most analytically demanding.

Data class What it is Latency profile Primary consumer Cost driver
Real-time streaming prices Tick-by-tick quotes and trades, order-book depth, from exchanges and venues Microseconds to milliseconds Trading, market making, execution algorithms Per-user and per-venue exchange fees; non-display fees
Delayed and end-of-day prices Closing prices, official settlement prices, daily marks Minutes to overnight Risk, finance, client reporting Volume of instruments covered
Evaluated pricing Modelled prices for instruments that did not trade Overnight, sometimes intraday Product control, risk, fund administration Instrument count and asset-class complexity
Instrument reference data Terms, identifiers, classifications, listings, ratings Daily, event-driven Every downstream system Universe size and depth of attributes
Corporate actions Coupons, dividends, splits, calls, mergers, restructurings Event-driven, deadline-bound Operations, custody, risk, tax Complexity and manual enrichment
Entity and counterparty data Legal entities, ownership hierarchies, LEIs, KYC attributes Slow, event-driven Credit, compliance, client onboarding Hierarchy depth and jurisdictional coverage
Index and benchmark data Constituents, weights, rebalance schedules, benchmark rates Daily, with scheduled events Index desks, structuring, asset management Licence tier and assets-linked pricing
Curves, surfaces and analytics Yield curves, volatility surfaces, credit spreads, derived risk factors Intraday and end-of-day Quant, XVA, risk, structuring Model complexity and source breadth
ESG and alternative data Emissions, taxonomy alignment, controversy scores, satellite and transaction data Quarterly to annual for ESG; variable for alternative Sustainable finance, research, structuring, compliance Coverage gaps, methodology licensing, exclusivity

Read down that table and one pattern dominates: the technically hardest class and the commercially most expensive class are not the same. Real-time streaming is the engineering challenge — hundreds of feeds, deterministic latency, no tolerance for gaps. Corporate actions is the operational challenge — unstructured, deadline-bound, and still substantially manual at most institutions. Entity data is the accuracy challenge, because a wrong hierarchy misstates the bank's exposure to a client group. Any architecture that optimises for one of these without regard to the others will fail somewhere else.

ESG data is the newest class and behaves unlike all the others

Sustainability data arrived as a commercial product and became a regulatory obligation, which is the reverse of the usual sequence. Banks first bought ESG ratings to answer investor questions; the Sustainable Finance Disclosure Regulation and the Corporate Sustainability Reporting Directive then made a subset of that content mandatory to collect, store and report. Content that was discretionary five years ago now sits in the same compliance perimeter as transaction reporting.

Three properties make it awkward for a data platform built around prices and instruments. It is sparse: coverage falls away sharply outside large listed issuers, and the gap has to be filled by estimation, which then has to be flagged as estimated. It is methodological rather than observed: two providers scoring the same company will disagree substantially, and the disagreement is not an error to be resolved by a tolerance rule but a genuine difference of definition. And it attaches to the issuer rather than the instrument, which means it depends entirely on the entity hierarchy being correct — the classic case where a new data class inherits the weaknesses of an old one.

Index and benchmark data is priced on a model unlike any other content

Index licensing is charged not on volume consumed but on what the licensee does with it, and often on the assets that reference it. A bank consuming index constituents for research pays modestly. A bank launching a structured product or an exchange-traded fund that tracks the index pays a licence linked to the assets under management or the notional issued, potentially for the life of the product. The provider market is a tight oligopoly — MSCI, S&P Dow Jones and FTSE Russell dominate — and for a structuring desk the index licence is frequently a larger constraint on product design than any technical consideration. It is the clearest case in this domain of a commercial term determining what the business is able to sell.

A bond, built from zero, because the rest of this module rides on one

Everything that follows uses one corporate bond as its example, so the bond itself has to be built from nothing. Readers who price instruments daily can skip this section. Readers arriving from consulting or technology should not, because the vocabulary here — coupon, yield, curve, spread, duration — is used without further explanation from Part 2 onwards.

A bond is a loan cut into tradeable pieces. A company that needs 500 million euro can borrow it from a bank, or it can issue a bond: a promise to pay a fixed amount of interest on fixed dates and to return the principal on a fixed date, split into standardised units that investors can buy and later sell to each other. NORDVOLT 4.25% 2031 is exactly that promise. The 4.25 per cent is the coupon, the annual interest rate applied to the face value. The 2031 is the maturity, the date the principal comes back. On 500 million euro of face value paying semi-annually, each coupon is 10.625 million euro, and there are eleven of them before the final payment that carries the last coupon plus the principal.

Figure 1 A bond is a fixed set of future payments; every other term describes what they are worth today

Once the payments are fixed, the only thing that can change is what someone will pay for them today, and that is the price. The coupon never moves. If market interest rates rise after issue, a promise to pay 4.25 per cent becomes less attractive than newly issued bonds paying more, so the price falls until the bond is competitive again. This is the single most important mechanic in fixed income: price and prevailing interest rates move in opposite directions, because the payments are frozen and only the price can adjust.

The yield is the return an investor actually earns by paying today's price and holding to maturity. Formally it is the discount rate that makes the present value of all those future payments equal to the price. It is the honest comparison measure: two bonds with different coupons, prices and maturities cannot be compared on coupon, but they can be compared on yield. Price and yield are two ways of saying the same thing, which is why market participants quote sometimes one and sometimes the other, and why a price record must state which it holds.

A yield curve is the same measurement taken across maturities. Plot the yield on essentially risk-free borrowing — government bonds, or the interest rate swap market — for one year, two years, five years, ten years, and the resulting line is the curve. It is the baseline cost of money over time, and it is the foundation on which every other instrument is priced.

The credit spread is what a specific borrower pays above that baseline. If the six-year point on the euro swap curve is 2.60 per cent and NORDVOLT yields 3.55 per cent, the spread is 0.95 per cent, conventionally quoted as 95 basis points — a basis point being one hundredth of a percentage point. That 95 basis points is the market's price for the risk that this particular company might not pay. The spread, not the yield, is what a credit trader actually watches, because it strips out the interest-rate movement that affects every bond equally and leaves the part that is about this issuer.

Duration measures how much the price moves when rates move. A bond with a duration of five means that a one percentage point rise in yields costs roughly five per cent of its price. Longer maturities and lower coupons produce higher duration. Duration is why a wrong maturity date or a missed call feature in reference data is not a cosmetic error: it changes the instrument's calculated sensitivity, and therefore its contribution to the bank's risk numbers and its capital requirement.

Most of what the bank holds does not trade, which is why prices must be constructed

When an instrument trades continuously on an exchange, its price is a fact. The closing price of a listed share is observed, published and beyond argument. The problem is that most of what a corporate and investment bank holds does not behave like that. Corporate bonds trade a handful of times a week or not at all. Bespoke derivatives never trade after inception. Private credit, structured notes and loan positions have no market at all in the ordinary sense.

For these, the price is not observed but constructed, and the construction is where the interesting work lives. An evaluated price is a modelled estimate of where an instrument would trade, built from whatever observable evidence exists: recent prints in the instrument itself, prints in similar instruments, dealer quotes, and a model that maps a benchmark curve plus a credit spread onto the specific terms of the security. Vendors sell evaluated prices at scale; banks also produce their own; and the independent price verification process compares the two and forces someone to explain the gap.

This is the module's reference case for the same reason an initial public offering is the reference case for equity capital markets: it touches everything. Producing a defensible evaluated price for a single corporate bond requires the instrument's full reference data, at least three market data sources, a curve, a mapping of the issuer to a legal entity and a rating, a model, a challenge process, a governance sign-off, and an audit trail that survives external scrutiny. Every stage of Part 2 will be illustrated against it.

The industry is large, growing and unusually concentrated

Global spending on financial market data and analysis reached a record 49.2 billion US dollars in 2025, up 6.5 per cent on the previous year, according to Burton-Taylor International Consulting. That figure has grown every year for well over a decade, through market conditions that were otherwise hostile to bank cost bases. The 2025 increase was driven by tariff-policy volatility early in the year followed by strong markets, which is the pattern described later in this part: turbulence raises consumption rather than lowering it.

The spend does not distribute evenly, and the shape tells you where the industry is moving. Real-time trading and data is the largest single segment at over 35 per cent of total revenues, with the strongest demand growth in portfolio management and analytics, and in pricing, reference and valuation data. The broader shift is away from static end-of-day reporting and towards real-time analytics, alternative data and AI-supported intelligence — which is why the nine content classes above are not a static taxonomy but a market whose centre of gravity is moving towards the top of the table.

Dimension 2025 position Direction
Total spend 49.2bn USD, up 6.5% Record; has risen every year for over a decade
Largest segment Real-time trading and data, over 35% of revenue Growing, alongside portfolio analytics and pricing/reference/valuation
Americas 50.3% of global spend Growing 5.8%
EMEA 31.5% of global spend Growing 7.0%
Asia 18.2% of global spend Fastest growing at 7.4%
Vendor leadership Bloomberg, LSEG, S&P Global Market Intelligence Stable at the top; SIX Financial posted the sharpest 2025 growth rate

Three suppliers dominate the top of the market: Bloomberg, LSEG and S&P Global Market Intelligence. Behind them sit FactSet, ICE Data Services, SIX, Moody's, MSCI and Morningstar, alongside the exchanges themselves, which sell their own proprietary feeds directly and have grown data revenue faster than trading revenue for most of the past decade. A wave of consolidation reshaped the top tier: LSEG completed its acquisition of Refinitiv in 2021, S&P Global merged with IHS Markit in 2022, and ICE assembled a data business through successive acquisitions. Each transaction reduced the number of credible alternatives available to a bank negotiating a renewal.

At the level of an individual institution the numbers are large enough to be a board matter. A tier-one corporate and investment bank typically spends in the range of several hundred million dollars a year on market and reference data across all categories, with the largest institutions above that. Set against a technology budget of a few billion, data is routinely among the three largest discretionary line items and the one with the least tractable growth rate. The internal reaction follows a predictable arc: the spend is first treated as a procurement exercise, then as a demand-management exercise, and only after those disappoint is it treated as an architecture exercise — which, as Part 4 argues, is where the tractable savings actually live.

Figure 2 The largest cost lines are the ones the bank controls least
Market feature Situation Consequence for the bank
Total industry spend Record 49.2bn USD in 2025, growing mid-single digits annually Data cost grows faster than most bank cost lines and resists cutting
Supplier concentration Three vendors dominate; consolidation reduced alternatives Weak negotiating position; switching cost is the real lock-in
Exchange-owned content Venues sell proprietary feeds directly and price them independently Exchange fees cannot be avoided by changing vendor
Licence complexity Display, non-display, derived-data and redistribution tiers priced separately Architecture decisions become licence decisions
Audit regime Vendors and exchanges audit clients for unlicensed use Entitlement records are a compliance artefact, not an IT convenience

League tables have an equivalent here, and it distorts behaviour in the same way

In capital markets, league-table position drives banker behaviour in ways that are not always aligned with economics. The data industry has a structurally similar metric: the terminal count and the user count. Vendor commercial models have historically been priced per named user, which turns a technical question — who needs access to this screen? — into a budget negotiation, and creates a persistent incentive to under-declare.

The distortion runs deeper than cost. Because display access is priced per user and non-display access — data consumed by an application rather than a human — is priced differently and often more punitively, banks make architectural choices to land on the cheaper side of a licence boundary. Caching strategies, the decision to compute a derived value rather than consume a published one, and even where a service is physically deployed are all shaped by licence definitions rather than by engineering judgement. This is one of the few areas of bank technology where the commercial contract is a first-order input to the system design, and engineers who do not know it produce designs that procurement has to reject.

The bank is also a producer of data, and this is the distinctly CIB part of the story

Everything so far has treated the bank as a buyer. It is also an originator, and for a corporate and investment bank that side of the ledger is growing faster than the purchasing side. A CIB sits on content that exists nowhere else and that other market participants will pay for, and the past decade has seen the largest institutions build genuine commercial businesses out of it. The client-facing data and analytics platforms operated by the major dealers — Goldman Sachs's Marquee, J.P. Morgan's DataQuery and Morgan Stanley's Matrix are the recognised examples — are the visible form of this, delivering pricing, analytics, research and risk tooling to clients through APIs and portals rather than through a salesperson.

Four categories of proprietary content matter. Research is the oldest and the most regulated, having been unbundled from execution commissions by MiFID II and therefore forced to be priced explicitly. Axes and indications of interest — the bank's own statements of what it wants to buy or sell — are valuable precisely because they reveal where liquidity sits in instruments that have no visible order book. Execution and transaction cost analysis data lets clients measure the quality of the fills they received, and a bank confident in its execution has a commercial interest in making that measurable. And proprietary indices, constructed by the bank and licensed into structured products, turn an internal methodology into a licensable asset with the same economics as any index provider.

The economics invert, and so do the controls. As a buyer the bank worries about entitlement inbound: is this consumer permitted to receive this data. As a producer it worries about entitlement outbound: which client is permitted to receive which content, on what commercial terms, and what has been derived from data the bank itself only licensed. That last question is the sharp one. A bank publishing an analytic built partly on a vendor's inputs is redistributing derived data, and the derived-data clause in the original contract governs whether it may do so at all.

The control problem is more serious than the commercial one. The bank's own flow data is a direct window into client positioning, and packaging it for sale runs immediately into confidentiality obligations to those clients and into market abuse rules. Aggregation and anonymisation thresholds — how many clients must contribute before a flow statistic can be published, and how large must each be — are set by compliance, not by the commercial team, and they are the mechanism that makes the product legal. The same information barrier architecture described in Part 3 applies, running now in the opposite direction: outbound rather than inbound.

The alternatives to buying from the major vendors are real but partial

Mutualisation is the most credible alternative and the least adopted. Managed reference-data services — SmartStream's Reference Data Utility, and the managed offerings from Alveo, NeoXam and GoldenSource — take on the acquisition, cleansing and cross-referencing work on behalf of multiple banks, on the argument that no institution gains competitive advantage from mastering a bond's coupon date more accurately than its peers. The logic is sound and the adoption has been slower than the logic implies, because banks are reluctant to depend on a shared utility for something so load-bearing, and because the migration cost is high.

Open and free identifier registries have genuinely changed part of the landscape. The Global Legal Entity Identifier Foundation publishes the full LEI reference file free of charge, which removed a class of licensed entity-identification content from the market. OpenFIGI provides an open, freely redistributable instrument identifier. Neither replaces commercial content, but both broke the assumption that identification itself must be purchased, and both give banks a licence-clean spine on which to hang commercial attributes.

The consolidated tape is the regulatory alternative, and it is arriving now. Under the MiFIR review, the European Securities and Markets Authority has been running a selection process for consolidated tape providers — a single aggregated view of trading data across European venues. ESMA selected Ediphy, operating as fairCT, for bonds, EuroCTP for shares and exchange-traded funds, and Etrading Software for over-the-counter derivatives, each to operate under ESMA's direct supervision for a five-year term. The equity tape is targeted to go live in 2026. Whether a tape materially reduces cost is contested, but it establishes a regulated reference point for what the market did, which has value independent of price.

Building everything internally is the alternative that survives on paper and rarely in practice. A bank can construct its own instrument master, its own curve library and its own evaluated pricing. Many do, for the instruments where their view is the differentiating one. Almost none do it across the full universe, because the marginal instrument is one nobody trades and the cost of covering it internally exceeds any vendor licence.

Where a bank buys, builds and mutualises — the line is a statement about competitive advantage

The business cycle here is muted, which changes the technology argument

Market data spend is famously counter-cyclical in its resilience. Trading revenue swings violently; data spend does not. It grew through the 2020 disruption, through the 2022 rates shock and through successive bank cost-reduction programmes, because the obligations that drive it — pricing the book, computing capital, reporting transactions, satisfying auditors — do not shrink when revenue does. If anything volatility increases consumption, since more volatile markets demand more frequent revaluation and generate more exceptions.

The recent cycle, roughly 2021 to 2026, has three defining features. Supplier consolidation concentrated the market and strengthened vendor pricing power. Regulatory demands intensified: the fundamental review of the trading book made the availability of real, observable price data a direct determinant of capital, and the transaction-reporting regimes hardened their reference-data requirements. And the cost conversation became a board-level one at most institutions, driven by the simple arithmetic that a line item growing at mid-single digits while the rest of the cost base is being cut eventually becomes conspicuous.

For technology investment, the implication is specific and often missed. Because demand is inelastic and supply is concentrated, the return on a data programme rarely comes from buying less. It comes from three places: eliminating duplicate consumption of the same content by different parts of the bank, moving usage to the cheaper side of licence boundaries where that is legitimate, and reducing the manual effort spent on exceptions. Programmes that promise savings from renegotiation alone tend to disappoint. Programmes that reduce the number of places a given piece of data is bought and stored tend to deliver.

  1. Why is financial data described as a manufactured product rather than a public good?

  2. What is the practical difference between market data and reference data, and why is reference data more dangerous when wrong?

  3. Name the five internal consumer roles and the single property each cares most about.

  4. Why is evaluated pricing chosen as this module's reference case?

  5. What makes the supplier relationship structurally unfavourable to the bank?

  6. How do licence categories influence technical architecture?

  7. Where does the return on a data programme actually come from?

  8. Explain price, yield, spread and duration in terms of a bond's fixed payments.

  9. Why does ESG data behave unlike the other content classes?

  10. Why is an index licence often a bigger constraint on a structuring desk than any technical consideration?

  11. How does the entitlement problem change when the bank acts as a data producer?

1

  1. Why is financial data a manufactured product, not a public good?

  2. Why is wrong reference data more dangerous than a wrong price?

  3. What does trading care about most in data?

2Following one instrument through eight stages of manufactured truth

This is the heart of the module. A single bond enters Meridian Bank as an unfamiliar identifier on a vendor file and leaves as a number in the audited accounts and a line in a regulatory submission. Between those two points sit eight stages, each with its own decisions, its own systems and its own ways of failing. The bond is ordinary. The machinery it passes through is not.

The eight stages and where the effort concentrates

The lifecycle is continuous rather than sequential, which is what makes it hard to reason about. An initial public offering has a beginning and an end. The data lifecycle runs permanently, with new instruments entering at every hour, existing instruments changing, and the whole estate being revalidated daily against a cut-off. The eight stages below describe the path a single piece of data takes; at any moment the bank is executing all eight simultaneously across millions of instruments.

Stage Typical duration Core output Who leads
1. Sourcing and licensing 3–9 months per new source Signed contract with defined usage rights Market data management, procurement, legal
2. Acquisition and ingestion Continuous Raw records landed and acknowledged Data engineering, feed operations
3. Normalisation and symbology Milliseconds to minutes Records mapped to internal identifiers and formats Data engineering, reference data operations
4. Mastering and golden copy Continuous, with daily cut-offs The bank's single authorised version of each record Reference data operations, data stewards
5. Validation and exception management Intraday plus end-of-day cut-off Approved records, escalated exceptions, audit trail Data quality, product control, risk
6. Distribution and entitlements Microseconds to overnight batch Data delivered only to permitted consumers Distribution engineering, market data management
7. Consumption Continuous Trading decisions, risk numbers, valuations, client output Front office, risk, finance, compliance
8. Reporting, archival and cost attribution T+1 to monthly Regulatory submissions, lineage evidence, chargeback Regulatory reporting, data governance, finance
Figure 3 The eight-stage lifecycle, with effort concentrated where the bank decides what it believes

Effort concentrates in the middle, and so does risk

Stages 4 and 5 absorb a disproportionate share of the work, the headcount and the failure. Acquisition is largely solved: feeds arrive, and when they do not, the failure is loud and immediate. Distribution is an engineering problem with well-understood solutions. But mastering and validation are where the bank exercises judgement, and judgement does not scale the way pipelines do. When three sources disagree about the price of NORDVOLT, a rule has to break the tie, someone has to own that rule, and someone has to review the cases where the rule produced an implausible answer.

The shape of the curve also explains a persistent misallocation of investment. Data programmes are frequently sold on the strength of their ingestion architecture — the new streaming platform, the cloud migration, the API layer — because those are visible, demonstrable and comfortable for engineers to build. The work that actually reduces risk sits in the middle of the curve, in rule quality, exception reduction and stewardship, and it is far less attractive to fund. A bank with a superb pipeline feeding a weak golden copy has industrialised the delivery of numbers it cannot defend.

Step 1 — The licence is negotiated before a single byte moves

Nothing enters the bank until somebody has agreed what the bank is allowed to do with it, and that agreement is more restrictive than most engineers expect. A market data contract does not simply grant access. It specifies which legal entities of the bank may use the data, in which locations, by how many named individuals, through which applications, for which purposes, whether the output of a calculation using that data may be shown to a client, and how long records may be retained. Each of those dimensions is priced.

The negotiation typically runs three to nine months for a significant new source, and involves four functions in tension. The business wants the content now. Procurement wants the lowest unit price. Legal wants defensible terms on liability and audit. Market data management wants terms the bank can actually monitor and comply with, which is frequently the constraint that the other three overlook. A contract the bank cannot evidence compliance with is a contract that will produce an audit finding regardless of how good the price was.

Licence categories are the hidden architecture of the data estate

Four categories carry almost all of the commercial weight, and confusing them is expensive.

Display use

Data shown to a human being on a screen. Priced per named user, per venue, per data set. This is the historical model, it remains the largest single line in most banks' market data budgets, and it is the reason terminal-count management is a permanent internal exercise. A user who moves desks, leaves the bank or changes role without the entitlement being updated is either a compliance exposure or a wasted fee, and usually the former is discovered by the vendor before the latter is discovered by the bank.

Non-display use

Data consumed by an application rather than read by a person: an automated pricing engine, a risk calculation, an algorithmic execution system. Exchanges introduced separate and often substantially higher non-display fees once they recognised that a single machine-readable feed could replace hundreds of screens. Non-display licensing is now one of the fastest-growing cost categories, and it is charged on categories of use — trading, risk, index calculation — that require the bank to declare and evidence what its applications actually do.

Derived data

A value computed from licensed data. A volatility surface built from option prices, an internal curve built from observed yields, a proprietary index built from constituent prices. Whether the derived value is free of the original licence depends entirely on the contract, and the tests vendors apply — whether the original value can be reverse-engineered from the derivative, how many inputs contributed — vary between suppliers. This is the single most common source of unintended licence breach, because a quantitative team building a curve is not thinking about the contract under which its inputs arrived.

Redistribution

Passing data outside the licensed legal entity: to a client portal, a research publication, an affiliate in another country, or a joint venture. Redistribution is the most tightly controlled and most expensively priced category, and it is the one that turns a straightforward internal system into a licensing project the moment somebody proposes showing the number to a customer.

Licence category What triggers it Typical pricing basis Where it bites in architecture
Display A human sees the value Per named user, per venue Joiner-mover-leaver processes; entitlement accuracy
Non-display An application consumes the value Per application, per use category Service decomposition; what counts as one application
Derived data A new value is computed from the original Negotiated; sometimes prohibited Quant libraries; curve construction; internal indices
Redistribution The value crosses the licensed entity boundary Per external recipient or flat fee Client portals; cross-border deployment; affiliate access

The internal gauntlet before a new source is admitted

A bank does not simply buy data; it onboards a third party, and that process has hardened considerably. Three gates typically stand between a business request and a live feed.

The sourcing and duplication gate asks whether the bank already has this content. In a fragmented estate the answer is frequently yes, purchased by another division under a different contract. A functioning data function maintains an inventory precisely so it can answer this question, and the savings from answering it correctly usually exceed anything procurement extracts in negotiation.

The third-party and operational resilience gate asks what happens when the supplier fails. Under the EU Digital Operational Resilience Act, which applies from January 2025, financial entities must maintain a register of information on all contractual arrangements with information and communication technology third-party providers, assess concentration risk, and ensure contracts contain specified provisions on access, audit, incident reporting and exit strategy. Critical data vendors fall squarely inside this. A market data contract is now a regulated outsourcing arrangement in substance, and the exit plan is a required artefact rather than a courtesy.

The data governance gate asks who will own the content once it is inside. Every incoming data set requires a named steward, a defined quality standard, a place in the golden-copy hierarchy and a retirement plan. Sources admitted without an owner become the orphan feeds that nobody can decommission a decade later because nobody knows what depends on them.

The vendor audit is the moment the licence stops being paperwork

Every significant market data contract grants the supplier a right to audit the client's use, and that right is exercised routinely rather than exceptionally. Large vendors and exchanges run continuous audit programmes; a major bank can expect several in any given year across its supplier base. Understanding how one runs is the difference between treating entitlement records as an IT convenience and treating them as a financial control.

The sequence is consistent across suppliers. A notice arrives, typically giving thirty to ninety days and naming the contracts and entities in scope. A look-back period is defined, commonly the preceding three to five years, occasionally longer where the contract permits. The bank is asked to self-declare — user counts by location and entity, applications consuming the data, downstream systems, any redistribution — and the auditor then tests those declarations against evidence: entitlement system extracts, access logs, joiner-mover-leaver records, architecture documentation and, increasingly, direct interviews with application owners.

Findings are settled as a true-up, and the arithmetic is unforgiving. Where unlicensed use is established, the bank pays the fees that would have been due for the full look-back period, frequently with interest and sometimes with a contractual uplift. A modest ongoing breach becomes a substantial one-off charge simply by being multiplied by the number of years it went undetected. A single application consuming a feed under the wrong category for four years produces a bill four times the annual delta before any penalty.

The defence is documentary, not argumentative. An auditor is not persuaded by an assertion that the bank believed its use was compliant. What settles a finding favourably is a contemporaneous, reconcilable record: an entitlement system whose state at any past date can be reproduced, joiner-mover-leaver evidence showing access was revoked when a person moved, an application inventory that maps each consuming system to a declared use category, and usage declarations that agree with the invoices. Where those three sources — contract, entitlement state and declared usage — disagree with each other, the auditor adopts the reading least favourable to the bank, and the bank has no basis on which to argue.

The systems supporting this stage are unglamorous and load-bearing. A market data inventory or contract management platform — MDSL, TRG Screen and Infomatch are the recognised names — holds the contracts, the entitlement rules, the declared usage and the invoices, and reconciles the three. An enterprise entitlement system, often driven from the identity and access management estate, translates contractual permissions into technical access. A usage-declaration and reporting capability produces the periodic statements vendors require. When a vendor audit arrives, these three systems are the defence, and if they disagree with each other the bank pays.

Step 2 — Acquisition is a solved problem that fails in interesting ways

Data arrives through three mechanisms, and the choice among them is dictated by latency and volume rather than preference. Streaming connections carry real-time prices over persistent sessions, delivering updates as they occur. Bulk file transfer carries reference data and end-of-day content, typically as a full or incremental file dropped on a schedule. Request-response interfaces, increasingly REST APIs, serve on-demand lookups for specific instruments or entities.

The engineering challenge in streaming acquisition is not throughput but determinism. A feed handler must process every update in the order it was sent, detect gaps, request retransmission where the protocol allows it, and do all of this without introducing variable delay. Exchanges publish native binary protocols — Nasdaq's ITCH, CME's market data protocol, and variants built on the FIX Simple Binary Encoding standard — precisely because a compact fixed-layout binary message can be decoded in a predictable number of nanoseconds, while a text-based format cannot. Latency-sensitive consumers connect to these directly; everyone else takes a normalised vendor feed and accepts the additional milliseconds.

The failure modes of acquisition are specific and worth naming

A gap is a missing sequence number, and it is the least dangerous failure because it announces itself. The feed handler detects the discontinuity, requests recovery, and either fills the gap or flags the period as incomplete. Systems are built for this.

Staleness is worse, because it is silent. A connection that remains open but stops delivering updates looks healthy to a naive monitor. The last price simply persists, and every downstream consumer continues to use a number that stopped being true. Detecting staleness requires a heartbeat mechanism or an expectation of update frequency per instrument, and instruments that legitimately trade rarely — NORDVOLT among them — make that expectation hard to set.

Silent truncation is the most dangerous, and it is characteristic of file-based delivery. A reference-data file that normally contains 900,000 rows arrives with 400,000 because the vendor's own upstream process failed partway. If the loading process has no row-count expectation, the bank ingests the file, and half a million instruments quietly retain yesterday's attributes with no error anywhere. Every mature loading process carries completeness assertions for exactly this reason, and every immature one has learned the lesson at least once.

The distinction that matters architecturally is between a feed and a file, and it is not merely technical. A feed is a stream of events where the unit of correctness is the individual message and the risk is ordering. A file is a snapshot where the unit of correctness is the whole file and the risk is completeness. Systems designed for one handle the other badly. A bank that loads reference data through its streaming platform without adding snapshot-level completeness checks has adopted the failure modes of both.

Step 3 — Symbology is the translation layer where most silent errors are born

Every source names the same instrument differently, and reconciling those names is a permanent, non-trivial operation. NORDVOLT has an ISIN assigned by the national numbering agency. It has a Bloomberg identifier. It has an LSEG instrument code. It has a FIGI. It may have a CUSIP if it was offered to United States investors, and a SEDOL if it was distributed through United Kingdom channels. Meridian, internally, assigns it a bank-specific instrument key, because it cannot make every internal system depend on an external identifier that a vendor might change.

Symbology — the discipline of cross-referencing these identifiers — is the load-bearing translation layer of the entire estate. When the risk engine says it holds a position in instrument 4471829 and the vendor file says it has a price for XS2612345678, something has to know these are the same thing. That mapping is maintained per source, per identifier type, and per point in time, because identifiers change: a company reorganises, a bond is exchanged, an exchange reassigns a code.

The mapping problem is harder than it appears for four specific reasons

Identifiers are not one-to-one. A single ISIN may correspond to several traded lines, one per venue, each with its own price. A single company may have dozens of LEIs across its subsidiaries. Mapping is therefore a relationship with cardinality, not a lookup, and modelling it as a simple key-value pair is the first mistake most implementations make.

Identifiers are time-dependent. A price recorded under a code that has since been reassigned must be interpreted using the mapping that was valid on the date of the record, not the mapping valid today. A symbology store without effective dating cannot correctly reprice history, which means it cannot support backtesting, model validation or a regulatory reconstruction.

Some instruments have no standard identifier at all. Over-the-counter derivatives had no equivalent of an ISIN until the introduction of the Unique Product Identifier under ISO 4914, administered by the ANNA Derivatives Service Bureau. Loans, private placements and internal structures frequently have only bank-assigned keys, which makes cross-source enrichment impossible and forces manual maintenance.

Vendors disagree. Two sources will occasionally map the same ISIN to different underlying instruments, usually around corporate events. The bank has to choose, and the choice has to be recorded.

Figure 4 Six identifiers on one instrument, each answering a different question

Deduplication is the hardest unglamorous problem in reference data, and it is worth understanding why it resists automation. The task is to decide whether two records describe the same real-world thing. For instruments with a shared ISIN it is trivial. For legal entities it is genuinely hard: "Nordvolt Industrial AB", "NORDVOLT INDUSTRIAL A.B." and "Nordvolt Ind. AB (publ)" may be one entity or three, and the correct answer depends on registry data the bank may not hold. Matching algorithms — string distance, address comparison, registry-number matching, ownership-graph consistency — get a bank most of the way, and the residual is reviewed by humans. That residual never reaches zero. Every large bank runs a permanent entity-matching queue, and the size of that queue is one of the better proxies for the health of its reference data operation.

Step 4 — The golden copy is where the bank decides what it believes

A golden copy is the single authorised version of a record that the whole institution uses. One price for NORDVOLT at the end of the day. One maturity date. One issuer hierarchy. Downstream systems consume the golden copy rather than the underlying sources, which is what makes consistency possible: the risk engine and the finance ledger produce comparable numbers because they started from the same input.

Constructing it requires a rule that resolves disagreement, and that rule is the most consequential piece of logic in the data estate. Three mechanisms do the work, usually in combination.

The source hierarchy establishes a default preference

For each data attribute, per asset class and often per region, the bank ranks its sources. For European corporate bond evaluated prices Meridian might rank source A first, source B second, and an internal model third. If A provides a value that passes validation, it wins. If A is absent, B is used. If neither is available, the internal model produces a value which is flagged as modelled rather than sourced.

The hierarchy is attribute-level, not source-level, and this distinction separates good implementations from crude ones. A vendor may be the best source for a bond's coupon and a poor source for its credit rating. Ranking whole vendors rather than individual fields imports every source's weaknesses along with its strengths, and produces a golden copy that is the best available on average and wrong in specific, predictable places.

Tolerance checks catch disagreement worth investigating

Where two sources both provide a value, the rule compares them. If they agree within a defined tolerance, the preferred source is taken and nothing further happens. If they diverge beyond tolerance, the record is held and routed to an exception queue for human adjudication.

Setting the tolerance is a genuine trade-off with no correct answer. Too tight and the exception queue fills with immaterial noise, stewards develop alert fatigue, and the meaningful exception is missed among the trivial ones. Too loose and material errors pass through silently. Mature operations set tolerances per instrument type and per liquidity band, and review them regularly against the outcomes — how many exceptions were genuine, how many escaped.

Approval converts a candidate value into an authorised one

The golden copy is not what the rule produced; it is what the bank approved. For most records approval is automatic because the rule succeeded and no tolerance was breached. For the remainder a data steward, a product controller or a valuation committee makes an explicit decision, which is recorded with a timestamp, an identity and a reason. That record is the difference between a number and a defensible number.

Golden copy mechanism What it does What it costs the bank if done badly
Source hierarchy Ranks sources per attribute, per asset class Systematically wrong values in predictable places
Tolerance check Escalates disagreement beyond a threshold Alert fatigue, or silent material error
Completeness assertion Requires every expected record to be present Positions valued on stale data with no alarm
Approval and override Records who authorised the final value and why No defence when the number is challenged
Effective dating Preserves what was believed on each historical date Inability to reconstruct or backtest

Step 5 — Validation, exceptions, and how an evaluated price is actually made

Validation runs continuously but has a hard daily focal point: the end-of-day cut-off. At a defined time, the bank freezes the official values that will drive the day's profit and loss, the risk numbers reported to committee, the margin calls issued to counterparties and the collateral posted to clearing houses. Everything before the cut-off is provisional; everything after is history. The cut-off is the moment the machinery has to be right.

The independent price verification process is the control that makes marks defensible

Independent price verification, universally abbreviated to IPV, is the process by which a function separate from the trading desk confirms that the desk's marks are reasonable. The separation is the point. A trader whose compensation depends on the reported value of a position should not be the sole source of that value, and the entire architecture of product control exists to enforce that.

The process compares the desk's mark against independent evidence, and requires explanation of any gap beyond tolerance. Evidence is ranked by observability: an executed trade in the instrument itself is strongest, a firm dealer quote next, an indicative quote after that, a consensus service contribution after that, and a model output weakest. This ranking is not merely internal practice — it maps directly onto the fair-value hierarchy in IFRS 13, where Level 1 inputs are quoted prices in active markets, Level 2 are other observable inputs, and Level 3 are unobservable inputs requiring the most disclosure and attracting the most audit scrutiny.

Figure 5 Triangulating an evaluated price for NORDVOLT from three imperfect sources
The matrix model, computed rather than asserted

The third leg of the triangle is a model, and it is worth working through because it shows exactly which reference-data fields carry the valuation. A matrix model prices an instrument that did not trade by reference to instruments that did. The logic is three steps.

Step one takes the risk-free baseline at the instrument's maturity. NORDVOLT matures in 2031; at the valuation date that is roughly six years away. The six-year euro swap rate is taken from the curve: 2.60 per cent in this illustration.

Step two adds the credit spread implied by comparable paper. The model selects bonds that did trade and that match NORDVOLT on the attributes that matter — same rating band, same sector, similar maturity, similar seniority — and extracts their spread over the same curve point. Suppose five comparable Nordic industrial names in the same rating band trade at spreads between 88 and 104 basis points, with a median of 95. The model applies 95 basis points, giving a yield of 2.60 plus 0.95, or 3.55 per cent.

Step three converts that yield back into a price using the instrument's own terms. Discounting eleven remaining semi-annual coupons of 2.125 per cent of face value plus the principal, at 3.55 per cent, produces a price of approximately 101.15. That is the model output in the triangulation.

Now notice which fields did the work. The maturity date selected the curve point. The rating and sector selected the comparable set. The coupon and its frequency determined the cash flows being discounted. The day-count convention determined the exact period lengths. The call schedule — had one been exercisable in the near term — would have truncated the cash flows entirely and produced a materially different answer. Five reference-data fields, none of them a price, together determine a valuation that flows into P&L, capital and the accounts. This is the cascade made concrete, and it is the reason the rest of this module treats reference data with the seriousness usually reserved for prices.

All figures in this calculation are illustrative.

Consensus pricing turns competitors into a valuation source

For instruments with no observable market at all, the industry solved the problem by having the market makers price each other's books anonymously. A consensus pricing service collects mid-market valuations from the banks that actually trade a product, aggregates them, and returns the result to every contributor. The dominant service, Totem, has operated since 1997 and draws on roughly 120 contributing institutions across equity, rates, repo, FX, commodity, credit, bond and valuation-adjustment markets, covering vanilla matrices as well as exotic and structured products.

The mechanism has an elegant property: contributors receive only the aggregate, never each other's individual submissions. A bank submits its own marks and receives back the consensus level together with distribution statistics — standard deviation, range, and explicit tenth and ninetieth percentile levels. It learns where it sits relative to the market without learning anything about any specific competitor, which is what makes the arrangement acceptable to both the participants and their regulators.

The incentive structure is what makes it work. A bank that submits deliberately distorted marks corrupts the consensus it will itself be measured against, and receives a distribution that no longer reflects reality. Since every contributor needs the output to be accurate for its own audit defence, the dominant strategy is honest submission. The service converts a set of competitors with no reason to help each other into a shared valuation utility.

The output does three jobs inside the bank. It is independent evidence in the IPV process for products where nothing else exists. It supports the observability argument under IFRS 13, helping move a valuation out of Level 3 where the consensus demonstrates that a market view exists. And the tenth and ninetieth percentile levels feed directly into prudent valuation — the regulatory calculation of additional valuation adjustments, which requires a bank to hold capital against the uncertainty in its own marks, and which needs a defensible measure of how wide the plausible range actually is.

Six dimensions define what "quality" actually means

"Data quality" is uselessly vague until it is decomposed, and the industry has converged on six dimensions that between them cover every rule a validation engine runs. Naming them matters because each is detected differently and each fails differently.

Dimension The question it asks Typical rule Characteristic failure
Completeness Is everything that should be here, here? Row-count assertion; every position has a price Silent truncation; positions valued on proxies
Accuracy Does the value match reality? Tolerance against an independent source A plausible but wrong mark
Consistency Do related values agree with each other? Maturity after issue date; price currency matches instrument currency Contradictory records that each pass in isolation
Timeliness Is the value current enough for its use? Age since last update, per instrument liquidity band Staleness — the most dangerous silent failure
Validity Does the value conform to its permitted form? Enumerated lists, format checks, permitted ranges The negative-price case: a rule that encodes an obsolete assumption
Uniqueness Does this thing appear exactly once? Deduplication on identifier and on entity attributes Double-counted exposure; split client hierarchies

The value of the taxonomy is that it exposes what a bank is not checking. Most institutions instrument completeness and validity thoroughly, because those are cheap to automate and fail loudly. Timeliness is checked unevenly, because setting an expected update frequency per instrument requires knowing how often each one should trade. Consistency and uniqueness are the weakest almost everywhere, because both require comparing records to each other rather than testing a record on its own — a materially harder query pattern and one that batch architectures handle badly.

Exception management is the operational reality of data quality

An exception is any record that failed a rule, and the exception queue is where the data function actually spends its day. Prices outside tolerance, missing attributes on newly issued instruments, entity matches the algorithm could not resolve, corporate actions with contradictory terms, files that failed a completeness assertion. A large bank generates tens of thousands of exceptions daily; the great majority are resolved automatically or by a steward in seconds.

Figure 6 The exception lifecycle, and the one path that must never be taken

The metric that matters is not the exception count but the auto-repair rate and the ageing profile. A rising exception count may simply mean the bank onboarded more instruments. A falling auto-repair rate means the rules are degrading relative to the data. An ageing profile with exceptions sitting unresolved for days means positions are being valued, risked and reported on data the bank has itself flagged as questionable, which is a materially worse condition than not having flagged it.

Step 6 — Distribution and entitlements decide who is permitted to know

Once the golden copy exists, it must reach every consumer that needs it, at the latency that consumer requires, and reach no consumer that is not entitled to it. Those two requirements pull in opposite directions. Distribution wants to broadcast; entitlement wants to restrict. The architecture that reconciles them is one of the more distinctive features of a bank's data estate.

Distribution operates in three tiers with different technology

The real-time tier carries live prices to trading systems over an internal message bus — commercially, platforms such as Solace or LSEG's real-time distribution system; in open technology, Kafka, or specialised low-latency transports such as Aeron where microseconds are decisive. The design goal is deterministic delivery with fan-out to many consumers without the publisher needing to know about them.

The query tier serves point-in-time lookups through APIs against an in-memory cache or a time-series store. Column-oriented tick databases such as kdb+ and OneTick exist because the characteristic query — give me every trade in this instrument between these two timestamps — is answered badly by conventional relational engines and well by a store designed around time.

The batch tier delivers end-of-day sets to risk engines, finance ledgers and regulatory reporting. This tier is the legacy default across most of the industry and the largest remaining source of overnight processing dependency. It is also where migration to streaming delivers its real benefit, discussed in Part 4.

Entitlement is the mechanism that turns a contract into enforced access

An entitlement system answers one question at high volume: is this consumer permitted to receive this data right now? The consumer may be a person, an application or a downstream system. The permission derives from the licence negotiated in Step 1, decomposed into rules the platform can evaluate.

Three properties make entitlement genuinely difficult. It must be enforced at the point of delivery rather than at the point of request, because a system that fetches data and then decides whether to display it has already redistributed it internally. It must be auditable, because the vendor will eventually ask for evidence and an assertion will not suffice. And it must be current, because the largest category of entitlement breach is not malice but a leaver whose access was never revoked.

Figure 7 Entitlement resolution, evaluated at the point of delivery

Client sub-distribution is the tranche that gets discovered last

The equivalent of a retail tranche in this domain is the client-facing channel, and it behaves the same way: it reaches the largest number of recipients, carries the most restrictive rules, and is designed last. Four routes carry bank data to clients, and each triggers a different licence question.

The client portal shows prices and valuations on a screen the client operates. This is display use by a non-employee, which most contracts do not cover at all under the internal display licence, and it is the most common trigger for a redistribution addendum.

Client reporting embeds values in statements, valuations and performance reports. The number has left the bank in a document, which some contracts treat as redistribution and some treat as an incidental use — the distinction is contractual, not logical, and has to be read per supplier.

Data APIs deliver machine-readable content directly into client systems. This is redistribution in its purest form, priced per recipient or as a flat licence, and the one route where the vendor will want to know the identity and number of downstream consumers.

Research and marketing material embeds prices, charts and index levels in published documents. Widely distributed, often publicly, and governed by clauses about attribution and about how much of the original data set may be reproduced.

Three mechanics govern all four routes, and they are the equivalent of the clawback rules in a retail offering. Delay is the first: serving a fifteen-minute delayed value rather than a real-time one moves the content into a category venues must provide free of charge, and a great many client-facing use cases are satisfied by it. Aggregation is the second: a value computed across enough underlying inputs may cease to be redistribution of any single source, but the threshold is set by the contract, not by the bank's judgement. Attribution is the third: many licences permit reproduction only with a specific credit line and prohibit any presentation implying the bank originated the data.

The entitlement estate is where identity and access management meets commercial contract law, and it is usually the least modernised part of a bank's data platform. The reason is structural: entitlement rules encode contract terms that change on renewal cycles, are written in natural language, and are interpreted differently by different vendors. There is no standard model to converge on. The practical consequence is that entitlement logic tends to accumulate as bank-specific rules maintained by a small team who understand both the contracts and the systems, and the departure of that team is a genuine operational risk that rarely appears on any risk register.

Step 7 — Consumption, and the point where data becomes a decision

The data has now become an input to something consequential, and each consumer transforms it differently. The trading system turns a price into a quote. The risk engine turns a curve into a sensitivity and a value-at-risk number. The finance system turns a mark into revenue. The collateral system turns a valuation into a margin call that moves actual cash. The regulatory reporting system turns a transaction plus its reference data into a submission.

The critical property at this stage is that consumption is where the data becomes irreversible. A margin call issued on a wrong valuation moves money that has to be recalled. A quote published on a stale price can be hit. A regulatory report submitted with a wrong identifier has been submitted. Upstream errors are correctable; consumed errors have consequences that persist after the correction.

Corporate actions are where consumption meets a deadline that does not move

A corporate action is any event initiated by an issuer that changes the terms, quantity or value of a holding. A coupon payment, a dividend, a stock split, a rights issue, a merger, an early call. For NORDVOLT the relevant events are its semi-annual coupon and, because the bond carries a call provision, the possibility of early redemption at the issuer's option.

Corporate actions are the one part of the data lifecycle with a hard external deadline and a real economic penalty for missing it. A voluntary corporate action — one where the holder must choose, such as electing cash or shares — has an election deadline set by the issuer and propagated through the custody chain. Miss it and the default option applies, which may be materially worse. The industry has automated a great deal of this and it remains one of the most manually intensive areas, because the source document is frequently a prospectus or a notice written in prose rather than a structured message.

The messaging standards here are precise and worth knowing by name. Under ISO 15022, the SWIFT MT564 message carries a corporate action notification and the MT566 carries the confirmation of a completed action. The successor standard, ISO 20022, replaces these with the seev message family — seev.031 for the notification, seev.036 for the confirmation — as part of the industry-wide migration to a richer, XML-based message structure. Both standards are in production simultaneously across the industry, which means most banks maintain translation between them.

Layer Standard Carries Timescale
Exchange native ITCH, MDP3, SBE binary encodings Order book updates, trades, venue status Nanoseconds to microseconds
Front office FIX, FAST Quotes, orders, executions, market data snapshots Microseconds to milliseconds
Internal distribution Vendor feeds, Kafka, Solace, Aeron Normalised prices, reference updates, events Milliseconds
Post-trade servicing ISO 15022 MT564 / MT566 → ISO 20022 seev.031 / seev.036 Corporate action notification and confirmation Minutes to days
Derivative terms FpML, ISDA Common Domain Model Contract terms, lifecycle events, valuations Minutes to days
Figure 8 Two messaging worlds — one optimised for speed, one for certainty

Step 8 — Reporting, archival, and proving what the bank believed

The final stage produces evidence rather than value, and it is the stage most likely to be under-resourced. Three outputs matter.

Regulatory submissions carry the data outside the bank under legal obligation. A MiFIR transaction report contains upwards of sixty fields, a substantial proportion of which are reference data: the instrument identifier, the venue, the issuer's LEI, the classification, the counterparty's LEI, the decision-maker identification. Each field has a validation rule at the regulator's end, and a report that fails validation is rejected and must be resubmitted. Persistent rejection rates are a supervisory signal.

Lineage evidence proves where a number came from. The regulator's question — reconstruct this number as of this date — requires the bank to retrieve not only the value but the sources, the rule version, the approvals and the identifiers as they stood on that date. Systems that overwrite rather than version cannot answer it.

Cost attribution charges consumption back to the consumers. Without it, data appears free at the point of use and demand is unbounded. With it, a desk that requests a new data set sees the cost against its own profit and loss, which is the only mechanism that has ever reliably reduced consumption.

The instrument lifecycle continues long after onboarding

An instrument is not finished when it is set up, and the maintenance tail is where quiet decay occurs. Four categories of change matter.

Terms are amended. A bond's covenants are renegotiated, a call schedule is modified, a maturity is extended in a restructuring. The reference data must be updated and effective-dated so that historical valuations remain reproducible under the old terms.

Ratings migrate. A downgrade changes the instrument's eligibility for certain portfolios, its regulatory capital treatment, its haircut in collateral arrangements and potentially its index membership. The rating is reference data with immediate operational consequence.

Index membership changes. Inclusion in or exclusion from a benchmark index moves demand mechanically as index-tracking funds rebalance, and rebalance dates are among the highest-volume, highest-scrutiny days in the market data calendar.

Instruments mature or are retired, and are then archived rather than deleted. A matured bond still appears in historical positions, in regulatory reports covering past periods, and in model backtests. Deleting it breaks all three. The retirement process marks the instrument inactive, blocks new positions, and preserves the record indefinitely under the retention period the regulator requires — typically five to seven years for transaction data, and longer for some jurisdictions.

Data retention is a more delicate problem than it appears, because two regulatory obligations point in opposite directions. Financial regulation requires records to be retained and reproducible for years. Data protection regulation — the General Data Protection Regulation in the EU and its equivalents elsewhere — requires personal data to be deleted when the purpose for holding it expires, and grants individuals a right to erasure. Counterparty and client reference data contains personal data: names of authorised signatories, beneficial owners, traders identified in transaction reports. Reconciling these is done through a legal-obligation basis for retention, pseudonymisation of fields not required for the regulatory purpose, and strict access control on the archive. It is a genuine architectural constraint, not a compliance formality, and it has to be designed in at the point the archive is built.

  1. Why is the effort curve highest at mastering and validation rather than at acquisition?

  2. Distinguish display, non-display, derived and redistribution licensing, and give an architectural consequence of each.

  3. What is silent truncation and why is it more dangerous than a feed gap?

  4. Why must a symbology store be effective-dated?

  5. Name the three mechanisms that construct a golden copy and state the trade-off in setting tolerance.

  6. What does independent price verification control against, and how is evidence ranked?

  7. Which SWIFT and ISO 20022 messages carry corporate actions, and why does a coupon payment reveal reference-data errors?

  8. Why is detection of a data exception without matching resolution capacity worse than no detection?

  9. Why are matured instruments archived rather than deleted, and what tension does that create?

  10. How does a vendor audit convert a small ongoing breach into a large charge, and what settles a finding favourably?

  11. Which five reference-data fields determine the matrix model output, and how?

  12. Why does consensus pricing produce honest submissions, and what three jobs does its output do?

  13. Which two of the six quality dimensions are weakest at most banks, and why?

  14. Why can a client-facing API built on the golden copy become unlicensable?

2

  1. Why is effort highest at mastering and validation?

  2. What is silent truncation?

  3. Which licence covers an application consuming data?

3Four camps want different things from the same data, and the bank stands between all of them

A process description tells you what happens. It does not tell you why the process is shaped the way it is. That explanation lives in the incentives of the parties involved — who owns the data, who pays for it, who is punished when it is wrong, and who is measured on something that conflicts with quality. This part maps the full cast and the tensions between them.

The supply chain has four camps and a referee

The originators create the data as a by-product of running a market. Exchanges and trading venues capture prices. Central securities depositories and custodians capture holdings and corporate action events. Issuers create the terms of the instruments themselves and announce the events that change them. National numbering agencies assign identifiers. None of these bodies exists to serve banks; each captures data because it is inherent to its function, and each has discovered that the data is a valuable asset in its own right.

The vendors and aggregators collect, normalise and resell. Bloomberg, LSEG, S&P Global Market Intelligence, ICE Data Services, FactSet, SIX and their peers buy or licence content from hundreds of originators, harmonise it into consistent formats and identifier schemes, add analytics and evaluated prices, and sell the result. Their value proposition is aggregation and normalisation, and their commercial model rewards breadth of coverage and depth of client integration, because both raise switching costs.

The bank's data function acquires, masters and distributes internally. It negotiates the contracts, runs the pipelines, owns the golden copy, enforces entitlements and answers for quality. It has no revenue and considerable cost, and its internal clients each believe their requirement should take precedence.

The internal consumers use the data to make decisions and are measured on those decisions rather than on the data. A trader is measured on profit and loss, a risk manager on the accuracy of risk assessment, a product controller on the integrity of reported revenue, a compliance officer on regulatory outcomes. None of them is measured on data quality, which is precisely why data quality has to be somebody's explicit job.

And the bank appears a second time, on the other side of the chain, as an originator in its own right. As Part 1 set out, a corporate and investment bank produces content nobody else has — research, axes and indications of interest, execution analytics, proprietary indices — and licenses it to clients through its own platforms. It also contributes to consensus pricing services, submitting its own marks and receiving back an aggregate it could not construct alone. This means the four-camp model is not a line but a loop: the bank buys from the vendors, and in specific products it competes with them. The practical consequence inside the institution is that the same entitlement machinery must run in both directions, and the same information barriers must prevent inbound client data from leaking into an outbound commercial product.

Regulators and standards bodies referee. Supervisors set the obligations that drive consumption and enforce the reporting that proves it. Standards bodies — the International Organization for Standardization, the Association of National Numbering Agencies, the Global LEI Foundation, the ANNA Derivatives Service Bureau — define the identifiers and message formats that make cross-institution communication possible at all.

Figure 9 The bank's data function is the only party in contact with every other party

Every major interaction in the chain, and when it happens

From To What flows Stage
Exchange / venue Vendor, and bank directly Real-time prices, order book, venue status Acquisition
Issuer / paying agent Custodian, then bank Corporate action notification and confirmation Consumption
National numbering agency ANNA, then vendors and bank ISIN assignment for a new instrument Symbology
GLEIF All parties, free of charge LEI reference file and ownership relationships Symbology, reporting
Vendor Bank data function Normalised feeds, reference sets, evaluated prices Acquisition, mastering
Bank data function Vendor and exchange Usage declarations, user counts, audit responses Licensing, reporting
Bank data function Internal consumers Golden copy, live prices, curves, entity hierarchies Distribution
Internal consumers Bank data function Exceptions, challenges, new instrument requests Validation
Product control Trading desk IPV challenge on marks outside tolerance Validation
Bank Regulator Transaction reports, risk submissions, lineage evidence Reporting
Regulator Bank Rejections, validation feedback, supervisory findings Reporting
Bank Consensus pricing service Its own mid-market marks on illiquid products Validation
Consensus service Bank and all contributors Aggregated level, distribution, 10th and 90th percentiles Validation
Bank Clients Research, axes, execution analytics, proprietary indices Consumption, redistribution

The consumer taxonomy determines almost every design trade-off

The single most useful thing to know about a bank's data architecture is which consumer it was optimised for. Data platforms cannot be simultaneously optimal for latency, completeness, accuracy and traceability, because those properties trade against each other. Every real design privileges some consumers and compromises others, and reading a platform in terms of who won explains most of its apparent oddities.

The trading desk needs speed and will accept imperfection to get it

A market maker quoting NORDVOLT needs the current state of the market with the lowest achievable delay, and its tolerance for missing or approximate data is higher than any other consumer's. A price that is 99 per cent right and arrives in 200 microseconds is more valuable than a price that is perfect and arrives in 200 milliseconds, because the trading opportunity has gone. Desks therefore consume direct venue feeds where the economics justify the connectivity cost, maintain their own local caches, and frequently run parallel data paths that bypass the enterprise platform entirely.

That bypass is a permanent source of tension and is not simply indiscipline. The enterprise platform adds latency by design — normalisation, entitlement checks, quality validation all cost time. A desk that routes around it is making a rational trade. The consequence is that the price the desk traded on and the price the golden copy recorded may differ, and reconciling that difference is a recurring exercise.

Risk management needs completeness above all

A risk number is defined by its worst-covered position, not its average. If Meridian holds 12,000 instruments and 11,940 are priced from good sources, the value-at-risk calculation still has to do something about the remaining 60. It will use a proxy: a similar instrument, a sector average, a model output. Each proxy is an assumption, and the aggregate of those assumptions is a known blind spot that the risk committee accepts explicitly or, more often, inherits without discussion.

Risk is also the consumer most sensitive to consistency across time. A value-at-risk model calibrated on historical data requires that history to have been recorded consistently. A change in the source hierarchy — switching the preferred vendor for a whole asset class — introduces a discontinuity in the time series that will show up as an apparent change in market risk when nothing about the market changed. Data changes have to be communicated to risk as model events, not as operational improvements.

Finance and product control need accuracy that survives external challenge

Their output is the audited income statement, which means their standard is not "is this a reasonable number" but "can I defend this number to someone whose job is to doubt it". This is why the IPV process is documented to a level that appears excessive from the outside, why every override is logged with a reason, and why the fair-value hierarchy classification matters so much: a Level 3 valuation, built on unobservable inputs, attracts disclosure requirements and audit attention that a Level 1 valuation does not.

Compliance and regulatory reporting need traceability above accuracy

This is counter-intuitive and worth stating plainly: the regulator's first question is rarely whether the number is right. It is whether the bank can show how the number was produced, demonstrate that the process was controlled, and reproduce it. A bank that reports a slightly wrong number through a well-controlled and fully evidenced process is in a materially better position than a bank that reports the right number through a process it cannot describe. Lineage is the deliverable.

Client-facing channels need licence cleanliness before anything else

The moment a number is shown to a client, it has left the bank, and the constraint becomes commercial rather than technical. Digital portals, research publications, client reporting and API products all trigger redistribution rights. The delivery team usually discovers this after the design is complete, and the resolution is either an expensive addendum or a fallback to delayed data.

Consumer Latency requirement Accuracy tolerance What breaks if the data is wrong
Trading desk Microseconds to milliseconds Higher — speed dominates A quote is hit at the wrong price; immediate, visible loss
Risk management Minutes to end of day Low on completeness, moderate on precision Exposure understated; limits breached without detection
Finance / product control End of day, month end Very low — audit standard Misstated revenue; restatement; auditor qualification
Compliance / reporting T+1 Low, but traceability dominates Rejected submissions; supervisory finding; fine
Client-facing channels Seconds to minutes Low on presentation, licence must be clean Client dispute; unlicensed redistribution; vendor claim

Information barriers exist here too, in a form specific to data

A bank knows things through its data platform that it must not allow all of itself to know, and the platform is what enforces the separation. Three distinct barrier problems arise.

Material non-public information travels through reference data

When Meridian's advisory business is engaged on an unannounced acquisition, the target company's identity is material non-public information. That information can leak through the data estate in ways that have nothing to do with a document being emailed. A newly created entity record with an unusual name. A sudden restricted-list entry. An access log showing which analysts pulled a particular company's data. The reference-data platform is an inference surface, and controlling it requires access restrictions on entity records and on the logs themselves, not merely on documents.

The restricted list is a data product with immediate trading consequences

A restricted list identifies instruments in which the bank or its employees may not deal, because the bank holds inside information or has a conflicting role. It is maintained by control room, distributed to trading systems, order management systems and personal-account-dealing platforms, and enforced at the point of order entry. It is reference data with the shortest possible tolerance for latency and error: an instrument added to the list must stop being tradeable within seconds, and an instrument wrongly on the list blocks legitimate business.

Client data cannot flow freely between divisions

Counterparty and client reference data is subject to confidentiality obligations that do not disappear because the data is in a shared platform. A client's exposure profile held for credit purposes cannot be visible to a trading desk that might act on it. A single master entity record shared by the whole bank is exactly the right architecture for consistency and exactly the wrong architecture for confidentiality, and the reconciliation is attribute-level access control: one entity record, many permission sets over its fields.

The systems that enforce data-side information barriers are a specific and often overlooked cluster. Attribute-level access control in the master data platform restricts which fields of an entity record a given role may read. The restricted-list distribution service pushes prohibitions into every order-entry path with sub-second propagation. Access logging over reference-data queries records who looked at what, and — critically — those logs are themselves restricted, because a log of who searched for a company is as revealing as the search. Wall-crossing records document the controlled exceptions where a person was deliberately brought inside a barrier. And personal-account-dealing surveillance cross-references employee trades against the restricted list and against the bank's own deal pipeline.

Six internal teams carry the function, split by the barrier

The data function is not one team, and the boundaries between its parts are where accountability is most often lost.

Team Division Core role Primary technology
Market data management COO / CDO office Contracts, licences, entitlement policy, usage declarations, cost Inventory and contract platforms (MDSL, TRG Screen); entitlement engine
Reference data operations Operations Instrument setup, golden copy stewardship, exception resolution Master data platform (GoldenSource, Alveo, NeoXam); workflow tooling
Client and counterparty data Operations / Compliance Legal entity records, hierarchies, LEI maintenance, KYC attributes Entity master; GLEIF integration; matching engines
Data engineering and distribution Technology Feed handlers, normalisation, message bus, APIs, tick stores Kafka, Solace, kdb+ / OneTick, cloud data platforms
Data governance and quality CDO office Policy, standards, ownership model, quality metrics, lineage Catalogue and lineage tooling; quality rule engines
Embedded desk data managers Front office Desk-specific requirements, direct feeds, local caches Desk analytics; direct venue connectivity

The barrier runs between the last row and all the others. Embedded desk data managers sit inside the business, are funded by it, are measured on its outcomes, and can therefore make decisions that are locally optimal and globally expensive — a direct venue feed duplicating content already licensed centrally, a local instrument cache diverging from the golden copy. Removing them entirely fails, because the enterprise function genuinely cannot respond at the speed a trading desk requires. The workable arrangement is a dotted reporting line into the central function, mandatory registration of any locally sourced data, and cost transparency that makes the duplication visible.

Mapping each team to a stage and a system

Stage 1, licensing, belongs to market data management, operating the contract inventory. Stage 2 and Stage 3, acquisition and symbology, belong to data engineering with reference data operations owning the mapping rules. Stage 4 and Stage 5, mastering and validation, belong jointly to reference data operations and, for prices specifically, to product control — this is the only stage with genuinely shared ownership, and it is the stage where accountability most often becomes ambiguous. Stage 6, distribution and entitlement, splits between data engineering for the pipes and market data management for the permissions. Stage 7, consumption, belongs to the consumers. Stage 8, reporting and cost, belongs to data governance and finance.

The joint ownership of Stage 5 between reference data operations and product control deserves scrutiny, because it is a structural weakness dressed as a sensible division of labour. Reference data operations owns whether a price passed its validation rules. Product control owns whether the mark is right. In principle these are different questions. In practice, when a mark is wrong and nobody caught it, both functions can point to a defensible reading of their own mandate: operations applied the rules correctly, control reviewed the exceptions it was shown. The gap is the population that never became an exception because the rule did not fire. Closing it requires an explicit owner for rule adequacy — distinct from rule execution — and that role is frequently unassigned.

  1. Why do internal consumers' incentives work against data quality?

  2. How does the consumer taxonomy determine architecture, and give an example of a design decision it explains.

  3. Why does risk management care about consistency across time as much as accuracy today?

  4. Why is traceability more important than accuracy for regulatory reporting?

  5. In what three ways do information barriers apply specifically to data rather than to documents?

  6. Why does removing embedded desk data managers usually fail, and what arrangement works instead?

  7. Why is the supply chain a loop rather than a line, and what does that require internally?

  8. What is the structural weakness in joint ownership of the validation stage?

3

  1. Why do consumers' incentives work against data quality?

  2. Why does risk care about consistency across time?

  3. Why does every platform privilege some consumers?

4The architecture is a set of six functions, and the vendor map is a map of competitive advantage

Naming vendors first is the fastest way to misunderstand a data platform. This part asks the clean question — what functions does any bank need, regardless of who supplies them — then maps the vendors onto those functions, follows one field end to end, opens the data model, and works through the identifiers, protocols, resilience requirements and regulations that constrain every design decision.

Six functional blocks, plus the capabilities that cut across all of them

Strip away the product names and every bank's data platform performs the same six functions in the same order. The differences between institutions are almost entirely in how each block is implemented and where the boundaries between them are drawn.

Functional block What it must do Non-negotiable property
Feed handling and ingestion Connect to sources, decode protocols, detect gaps, land raw records Determinism and completeness detection
Normalisation and symbology Translate formats and identifiers into the internal model, effective-dated Historical reproducibility of mappings
Mastering and golden copy Apply source hierarchy, tolerance and approval to produce one authorised value Recorded provenance for every field
Quality and exception management Detect, route, age and resolve records that failed a rule Resolution capacity matched to detection volume
Distribution and caching Deliver to every consumer at its required latency, with fan-out Deterministic delivery and back-pressure handling
Entitlement and billing Enforce licence terms at delivery; evidence usage; attribute cost Auditability against the contract
Figure 10 The six functional blocks and the capabilities that cut across all of them

Six capabilities cut across every block and cannot be bolted on afterwards. Lineage and audit must be captured as data moves, not reconstructed later. Governance and stewardship assign a named owner to every data domain. Security and access control enforce the information barriers described in Part 3. Resilience and failover keep the platform available at the moments that cannot be repeated. Cost attribution charges consumption back to consumers. Catalogue and discovery let people find what exists — the capability most often missing, and the reason banks routinely buy content they already own.

The vendor landscape maps onto those blocks unevenly

Some blocks have mature commodity markets and some have none, and the pattern tells you where banks believe their advantage lies.

Functional block Typical suppliers Buy or build Why
Feed handling Vendor feed platforms; exchange-native handlers; specialist ticker plant providers Overwhelmingly buy Protocol decoding is commodity engineering with no differentiation
Normalisation and symbology Vendor cross-reference services; OpenFIGI; ANNA; internal mapping layer Buy the reference sets, build the internal layer The reference sets are shared truth; the internal key model is bank-specific
Mastering and golden copy GoldenSource, Alveo, NeoXam, S&P Global EDM Buy the platform, build the rules The engine is commodity; the source hierarchy and tolerances encode the bank's judgement
Quality and exception management Platform-embedded modules; workflow tooling Buy the tooling, build the rules and the operating model Same reason — the rules are the asset
Distribution and caching Solace, Kafka, Aeron, kdb+, OneTick, LSEG RTDS Buy the components, build the topology Components are commodity; the topology reflects the bank's latency and cost trade-offs
Entitlement and billing MDSL, TRG Screen, Infomatch; internal entitlement engines Buy the inventory, build the enforcement Contract terms are bank-specific and change on renewal

The buy-build-mutualise line as a statement about where the bank thinks its edge is

Read the right-hand column and a consistent principle emerges: banks buy engines and build rules. No institution gains advantage from writing its own FIX decoder or its own message bus. Every institution's source hierarchy, tolerance calibration, entity hierarchy conventions and entitlement interpretation are specific to its business mix, its regulatory footprint and its contracts. A programme that proposes to buy a platform and adopt its default rules has bought the commodity and discarded the asset.

Time is a regulated input, and everything else stands on it

Every claim made in this module depends on a timestamp, and the timestamp has to come from somewhere. Staleness detection compares an arrival time to now. Sequencing depends on ordering. The end-of-day cut-off is a moment. A regulatory reconstruction asks what was true at a stated instant. Yet the question of where the bank's notion of "now" originates, and how accurate it is, is one that architecture discussions routinely skip.

Under MiFID II it is not a design choice but a legal requirement with numbers attached. Regulatory technical standard 25 requires operators of trading venues and their members to synchronise their business clocks to Coordinated Universal Time, and specifies the tolerances. The maximum permitted divergence from UTC depends on the gateway-to-gateway latency of the trading system: one millisecond where that latency exceeds one millisecond, and 100 microseconds where it is one millisecond or less. Timestamp granularity follows the same split, at one millisecond or one microsecond respectively. Firms engaged in high-frequency algorithmic trading sit at the demanding end of both: 100 microseconds of maximum divergence and one microsecond of granularity. Voice trading and other slower activity classes are permitted materially wider tolerances, up to one second.

Meeting a 100 microsecond tolerance is a network engineering problem, not a configuration setting. The Network Time Protocol, adequate for ordinary corporate systems, cannot reliably deliver it across a data centre. The industry standard answer is the Precision Time Protocol, which timestamps synchronisation messages in network hardware rather than in software and thereby removes the variable delay introduced by operating system scheduling. A compliant estate typically runs a grandmaster clock disciplined by a satellite time source, distributes time over PTP through switches that support hardware timestamping, and continuously monitors and records the measured divergence — because the obligation is not only to be accurate but to be able to demonstrate that you were accurate on any past date.

The volumes, so that the engineering problem is felt rather than asserted

Assertions that market data is high volume are worthless without an order of magnitude, so here is one. The Options Price Reporting Authority, which consolidates quotes and trades across the United States options exchanges, is the reference case for extreme market data throughput.

During the April 2025 market sell-off, peak one-millisecond bursts on OPRA exceeded 23.7 million packets per second, equating to more than 187 million messages per second. Practitioners building feed handlers for it are advised to design for microburst peak rates in the order of 75 million messages per second — meaning 75,000 messages arriving inside a single millisecond. Peak bursts can exceed 50 gigabits per second, which is why a ten gigabit connection does not merely slow down but drops packets, since the data physically does not fit in the pipe.

The infrastructure has been rebuilt around these numbers. OPRA now distributes its consolidated output across 96 multicast lines, using symbol-based routing to assign messages to a specific line, and the expansion delivered a material latency improvement: median latency now sits under 18 microseconds and daily maximum latency reliably below 60 microseconds. Consuming the feed in full is therefore a commitment to 96 simultaneous multicast subscriptions, not one connection.

Two consequences follow for anyone designing in this space. The first is that capacity must be sized on the microburst, not the average: a system provisioned for a comfortable daily mean will fail in the minutes that matter, which are precisely the volatile minutes when the data is most valuable. The second is that these volumes make the licensing discussion concrete. Consuming OPRA in full is an infrastructure commitment of real substance, which is why many firms consume a filtered or vendor-normalised subset and accept the added latency — an architecture decision driven jointly by physics and by price.

Following one field from origination to regulatory submission

The cascade principle is simple to state and repeatedly underestimated in practice: an error introduced upstream reaches every system downstream, and the systems downstream have no way to detect it. A wrong maturity date on NORDVOLT does not look wrong to the risk engine. It produces a plausible sensitivity, a plausible value-at-risk contribution and a plausible capital number. Nothing in the calculation fails. The output is simply wrong in a way that is invisible until an external event — a coupon reconciliation, a counterparty dispute, an auditor's sample — forces a comparison against reality.

Stage System Data object created What it inherits
Origination Exchange, APA, issuer notice Raw print or event record with the source's own identifier The source's own errors and conventions
Ingestion Feed handler Landed record with arrival timestamp and sequence Any gap or truncation not detected
Symbology Cross-reference service Record bound to the internal instrument key Any mapping error, permanently
Mastering Golden copy platform Authorised value with provenance and approval The source hierarchy's blind spots
Valuation Pricing and curve engine Mark, curve point, sensitivity Every upstream defect, silently
Risk Risk engine VaR contribution, exposure, capital charge The valuation's defects, amplified by aggregation
Finance Ledger and product control Reported revenue, fair-value hierarchy classification The mark, now in the audited accounts
Reporting Regulatory reporting engine MiFIR / EMIR submission with LEI, ISIN, UPI Every identifier error, now visible to the supervisor
Figure 11 One field, eight systems — nothing downstream can detect an upstream error

The anatomy of the core data objects

Two objects carry most of the weight in this domain: the instrument record and the price record. Understanding their fields, and what each field drives downstream, converts an abstract discussion about data quality into a specific one about consequence.

The instrument record

Field group Representative fields What it drives downstream
Identity ISIN, internal key, FIGI, CUSIP, SEDOL, ticker Every join between systems; reconciliation; reporting
Issuer Issuer name, issuer LEI, ultimate parent, country of risk Credit exposure aggregation; concentration limits; capital
Classification CFI code, asset class, seniority, sector, instrument type Risk model selection; regulatory treatment; eligibility
Economic terms Coupon, coupon frequency, day-count convention, maturity, call schedule Accrual, cash flow projection, valuation, duration
Trading Listing venue MIC, currency, minimum piece, settlement convention Settlement instruction; venue reporting; liquidity assessment
Credit Rating by agency, rating date, watch status Portfolio eligibility; haircut; capital; index membership
Lifecycle Issue date, first settlement, status, effective-dated amendments Historical reproducibility; retirement; archival

Three of those fields deserve specific attention because they cause disproportionate damage when wrong. The day-count convention determines how accrued interest is calculated, and an error produces a small daily distortion that persists undetected until a coupon reconciliation. The issuer LEI is the join key for credit exposure aggregation across the whole bank, and a wrong or missing LEI means a client group's exposure is understated with no error anywhere. The call schedule changes the instrument's effective duration entirely, and a missing call feature makes a bond look longer-dated and less interest-rate sensitive than it is.

The price record

Field Why it exists Consequence if wrong or absent
Instrument key Binds the price to the instrument Price applied to the wrong position
Value and price type Distinguishes clean, dirty, yield, spread Double-counting or omitting accrued interest
Currency The unit of the value Wrong translation into reporting currency
Timestamp and time zone Establishes when the value was true Stale data used as current; cut-off breaches
Source and source rank Records where it came from and its precedence No provenance; no defence under challenge
Quality flag and observability Marks it as traded, quoted, consensus or modelled Wrong fair-value hierarchy classification under IFRS 13
Approval identity and reason Records who authorised it and why No audit trail; override invisible
Effective date and version Preserves what was believed on each date Historical reconstruction impossible

Two time axes, because one is not enough

The last row of that table hides the hardest modelling problem in the domain, and it deserves its proper name. A single date column cannot answer the question a regulator or an auditor actually asks. Consider two genuinely different questions about NORDVOLT. What was the price on 30 June? And: what did the bank report as the price on 30 June, at the time it produced its July submission? If a correction was booked in August, those questions have different answers, and a record with one date can only serve one of them.

A bitemporal model carries two independent time axes. Valid time records when a value was true in the world — the price as at 30 June. Transaction time records when the bank knew it — booked on 30 June, superseded on 12 August. Every stored fact carries both, and no fact is ever overwritten; a correction inserts a new version with a later transaction time and the same valid time.

Figure 12 One valid date, two recorded truths — the only structure that answers both questions

With both axes present, three otherwise impossible queries become routine. The bank can reproduce its July regulatory submission exactly as filed, by querying valid time 30 June as at transaction time 20 July, which is what a supervisor asking about that submission requires. It can state its current best understanding of 30 June, by querying valid time 30 June as at now, which is what the accounts require. And it can explain the difference between the two, which is what an auditor asking why a restatement occurred requires.

The cost is real and is the reason many implementations avoid it. Storage roughly doubles, every query gains a second predicate that developers forget, and the join semantics between two bitemporal tables are genuinely difficult to reason about. Institutions that skipped it typically discover the gap during their first serious regulatory reconstruction, at which point the missing history cannot be recreated — the information a bitemporal model would have preserved was destroyed by each in-place update, years earlier.

Identifiers are the spine, and no single one is sufficient

Six identifier standards do the load-bearing work, and each answers a question the others cannot.

Identifier Full name and standard What it identifies Where it bites
ISIN International Securities Identification Number, ISO 6166, assigned by national numbering agencies coordinated by ANNA The security itself, globally unique The primary key for most securities reporting; does not distinguish trading venue
LEI Legal Entity Identifier, ISO 17442, issued under GLEIF, published free of charge The legal entity — issuer, counterparty, client Mandatory in EMIR and MiFIR reporting; the join key for exposure aggregation
FIGI Financial Instrument Global Identifier, open standard, freely redistributable The traded line — instrument at a specific venue Distinguishes the same ISIN listed on several venues; licence-clean
MIC Market Identifier Code, ISO 10383 The trading venue or segment Required in transaction reporting; determines applicable venue rules
CFI Classification of Financial Instruments, ISO 10962 The instrument's type and characteristics Drives regulatory classification and eligibility tests
UPI Unique Product Identifier, ISO 4914, administered by the ANNA Derivatives Service Bureau An over-the-counter derivative product type Fills the gap where no ISIN exists; required for derivative reporting

The relationship between ISIN and FIGI is worth understanding precisely, because it is the most common source of confusion. One ISIN can correspond to several FIGIs — one per venue where the security trades — because a share listed in Frankfurt and in Paris is the same security but two distinct traded lines with distinct prices and distinct liquidity. Reporting a transaction requires the ISIN and the MIC. Storing a price requires knowing which line it came from. A data model that treats ISIN as the sole instrument key has collapsed a one-to-many relationship and will produce prices that appear to contradict each other.

The LEI deserves particular emphasis because it is the identifier that regulators have made non-negotiable. No LEI, no trade: the "no LEI, no trade" principle under MiFIR means an investment firm may not execute a transaction on behalf of a client eligible for an LEI unless that LEI has been obtained. It is also the only major identifier published entirely free of charge, with the full reference file and the parent-relationship data available for download from GLEIF, which makes it the natural spine for entity hierarchies.

The LEI exists because of a single question that nobody could answer in September 2008. When Lehman Brothers failed, supervisors around the world asked financial institutions what their aggregate exposure to Lehman was, and received a consistent reply: we do not know, and finding out will take weeks or months. The reason was mundane and total. Counterparties were recorded in most systems as a name in a text field. Lehman was hundreds of legal entities across dozens of jurisdictions, and no institution — and no regulator — had a reliable way to determine which of those entities its various desks had faced, or which entities belonged to the same group.

The Financial Stability Board recommended a global identifier in 2011, the G20 endorsed it in 2012, and GLEIF was established in 2014 to operate the system. The LEI is therefore not a technical convenience that regulation later adopted. It is a direct, deliberate answer to a specific failure, and its two components reflect that origin exactly: a unique code for each legal entity, and — the part institutions consistently underinvest in — the parent relationships that let those entities be rolled up into a group.

This history is worth carrying because it explains the emphasis. The worked example in Part 2, where a missing hierarchy link understates group exposure by 18 million euro while nothing errors, is Lehman in miniature. The identifier solved the naming problem. The hierarchy, which is where the aggregation actually happens, remains the part banks get wrong.

The other identifiers carry comparable origin logic, briefly worth knowing. The ISIN was standardised in the 1980s because cross-border settlement failed when each national market used its own code and no mapping existed. The UPI was created in the 2010s for the opposite reason: derivative reporting was mandated for instruments that had no identifier at all, so one had to be invented to make the regime executable. And BCBS 239 was published in 2013 because supervisors reviewing the crisis concluded that the inability to aggregate risk data quickly was not an operational inconvenience but a cause of the crisis being worse than it needed to be. In each case the standard is a response to a specific failure, which is why reading them as arbitrary compliance burdens produces bad architecture.

Integration patterns, and why streaming is not simply faster batch

Three patterns move data between systems, and the industry is midway through a migration between them.

Batch is the legacy default and remains the largest volume. A file is produced on a schedule, transferred, loaded and processed. It is simple, robust, easy to reason about and easy to restart. Its weakness is latency measured in hours and, more importantly, that every consumer holds its own copy which must then be reconciled against every other copy.

Request-response through APIs is the modernisation most banks have completed. A consumer asks for what it needs when it needs it. This removes the staleness problem and reduces unnecessary movement of data, but it inverts the load profile: a thousand consumers each polling for updates generate far more traffic than a single push, and it does not remove the need for local caching.

Event streaming pushes changes to subscribers as they occur, and its significance is architectural rather than merely performance. In a Kafka-style topology the golden copy publishes a change once and every interested system receives it in order. Each consumer maintains its own view derived from the same ordered event log.

A fourth pattern is organisational rather than technical, and it is the one currently changing practice fastest. A data contract is an explicit, versioned, machine-checkable agreement between a data producer and its consumers: the schema, the semantics of each field, the freshness guarantee, the quality thresholds and the process for making a breaking change. It exists because the classic failure in a large estate is not a system going down but a producer changing a field's meaning without telling anyone, and every downstream consumer continuing to run against a definition that silently moved. Enforcing the contract in the pipeline — rejecting a publication that violates its own declared schema — converts an organisational discipline problem into an automated test.

Figure 13 Why the migration removes reconciliation rather than merely accelerating delivery

The reason the batch-to-streaming migration matters more than the latency improvement suggests is that it changes what reconciliation is for. Under batch, every consuming system holds an independently loaded copy of the data, and the bank runs reconciliation processes to detect where those copies have diverged — an entire category of operational work, staffed and funded, that exists solely because the architecture created the possibility of divergence. Under an ordered event log, every consumer derives its state from the same sequence of changes, so divergence is either impossible or immediately detectable as a consumer lagging the log. The saving is not faster data. It is the elimination of a whole class of reconciliation work and the operational risk that comes with it. Banks that migrate to streaming while retaining batch-era reconciliation processes have paid for the platform and kept the cost it was meant to remove.

Immutability, audit and the parts that are always proprietary

Compliance-critical records must be append-only, and the reason is evidential rather than technical. If a price record can be updated in place, then the record of what the bank believed yesterday can be silently changed today, and no assertion about historical values is credible. Append-only storage, where a correction creates a new version and the previous version remains retrievable, is what makes the statement "this is what we believed on 30 June" defensible. The same principle governs approval logs, entitlement changes and rule versions.

Rule versioning is the part most often overlooked. Reproducing a historical value requires not only the source data as it stood but the rule that was applied to it. A source hierarchy that was changed in March cannot be used to explain a February value. Mature platforms version the rules alongside the data and record which rule version produced each stored value.

The golden-copy rule engine is always proprietary, in the sense that its content is never bought. The platform is purchased; the hierarchy, the tolerances, the escalation thresholds and the approval matrix are written by the bank, encode its judgement about its own business, and constitute one of the few genuinely bank-specific assets in the data estate. This is the direct analogue of the allocation engine in a capital markets platform: the machinery is a commodity, the policy embedded in it is not.

Where the machines physically sit, for the content where microseconds decide

For the latency-sensitive tier, geography stops being an abstraction. Light travels roughly 200 metres per microsecond through optical fibre, so physical distance to a matching engine is a hard floor on how quickly a firm can see a price and respond to it. Three arrangements follow from that arithmetic.

Colocation places the firm's servers in the same data centre as the venue's matching engine, rented by the rack from the exchange, with standardised cable lengths so that no participant in the hall is nearer than another. Cross-connects are the direct physical cables between a firm's rack and the exchange's access switches, sold as a monthly-fee product and forming a meaningful revenue line for venues. And between geographically separated centres — the Chicago-to-New Jersey corridor being the canonical example — microwave and millimetre-wave links carry data faster than fibre, because a straight line through air beats the refractive index and the routing detours of buried cable, at the cost of far lower bandwidth and vulnerability to weather.

Two points make this relevant beyond the trading desk. First, colocation and cross-connect fees are market data costs in substance even though they arrive on an infrastructure invoice, and a bank measuring its data spend without them is understating it. Second, the same physics constrains the enterprise platform: a normalisation and entitlement layer that adds a millisecond is adding five times the entire budget of a colocated firm's decision loop, which is the concrete reason the trading desks described in Part 3 route around it.

Non-production environments are a licensing problem before they are a technical one

A bank cannot properly test a data platform without realistic data, and it usually may not use the real data to do so. Market data licences are granted for defined purposes, and development, testing and training environments frequently fall outside them or attract a separate fee. Copying a production feed into a test environment is, under many contracts, an additional instance of consumption requiring an additional licence, and it is a recurring audit finding.

Three approaches are used and each has a real limitation. Some suppliers offer a test or evaluation licence, usually with restricted volumes and a prohibition on any use of the output in a business decision — clean, but not always available and rarely covering the full estate. Masked or obfuscated production data preserves structure and volume while altering values, which works well for load and integration testing but destroys the correctness testing that matters most, since the whole point is often whether a value was computed correctly. Synthetic generation constructs artificial instruments, prices and events statistically resembling the real ones, which is licence-clean and increasingly viable, but a generator built by the same team that built the system tends to reproduce that team's assumptions and therefore misses precisely the edge cases worth testing.

Operational resilience, and the moments that cannot be repeated

Most of the data day is forgiving. A handful of moments are not. Understanding which is which is what separates a resilience design from a generic high-availability posture.

The market open is unforgiving because the bank must quote from the first second. A feed that arrives late means the desk is either not quoting, which loses business, or quoting on stale data, which loses money.

The end-of-day valuation cut-off is unforgiving because it is a fixed point that other processes depend upon. Risk reporting, margin calls, collateral movements and the ledger all key off it. A cut-off missed by an hour does not delay one process; it delays a chain.

Index rebalance dates are unforgiving because volume concentrates and errors are expensive. When a benchmark index changes constituents, index-tracking funds must trade to match, and the volume arrives in a narrow window with unusual price behaviour.

Month-end and period-end close is unforgiving because the accounts must be produced. The valuation feeding the reported income statement cannot be provisional.

Regulatory reporting deadlines are unforgiving because they are legal. A MiFIR transaction report is due by close of the working day following the transaction. A missed deadline is a breach regardless of the reason.

Concept What it guarantees Why it matters at the critical moment
Redundancy A second instance of a component exists Necessary but insufficient — a standby that has never been exercised is a hypothesis
Automatic failover Traffic moves to the standby without human action At market open there is no time for a person to decide and act
Recovery time objective The maximum tolerable duration of an outage Determines whether the cut-off can still be met after a failure
Recovery point objective The maximum tolerable amount of data loss Determines whether the day's history is reconstructable
Disaster recovery rehearsal Proof that failover works under realistic conditions Converts an assumption into evidence, and is what supervisors ask for
Business continuity The process continues by other means, including manual Covers the case where technology recovery is not fast enough
Source diversity A second independent data source for critical content The only defence against a vendor's own outage, which redundancy inside the bank cannot address

Six regulatory regimes read as system specifications

Regulation shapes this domain more directly than almost any other in banking, because several regimes specify data requirements at field level. Read as specifications rather than legal summaries, six regimes account for most of the constraints.

MiFID II and MiFIR — the transparency and reporting specification

The requirement is that the bank reports every reportable transaction to its supervisor by close of the following working day, with complete and accurate reference data. The specification consequences are concrete: mandatory LEIs for counterparties and issuers, mandatory instrument identification, venue identification by MIC, and a validation regime at the regulator that rejects non-conforming reports. MiFIR also requires venues to publish market data on a reasonable commercial basis and to make it free of charge fifteen minutes after publication, which is why every bank's architecture contains a delayed-data fallback path. RTS 25, discussed above, sits inside this regime. The consolidated tape provisions, now moving into implementation with providers selected for bonds, equities and over-the-counter derivatives, add a further authoritative reference source.

BCBS 239 — the risk data aggregation specification

The Basel Committee's principles for effective risk data aggregation and risk reporting are, in substance, an architecture standard. They require accuracy and integrity, completeness across all material risk, timeliness, adaptability to ad-hoc requests, and — the demanding one — the ability to aggregate risk data across the whole group on a consistent basis, particularly during stress. That last requirement is what makes fragmented instrument masters a supervisory issue rather than an internal inefficiency, and BCBS 239 findings remain open at many large institutions more than a decade after publication.

EMIR and the derivative reporting regimes — the identifier specification

Derivative reporting requires both counterparties to report every contract, with matching identifiers. This drove the mandatory adoption of the LEI, the creation of the Unique Transaction Identifier to pair the two sides of a trade, and the Unique Product Identifier to classify products that have no ISIN. The specification consequence is that a bank must be able to generate, exchange and store identifiers agreed bilaterally with counterparties — a reference-data problem that did not exist before the regime created it.

The Fundamental Review of the Trading Book — the observability specification

FRTB makes the availability of real, observable price data a direct determinant of regulatory capital, which is unusual and consequential. Under the internal models approach, a risk factor must pass a modellability test based on the number of real price observations available over a defined window. Risk factors that fail become non-modellable and attract a substantially more punitive capital charge computed through a stressed-scenario approach. The practical consequence is that a bank's ability to source, evidence and retain observable transaction data translates directly into capital, which changed the internal economics of buying data: the cost of a data source can now be weighed against a capital saving.

IFRS 13 and IFRS 9 — the valuation input specification

Accounting standards classify fair-value measurements by the observability of their inputs, which makes observability itself a reportable attribute. Level 1 uses quoted prices in active markets, Level 2 uses other observable inputs, and Level 3 uses unobservable inputs and carries the heaviest disclosure and audit burden. The specification consequence is that every price record must carry a quality flag recording how it was derived, because the classification of the resulting valuation depends on it. A price stored without observability metadata cannot be correctly classified, and consensus pricing earns part of its value precisely by supplying the evidence that keeps a valuation out of Level 3.

The Benchmarks Regulation — the specification for the numbers everything else references

A benchmark is a published number that other contracts point at, and regulating it is different from regulating a price. A trade reports what happened. A benchmark determines what someone owes: a floating-rate loan, a swap, a structured note and an index fund all settle by reference to a value the counterparties did not negotiate and cannot observe directly. That makes the benchmark a piece of reference data with contractual force, and it is why the EU Benchmarks Regulation, Regulation (EU) 2016/1011, imposes obligations not only on the administrator that produces the number but on the supervised entity that uses it.

The regime was substantially narrowed in 2025 and the change matters operationally. Regulation (EU) 2025/914 reduced the scope so that the BMR now applies to critical and significant benchmarks, EU Climate Transition and Paris-Aligned benchmarks, and certain commodity benchmarks, rather than to essentially every benchmark used in the Union. Thresholds for the "significant" category were raised, with designation and voluntary opt-in available. Third-country benchmarks that fall outside the new scope may now be used freely by EU supervised entities without equivalence, recognition or endorsement; those that remain in scope must, from January 2026, come through one of those three routes. For a bank, this converts a blanket compliance question into a per-benchmark one, and the operational consequence is a specific reference-data attribute: every benchmark the bank uses now needs a recorded regulatory status, refreshed as designations change.

Three obligations translate directly into system requirements. The administrator must be authorised or registered and appear on the ESMA register, which means the bank needs a control that checks a benchmark's administrator status before a new product references it. Written fallback provisions are required, specifying what happens if the benchmark materially changes or ceases — and a fallback is only executable if the instrument's reference data records which fallback applies. And benchmark statements defining the market a benchmark measures must be available to users, which is what allows a bank to demonstrate the benchmark was appropriate for the exposure it was applied to.

The LIBOR transition showed what happens when the number itself changes shape

Between 2021 and 2024 the industry replaced its most widely referenced benchmark, and the exercise is the clearest illustration in this module of reference data as an operational risk. Most LIBOR settings ceased at the end of 2021. The US dollar panel ceased in June 2023, after which one-, three- and six-month settings continued on a synthetic basis — calculated from CME Term SOFR plus a fixed ISDA spread adjustment — purely as a bridge for legacy contracts. The final synthetic settings were published on 30 September 2024, and LIBOR ended.

The replacement is not a like-for-like substitution, which is the part that broke systems. LIBOR was a forward-looking term rate: the three-month rate was known at the start of the period, produced from panel bank submissions that embodied expert judgement about unsecured funding costs. The risk-free rates that replaced it — SOFR in dollars, €STR in euro, SONIA in sterling, TONA in yen, SARON in Swiss francs — are overnight, transaction-based and backward-looking. Deriving a three-month rate from them means compounding the daily fixings in arrears, so the rate for a period is not known until that period has almost ended.

Every consequence of that is a data consequence. A cash flow that could previously be projected at the start of a period now cannot be, which changes the timing of settlement calculations and the design of the systems that produce them. Compounding requires a full history of daily fixings rather than a single published figure, so the volume of rate data rose sharply. The conventions used to make the arrears mechanic workable — lookback periods, observation shifts, lockouts — are themselves reference-data attributes that must be stored per contract, because two otherwise identical swaps with different lookback conventions produce different cash flows. And each currency migrated to a different rate on a different timetable under different conventions, so the instrument master had to carry the transition state of every affected contract for several years.

A seventh regime overlays all of these without being specific to data. The General Data Protection Regulation applies wherever counterparty and client reference data contains personal data, which it routinely does. Its requirements — lawful basis, purpose limitation, minimisation, and the right to erasure — sit in direct tension with the multi-year retention obligations of the financial regimes above, and the reconciliation described in Part 2 is an architectural constraint on the archive rather than a policy footnote.

The six risks of the data function, sorted by whether the bank can engineer them away

  1. What principle governs the buy-versus-build line in this domain?

  2. Explain the cascade asymmetry and its architectural implication. — Detection capability decreases moving downstream while consequence increases: the feed handler can detect a gap but the risk engine cannot detect a wrong maturity date, the ledger cannot detect a wrong mark and the report cannot detect a wrong LEI. Therefore validation should sit as close to ingestion as the available context permits, and upstream control is worth more than equivalent downstream investment.

  3. Which three instrument fields cause disproportionate damage when wrong, and why?

  4. Why is the relationship between ISIN and FIGI one-to-many, and what breaks if a model ignores this?

  5. Why does the batch-to-streaming migration matter more than the latency improvement suggests?

  6. What is the difference between FIX and the exchange-native binary protocols, and where does ISO 20022 sit?

  7. Why is redundancy alone insufficient for data resilience?

  8. How does FRTB change the economics of buying data?

  9. Why must every price record carry an observability flag?

  10. What does RTS 25 require, and why can NTP not satisfy it?

  11. Why must capacity be sized on the microburst rather than the daily average?

  12. What are the two axes of a bitemporal model and which three queries do they enable?

  13. What specific failure produced the LEI, and which half do banks still get wrong?

  14. Why is a non-production environment a licensing problem, and what is the limitation of each of the three solutions?

  15. Why does the Benchmarks Regulation apply to the user of a benchmark and not only its administrator, and what did the 2025 amendment change operationally?

  16. Why did replacing LIBOR break systems rather than simply changing a value?

  17. What does a data contract enforce, and what failure does it prevent?

4

  1. What governs the buy-versus-build line?

  2. What is the cascade asymmetry?

  3. Which field is the join key for group-wide credit exposure?

5Six risks, three real failures, and the transformations that actually move something

The data function carries risks that rarely appear on a trading risk report and occasionally cost more than one. This part sets out the taxonomy, gives the metrics that distinguish a well-run operation from a lucky one, examines three real cases covering three distinct failure modes, and applies a strict filter to the transformation agenda — including generative AI, which belongs here on merit and needs governing on merit.

The risk taxonomy is a map of what the bank can and cannot engineer away

Six risk types cover the domain without overlap, and sorting them by controllability is more useful than sorting them by severity.

Risk type What it is Who owns it Main defence
Data quality Values that are wrong, stale, incomplete or mismapped Reference data operations; data governance Upstream validation, tolerance rules, exception resolution capacity
Valuation and model input A defensible-looking mark built on bad inputs or an unchallenged model Product control; independent model validation IPV, consensus pricing comparison, prudent valuation, fair-value hierarchy discipline, model risk governance
Operational and continuity Feed outage, latency degradation, missed cut-off, platform failure Data engineering; operational resilience Failover, rehearsed disaster recovery, source diversity
Licence and entitlement Use of data beyond contracted rights, discovered on audit Market data management Entitlement enforcement at delivery, usage evidence, contract inventory
Cost and concentration Spend growing faster than the cost base; dependency on few suppliers COO / CDO office; procurement Cost attribution, duplication elimination, source diversity, exit planning
Regulatory and reporting Rejected or inaccurate submissions; supervisory findings Regulatory reporting; compliance Field-level validation, lineage evidence, reconciliation to source

The six risks sorted by whether better architecture can fix them

Telling a well-run data operation from a lucky one

A data function that has not failed recently may be well controlled or may simply not have been tested. Six metrics distinguish the two, and none of them is the exception count.

Metric What it measures What good looks like Why it discriminates
Golden copy completeness Share of positions priced from an approved source at cut-off, without proxy Consistently high, with the proxy population named and reviewed A high headline figure with an unexamined proxy tail is a hidden blind spot
Auto-repair rate Share of exceptions resolved by rule without human touch Rising over time as rules improve A falling rate means rules are degrading relative to the data, before anything visibly breaks
Exception ageing How long unresolved exceptions sit before the cut-off Almost nothing crossing the cut-off unresolved Detection without resolution capacity is documented negligence
Time to onboard a new instrument Elapsed time from first request to fully attributed golden-copy record Same day for standard instruments Slow onboarding forces desks to trade on local, unmastered data
Data-attributable P&L breaks Value of profit and loss differences traced to a data cause Small and falling, with root causes categorised The only metric that converts data quality into money the business recognises
Regulatory rejection rate Share of submissions rejected on validation by the supervisor Very low and stable A persistently elevated rate is a supervisory signal long before it is a finding

Three real cases, three distinct failure modes

The cases below are not a selection of interesting stories. They are the three ways this function fails: the model of the world is wrong, the commercial structure becomes unsustainable, or the evidence does not stand up.

Case one — negative oil prices, April 2020: the data model was wrong about what was possible

On 20 April 2020, the front-month West Texas Intermediate crude futures contract settled below zero, trading as low as minus 40.32 dollars a barrel, as storage capacity constraints and collapsing demand met the expiry of the May contract. The market event itself was extraordinary. The systems failure that accompanied it is the instructive part.

A negative price had always been theoretically possible and was almost universally excluded by system design. Price fields were defined as unsigned or validated as positive. Option pricing models assumed lognormal distributions in which prices cannot go below zero. Risk systems, margin calculators and display layers all embedded the same assumption at different depths.

CME Group had anticipated the possibility and prepared. Around 1 April the exchange advised the Commodity Futures Trading Commission that it was taking operational steps to support negative pricing for futures and negative strikes for options on certain energy products, and issued public notices to members on 3, 8, 13 and 15 April, including a test environment. Support for zero and negative prices was in place across CME systems before the event.

Not every participant used the warning. Interactive Brokers failed to deploy the necessary system changes before negative prices occurred, producing two failures on 20 April: negative prices were not displayed to customers and customers could not place negative-priced limit orders, and internal minimum margin requirements were not correctly enforced before trade execution. Hundreds of accounts holding long positions into settlement suffered losses the firm initially determined to exceed 82.57 million dollars. The CFTC ordered a civil monetary penalty of 1.75 million dollars and restitution of 82.57 million dollars to customers.

The lesson is about validation rules as embedded assumptions. Every constraint in a data model — a positive-price check, a maximum tenor, an assumed currency set, a fixed rating scale — is a statement about what the modellers believed possible. Those statements age. The failure was not that anyone thought negative oil prices likely; it was that the assumption was distributed across dozens of systems with no inventory of where it lived, so when the exchange gave three weeks' warning, one firm could not find and change all the places in time.

Case two — the cost and concentration of market data: the commercial structure became the risk

Global spending on financial market data reached a record 49.2 billion dollars in 2025, growing 6.5 per cent, having grown every year through a period in which banks were cutting cost almost everywhere else. This is not a single incident; it is a slow, structural failure of negotiating position, and it belongs in a failure taxonomy because its consequences are as material as any operational event.

Three forces compounded. Consolidation reduced the number of credible suppliers, with LSEG acquiring Refinitiv in 2021, S&P Global merging with IHS Markit in 2022, and ICE assembling a data business through successive transactions. Exchanges grew data revenue faster than trading revenue and priced proprietary feeds independently of any vendor relationship. And licence structures proliferated into display, non-display, derived and redistribution tiers, each separately priced, with new categories appearing at renewal.

The regulatory response has been persistent and, so far, of limited effect. MiFID II required venues to publish market data on a reasonable commercial basis and free of charge fifteen minutes after publication. ESMA has consulted repeatedly on the cost of market data; users argue prices have not fallen since the regime began in 2018 and in some cases have risen, venues argue their pricing is reasonable, and the two positions have not converged. The consolidated tape provisions of the MiFIR review are the more substantive intervention, with ESMA selecting Ediphy for bonds, EuroCTP for shares and exchange-traded funds, and Etrading Software for over-the-counter derivatives, the equity tape targeting a 2026 go-live under ESMA's direct supervision.

The lesson is that a risk with no incident date still requires a plan. Because the cost escalation has no dramatic moment, it is easy to treat as a procurement matter rather than a risk. The institutions that have managed it best treated concentration as a resilience question — mapping which content has no alternative source, maintaining exit plans under the DORA third-party requirements, and eliminating duplicate consumption — rather than as an annual negotiation.

Case three — transaction reporting fines: the evidence did not stand up

In 2019 the Financial Conduct Authority fined Goldman Sachs International 34,344,700 pounds for failures relating to 220.2 million transaction reports between November 2007 and March 2017, and fined UBS AG 27,599,400 pounds for failures relating to 135.8 million transaction reports between November 2007 and May 2017. Both were breaches of MiFID transaction reporting obligations and of Principle 3 of the FCA's Principles for Businesses, requiring firms to organise and control their affairs responsibly with adequate risk management systems. More than a dozen other firms have been fined for equivalent failures.

The stated causes are precisely the subject of this module. The FCA identified failings in change management processes, in the maintenance of counterparty reference data used in reporting, and in how the firms tested whether the transactions they reported were accurate and complete. Not a trading failure, not a market abuse, not a control breakdown on a desk: a reference-data and assurance failure, sustained across a decade, at institutions with substantial resources.

Two features make this case the most transferable of the three. The duration is the first: nearly ten years, which means the failure survived multiple audit cycles, multiple system changes and multiple regulatory examinations. The absence of a triggering event is the second: nothing went wrong in a way that produced a loss or a customer complaint. The reports were submitted and accepted. The defect was only visible to someone who reconciled the reports back against the underlying transactions and the reference data — the assurance activity whose inadequacy was itself part of the finding.

Case Failure type Stage it maps to Lesson
Negative WTI prices, April 2020 The data model was wrong about what was possible Stage 3 and 4 — normalisation and validation rules Validation constraints are undocumented assumptions with no inventory; they age and cannot be found when they must change
Market data cost and concentration The commercial structure became unsustainable Stage 1 — sourcing and licensing A risk with no incident date is still a risk; concentration is a resilience question, not a procurement one
FCA transaction reporting fines, 2019 The evidence did not stand up Stage 8 — reporting and assurance Reference-data defects can persist for a decade without any event revealing them; only reconciliation back to source finds them

Generative AI has a genuine place here, and a governance problem of the same size

Three use cases justify themselves on economics rather than novelty.

Corporate action extraction from unstructured documents is the strongest. A large share of corporate action processing still begins with a prospectus, a notice or an agent letter written in prose, which an operations analyst reads and converts into structured fields. Large language models are well suited to this: the task is extraction from text into a defined schema, the schema is stable, and — critically — the output is verifiable against a subsequent structured message. The economics are direct, because the manual effort being displaced is measurable.

Entity resolution and counterparty matching is the second. The residual population that deterministic and fuzzy matching cannot resolve — the "Nordvolt Industrial AB" versus "Nordvolt Ind. AB (publ)" cases from Part 2 — requires contextual judgement about whether two descriptions refer to the same organisation. Language models handle context better than string-distance algorithms, and the population is small enough that human review of the model's proposals remains affordable.

Data quality rule generation and anomaly explanation is the third. Writing validation rules is skilled, slow work, and explaining why a particular record was flagged is a recurring cost across every exception queue. A model that proposes candidate rules from observed data patterns, and that produces a plain-language explanation of an anomaly for the steward reviewing it, reduces both the time to write rules and the time to action an exception.

A fourth use case, natural-language discovery over the data catalogue, is genuinely useful and requires care. Letting an analyst ask "which instruments have no approved price this month" is a real productivity gain and, as flagged in Part 3, an information-barrier surface that did not previously exist.

Use of generative AI The gain The governing risk
Corporate action extraction from documents Removes measurable manual effort; output verifiable against a later structured message Hallucinated terms entering the golden copy, where they cascade to every downstream system
Entity resolution and counterparty matching Resolves the contextual residual that string matching cannot An incorrect merge misstates group-wide credit exposure with no error signal anywhere
Quality rule generation and anomaly explanation Reduces time to write rules and time to action exceptions Rules that appear reasonable but encode a spurious pattern; automation bias in stewards
Natural-language catalogue discovery Makes existing data findable, reducing duplicate purchase Inference across information barriers; access logs that are themselves revealing

Three risks govern all of them

Confidentiality and material non-public information is the first and hardest. Reference data contains client identities, deal pipeline inferences and counterparty exposures. Submitting that content to a model — particularly a hosted one — is a disclosure, and vendor contracts routinely prohibit the use of licensed data for model training outright, which is a licence question before it is a privacy one. The controls are deployment topology, contractual clarity with the model provider, and technical prevention of restricted content reaching a prompt.

Hallucination is more consequential here than in most applications, because of the cascade. A wrong answer from a model in a chat interface is an inconvenience. A hallucinated coupon date written into the golden copy propagates to accrual, valuation, risk, capital and the regulatory report, and nothing downstream can detect it. The mitigation is structural rather than behavioural: model output enters as a proposal requiring verification, never as an authoritative value, and any field a model contributed to carries a provenance flag exactly as a vendor source would.

Explainability and model risk governance is the third. A model whose output influences a valuation, a capital number or a regulatory submission is a model in the supervisory sense and falls under the bank's model risk management framework — validation, documentation, performance monitoring, periodic review. The instinct to treat a language model as a productivity tool outside that perimeter is the mistake supervisors are most alert to. The question a supervisor asks is not whether the model is accurate but whether the bank can explain what it did and demonstrate that it monitors it.

  1. Why is the risk taxonomy sorted by controllability rather than severity?

  2. Why is the absence of incidents a poor measure of a data operation?

  3. What does the April 2020 negative oil price event teach about validation rules?

  4. Why does market data cost escalation belong in a failure taxonomy at all?

  5. What makes the FCA transaction reporting fines the most transferable of the three cases?

  6. Apply the transformation filter to data mesh.

  7. Why is hallucination more dangerous in this domain than in most applications?

  8. Why is the human in the loop described as a control rather than a limitation?

5

  1. What does April 2020's negative oil price teach?

  2. Why is the absence of incidents a poor measure?

  3. Why sort risks by controllability rather than severity?

Module summary — the six ideas that matter most

Everything in this module reduces to six claims. If nothing else survives, these should.

Idea 1 — The data layer defines what the bank can do, not the size of the bank

Every capability above the data layer is bounded by it. A bank cannot run a risk model on instruments it cannot price, cannot aggregate exposure to a client group whose hierarchy it has not mastered, cannot report transactions whose identifiers it does not hold, and cannot pass a supervisory examination on numbers it cannot reproduce. Scale does not substitute for this. A large institution with a fragmented instrument master is less capable, in the ways that are examined, than a smaller one with a coherent golden copy, because the constraint is coherence rather than volume.

Idea 2 — Data cascades, so errors are never contained

Detection capability falls as you move downstream while consequence rises, and this asymmetry governs everything. The feed handler can detect a gap and the golden copy can detect disagreement between sources, but the risk engine cannot detect a wrong maturity date, the ledger cannot detect a wrong mark, and the regulatory report cannot detect a wrong LEI. Each downstream system faithfully processes what it is given and produces a plausible answer.

The arithmetic in Part 2 is the whole argument in two numbers. A day-count convention recorded as Actual/365 instead of 30/360 costs 3,615 euro a day on a single 25 million euro position — far below any tolerance, present every day, reversing only when the coupon forces a cash reconciliation. A financing subsidiary not linked to its parent in the entity hierarchy reports group exposure of 65 million against a true 83 million and a limit of 75 million, with every individual exposure correct and nothing erroring anywhere. Both figures are illustrative; both failure shapes are exact. Errors that are too small to trip a threshold and errors that omit a row from an aggregation are the two that survive, and neither is visible downstream. Validation belongs as close to ingestion as context allows, because it is the only place it can work.

Idea 3 — The golden copy is where the price is actually made

For anything that does not trade continuously, the price is constructed rather than observed, and the construction is the interesting part. A source hierarchy ranked per attribute decides which evidence takes precedence. A tolerance decides what disagreement is worth escalating. An approval, with an identity and a reason, converts a candidate value into an authorised one. Independent price verification separates the mark from the person whose compensation depends on it, ranking evidence by observability in a hierarchy that maps directly onto the IFRS 13 fair-value levels.

Two mechanisms show how far the construction reaches. The matrix model derives a price from five reference-data fields and a curve, none of which is itself a price — the maturity picks the curve point, the rating and sector pick the comparable set, the coupon and day-count define the cash flows, and the call schedule can truncate them entirely. Consensus pricing goes further still, constituting a market where none exists by aggregating the anonymised submissions of around 120 competing contributors, and returning percentile levels that feed straight into prudent valuation. The bank's belief about NORDVOLT at 101.18 is not a fact it discovered. It is a decision it made, bounded by rules it wrote, that it must be able to defend.

Idea 4 — Compliance is core architecture, not an afterthought

Six regulatory regimes act as field-level system specifications, and one of them converts data directly into capital. MiFIR sets the reporting fields and the delayed-data fallback. BCBS 239 is an architecture standard for group-wide aggregation. The derivative reporting regimes created the LEI, the UTI and the UPI as mandatory reference data. IFRS 13 makes observability itself a stored attribute of every price. FRTB makes the count of real price observations a determinant of the capital charge, which means a data source can be justified against a capital saving. And the Benchmarks Regulation governs the numbers every other contract points at — the LIBOR transition, ending with the final synthetic settings on 30 September 2024, proved that a reference-data field can change shape rather than value and take millions of contracts with it.

Two often-overlooked requirements belong in the same category, and both are load-bearing. Clock synchronisation under RTS 25 fixes the tolerances within which the bank's notion of "now" must sit — one millisecond, or 100 microseconds for the fastest activity — which is what makes every timestamp in the estate comparable and every reconstruction meaningful. And the bitemporal record, with valid time separated from transaction time, is what allows the bank to reproduce a past submission exactly as filed while simultaneously stating its current best understanding of the same date. Neither looks like architecture when a programme is being scoped. Both are the reason a reconstruction succeeds or fails. Alongside them sit the messaging standards — FIX and exchange-native binary encodings in the microsecond world, ISO 15022 migrating to ISO 20022 in the post-trade world — and resilience requirements that only become tractable when expressed as a recovery time objective against a specific irreversible deadline.

Idea 5 — The vendor's commercial model quietly reshapes the bank's architecture

This is the league-table equivalent, and it is the one most often missed by people who arrive from a purely technical background. Display, non-display, derived-data and redistribution rights are priced separately, so decomposing a service into microservices can multiply a licence, building a curve from licensed inputs can breach a contract, and showing a number to a client can turn a finished system into a licensing project. Consumption is driven by non-discretionary obligations, so the bank cannot exit or meaningfully reduce volume, and a decade of consolidation has left three suppliers dominating a market that reached a record 49.2 billion dollars in 2025, of which a tier-one institution carries several hundred million.

The vendor audit is where this stops being theoretical, and the direction reverses when the bank sells. An audit charges unlicensed use across a look-back of three to five years, so a small ongoing misclassification becomes a large one-off bill purely through duration, and what settles it is documentary evidence rather than argument. Meanwhile the same bank licensing content out — research, axes, execution analytics, proprietary indices — discovers that its ability to sell an analytic is decided by derived-data clauses agreed years earlier by someone buying an input. Both directions lead to the same conclusion: the contract has to be present in the design review, not discovered at the launch review.

Idea 6 — Real transformation moves cost or risk, and generative AI is no exception

The filter is narrow and worth applying literally: after this change, does a named cost line fall or a named risk reduce? Event streaming passes, because it removes a class of reconciliation work rather than merely accelerating delivery. Machine learning for exception management passes, because it attacks the largest remaining manual queue. Mutualisation passes on economics and struggles on adoption. Cloud delivery passes for some content and not for latency-sensitive content. Data mesh passes only if budget, headcount and rule-setting authority move with the responsibility. Generative AI passes in three specific places — corporate action extraction, entity resolution, and quality rule generation — where the task is extraction or classification into a defined schema, the output is verifiable against an independent source, and a human remains the approver of record. That human is a control, not a limitation, and it is the same control that makes every other number in this module defensible.

Process Cheat Sheet — The Whole Data Lifecycle on One Page

A single reference for the entire market data and reference data flow, from vendor contract to regulatory submission.

Stage Timing Core activity Key output Lead and systems
1. Sourcing and licensing 3–9 months per source Negotiate content, usage rights, entity scope, audit terms Contract with defined display, non-display, derived and redistribution rights Market data management; contract inventory (MDSL, TRG Screen)
2. Acquisition and ingestion Continuous Connect, decode, sequence, detect gaps and truncation Landed raw records with arrival timestamps Data engineering; feed handlers, exchange-native and vendor feeds
3. Normalisation and symbology Milliseconds to minutes Map identifiers and formats to the internal model, effective-dated Records bound to the internal instrument key Data engineering, reference data ops; cross-reference services, OpenFIGI
4. Mastering and golden copy Continuous, daily cut-offs Apply source hierarchy, tolerance, approval One authorised value per field, with provenance Reference data ops; GoldenSource, Alveo, NeoXam
5. Validation and exceptions Intraday plus EOD cut-off Detect, route, age and resolve failures; run IPV Approved marks, escalations, audit trail Reference data ops and product control; quality rule engines
6. Distribution and entitlements Microseconds to overnight Deliver at required latency; enforce licence at point of delivery Data in the hands of permitted consumers only Data engineering, market data management; Kafka, Solace, kdb+, entitlement engine
7. Consumption Continuous Quote, risk, value, report, show to clients Trading decisions, risk numbers, marks, client output Front office, risk, finance, compliance
8. Reporting, archival and cost T+1 to monthly Submit, evidence lineage, attribute cost, archive Regulatory submissions, reproducible history, chargeback Regulatory reporting, data governance, finance

Key numbers and rules of thumb

Metric Rule of thumb
Global market data spend 49.2bn USD in 2025, up 6.5%; grows every year regardless of market conditions
Tier-one bank spend Several hundred million USD a year; routinely a top-three discretionary technology line
Clock synchronisation (RTS 25) Max divergence from UTC 1ms, or 100µs where gateway-to-gateway latency ≤ 1ms; granularity 1ms or 1µs; HFT at the 100µs / 1µs end
Peak market data volume OPRA bursts above 187 million messages per second (April 2025); design for ~75m/sec microbursts; peaks over 50 Gbps
Consensus pricing Totem, operating since 1997, around 120 contributing institutions; contributors see only the aggregate
Vendor audit look-back Commonly 3–5 years; unlicensed use charged for the full period, often with interest or uplift
Regional split of spend Americas roughly half, EMEA just under a third, Asia the remainder and growing fastest
Top of the vendor market Bloomberg, LSEG and S&P Global Market Intelligence dominate; consolidation since 2021 reduced alternatives
Time to onboard a new data source 3–9 months for a significant contract; the licence, not the pipeline, is the critical path
Time to onboard a standard instrument Same day is the benchmark; slower forces desks onto local, unmastered data
Delayed-data fallback Venues must publish market data free of charge 15 minutes after publication under MiFID II
MiFIR transaction reporting deadline Close of the working day following the transaction
Typical IPV tolerance on a liquid corporate bond Illustrative ±0.25 points around the independent evaluated price; tighter for liquid, wider for illiquid
Fair-value hierarchy Level 1 quoted in active markets, Level 2 other observable inputs, Level 3 unobservable and most scrutinised
Corporate action notification lead time Commonly around 20 business days before payment or election deadline
Corporate action messaging MT564 notifies, MT566 confirms (ISO 15022); seev.031 and seev.036 are the ISO 20022 successors
Benchmark transition LIBOR ended 30 September 2024 with the final synthetic settings; replacements are overnight, backward-looking, compounded in arrears
BMR scope since 2025 Regulation (EU) 2025/914 narrows scope to critical, significant, EU Climate and certain commodity benchmarks
Regulatory retention Typically 5–7 years for transaction data, longer in some jurisdictions; archive, never delete
Exception health Judge by auto-repair rate and ageing, not by exception count
Day-count error cost Illustrative 3,615 euro a day on a 25m position — below every tolerance, present every day
The number that matters most Data-attributable P&L breaks — the only metric that converts data quality into money the business recognises

The one-sentence version: A bank buys observations of the market under contracts that constrain what it may do with them, translates every source's names into one internal language, resolves the inevitable disagreements into a single authorised value through rules it wrote and approvals it recorded, delivers that value only to consumers entitled to receive it, and then proves — years later if asked — exactly how it was made.