Renting market data is easier than holding it. A vendor API answers a question in milliseconds, requires no storage, and stays current without any effort on your part. Building your own store means ingestion, schema decisions, disk, backups, and a class of problem that has nothing to do with markets.
We hold our own anyway. Here is why the trade is worth it.
A vendor’s history can change
The most important property of historical data is not accuracy. It is stability.
Vendors correct their data, and they should. A bad print gets fixed, a corporate action gets applied retroactively, a calculation is improved. Each correction makes the dataset more accurate and makes every result computed against the previous version unreproducible.
If your research queries a vendor on demand, you cannot tell the difference between “the code changed” and “the data changed”. A result from six months ago that no longer reproduces might be a regression you introduced, or a revision they applied. Both look identical from where you are standing, and only one is your problem.
Holding the data collapses that ambiguity. The bytes we computed against are the bytes we still have.
Research needs to see what was knowable, not what is known
A backtest is a claim about what a decision process would have done at a particular moment. That requires knowing what was actually observable at that moment, not what the best current understanding of that moment is.
These differ more than they seem to. Late prints, revised settlements, and adjustments applied after the fact are all improvements to the historical record and all forms of information the decision could not have had. A store organized around what was known at each point in time makes the distinction enforceable rather than aspirational: the code evaluating a decision cannot reach data that did not exist yet, because the store will not serve it.
That is the difference between a rule and an architecture. A rule survives until the afternoon someone is in a hurry.
Scale changes what questions you can ask
Options data is large in a way that surprises people arriving from equities. A single trading day of a single index’s option chain, at full granularity across every strike and expiry, runs to millions of rows.
At that size, the cost of a question stops being intellectual and starts being logistical. A study that requires a full re-read of several years of chain data is a study you will not run casually, and “will not run casually” quietly becomes “will not run”, which means the questions that get asked are the cheap ones rather than the important ones.
Holding the data locally, in a columnar format organized for the access patterns research actually uses, moves that cost down far enough that the expensive questions become askable. The change is not that any individual query gets faster. It is that a whole category of investigation stops being a project and becomes an afternoon.
Live data is pushed, not polled
The same argument applies at the other end of the timescale, in a different form.
Live prices and positions arrive over a streaming connection to the broker rather than by repeatedly asking. Polling has a structural problem beyond inefficiency: it can only tell you the state at the moments you happened to ask, and everything between two polls is invisible. For a system whose job includes noticing when something has changed, that gap is the thing you care about.
A stream also fails more honestly. A polling loop that has silently stopped receiving useful data looks exactly like a polling loop in a quiet market. A dropped connection announces itself.
We still reconcile against the broker’s own record periodically. A stream is the primary source of truth and reconciliation is the insurance, because the failure mode of a stream, a message that never arrives, is precisely the one the stream cannot report.
What it costs
This is not free, and it is worth being clear about that.
Holding your own data means owning ingestion, storage, integrity checking and backup. It means storage that grows every day, and it means a class of failure (a gap nobody noticed, a schema that no longer matches) that a vendor would otherwise have absorbed on your behalf.
The reason to accept those costs is not that they are small. It is that the alternative puts an opaque dependency underneath every number the firm produces, and a firm whose entire claim is that it measures carefully cannot afford one of those in the foundation.