I’m working on a real-time market data pipeline and currently only persist 1-minute candles and higher-timeframe aggregates, while consuming live tick data in memory.
The tick stream itself is already available via WebSockets, but I’ve intentionally avoided storing raw ticks in a traditional relational database because of write volume, storage cost, and long-term maintenance concerns.
I’m trying to decide what the most optimal long-term approach is for things like:
* historical replay
* research and strategy development
One approach I’ve been considering:
* append-only flat files (per symbol / per day)
For those of you who work with tick data in production or research environments:
* Do you store every tick?
* if yes, where and in what format?
* If not, what do you discard and why?
I’m mainly interested in real-world tradeoffs and lessons learned rather than theoretical answers.