u/DoomsdayMcDoom ·
Reddit — r/algotrading
· May 21, 2026 at 23:12
· ⬆ 16 pts
· 💬 41 comments
| View on Reddit ↗
AI Summary
Summary
The post describes an algorithmic trading system design using GCP, storing 20 years of historical data across all timeframes and building hundreds of feature tables.
The author is focused on pipeline efficiency and cost, not on specific trade ideas or market direction.
Quality assessment: This is a technical infrastructure discussion, not an investment analysis or DD. It is noise from a trading perspective.
Score16
Comments41
Upvote %84%
▶ Full Post Text
Anyone care to share some tips on system design?
I finally went to GCP with 20 years of historic data across all time frames from 1 second to 1 quarter. I loaded the raw data to a storage bucket for my data lake.
For another layer I have hundreds of feature tables across all time frames joined on the key ticker/contract, timeframe, window start, window close, and date.
I then built a massively wide feature table across all timeframes. For realtime data I’m using dataflow/apache beam orchestrated with airflow.
I’m using this data to locate repeatable signals across timeframes and a combination of features. Once a repeatable signal is found I build it into the neural network for regime detection but if it’s repeatable between multiple timeframes I have a separate neural network on those time frames.
My issue is building the features and gold layer is taking forever. I mean multiple days using cloud run and it’s costing quite a bit. I tried loading the data into bigquery and building the gold layer there but it’s a lot more expensive than cloud run.
I’m open to suggestions on how to improve my pipeline and I’m curious as to what system design many of you are using?