System Design

Anyone care to share some tips on system design? I finally went to GCP with 20 years of historic data across all time frames from 1 second to 1 quarter. I loaded the raw data to a storage bucket for my data lake. For another layer I have hundreds of feature tables across all time frames joined on the key ticker/contract, timeframe, window start, window close, and date. I then built a massively wide feature table across all timeframes. For realtime data I’m using dataflow/apache beam orchestrated with airflow. I’m using this data to locate repeatable signals across timeframes and a combination of features. Once a repeatable signal is found I build it into the neural network for regime detection but if it’s repeatable between multiple timeframes I have a separate neural network on those time frames. My issue is building the features and gold layer is taking forever. I mean multiple days using cloud run and it’s costing quite a bit. I tried loading the data into bigquery and building the gold layer there but it’s a lot more expensive than cloud run. I’m open to suggestions on how to improve my pipeline and I’m curious as to what system design many of you are using?