▶ Full Post Text
Hey all, in this post I will be outlining the approach I've taken to my current infrastructure, data, and strategy, along with how I tested and how I've verified there's no alpha, for two reasons:
1. To help other algo quant devs to avoid my mistakes
2. Look into insight from smarter people than me.
**So first things first, The Data Approach:**
I started off downloading 1 minute data over all 13,000 tickers in the US stock market over the last 20 years, including some other macros such as Oil, Gold, Silver, some international ETFs, US ETFs, and VIX. This is effectively (2005 - 2026). This is my data I am training everything on.
From there I built parquet files, and caches for the 1 minute and 1 day time frames. Incorporated company splits, M&A, ticker renames, point in universe (keeping track of dropped and newly added tickers) in the S&P 500 for example. Validated data is clean.
**Next, The BackTesting Approach:**
I used both Combinatorial Purged Cross Validation, as well as Walk Forward Optimization (all built in house), to test my strategy. I would then also track deflated sharpe ratio, sharpe ratio, Max Drawdown, Cum Return, CAGR, amongst other metrics. I then developed a triple barrier labelling (which is based on the AFML book, and takes into account 3 barriers (profit taking and stop loss barriers, which are daily computed based on ticker volatility), and a third barrier \~ time (which I arbitrarily chose as 10 days) for a daily based trading strategy.
I also ran 4 models as baselines (S&P 500 Buy and Hold, Mom\_12 (monthly rotating of highest momentum ticker per sector), and two others). S&P 500 proved to be the highest sharpe ratio and cumulative return, so that effectively is my baseline I need to beat, with a sharpe ratio of about \~0.5.
**Next, Feature Set:**
With the backtesting framework setup complete, I developed a set of 60 features, most of them technical or statistical indicators including (price, volatility, volume, return vs. stock's own return in a given period, return vs. s&p 500, return vs. sector average, and multiple other cross-asset correlation features).
**Next, Models:**
I only built two models to test up until this phase of the project. I used a LightGBM model in a supervised learning capacity, attempting to classify the daily labels across every 150 selected tickers, across my 20 year dataset. Keep in mind the triple barrier labels were computed pre-hand. CPCV would take care of look ahead bias.
I also built a linear regression model to attempt to estimate the time at which one of the 3 barriers would touch.
**Next, The Dissappointment:**
I ran my model with default hyperarameters, just to see how well it would be able to classify my labels. In all honesty, I anticipated it would be somwhere in the 60-70% accuracy and recall range, then with Optuna hyperparam tuning I could maybe get it up to 70-85%. These numbers are very humble comared to my grad school work where training on classification problems such as image classification, etc. would easily grant me 90%+ accuracy scores.
To my surprise, my model was only able to achieve around 50.5% accuracy, essentially a coinflip \~ zero alpha. In-sample validation showed 70% accuracy, and to further investigate, I tested which epoch gave me the best generalization accuracy \~ turned out to be epoch 2. Anything after that was overfitting heavily.
The linear regression model wasn't much better, effectively too much error to reliably generalize.
Of course there was a lot more future work to do in my algorithm, outlined in the next section, but I wanted to see even SOME promise from my classifier to be able to continue. Right now I feel completely devastated by these results.
**Future Phases of my Project (On Hold for now until I decide next pivot):**
1. Meta-labeling (based on AFML), a second layer on top of the models classification results
2. Optuna based hyper tuning of parameters
3. SHAP for interoperability of feature importance and model performance
4. Other interesting models (Transformers, Hidden Markov Models, Random Forests, etc.)
5. Risk Management Models
6. Execution Models (L2 based execution and fills)
**FINALLY, Where I think I went wrong, What could be done better, And Opening the floor for discussion**
1. AFML strictly talks about how time-based data such as (minute, hour, daily) etc. carries no significant alpha, and instead we should be looking at event driven information, which carries more information entropy.
2. I've seen a few people talk about tick-level data as where they've found success, rather than minute or hourly or daily time based data
3. Is my approach completely wrong? Is trying to predict triple barrier labels at 10 days out just a genuinely wrong approach given my feature set? What are typical classification predictions you try to make in your own algos? (Price, volatility, volume, imbalances, etc.)?
4. Finally, maybe I don't really need high classification accuracy, as Citadel I believe only achieves 51.5% accuracy, but at millions of trades, they're profitable in the billions. Maybe the real alpha is in the execution and risk management side of the algorithm?
5. I also tested across 20 years of 1 minute data across 150 tickers. Maybe sizing down my dataset could help?
I appreciate any, and all insight, PREFERABLY from smarter people than me who have ACTUALLY managed to produce profitable algorithms that trade in real markets.
(I'm not interested in how good your backtests are, I'm interested in insight from real-trading algorithms in the markets)
\- Thank you for reading my long post. You are a real one if you've got this far