▶ Full Post Text
Hi,
I'm looking to understand if my strategy development process is guarding me against over fitting or whether I'm over optimising.
I'd really appreciate any constructive comments and advice on how I can improve my process.
This is the high level principles of my strategy development process assuming that I am developing a strategy for US futures indices (e.g. ES, NQ, etc.):
**Step 1. Initial strategy testing on in-sample data (2019-2021 = 3 years).**
* This is where I test a strategy for profitability.
* I will only take the strategy forward for further consideration if it hits my performance target, in this case the Drawdown ratio must be greater than 2 (for clarity I consider DD ratio to be net profit / maximum drawdown).
* I generally do not optimise indicator variables, I prefer to use the default variables for a given indicator e.g. 14-period RSI, etc.
* During this step I only optimise my profit and stop targets which are typically tick, time or indicator based.
**Step 2. Performance check on out-of-sample data (2022-2024 = 3 years).**
* This is where I test the strategy against additional data.
* I will take the strategy forward if it hits my performance target, in this case the Drawdown ratio must be greater than 2.
* If the strategy does not meet my performance target then I will allow myself to revisit step 1 for an additional 2 times, so 3 times in total.
* If after the 3rd iteration the strategy does not meet my performance target then I bin the idea and stop development.
* If the strategy is profitable at this stage I lock in my parameters such as the signal variables, profit and stop targets, exit indicators, etc. I do not change these at any other stage.
**Step 3. Performance check on unseen data (2016-2018 = 3 years).**
* During this step I am looking to see how my strategy stands up against completely unseen data on a different market regime.
* All I want to see here is that the strategy is profitable i.e. net profit > $0.
**Step 4. Performance check on unseen data (2025 = 1 year).**
* Similar to step 3, this is a check on completely unseen data against a different market regime.
* All I want to see here is that the strategy is profitable i.e. net profit > $0.
**Step 5. Walk Forward Analysis (2019 - 2025 = 7 years)**
* As I do not optimise my indicator variables, I use WFO to understand the sensitivity of my strategy against varying profit & stop targets.
* My optimisation period is 1 year and my test period is 1 year.
* All I want to see here is that my strategy does not fall apart and that the spread of my profit is distributed relatively evenly across the total period.
* If I see one year that has significantly out-performed the others then I will drop the strategy.
* If my strategy gets this far in my process, it is likely to pass the Walk Forward Optimisation check.
**Step 6. Monte-Carlo Simulation (2022-2025 = 4 years)**
* For the Monte Carlo Simulation I only use trades on out-of-sample data, I do not use my in-sample trades from 2019 -2021.
* On this step I want to see how the DD ratio and maximum DD compare to the backtest data from steps 2 & 4. If they are broadly similar or better then I consider this a pass.
* If my strategy gets this far in my process, it is likely to pass the Monte-Carlo check.
**Step 7. Forward test in simulation account (1-3 months)**
* Depending on the timeframe of my strategy I will run my strategy in a simulation account. For intraday systems, this will typically be 1 month, for swing systems I'd stretch this to 3 months.
* I will move the strategy to my live account if the strategy is profitable.