Watched a couple "validated" strategies come apart today, and it had nothing to do with the signal

Today was a decent gut check (Nasdaq down about 4%). The entries were fine. What broke was everything the backtest waves away. Fills was the first thing I noticed. The sim was marking trades at prices that didn't exist in any real size once things were moving, and the limits that "filled instantly" in the backtest were the exact ones getting run over live. You only get the passive fill when someone's about to trade through you, so on a day like today your passive edge doesn't shrink, it flips sign, and a clean queue model never shows you that. Also, the "just stress test against 2020 and 2022" advice doesn't save anyone either. That's three data points. Tune a system to survive those specific days and you've memorized them, not learned anything, and the next one won't rhyme. Replaying old crashes is curve-fitting with a scarier dataset. Here's the part that actually matters: your costs and your edge blow up together. Spread and depth fall apart on the same volspike that's firing your signal, so a flat slippage number is most wrong exactly when you're trading the most. If your cost model isn't conditioned on live book state, it's lying to you on the only days that decide whether you survive. So if you want to know whether a strategy is real, look at how it behaves on the worst handful of vol days, model fills off real book depth, and measure correlations under stress rather than over ten calm years. That's the difference between a system that survives a morning like this and one that just hadn't met it yet. I build validation tooling, so I stare at this daily. Today was just a reminder of which half of the work everyone skips.