Letting an LLM write your backtest? Check for this one-line look-ahead bug first

Vibe-coding strategies is everywhere now. Describe an idea, get a backtest in Python, run it, admire the equity curve. The problem usually isn't the strategy idea. It's that the generated harness is quietly wrong in a way that inflates returns, and the code runs clean, so nobody looks closer. The one I see most often is an unlagged signal: df\['signal'\] = (df\['close'\] > df\['close'\].rolling(50).mean()).astype(int) df\['ret'\] = df\['close'\].pct\_change() df\['strat'\] = df\['signal'\] \* df\['ret'\] # look-ahead (1 + df\['strat'\]).cumprod().plot() The signal is only known at the close of bar t. But ret at bar t is the move into that close. Multiply them and you're capturing a return you could only have earned by acting before the bar formed. You didn't. You're trading on information you didn't have yet. The fix is one line: df\['strat'\] = df\['signal'\].shift(1) \* df\['ret'\] In a quick illustrative test (hypothetical, not a live result), a plain 50-period crossover that should sit near breakeven after costs printed a Sharpe well north of 2 with the bug. Add the .shift(1) and the edge evaporates. Same idea, same data, the only difference was one bar of timing. Why LLMs produce this so reliably: they pattern-match to tutorial code that has the same bug, they optimize for "runs and looks plausible," and they have no idea what the timestamp on your bar actually means. "The code executed and the chart looks great" isn't validation. It's the most convincing way to be wrong. A few other quiet ones I've caught in generated code: \- Fitting a scaler/normalizer on the full series before the train/test split (leakage). \- Entering at a bar's high or low as if you'd have known the intrabar extreme. \- resample() or merge\_asof pulling a value forward across the timestamp it belongs to. \- dropna() after an indicator that silently misaligns the signal and price index. don't get me wrong, LLMs are genuinely decent for boilerplate, plotting, refactors. The point is narrower: generated backtest code has to be read line by line for timing and fill logic, because that's exactly where it breaks and exactly where it looks fine. What's the worst LLM-generated backtest bug you've caught? And does anyone run a fixed checklist over generated code before trusting a curve from it?