Hi everyone,
I’ve been working on a side project focused on forecast evaluation rather than model building.
In finance (and other decision-driven domains), I kept running into the same issue:
a model can look great on MSE, MAE, or R² and still be useless or harmful in practice.
Example:
Predict $101, actual is $99.
MSE or RMSE says “close”.
In reality, you lost money.
So I built an evaluator that scores predictions based on decision utility rather than proximity, using things like:
\- directional correctness
\- alignment over time
\- asymmetric downside risk
\- whether a naïve strategy based on the signal would have worked
Two core metrics (both model-agnostic and scale-invariant):
\- \*\*FIS\*\*: measures whether a forecast behaves like a usable signal relative to the realized data
(directional correctness, consistency, and outcome alignment matter more than small numerical error)
\- \*\*CER\*\*: measures how efficiently confidence is earned relative to error
(strong predictions are only rewarded if they justify their risk)
The math goes fairly deep (event-based weighting, regime sensitivity, etc.), but I’ve sanity-checked it using Monte Carlo simulations as well as real model outputs across different datasets.
When using these metrics to select between models on real datasets, the resulting strategies tended to behave materially better out-of-sample than those selected purely by error-based metrics, but I’m deliberately not claiming this as a trading edge, just an evaluation signal.
This is early and intentionally narrow, and I’m not selling anything.
I’d really value feedback from people here:
\- Does this framing make sense?
\- What obvious pitfalls should I watch out for?
\- Are there known approaches that already do this well?
If useful, I’m happy to explain details or share examples.
Demo and explanation: [https://quantsynth.org](https://quantsynth.org)