A 35%-accurate model that still ranked well, and adding more features made it worse (point-in-time equity backtest) [P]

I built a calibrated predictive model on point-in-time fundamentals and two results surprised me enough to write up. Sharing the project and the method, happy to have the methodology picked apart.

Setup: ~250 S&P 500 companies graded from their 10-K filings on qualitative factors (moat, market position, leadership, capital allocation). Grading is point-in-time, the grader sees only the filing for that vintage, across 2014/2017/2020. Index membership reconstructed as-of each vintage, so no survivorship. Cross-validation folds grouped by ticker so a company never sits on both sides. The predictor is a predictive database, so there is no training step; the calibrated probability comes back from a query with a per-feature contribution breakdown.

Result 1, ranking beat accuracy. Exact-outcome accuracy was 35% vs a 27% base rate. Unimpressive alone, but the ranking separated a top-20 bucket returning 20.6%/yr from a bottom-20 that lost money (market 7.9%). Brier 0.181, and the calibration curve tracks the diagonal across deciles. For a "rank and act on the top slice" decision, calibrated ordering is the metric, not argmax accuracy.

Result 2, more features hurt. Information gain dropped from 0.107 on 6 decorrelated features to 0.023 with all 16 in. Not redundancy, correlated features are already collapsed, just weak signals accumulating as noise.

Honest limits, since this crowd will rightly push: small sample, 12-year horizon, no transaction costs, and the winning sector (semiconductors) is visible in the training outcomes, so the backtest says what would have worked, not what you could have known in 2015. The LLM grades also risk a halo from already-winning firms; the headline model leans on the less-haloed features and passes a per-vintage drift check, but I cannot fully rule it out.

Writeup with charts: https://aito.ai/blog/value-quality-or-growth-who-was-right

Playable demo, no signup: https://demos.aito.ai/equity

Not claiming alpha or investment advice, equities are just a domain everyone can check. Happy to get into the fold construction, the point-in-time grading, or where you think it still leaks.