It's time to desk reject papers that don't include code that can reproduce the results [D]

As review season for NeurIPS wraps up, I have now reviewed for 3 major conferences this year. And I'm noticing a worrying trend:

Out of the 12 papers I reviewed this year, only 1 provided full code (that runs the whole training pipeline from input dataset to output AUROC). 4 provided partial code with fragments of their method, but no ability to run the experiment end to end. And 7 provided no code.

This is really bad for ensuring quality and reproducibility. Of the 5 papers that provided at least some code, 3 of them contained obvious bugs that completely invalidated the results. ML is highly technical and small bugs can have huge impacts if they are in the wrong place. Who knows what was going on in the remaining 7 papers.

The fundamental issue here is of incentives: there is almost no cost to hiding code during the review process. Releasing code only increases odds of rejection due to reviewers finding bugs. The only way to fix this is to change the game by imposing real penalties on hiding code.