We’re excited to share our paper on reward model serving for RLHF: How Fast Can Reward Models Score? A Systems Study of C++ and PyTorch Inference Runtimes for RLHF. We built and benchmarked a native C++ ONNX Runtime engine against PyTorch eager mode, torch.compile, and FastAPI on CPU and GPU, with repeated independent launches for statistical rigor. The paper shows that on CPU the C++/ONNX Runtime path clearly outperforms PyTorch baselines, while on GPU torch.compile is faster at both median and tail latency. We also found that batching strategy matters more than language wrapper choice: naive padding hurts throughput badly, while length-aware bucketing helps only on GPU.</p>\n","updatedAt":"2026-07-29T19:35:28.165Z","author":{"_id":"69b46ad9488d75cf4201a24d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/69b46ad9488d75cf4201a24d/KA0Bob_6VM8xijpFETzQX.jpeg","fullname":"Vishnu Pulipaka","name":"pulipakav-1","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8994534015655518},"editors":["pulipakav-1"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/69b46ad9488d75cf4201a24d/KA0Bob_6VM8xijpFETzQX.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2607.19712","authors":[{"_id":"6a6a54c84463a8a84bdc3e83","name":"Venkata Naga Sai Vishnu Rohit Pulipaka","hidden":false},{"_id":"6a6a54c84463a8a84bdc3e84","name":"Anish Katta","hidden":false},{"_id":"6a6a54c84463a8a84bdc3e85","name":"Deva Rohit Reddy Peddireddy","hidden":false}],"publishedAt":"2026-07-22T00:00:00.000Z","submittedOnDailyAt":"2026-07-29T00:00:00.000Z","title":"How Fast Can Reward Models Score? A Systems Study of C++ and PyTorch Inference Runtimes for RLHF","submittedOnDailyBy":{"_id":"69b46ad9488d75cf4201a24d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/69b46ad9488d75cf4201a24d/KA0Bob_6VM8xijpFETzQX.jpeg","isPro":true,"fullname":"Vishnu Pulipaka","user":"pulipakav-1","type":"user","name":"pulipakav-1"},"summary":"In RLHF pipelines, reward scoring blocks policy updates. Slow scoring bottlenecks the entire loop, since no update runs until every rollout gets a score. And yet most setups just default to PyTorch eager mode or torch.compile, no one checks if that's actually fastest. Scoring itself is small. Rollout generation eats far more of a typical RLHF step. But scoring and generation fight over the same CPU and GPU resources, so a faster scoring engine doesn't shrink step time on its own. It mainly frees up capacity generation can use instead. We built a native C++ inference engine on ONNX Runtime. First step: confirm correctness. Output matched the PyTorch reference to 5.7 x 10^-6 on CPU and 4.2 x 10^-3 on GPU, close enough to trust. Then we tested it against PyTorch eager mode, torch.compile, and FastAPI, on both CPU and GPU. CPU was decisive. Our engine beat every baseline, confidence intervals didn't even overlap. GPU gave a different view: we beat PyTorch and FastAPI, but torch.compile came out ahead. Further testing traced the speedup to ONNX Runtime itself, not C++ as a language. And batching strategy mattered more than either the language or the runtime choice, more than we expected. The results are from repeated, independent runs, since single runs just aren't reliable enough to trust.","upvotes":2,"discussionId":"6a6a54c84463a8a84bdc3e86","projectPage":"https://arxiv.org/abs/2607.19712","githubRepo":"https://github.com/vishnup22/reward-model-benchmarks","githubRepoAddedBy":"user","githubStars":0},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"69b46ad9488d75cf4201a24d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/69b46ad9488d75cf4201a24d/KA0Bob_6VM8xijpFETzQX.jpeg","isPro":true,"fullname":"Vishnu Pulipaka","user":"pulipakav-1","type":"user"},{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2607/2607.19712.md","query":{}}">
How Fast Can Reward Models Score? A Systems Study of C++ and PyTorch Inference Runtimes for RLHF
Abstract
In RLHF pipelines, reward scoring blocks policy updates. Slow scoring bottlenecks the entire loop, since no update runs until every rollout gets a score. And yet most setups just default to PyTorch eager mode or torch.compile, no one checks if that's actually fastest. Scoring itself is small. Rollout generation eats far more of a typical RLHF step. But scoring and generation fight over the same CPU and GPU resources, so a faster scoring engine doesn't shrink step time on its own. It mainly frees up capacity generation can use instead. We built a native C++ inference engine on ONNX Runtime. First step: confirm correctness. Output matched the PyTorch reference to 5.7 x 10^-6 on CPU and 4.2 x 10^-3 on GPU, close enough to trust. Then we tested it against PyTorch eager mode, torch.compile, and FastAPI, on both CPU and GPU. CPU was decisive. Our engine beat every baseline, confidence intervals didn't even overlap. GPU gave a different view: we beat PyTorch and FastAPI, but torch.compile came out ahead. Further testing traced the speedup to ONNX Runtime itself, not C++ as a language. And batching strategy mattered more than either the language or the runtime choice, more than we expected. The results are from repeated, independent runs, since single runs just aren't reliable enough to trust.
Community
We’re excited to share our paper on reward model serving for RLHF: How Fast Can Reward Models Score? A Systems Study of C++ and PyTorch Inference Runtimes for RLHF. We built and benchmarked a native C++ ONNX Runtime engine against PyTorch eager mode, torch.compile, and FastAPI on CPU and GPU, with repeated independent launches for statistical rigor. The paper shows that on CPU the C++/ONNX Runtime path clearly outperforms PyTorch baselines, while on GPU torch.compile is faster at both median and tail latency. We also found that batching strategy matters more than language wrapper choice: naive padding hurts throughput badly, while length-aware bucketing helps only on GPU.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2607.19712 in a model README.md to link it from this page.
Cite arxiv.org/abs/2607.19712 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2607.19712 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.