iFAN: Inference-Aware Learning for Plain Mask Transformers</p>\n","updatedAt":"2026-08-12T03:53:16.333Z","author":{"_id":"630f612fcc8ed75decb4796e","avatarUrl":"/avatars/261f03bb0c926d66993df9560abb74fc.svg","fullname":"Lucas","name":"xxlucas","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.3768432140350342},"editors":["xxlucas"],"editorAvatarUrls":["/avatars/261f03bb0c926d66993df9560abb74fc.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.03216","authors":[{"_id":"6a7bedcc1653ef87c6af1cb6","name":"Fang Li","hidden":false},{"_id":"6a7bedcc1653ef87c6af1cb7","name":"Yu He","hidden":false},{"_id":"6a7bedcc1653ef87c6af1cb8","name":"Haoyang Tong","hidden":false},{"_id":"6a7bedcc1653ef87c6af1cb9","name":"Lichen Ma","hidden":false},{"_id":"6a7bedcc1653ef87c6af1cba","name":"Jingling Fu","hidden":false},{"_id":"6a7bedcc1653ef87c6af1cbb","name":"Wenxiao Fan","hidden":false},{"_id":"6a7bedcc1653ef87c6af1cbc","name":"Tongxuan Liu","hidden":false},{"_id":"6a7bedcc1653ef87c6af1cbd","name":"Luohang Liu","hidden":false},{"_id":"6a7bedcc1653ef87c6af1cbe","name":"Ke Zhang","hidden":false},{"_id":"6a7bedcc1653ef87c6af1cbf","name":"Junshi Huang","hidden":false}],"publishedAt":"2026-08-07T00:00:00.000Z","submittedOnDailyAt":"2026-08-12T00:00:00.000Z","title":"iFAN: Inference-Aware Learning for Plain Mask Transformers","submittedOnDailyBy":{"_id":"630f612fcc8ed75decb4796e","avatarUrl":"/avatars/261f03bb0c926d66993df9560abb74fc.svg","isPro":true,"fullname":"Lucas","user":"xxlucas","type":"user","name":"xxlucas"},"summary":"Query-based mask transformers assemble segmentation outputs through pixel-wise competition among query predictions of the final layer, yet this inference process is not explicitly optimized during training. We identify two key mismatches: the query with the highest probability-mask score does not necessarily produce the most accurate mask, and final-layer decoding may discard superior predictions from intermediate layers. To address these issues, we propose Inference-Aware Learning (iFAN), a general training framework for plain mask transformers. iFAN introduces Adjusted Probability-Mask Ranking (APMR), which aligns query competition with predicted mask quality and suppresses high-confidence but inaccurate competitors. We further employ Cross-Layer Self-Distillation (CLSD) to transfer stronger intermediate predictions to the final layer. The ranking and distillation objectives are training-only, while inference retains efficient final-layer decoding. Experiments on COCO, ADE20K, and Cityscapes demonstrate consistent improvements across panoptic, instance, and semantic segmentation, as well as across different architectures, backbone scales, and input resolutions. Overall, iFAN improves performance by an average of 1.20 PQ, 1.30 AP, and 0.63 mIoU, with negligible additional parameters, FLOPs and inference latency.","upvotes":3,"discussionId":"6a7bedcc1653ef87c6af1cc0","projectPage":"https://neesky163.github.io/iFAN/","ai_summary":"A training framework called iFAN improves mask transformers by aligning query ranking with mask quality and distilling stronger intermediate predictions to the final layer.","ai_keywords":["query-based mask transformers","pixel-wise competition","Adjusted Probability-Mask Ranking","Cross-Layer Self-Distillation","inference-aware learning","panoptic segmentation","instance segmentation","semantic segmentation"],"ai_summary_model":"thinkingmachines/Inkling-Small"},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"630f612fcc8ed75decb4796e","avatarUrl":"/avatars/261f03bb0c926d66993df9560abb74fc.svg","isPro":true,"fullname":"Lucas","user":"xxlucas","type":"user"},{"_id":"69ccb51c92e44910c0fe5770","avatarUrl":"/avatars/b441c6523493ad04c76fae16fb553688.svg","isPro":false,"fullname":"Григорьев Никита","user":"dylangarcial22","type":"user"},{"_id":"69a3f55823713679f5174978","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/pTt6-L7RqrI9qJZhnSXla.jpeg","isPro":false,"fullname":"山崎翔太","user":"isaacgarcia","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.03216.md","query":{}}">
iFAN: Inference-Aware Learning for Plain Mask Transformers
Published on Aug 7
· Submitted by Lucas on Aug 12 Abstract
A training framework called iFAN improves mask transformers by aligning query ranking with mask quality and distilling stronger intermediate predictions to the final layer.
Query-based mask transformers assemble segmentation outputs through pixel-wise competition among query predictions of the final layer, yet this inference process is not explicitly optimized during training. We identify two key mismatches: the query with the highest probability-mask score does not necessarily produce the most accurate mask, and final-layer decoding may discard superior predictions from intermediate layers. To address these issues, we propose Inference-Aware Learning (iFAN), a general training framework for plain mask transformers. iFAN introduces Adjusted Probability-Mask Ranking (APMR), which aligns query competition with predicted mask quality and suppresses high-confidence but inaccurate competitors. We further employ Cross-Layer Self-Distillation (CLSD) to transfer stronger intermediate predictions to the final layer. The ranking and distillation objectives are training-only, while inference retains efficient final-layer decoding. Experiments on COCO, ADE20K, and Cityscapes demonstrate consistent improvements across panoptic, instance, and semantic segmentation, as well as across different architectures, backbone scales, and input resolutions. Overall, iFAN improves performance by an average of 1.20 PQ, 1.30 AP, and 0.63 mIoU, with negligible additional parameters, FLOPs and inference latency.
Community
iFAN: Inference-Aware Learning for Plain Mask Transformers
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2608.03216 in a model README.md to link it from this page.
Cite arxiv.org/abs/2608.03216 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2608.03216 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.