We propose Functional Attention (FUNCATTN), a new attention mechanism for operator learning inspired by functional maps. Instead of computing pointwise softmax affinities between tokens, we reframe attention as a compact linear operator between learned functional spaces, reducing complexity from O(n²) to O(ndk). FUNCATTN achieves SOTA on 5/6 PDE benchmarks, 3D point cloud segmentation, and OOD generalization, while remaining resolution-invariant. Accepted to ICML 2026.</p>\n","updatedAt":"2026-06-04T16:37:00.961Z","author":{"_id":"6a1ff199c30665929c7c5ccb","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/XykYIkOjNjFwZsAJWXeCX.png","fullname":"Simon Weber","name":"SimonWeber","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":1,"identifiedLanguage":{"language":"en","probability":0.8486101627349854},"editors":["SimonWeber"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/XykYIkOjNjFwZsAJWXeCX.png"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2605.31559","authors":[{"_id":"6a203de715100c5272a843f7","name":"Jiefang Xiao","hidden":false},{"_id":"6a203de715100c5272a843f8","name":"Maolin Gao","hidden":false},{"_id":"6a203de715100c5272a843f9","user":{"_id":"6a1ff199c30665929c7c5ccb","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/XykYIkOjNjFwZsAJWXeCX.png","isPro":false,"fullname":"Simon Weber","user":"SimonWeber","type":"user","name":"SimonWeber"},"name":"Simon Weber","status":"claimed_verified","statusLastChangedAt":"2026-06-04T12:41:21.497Z","hidden":false},{"_id":"6a203de715100c5272a843fa","name":"Guandao Yang","hidden":false},{"_id":"6a203de715100c5272a843fb","name":"Daniel Cremers","hidden":false}],"publishedAt":"2026-05-29T00:00:00.000Z","submittedOnDailyAt":"2026-06-04T00:00:00.000Z","title":"Functional Attention: From Pairwise Affinities to Functional Correspondences","submittedOnDailyBy":{"_id":"6a1ff199c30665929c7c5ccb","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/XykYIkOjNjFwZsAJWXeCX.png","isPro":false,"fullname":"Simon Weber","user":"SimonWeber","type":"user","name":"SimonWeber"},"summary":"Learning mappings between infinite-dimensional function spaces, or operator learning, is essential for many machine learning applications. Although transformer-based operators are popular, they often rely on token-wise attention. These methods treat continuous fields as discrete tokens and usually ignore the global functional structure. We introduce Functional Attention, which reinterprets attention as a functional correspondence between adaptive bases. Inspired by geometric functional maps, our method replaces softmax affinities with structured linear operators. This yields a compact, generalizable, resolution-invariant representation that explicitly captures global dependencies. Experiments demonstrate that Functional Attention can match state-of-the-art performance in many operator learning tasks, including solving PDEs, 3D segmentation, and regression, while remaining robust to varying discretizations. Project page is available at https://github.com/xjffff/FUNCATTN.","upvotes":0,"discussionId":"6a203de715100c5272a843fc","projectPage":"https://xjffff.github.io/funcattn/","githubRepo":"https://github.com/xjffff/FUNCATTN","githubRepoAddedBy":"user","ai_summary":"Functional Attention reinterprets attention as functional correspondence between adaptive bases, enabling compact and resolution-invariant operator learning for PDE solving and 3D segmentation.","ai_keywords":["operator learning","transformer-based operators","token-wise attention","functional correspondence","adaptive bases","geometric functional maps","softmax affinities","structured linear operators","global dependencies","PDE solving","3D segmentation","regression"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":6,"organization":{"_id":"61fae781e68759322b9767be","name":"TUM","fullname":"Technical University of Munich","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1661167219960-629521a0f937190946e15d7f.jpeg"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[],"acceptLanguages":["en"],"organization":{"_id":"61fae781e68759322b9767be","name":"TUM","fullname":"Technical University of Munich","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1661167219960-629521a0f937190946e15d7f.jpeg"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2605/2605.31559.md"}">
Functional Attention: From Pairwise Affinities to Functional Correspondences
Abstract
Functional Attention reinterprets attention as functional correspondence between adaptive bases, enabling compact and resolution-invariant operator learning for PDE solving and 3D segmentation.
Learning mappings between infinite-dimensional function spaces, or operator learning, is essential for many machine learning applications. Although transformer-based operators are popular, they often rely on token-wise attention. These methods treat continuous fields as discrete tokens and usually ignore the global functional structure. We introduce Functional Attention, which reinterprets attention as a functional correspondence between adaptive bases. Inspired by geometric functional maps, our method replaces softmax affinities with structured linear operators. This yields a compact, generalizable, resolution-invariant representation that explicitly captures global dependencies. Experiments demonstrate that Functional Attention can match state-of-the-art performance in many operator learning tasks, including solving PDEs, 3D segmentation, and regression, while remaining robust to varying discretizations. Project page is available at https://github.com/xjffff/FUNCATTN.
Community
We propose Functional Attention (FUNCATTN), a new attention mechanism for operator learning inspired by functional maps. Instead of computing pointwise softmax affinities between tokens, we reframe attention as a compact linear operator between learned functional spaces, reducing complexity from O(n²) to O(ndk). FUNCATTN achieves SOTA on 5/6 PDE benchmarks, 3D point cloud segmentation, and OOD generalization, while remaining resolution-invariant. Accepted to ICML 2026.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2605.31559 in a model README.md to link it from this page.
Cite arxiv.org/abs/2605.31559 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2605.31559 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.