In-context learning (ICL) is the standard method for low-resource classification, yet its efficacy in specialized domains remains largely unexplored. We address the challenge of classifying semantically complex, multi-party B2B conversations, where traditional ICL encounters significant limitations, especially as context length increases due to the concatenation of multiple few-shot examples. We introduce the Call Playbook dataset, featuring five classification tasks derived from real-world B2B conversations targeting core sales concepts. To bridge the gap between performance and practical utility, we propose novel knowledge extraction methods that distill verbose examples into compact, interpretable representations of structured classification criteria and precise task descriptions. Our approach achieves a 99% reduction in token usage and improves macro-averaged AUC by up to 7% over traditional ICL. Notably, it remains robust as context grows, unlike advanced token compression baselines which degrade by over 9 F1 points. Importantly, our framework enables direct refinement of classification logic, addressing critical needs for transparency, efficiency, and user interaction in real-world NLP applications.</p>\n","updatedAt":"2026-06-22T05:39:39.342Z","author":{"_id":"630733614d2c7796a4f3d27d","avatarUrl":"/avatars/98f2b5dbeca5fafd9b205a9ac6a83d2c.svg","fullname":"Guy Rotman","name":"guyrotman","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9063892364501953},"editors":["guyrotman"],"editorAvatarUrls":["/avatars/98f2b5dbeca5fafd9b205a9ac6a83d2c.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.15641","authors":[{"_id":"6a38c9ecdb23715e9da139ac","name":"Guy Rotman","hidden":false},{"_id":"6a38c9ecdb23715e9da139ad","name":"Adi Kopilov","hidden":false},{"_id":"6a38c9ecdb23715e9da139ae","name":"Danit Berger Zalmanson","hidden":false},{"_id":"6a38c9ecdb23715e9da139af","name":"Omri Allouche","hidden":false}],"mediaUrls":["https://cdn-uploads.huggingface.co/production/uploads/630733614d2c7796a4f3d27d/KLbUBkAGFj0oN7_a3U-F_.png"],"publishedAt":"2026-06-14T00:00:00.000Z","submittedOnDailyAt":"2026-06-22T00:00:00.000Z","title":"Distilling Examples into Task Instructions: Enhanced In-Context Learning for Real-World B2B Conversations","submittedOnDailyBy":{"_id":"630733614d2c7796a4f3d27d","avatarUrl":"/avatars/98f2b5dbeca5fafd9b205a9ac6a83d2c.svg","isPro":false,"fullname":"Guy Rotman","user":"guyrotman","type":"user","name":"guyrotman"},"summary":"In-context learning (ICL) is the standard method for low-resource classification, yet its efficacy in specialized domains remains largely unexplored. We address the challenge of classifying semantically complex, multi-party B2B conversations, where traditional ICL encounters significant limitations, especially as context length increases due to the concatenation of multiple few-shot examples. We introduce the Call Playbook dataset, featuring five classification tasks derived from real-world B2B conversations targeting core sales concepts. To bridge the gap between performance and practical utility, we propose novel knowledge extraction methods that distill verbose examples into compact, interpretable representations of structured classification criteria and precise task descriptions. Our approach achieves a 99\\% reduction in token usage and improves macro-averaged AUC by up to 7\\% over traditional ICL. Notably, it remains robust as context grows, unlike advanced token compression baselines which degrade by over 9 F1 points. Importantly, our framework enables direct refinement of classification logic, addressing critical needs for transparency, efficiency, and user interaction in real-world NLP applications.","upvotes":1,"discussionId":"6a38c9eddb23715e9da139b0","projectPage":"https://github.com/gong-io/call-playbook/","githubRepo":"https://github.com/gong-io/call-playbook","githubRepoAddedBy":"user","ai_summary":"A novel approach for B2B conversation classification that reduces token usage by 99% while improving performance and maintaining robustness as context length increases.","ai_keywords":["in-context learning","few-shot examples","token compression","classification logic","semantic complexity","multi-party conversations","B2B conversations","macro-averaged AUC","F1 score"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":0},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"630733614d2c7796a4f3d27d","avatarUrl":"/avatars/98f2b5dbeca5fafd9b205a9ac6a83d2c.svg","isPro":false,"fullname":"Guy Rotman","user":"guyrotman","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"query":{}}">
Distilling Examples into Task Instructions: Enhanced In-Context Learning for Real-World B2B Conversations
Abstract
A novel approach for B2B conversation classification that reduces token usage by 99% while improving performance and maintaining robustness as context length increases.
In-context learning (ICL) is the standard method for low-resource classification, yet its efficacy in specialized domains remains largely unexplored. We address the challenge of classifying semantically complex, multi-party B2B conversations, where traditional ICL encounters significant limitations, especially as context length increases due to the concatenation of multiple few-shot examples. We introduce the Call Playbook dataset, featuring five classification tasks derived from real-world B2B conversations targeting core sales concepts. To bridge the gap between performance and practical utility, we propose novel knowledge extraction methods that distill verbose examples into compact, interpretable representations of structured classification criteria and precise task descriptions. Our approach achieves a 99\% reduction in token usage and improves macro-averaged AUC by up to 7\% over traditional ICL. Notably, it remains robust as context grows, unlike advanced token compression baselines which degrade by over 9 F1 points. Importantly, our framework enables direct refinement of classification logic, addressing critical needs for transparency, efficiency, and user interaction in real-world NLP applications.
Community
In-context learning (ICL) is the standard method for low-resource classification, yet its efficacy in specialized domains remains largely unexplored. We address the challenge of classifying semantically complex, multi-party B2B conversations, where traditional ICL encounters significant limitations, especially as context length increases due to the concatenation of multiple few-shot examples. We introduce the Call Playbook dataset, featuring five classification tasks derived from real-world B2B conversations targeting core sales concepts. To bridge the gap between performance and practical utility, we propose novel knowledge extraction methods that distill verbose examples into compact, interpretable representations of structured classification criteria and precise task descriptions. Our approach achieves a 99% reduction in token usage and improves macro-averaged AUC by up to 7% over traditional ICL. Notably, it remains robust as context grows, unlike advanced token compression baselines which degrade by over 9 F1 points. Importantly, our framework enables direct refinement of classification logic, addressing critical needs for transparency, efficiency, and user interaction in real-world NLP applications.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2606.15641 in a model README.md to link it from this page.
Cite arxiv.org/abs/2606.15641 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2606.15641 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.