AI-assisted pre-review of open-source software submissions: an experience report from BOSC 2026
Mirrored from arXiv — NLP / Computation & Language for archival readability. Support the source by reading on the original site.
Computer Science > Computation and Language
Title:AI-assisted pre-review of open-source software submissions: an experience report from BOSC 2026
Abstract:Most conferences rely on peer-review of submissions, but as generative AI makes it easier than ever to prepare submission materials, some conferences are seeing an overwhelming surge of submissions. We wanted to see if generative AI could help our conference's volunteer reviewers by pre-reviewing abstracts for certain criteria. The Bioinformatics Open Source Conference (BOSC) was well-positioned to experiment with this, as we already had a detailed rubric used by reviewers to evaluate submitted abstracts on multiple criteria, including openness (public availability of the code or other content associated with the project), valid open source license, and "runnability" (how easy it is to download, build, and run the project - an important measure of reusability). For BOSC 2026, we built bosc-pre-review, an agentic skill that assessed six review criteria, and Runabilly, which builds and tests each project in a disposable Docker container for safety. The AI only gathered evidence to present to the reviewers; humans made every decision regarding the acceptance of the abstracts. After the review period, we surveyed the reviewers to determine how useful they found the pre-review. Most of those who responded said they found it useful, but they preferred to check the AI's conclusions against their own, rather than accepting the AI results unquestioningly.
| Comments: | 18 pages, 4 figures |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Digital Libraries (cs.DL) |
| Cite as: | arXiv:2607.27228 [cs.CL] |
| (or arXiv:2607.27228v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.27228
arXiv-issued DOI via DataCite
|
Access Paper:
- View PDF
Current browse context:
References & Citations
Bookmark
Bibliographic and Citation Tools
Code, Data and Media Associated with this Article
Demos
Recommenders and Search Tools
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
More from arXiv — NLP / Computation & Language
-
Geometric and Behavioral Stratification in Transformer Residual Streams
Aug 14
-
Perturbation-based Regional Interpretability through Subtraction Mapping (PRISM): naming-error dissociations in language models and post-stroke aphasia
Aug 14
-
I-SDPO: Instance-Level Adaptive Self-Distillation Policy Optimization
Aug 14
-
Comment on "Modeling rapid language learning by distilling Bayesian priors into artificial neural networks"
Aug 14
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.