[Paper] Towards Scalable Lifelong Knowledge Editing with Selective Knowledge Suppression

In this paper, the authors tackle continued pretraining without the risk of catastrophic forgetting, by identifying parameters which can safely be changed without risking identified concepts, and freezing the rest:

https://arxiv.org/abs/2604.19089v1

Current practice is to mix new datasets into comprehensive datasets to facilitate pretraining without catastrophic forgetting, which works but at the cost of an order of magnitude or more higher training costs (since it is not only training on the new data, but also on old data which reinforces the existing knowledge/skills).

The authors' method might render mixing new data into comprehensive data unnecessary, because the model could be trained on only the new data, without risking old knowledge.

Edited: Fixed typo