UrgenT Help Detecting Performance Regressions Using Machine Learning and Hardware Counters [P]

I’m working on performance regression detection using machine learning/anomaly detection.

My setup is basically:

• Healthy runs are used to learn normal behaviour
• Regression runs are used to see whether the model detects the anomaly
• For each counter group I only have about 10 healthy samples
• I’m currently using leave-one-out on the healthy data to set the detection threshold
• The regression samples are not used during training or threshold selection

I’m confused about a few things:

• Do I still need a normal train/validation/test split for this type of one-class anomaly detection?
• With only 10 healthy samples, is leave-one-out better than splitting them into something like 60/20/20?
• Can the regression samples simply act as the unseen test set?
• Would it be better to collect a second independent healthy dataset and use that as a final test for false positives?
• For evaluation, should I mainly use false-positive rate and detection rate/recall rather than MSE/MAE, since I’m not predicting a continuous value?

Just trying to make sure the evaluation setup is correct before I finalise it.