Speaker
Description
As a biologist with a new dataset the cost of labelling and training a new ML model is prohibitive to downstream analysis. One solution is to utilise pre-trained networks in a transfer learning approach. Community efforts such as the BioImage-Model-Zoo have increased the availability of pre-trained models. However, how should the best pre-trained model for a particular task and dataset be selected? Qualitative approaches comparing datasets are common practice, but unreliable, as imperceptible differences in dataset distributions impact transfer success and they fail to consider the properties of the model itself. A quantified approach factoring in both datasets and models would enable a more systematic selection process, facilitating more effective reuse of existing models. I will present preliminary work on an unsupervised transferability heuristic that aims to rank pre-trained models with respect to their direct transfer performance on a target dataset. The approach centres on probing model prediction consistency to test time perturbations of the target data. Thus providing a conveniently obtainable unsupervised consistency score that correlates with model performance, allowing for suitable model selection for the target task.
| Authors | Joshua Talks*, Anna Kreshuk |
|---|---|
| Keywords | Model Selection, Transferability Heuristic, Transfer Learning |
