Impact of Transfer Learning on Transformers Networks for Prostate Image Segmentation

The segmentation of the prostate in magnetic resonance images (MRI) plays a crucial role in detection and treatment planning of prostate cancer and other types of disease. Recently, vision transformers (ViT) have gained great success in automating the segmentation of the different zones of the prostate. Specifically, ViTs excel in capturing long-range dependencies within an image through their self-attention mechanisms. However, ViTs demand large training datasets for effective performance, posing a challenge in medical applications where acquiring such datasets is costly and time-consuming. Transfer learning offers a solution by pre-training a ViT on natural image dataset and fine-tuning it for the specific segmentation task at hand. In this work, we statistically analyze how transfer learning from natural image datasets impacts the performance of ViTs in prostate MRI segmentation. We evaluate three ViT architectures, both with and without transfer learning, using the prostate dataset from the Medical Segmentation Decathlon database, with the Dice coefficient as evaluation metric. Through a paired t-test, our analysis reveals that applying transfer learning from natural images does not improve the segmentation performance of the peripheral zone or transition zone of the prostate. This suggests that features learned from natural images are not readily transferable to medical imaging tasks. Moreover, our experiments also indicate that pre-training does not speed optimization convergence during training.