Med-FastSAM: Improving Transfer Efficiency of SAM to Domain-Generalised Medical Image Segmentation
Medical image segmentation is a crucial computer vision task in medical image analysis. Recently, the Segment Anything Model (SAM) has made significant advancements in natural image segmentation. Despite current studies indicating the potential of SAM to revolutionise medical image segmentation using parameter- efficient fine-tuning techniques, it still faces three primary challenges. Firstly, these methods still rely on the large vision transformer of SAM, which is computationally expensive. Secondly, the point and box prompt modes of SAM demand manual annotations, which are time-consuming and expensive in medical sce- narios and reduce their clinical applicability. Thirdly, SAM leverages large-size patches to predict masks, resulting in the loss of fine-grained details. To address these limitations, in this paper, we propose a fast-transferring architecture for adapting SAM to domain-generalised medical image segmentation, named Med- FastSAM. Specifically, we introduce a lightweight knowledge aggregation encoder that combines the distilled natural image knowledge with learned medical-specific information for producing feature representation. Moreover, we devise a coarse prompt module to automatically generate coarse masks for guiding segmenta- tion decoding. Furthermore, we design a multi-scale feature decoder to produce precise segmentation masks. Eventually, extensive experiments on four bench- mark datasets have been conducted to evaluate the proposed model. The result demonstrates that Med-FastSAM outperforms state-of-the-art methods without any manual prompts. Especially, our model shows excellent zero-shot domain generalisation performance by using only 15.45% parameters compared to the standard SAM. The code for our work and more technical details can be found at https://github.com/GalacticHogrider/Med-FastSAM.
History
School affiliated with
- College of Health and Science (Research Outputs)
Publication Title
Advances in Neural Information Processing Systems 37 (NeurIPS 2024)External DOI
Date Accepted
2024-10-10Event Name
NeurIPS 2024: The Thirty-Eighth Annual Conference on Neural Information Processing SystemsEvent Dates
10 - 15 December 2024Will your conference paper be published in proceedings?
- Yes