University of Lincoln
Browse

Med-FastSAM: Improving Transfer Efficiency of SAM to Domain-Generalised Medical Image Segmentation

Download (796.95 kB)
Version 2 2025-01-20, 17:25
Version 1 2025-01-17, 12:52
conference contribution
posted on 2025-01-20, 17:25 authored by Yuxiang Luo, qing xu, Jinglei Feng, Guangwu Qian, Wenting DuanWenting Duan

Medical image segmentation is a crucial computer vision task in medical image analysis. Recently, the Segment Anything Model (SAM) has made significant advancements in natural image segmentation. Despite current studies indicating the potential of SAM to revolutionise medical image segmentation using parameter- efficient fine-tuning techniques, it still faces three primary challenges. Firstly, these methods still rely on the large vision transformer of SAM, which is computationally expensive. Secondly, the point and box prompt modes of SAM demand manual annotations, which are time-consuming and expensive in medical sce- narios and reduce their clinical applicability. Thirdly, SAM leverages large-size patches to predict masks, resulting in the loss of fine-grained details. To address these limitations, in this paper, we propose a fast-transferring architecture for adapting SAM to domain-generalised medical image segmentation, named Med- FastSAM. Specifically, we introduce a lightweight knowledge aggregation encoder that combines the distilled natural image knowledge with learned medical-specific information for producing feature representation. Moreover, we devise a coarse prompt module to automatically generate coarse masks for guiding segmenta- tion decoding. Furthermore, we design a multi-scale feature decoder to produce precise segmentation masks. Eventually, extensive experiments on four bench- mark datasets have been conducted to evaluate the proposed model. The result demonstrates that Med-FastSAM outperforms state-of-the-art methods without any manual prompts. Especially, our model shows excellent zero-shot domain generalisation performance by using only 15.45% parameters compared to the standard SAM. The code for our work and more technical details can be found at https://github.com/GalacticHogrider/Med-FastSAM.

History

School affiliated with

  • College of Health and Science (Research Outputs)

Publication Title

Advances in Neural Information Processing Systems 37 (NeurIPS 2024)

Date Accepted

2024-10-10

Event Name

NeurIPS 2024: The Thirty-Eighth Annual Conference on Neural Information Processing Systems

Event Dates

10 - 15 December 2024

Will your conference paper be published in proceedings?

  • Yes

Usage metrics

    University of Lincoln (Research Outputs)

    Licence

    Exports

    RefWorks
    BibTeX
    Ref. manager
    Endnote
    DataCite
    NLM
    DC