Predicting breast cancer types on and beyond molecular level in a multi-modal fashion

Tianyu Zhang, Tao Tan*, Luyi Han, Linda Appelman, Jeroen Veltman, Ronni Wessels, Katya M. Duvivier, Claudette Loo, Yuan Gao, Xin Wang, Hugo M. Horlings, Regina G.H. Beets-Tan, Ritse M. Mann

*Corresponding author for this work

Research output: Contribution to journalArticleAcademicpeer-review

3 Citations (Scopus)
5 Downloads (Pure)


Accurately determining the molecular subtypes of breast cancer is important for the prognosis of breast cancer patients and can guide treatment selection. In this study, we develop a deep learning-based model for predicting the molecular subtypes of breast cancer directly from the diagnostic mammography and ultrasound images. Multi-modal deep learning with intra- and inter-modality attention modules (MDL-IIA) is proposed to extract important relations between mammography and ultrasound for this task. MDL-IIA leads to the best diagnostic performance compared to other cohort models in predicting 4-category molecular subtypes with Matthews correlation coefficient (MCC) of 0.837 (95% confidence interval [CI]: 0.803, 0.870). The MDL-IIA model can also discriminate between Luminal and Non-Luminal disease with an area under the receiver operating characteristic curve of 0.929 (95% CI: 0.903, 0.951). These results significantly outperform clinicians’ predictions based on radiographic imaging. Beyond molecular-level test, based on gene-level ground truth, our method can bypass the inherent uncertainty from immunohistochemistry test. This work thus provides a noninvasive method to predict the molecular subtypes of breast cancer, potentially guiding treatment selection for breast cancer patients and providing decision support for clinicians.

Original languageEnglish
Article number16
Journalnpj Breast Cancer
Issue number1
Early online date22 Mar 2023
Publication statusE-pub ahead of print/First online - 22 Mar 2023

Cite this