MULTIMODAL AI integrating skin images with structured clinical information improved skin lesion classification performance across three datasets, achieving balanced accuracy of up to 99.6% in an institutional dataset. The findings highlight the potential of combining visual and patient-specific information to strengthen AI-assisted dermatological assessment.
The retrospective diagnostic study developed and evaluated a multimodal deep learning model designed to integrate skin images with structured clinical metadata. Researchers aimed to address limitations in existing approaches, which can rely on computationally intensive fusion mechanisms or relatively simple integration strategies that may not adequately capture relationships between imaging and clinical information.
Model performance was assessed separately across two publicly available datasets and one institutional dataset. These included PAD-UFES-20, comprising 2,298 clinical images with 21 structured clinical variables across six lesion classes, and ISIC 2019, which contained 25,331 dermoscopic images spanning eight diagnostic categories.
The institutional PKU-Skin dataset included 800 clinical images from 643 patients, alongside corresponding clinical metadata. This dataset focused on binary classification of psoriasis and basal cell carcinoma.
Balanced accuracy was the primary evaluation metric, with sensitivity, specificity, precision, F1 score, and overall accuracy assessed as secondary measures.
On PAD-UFES-20, the proposed multimodal model achieved a balanced accuracy of 85.6%, outperforming a previous multimodal baseline model, which achieved 75.1%. Sensitivity was 76.1%, specificity was 95.2%, and the F1 score was 78.1%, with overall accuracy reaching 78.3%.
Performance was similarly promising on ISIC 2019, where the model achieved 86.0% balanced accuracy, an F1 score of 84.3%, and overall accuracy of 84.5%.
The highest results were observed in the PKU-Skin dataset, with balanced accuracy of 99.6% and overall accuracy of 99.2%. However, this dataset represented a separate binary classification task. Consequently, these findings provide supportive evidence from an institutional setting rather than formal external validation of a single fixed model.
The findings suggest that integrating structured clinical metadata with skin images may improve the performance of AI-based skin lesion classification compared with comparator models. Incorporating patient-specific clinical information alongside visual characteristics could provide a more comprehensive approach to automated dermatological assessment.
For healthcare professionals, multimodal AI offers potential as a decision-support tool, particularly where clinical information and lesion appearance can be assessed together.
Reference
Wang R et al. Multimodal deep learning with clinical metadata for skin lesion classification. Sci Rep. 2026; DOI:https://doi.org/10.1038/s41598-026-78124-x.
Featured image: Kanishka on Adobe Stock.
Each article is made available under the terms of the Creative Commons Attribution-Non Commercial 4.0 License.
