A novel switching of artificial intelligence to generate simultaneously multimodal images to assess inflammation and predict outcomes in Ulcerative Colitis – (with video)

Objectives: Virtual Chromoendoscopy (VCE) is pivotal for assessing activity and predicting outcomes in Ulcerative Colitis (UC), though inter- and intra-observer variability and need for expertise persist. Artificial intelligence (AI) offers standardised VCE-based assessment. This study introduces a novel AI model to detect and simultaneously generate various endoscopic modalities, enhancing AI-driven inflammation assessment and outcome prediction in UC.

Methods: Endoscopic videos in high-definition white-light, iScan2, iScan3 and NBI modalities from UC patients of the international PICaSSO iScan and NBI cohort (302 and 54 patients, respectively) were used to develop a neural network (NN) able to identify the acquisition modality of each frame and for inter-modality image switching. 2535 frames from 169 videos of the iScan cohort were switched to different modalities and used to train a deep-learning model for inflammation assessment. Subsequently, the model was tested on a subset of the iScan and NBI cohort (72 and 51 videos, respectively). Its performance in predicting endoscopic and histological activity and outcomes was evaluated.

Results: The model efficiently classified and converted images across modalities (92% NN classifier accuracy). It showed excellent performance in predicting endoscopic and histological remission, especially when different modalities were combined in both iScan (accuracy 81.3% and 89.6%; AUROC 0.92 and 0.89 by UCEIS and PICaSSO, respectively) and NBI cohort. Moreover, it showed remarkable ability in predicting clinical outcomes in both cohorts.
Conclusions: Our multimodal "AI-switching" model innovatively detects and transitions between different endoscopic enhancement modalities, refining inflammation assessment and outcome pre-diction in UC by integrating model-derived images.