Impact of hybrid supervision approaches on the performance of artificial intelligence for the classification of chest radiographs.

Abstract

To evaluate the impact of different supervision regimens on the training of artificial intelligence (AI) in the classification of chest radiographs as normal or abnormal in a moderately sized cohort of individuals more likely to be outpatients.In a retrospective study, 7000 consecutive two-view chest radiographs obtained from 2012 to 2015 were labeled as normal or abnormal based on clinical reports. A convolutional neural network (CNN) was trained on this dataset and then evaluated with an unseen subset of 500 radiographs. Five different training approaches were tested: (1) weak supervision and four hybrid approaches combining weak supervision and extra supervision with annotation in (2) an unbalanced set of normal and abnormal cases, (3) a set of only abnormal cases, (4) a set of only normal cases, and (5) a balanced set of normal and abnormal cases. Standard binary classification metrics were assessed.The weakly supervised model achieved an accuracy of 82%, but yielded 75 false negative cases, at a sensitivity of 70.0% and a negative predictive value (NPV) of 75.5%. Extra supervision increased NPV at the expense of the false positive rate and overall accuracy. Extra supervision with training using a balance of abnormal and normal radiographs resulted in the greatest increase in NPV (87.2%), improved sensitivity (92.8%), and reduced the number of false negatives by more than fourfold (18 compared to 75 cases).Extra supervision using a balance of annotated normal and abnormal cases applied to a weakly supervised model can minimize the number of false negative cases when classifying two-view chest radiographs. Further refinement of such hybrid training approaches for AI is warranted to refine models for practical clinical applications.