Faculty Bibliography

OBJECTIVE: To measure agreement of plus disease diagnosis among retinopathy of prematurity (ROP) experts. METHODS: A set of 34 wide-angle retinal photographs from infants with ROP was compiled on a secure Web site and was interpreted independently by 22 recognized ROP experts. Diagnostic agreement was analyzed using 3-level (plus, pre-plus, or neither) and 2-level (plus or not plus) categorizations. RESULTS: In the 3-level categorization, all experts agreed on the same diagnosis in 4 of 34 images (12%), and the mean weighted kappa statistic for each expert compared with all others was between 0.21 and 0.40 (fair agreement) for 7 experts (32%) and between 0.41 and 0.60 (moderate agreement) for 15 experts (68%). In the 2-level categorization, all experts who provided a diagnosis agreed in 7 of 34 images (21%), and the mean kappa statistic for each expert compared with all others was between 0 and 0.20 (slight agreement) for 1 expert (5%), between 0.21 and 0.40 (fair agreement) for 3 experts (14%), between 0.41 and 0.60 (moderate agreement) for 12 experts (55%), and between 0.61 and 0.80 (substantial agreement) for 6 experts (27%). CONCLUSIONS: Interexpert agreement of plus disease diagnosis is imperfect. This may have important implications for clinical ROP management, continued refinement of the international ROP classification system, development of computer-based diagnostic algorithms, and implementation of ROP telemedicine systems.

PURPOSE: To measure agreement and accuracy of plus disease diagnosis among retinopathy of prematurity (ROP) experts; and to compare expert performance to that of a computer-based analysis system, Retinal Image multiScale Analysis. METHODS: Twenty-two recognized ROP experts independently interpreted a set of 34 wide-angle retinal photographs for presence of plus disease. Diagnostic agreement was analyzed. A reference standard was defined based on majority vote of experts. Images were analyzed using individual and linear combinations of computer-based system parameters for arterioles and venules: integrated curvature (IC), diameter, and tortuosity index (TI). Sensitivity, specificity, and receiver operating characteristic areas under the curve (AUC) for plus disease diagnosis were determined for each expert and for the computer-based system. RESULTS: Mean kappa statistic for each expert compared to all others was between 0 and 0.20 (slight agreement) in 1 expert (4.5%), 0.21 and 0.40 (fair agreement) in 3 experts (13.6%), 0.41 and 0.60 (moderate agreement) in 12 experts (54.5%), and 0.61 and 0.80 (substantial agreement) in 6 experts (27.3%). For the 22 experts, sensitivity compared to the reference standard ranged from 0.308 to 1.000, specificity from 0.571 to 1.000, and AUC from 0.784 to 1.000. Among individual computer system parameters compared to the reference standard, venular IC had highest AUC (0.853). Among linear combinations of parameters, the combination of arteriolar IC, arteriolar TI, venular IC, venular diameter, and venular TI had highest AUC (0.967). CONCLUSION: Agreement and accuracy of plus disease diagnosis among ROP experts are imperfect. A computer-based system has potential to perform with comparable or better accuracy than human experts, but further validation is required.

PURPOSE: To evaluate a semiautomated image analysis software package, Retinal Image multiScale Analysis (RISA), for the diagnosis of plus disease in preterm infants with retinopathy of prematurity (ROP). METHODS: Digital images of the posterior pole showing both disc and macula in preterm infants with ROP were analyzed with an enhanced version of RISA. Venules (N = 106) and arterioles (N = 44) were identified, and integrated curvature, diameter, and tortuosity of the vessels were calculated. After the RISA calculations were completed, the origins of the vessels were determined to be 32 eyes in 16 infants (12 eyes with plus disease, 20 with no plus disease, as diagnosed by ophthalmic examination). Vessels were sorted into two groups-plus disease and no plus disease-and each RISA parameter was compared using the Mann-Whitney test. For each parameter, sensitivity and specificity were plotted as a function of cutoff criterion, receiver operating characteristic (ROC) curves were constructed, and the areas under the curve (AUC) were calculated. RESULTS: For both arterioles and venules, each of the three parameters was significantly larger for the plus disease group. For instance, the median estimated arteriolar and venular diameters were approximately 12 mum greater in plus disease. Sensitivity and specificity plots indicated good accuracy of each parameter for the diagnosis of plus disease. The AUC showed that curvature had the highest diagnostic accuracy (0.911 for arterioles, 0.824 for venules). CONCLUSIONS: The strong performance of RISA parameters in this sample suggests that RISA may be useful for diagnosing plus disease in preterm infants with ROP.