Towards fair decentralized benchmarking of healthcare AI algorithms with the Federated Tumor Segmentation (FeTS) challenge
Zenk, Maximilian; Baid, Ujjwal; Pati, Sarthak; Linardos, Akis; Edwards, Brandon; Sheller, Micah; Foley, Patrick; Aristizabal, Alejandro; Zimmerer, David; Gruzdev, Alexey; Martin, Jason; Shinohara, Russell T; Reinke, Annika; Isensee, Fabian; Parampottupadam, Santhosh; Parekh, Kaushal; Floca, Ralf; Kassem, Hasan; Baheti, Bhakti; Thakur, Siddhesh; Chung, Verena; Kushibar, Kaisar; Lekadir, Karim; Jiang, Meirui; Yin, Youtan; Yang, Hongzheng; Liu, Quande; Chen, Cheng; Dou, Qi; Heng, Pheng-Ann; Zhang, Xiaofan; Zhang, Shaoting; Khan, Muhammad Irfan; Azeem, Mohammad Ayyaz; Jafaritadi, Mojtaba; Alhoniemi, Esa; Kontio, Elina; Khan, Suleiman A; Mächler, Leon; Ezhov, Ivan; Kofler, Florian; Shit, Suprosanna; Paetzold, Johannes C; Loehr, Timo; Wiestler, Benedikt; Peiris, Himashi; Pawar, Kamlesh; Zhong, Shenjun; Chen, Zhaolin; Hayat, Munawar; Egan, Gary; Harandi, Mehrtash; Isik Polat, Ece; Polat, Gorkem; Kocyigit, Altan; Temizel, Alptekin; Tuladhar, Anup; Tyagi, Lakshay; Souza, Raissa; Forkert, Nils D; Mouches, Pauline; Wilms, Matthias; Shambhat, Vishruth; Maurya, Akansh; Danannavar, Shubham Subhas; Kalla, Rohit; Anand, Vikas Kumar; Krishnamurthi, Ganapathy; Nalawade, Sahil; Ganesh, Chandan; Wagner, Ben; Reddy, Divya; Das, Yudhajit; Yu, Fang F; Fei, Baowei; Madhuranthakam, Ananth J; Maldjian, Joseph; Singh, Gaurav; Ren, Jianxun; Zhang, Wei; An, Ning; Hu, Qingyu; Zhang, Youjia; Zhou, Ying; Siomos, Vasilis; Tarroni, Giacomo; Passerrat-Palmbach, Jonathan; Rawat, Ambrish; Zizzo, Giulio; Kadhe, Swanand Ravindra; Epperlein, Jonathan P; Braghin, Stefano; Wang, Yuan; Kanagavelu, Renuga; Wei, Qingsong; Yang, Yechao; Liu, Yong; Kotowski, Krzysztof; Adamski, Szymon; Machura, Bartosz; Malara, Wojciech; Zarudzki, Lukasz; Nalepa, Jakub; Shi, Yaying; Gao, Hongjian; Avestimehr, Salman; Yan, Yonghong; Akbar, Agus S; Kondrateva, Ekaterina; Yang, Hua; Li, Zhaopei; Wu, Hung-Yu; Roth, Johannes; Saueressig, Camillo; Milesi, Alexandre; Nguyen, Quoc D; Gruenhagen, Nathan J; Huang, Tsung-Ming; Ma, Jun; Singh, Har Shwinder H; Pan, Nai-Yu; Zhang, Dingwen; Zeineldin, Ramy A; Futrega, Michal; Yuan, Yading; Conte, Gian Marco; Feng, Xue; Pham, Quan D; Xia, Yong; Jiang, Zhifan; Luu, Huan Minh; Dobko, Mariia; Carré, Alexandre; Tuchinov, Bair; Mohy-Ud-Din, Hassan; Alam, Saruar; Singh, Anup; Shah, Nameeta; Wang, Weichung; Sako, Chiharu; Bilello, Michel; Ghodasara, Satyam; Mohan, Suyash; Davatzikos, Christos; Calabrese, Evan; Rudie, Jeffrey; Villanueva-Meyer, Javier; Cha, Soonmee; Hess, Christopher; Mongan, John; Ingalhalikar, Madhura; Jadhav, Manali; Pandey, Umang; Saini, Jitender; Huang, Raymond Y; Chang, Ken; To, Minh-Son; Bhardwaj, Sargam; Chong, Chee; Agzarian, Marc; Kozubek, Michal; Lux, Filip; Michálek, Jan; Matula, Petr; Ker Kovský, Miloš; Kopr Ivová, Tereza; Dostál, Marek; Vybíhal, Václav; Pinho, Marco C; Holcomb, James; Metz, Marie; Jain, Rajan; Lee, Matthew D; Lui, Yvonne W; Tiwari, Pallavi; Verma, Ruchika; Bareja, Rohan; Yadav, Ipsa; Chen, Jonathan; Kumar, Neeraj; Gusev, Yuriy; Bhuvaneshwar, Krithika; Sayah, Anousheh; Bencheqroun, Camelia; Belouali, Anas; Madhavan, Subha; Colen, Rivka R; Kotrotsou, Aikaterini; Vollmuth, Philipp; Brugnara, Gianluca; Preetha, Chandrakanth J; Sahm, Felix; Bendszus, Martin; Wick, Wolfgang; Mahajan, Abhishek; Balaña, Carmen; Capellades, Jaume; Puig, Josep; Choi, Yoon Seong; Lee, Seung-Koo; Chang, Jong Hee; Ahn, Sung Soo; Shaykh, Hassan F; Herrera-Trujillo, Alejandro; Trujillo, Maria; Escobar, William; Abello, Ana; Bernal, Jose; Gómez, Jhon; LaMontagne, Pamela; Marcus, Daniel S; Milchenko, Mikhail; Nazeri, Arash; Landman, Bennett; Ramadass, Karthik; Xu, Kaiwen; Chotai, Silky; Chambless, Lola B; Mistry, Akshitkumar; Thompson, Reid C; Srinivasan, Ashok; Bapuraj, J Rajiv; Rao, Arvind; Wang, Nicholas; Yoshiaki, Ota; Moritani, Toshio; Turk, Sevcan; Lee, Joonsang; Prabhudesai, Snehal; Garrett, John; Larson, Matthew; Jeraj, Robert; Li, Hongwei; Weiss, Tobias; Weller, Michael; Bink, Andrea; Pouymayou, Bertrand; Sharma, Sonam; Tseng, Tzu-Chi; Adabi, Saba; Xavier Falcão, Alexandre; Martins, Samuel B; Teixeira, Bernardo C A; Sprenger, Flávia; Menotti, David; Lucio, Diego R; Niclou, Simone P; Keunen, Olivier; Hau, Ann-Christin; Pelaez, Enrique; Franco-Maldonado, Heydy; Loayza, Francis; Quevedo, Sebastian; McKinley, Richard; Slotboom, Johannes; Radojewski, Piotr; Meier, Raphael; Wiest, Roland; Trenkler, Johannes; Pichler, Josef; Necker, Georg; Haunschmidt, Andreas; Meckel, Stephan; Guevara, Pamela; Torche, Esteban; Mendoza, Cristobal; Vera, Franco; Ríos, Elvis; López, Eduardo; Velastin, Sergio A; Choi, Joseph; Baek, Stephen; Kim, Yusung; Ismael, Heba; Allen, Bryan; Buatti, John M; Zampakis, Peter; Panagiotopoulos, Vasileios; Tsiganos, Panagiotis; Alexiou, Sotiris; Haliassos, Ilias; Zacharaki, Evangelia I; Moustakas, Konstantinos; Kalogeropoulou, Christina; Kardamakis, Dimitrios M; Luo, Bing; Poisson, Laila M; Wen, Ning; Vallières, Martin; Loutfi, Mahdi Ait Lhaj; Fortin, David; Lepage, Martin; Morón, Fanny; Mandel, Jacob; Shukla, Gaurav; Liem, Spencer; Alexandre, Gregory S; Lombardo, Joseph; Palmer, Joshua D; Flanders, Adam E; Dicker, Adam P; Ogbole, Godwin; Oyekunle, Dotun; Odafe-Oyibotha, Olubunmi; Osobu, Babatunde; Shu'aibu Hikima, Mustapha; Soneye, Mayowa; Dako, Farouk; Dorcas, Adeleye; Murcia, Derrick; Fu, Eric; Haas, Rourke; Thompson, John A; Ormond, David Ryan; Currie, Stuart; Fatania, Kavi; Frood, Russell; Simpson, Amber L; Peoples, Jacob J; Hu, Ricky; Cutler, Danielle; Moraes, Fabio Y; Tran, Anh; Hamghalam, Mohammad; Boss, Michael A; Gimpel, James; Kattil Veettil, Deepak; Schmidt, Kendall; Cimino, Lisa; Price, Cynthia; Bialecki, Brian; Marella, Sailaja; Apgar, Charles; Jakab, Andras; Weber, Marc-André; Colak, Errol; Kleesiek, Jens; Freymann, John B; Kirby, Justin S; Maier-Hein, Lena; Albrecht, Jake; Mattson, Peter; Karargyris, Alexandros; Shah, Prashant; Menze, Bjoern; Maier-Hein, Klaus; Bakas, Spyridon
Computational competitions are the standard for benchmarking medical image analysis algorithms, but they typically use small curated test datasets acquired at a few centers, leaving a gap to the reality of diverse multicentric patient data. To this end, the Federated Tumor Segmentation (FeTS) Challenge represents the paradigm for real-world algorithmic performance evaluation. The FeTS challenge is a competition to benchmark (i) federated learning aggregation algorithms and (ii) state-of-the-art segmentation algorithms, across multiple international sites. Weight aggregation and client selection techniques were compared using a multicentric brain tumor dataset in realistic federated learning simulations, yielding benefits for adaptive weight aggregation, and efficiency gains through client sampling. Quantitative performance evaluation of state-of-the-art segmentation algorithms on data distributed internationally across 32 institutions yielded good generalization on average, albeit the worst-case performance revealed data-specific modes of failure. Similar multi-site setups can help validate the real-world utility of healthcare AI algorithms in the future.
PMCID:12238412
PMID: 40628696
ISSN: 2041-1723
CID: 5890702
Considerations and recommendations from the ISMRM diffusion study group for preclinical diffusion MRI: Part 2-Ex vivo imaging: Added value and acquisition
Schilling, Kurt G; Grussu, Francesco; Ianus, Andrada; Hansen, Brian; Howard, Amy F D; Barrett, Rachel L C; Aggarwal, Manisha; Michielse, Stijn; Nasrallah, Fatima; Syeda, Warda; Wang, Nian; Veraart, Jelle; Roebroeck, Alard; Bagdasarian, Andrew F; Eichner, Cornelius; Sepehrband, Farshid; Zimmermann, Jan; Soustelle, Lucas; Bowman, Christien; Tendler, Benjamin C; Hertanu, Andreea; Jeurissen, Ben; Verhoye, Marleen; Frydman, Lucio; van de Looij, Yohan; Hike, David; Dunn, Jeff F; Miller, Karla; Landman, Bennett A; Shemesh, Noam; Anderson, Adam; McKinnon, Emilie; Farquharson, Shawna; Dell'Acqua, Flavio; Pierpaoli, Carlo; Drobnjak, Ivana; Leemans, Alexander; Harkins, Kevin D; Descoteaux, Maxime; Xu, Duan; Huang, Hao; Santin, Mathieu D; Grant, Samuel C; Obenaus, Andre; Kim, Gene S; Wu, Dan; Le Bihan, Denis; Blackband, Stephen J; Ciobanu, Luisa; Fieremans, Els; Bai, Ruiliang; Leergaard, Trygve B; Zhang, Jiangyang; Dyrby, Tim B; Johnson, G Allan; Cohen-Adad, Julien; Budde, Matthew D; Jelescu, Ileana O
The value of preclinical diffusion MRI (dMRI) is substantial. While dMRI enables in vivo non-invasive characterization of tissue, ex vivo dMRI is increasingly being used to probe tissue microstructure and brain connectivity. Ex vivo dMRI has several experimental advantages including higher SNR and spatial resolution compared to in vivo studies, and enabling more advanced diffusion contrasts for improved microstructure and connectivity characterization. Another major advantage of ex vivo dMRI is the direct comparison with histological data, as a crucial methodological validation. However, there are a number of considerations that must be made when performing ex vivo experiments. The steps from tissue preparation, image acquisition and processing, and interpretation of results are complex, with many decisions that not only differ dramatically from in vivo imaging of small animals, but ultimately affect what questions can be answered using the data. This work represents "Part 2" of a three-part series of recommendations and considerations for preclinical dMRI. We describe best practices for dMRI of ex vivo tissue, with a focus on the value that ex vivo imaging adds to the field of dMRI and considerations in ex vivo image acquisition. We first give general considerations and foundational knowledge that must be considered when designing experiments. We briefly describe differences in specimens and models and discuss why some may be more or less appropriate for different studies. We then give guidelines for ex vivo protocols, including tissue fixation, sample preparation, and MR scanning. In each section, we attempt to provide guidelines and recommendations, but also highlight areas for which no guidelines exist (and why), and where future work should lie. An overarching goal herein is to enhance the rigor and reproducibility of ex vivo dMRI acquisitions and analyses, and thereby advance biomedical knowledge.
PMCID:11971501
PMID: 40035293
ISSN: 1522-2594
CID: 5818552