MRBrainS Challenge: Online Evaluation Framework for Brain Image Segmentation in 3T MRI Scans

Table 1

Results of the 11 evaluated algorithms presented at the workshop and the evaluated freeware packages on the 15 test datasets. The algorithms are ranked () based on their overall score () by using (5). This score is based on the ranks of the gray matter (GM), white matter (WM), and cerebrospinal fluid (CSF) segmentation and the three evaluated measures: Dice coefficient ( in %), 95th-percentile Hausdorff distance ( in mm), and the absolute volume difference (AVD in %). The rank denotes the rank based on the mean () over all 15 test datasets for each measure (0: , 1: , and 2: AVD) and component (0: GM, 1: WM, 2: CSF, 3: brain (WM + GM), and 4: intracranial volume (ICV = WM + GM + CSF)). Teams BIGR2 and UofL BioImaging, and FreeSurfer and Jedi Mind Meld have equal scores based on the mean (); therefore the ranking based on the standard deviation () is taken into account to determine the final rank (BIGR2: rank 4, UofL BioImaging: rank 8, FreeSurfer: rank 13, and Jedi Mind Meld: rank 17). Columns 2 and 3 present the average runtime per scan in seconds (s), minutes (m), or hours (h) and the scans (T1: T1-weighted scan, 3D T1: 3D T1-weighted scan, IR: T1-weighted inversion recovery (IR), and F: T1-weighted FLAIR) that are used for processing.

	Team		Scans	GM			WM			CSF				Brain			ICV
						AVD			AVD			AVD				AVD			AVD
				()	()	()	()	()	()	()	()	()		()	()	()	()	()	()
				1	BIGR2	35 m	T1, IR, F	1	2	4	2	2		4	4	5	14	38	1	2	12	8	4	12
84.7	1.9	6.1	88.4					2.4	6.0	78.3	3.2	23	95.1	2.7	3.2	96.0	3.9		5.2
(1.3)	(0.4)	(3.3)	(1.2)					(0.5)	(5.1)	(5.0)	(0.8)	(17)	(0.5)	(0.8)	(1.6)	(1.3)	(1.1)		(3.0)
2	UofL BioImaging	6 s	T1	5	1	9	4	1	13	2	2	1	38	2	1	12	5	2	5
				83.0	1.7	8.6	87.9	2.2	8.7	78.9	2.7	9.7		94.9	2.4	3.9	96.7	3.4	1.8
				(1.5)	(0.3)	(5.4)	(2.0)	(0.6)	(6.6)	(4.2)	(0.5)	(10)		(0.6)	(0.5)	(2.0)	(0.8)	(0.6)	(2.0)
3	CMIV	3 m	T1, F	6	7	5	5	3	10	3	3	8	50	5	7	2	4	3	11
				82.4	2.7	6.8	87.7	2.4	7.3	78.6	3.0	14		94.5	3.8	2.6	96.8	3.8	4.9
				(1.4)	(0.4)	(4.0)	(1.6)	(0.4)	(3.8)	(3.1)	(0.4)	(5.9)		(0.5)	(1.1)	(2.2)	(0.8)	(1.3)	(2.3)
4	UB VPML Med	30 m	T1, IR, F	4	4	2	1	5	7	8	13	17	61	4	4	1	10	11	15
				83.3	2.1	5.9	88.6	2.7	7.1	74.8	4.3	31		94.6	2.8	2.4	94.8	6.6	7.7
				(1.3)	(0.3)	(5.3)	(1.7)	(0.4)	(3.8)	(7.1)	(1.7)	(19)		(0.6)	(0.4)	(1.8)	(2.0)	(2.0)	(4.1)
5	Bigr_neuro	2 h	T1, F	7	13	3	6	6	9	6	4	10	64	7	10	10	7	5	8
				81.5	3.7	5.9	87.3	3.0	7.3	78.2	3.2	16		94.0	4.6	3.6	96.3	3.9	3.5
				(1.7)	(0.9)	(4.2)	(1.4)	(0.4)	(3.8)	(4.7)	(0.6)	(14)		(0.8)	(1.4)	(2.4)	(1.2)	(0.9)	(2.7)
6	Robarts	16 m	3D T1, IR	11	3	15	8	8	6	1	1	13	66	16	3	18	1	1	1
				79.7	2.0	9.8	86.2	3.1	7.1	80.3	2.7	20		93.1	2.8	7.9	97.9	2.6	0.9
				(2.4)	(0.1)	(7.3)	(1.3)	(0.4)	(6.2)	(4.1)	(0.5)	(13)		(1.6)	(0.5)	(3.6)	(0.3)	(0.4)	(0.7)
7	Narsil	2 m	T1, F	3	5	1	7	11	2	17	18	7	71	3	5	3	16	18	9
				83.5	2.3	5.5	87.1	3.3	5.8	66.6	13.3	14		94.8	2.9	2.9	92.5	24	3.7
				(1.8)	(0.4)	(4.4)	(1.3)	(0.9)	(5.3)	(2.4)	(5.4)	(9.5)		(0.5)	(0.5)	(2.0)	(0.5)	(8.9)	(1.7)
8	SPM_T1_F	3 m	T1, F	8	9	16	9	7	1	9	14	2	75	9	14	14	6	14	4
				81.2	2.9	10	86.0	3.0	5.2	74.1	4.6	10		93.9	5.8	5.3	96.6	8.2	1.5
				(2.2)	(0.3)	(8.5)	(1.5)	(0.1)	(3.8)	(3.4)	(0.6)	(4.7)		(1.0)	(2.2)	(3.8)	(0.2)	(3.0)	(1.0)
9	SPM_T1_IR	3 m	T1, IR	12	11	7	16	12	5	5	10	3	81	8	11	7	2	8	2
				79.4	3.0	7.2	83.5	3.6	6.3	78.3	4.0	10		93.9	4.6	3.4	97.7	6.5	1.0
				(2.1)	(0.4)	(6.3)	(2.1)	(0.3)	(4.6)	(3.8)	(0.6)	(5.7)		(0.8)	(1.2)	(2.8)	(0.2)	(1.3)	(0.8)
10	MNAB	15 m	T1, IR, F	2	8	12	3	4	11	15	15	16	86	6	9	11	15	12	17
				83.9	2.8	9.1	88.0	2.7	7.8	68.1	4.9	29		94.5	4.5	3.8	92.5	7.1	9.7
				(2.1)	(0.9)	(6.5)	(1.2)	(0.8)	(4.0)	(4.0)	(2.2)	(21)		(1.0)	(2.0)	(3.2)	(1.1)	(4.2)	(4.7)
11	SPM_T1	3 m	T1	9	10	6	11	10	3	11	16	15	91	10	8	6	9	13	13
				80.3	3.0	6.9	85.6	3.1	6.0	70.7	5.3	23		93.9	4.4	3.2	95.3	8.1	5.5
				(2.4)	(0.5)	(6.8)	(1.7)	(0.1)	(4.1)	(3.8)	(1.5)	(15.7)		(0.9)	(1.6)	(2.9)	(0.9)	(3.7)	(3.7)
12	FSL_Seg	10 m	T1	13	16	10	10	13	14	12	6	5	99	13	13	4	12	6	6
				78.7	4.3	8.6	86.0	3.7	11.5	69.9	3.4	12		93.3	5.5	3.0	94.2	5.3	3.4
				(2.2)	(1.2)	(6.3)	(2.6)	(0.8)	(6.3)	(2.8)	(0.2)	(10.3)		(0.8)	(1.4)	(1.5)	(0.8)	(1.1)	(1.5)
13	SPM_T1_IR_F	4 m	T1, IR, F	10	12	18	15	14	12	7	11	6	105	12	15	13	3	15	3
				80.1	3.0	13.9	83.6	3.8	8.4	76.9	4.1	12		93.6	5.9	5.1	97.7	8.2	1.2
				(2.4)	(0.2)	(9.6)	(2.1)	(0.5)	(5.2)	(3.1)	(0.5)	(6.0)		(1.1)	(1.8)	(3.6)	(0.2)	(1.8)	(0.9)
14	FSL _PVSeg	10 m	T1	15	15	8	13	15	17	13	7	4	107	11	16	9	13	7	7
				77.7	4.3	8.4	84.8	3.8	19.7	69.5	3.4	11		93.6	6.1	3.5	94.2	5.3	3.4
				(2.6)	(1.3)	(6.5)	(3.2)	(0.9)	(10)	(2.2)	(0.3)	(5.8)		(0.9)	(1.6)	(3.0)	(0.8)	(1.1)	(1.5)
15	FreeSurfer	1 h	3D T1	16	6	17	12	9	8	18	12	18	116	17	6	16	17	10	18
				77.4	2.3	12.1	85.2	3.1	7.2	65.8	4.3	50		92.3	3.6	6.3	92.4	6.6	10
				(2.0)	(0.6)	(6.0)	(2.2)	(0.5)	(4.6)	(3.7)	(0.6)	(19.6)		(0.8)	(0.6)	(3.4)	(0.9)	(0.4)	(4.5)
16	Jedi Mind Meld	27 s	T1, IR, F	14	14	11	17	16	15	10	8	11	116	15	12	8	11	16	14
				77.8	3.9	8.9	80.6	4.5	13.7	73.3	3.7	18		93.2	5.4	3.5	94.3	20	6.8
				(5.9)	(2.8)	(7.6)	(9.6)	(3.6)	(12)	(3.5)	(0.8)	(9.0)		(1.7)	(4.8)	(3.0)	(1.7)	(3.7)	(3.1)
17	S2_QM	1.5 h	T1, IR, F	17	17	13	14	17	18	16	9	9	130	14	17	15	14	9	10
				76.4	5.5	9.3	83.9	4.9	24.8	67.9	3.8	14		93.3	7.0	5.5	93.7	6.5	4.0
				(3.4)	(3.0)	(6.4)	(3.4)	(3.2)	(10)	(2.3)	(0.6)	(9.9)		(1.4)	(3.5)	(4.5)	(1.1)	(1.4)	(2.2)
18	LNMBrains	5 m	T1, IR, F	18	18	14	18	18	16	14	17	12	145	18	18	17	18	17	16
				72.8	6.8	9.5	78.3	6.8	15.3	68.8	7.7	20		88.5	8.6	7.4	89.2	20.4	7.9
				(5.3)	(2.3)	(7.6)	(6.4)	(3.5)	(13)	(6.6)	(2.4)	(13)		(4.5)	(2.8)	(7.7)	(4.8)	(3.7)	(8.3)

Semiautomatic due to per scan parameter tuning.