A Semi-Automated Usability Evaluation Framework for Interactive Image Segmentation Systems

Table 1

Overview of seed point location selection methods for a set of influential publications in the field of interactive image segmentation. Additional unordered seed information can be retrieved in arbitrary order by (a) manually drawn seeds or (b) randomly generated seeds. Seeds can be inferred rule-based from the ground truth segmentation by (c) sampling the binary mask image, (d) from provided bounding box mask images, (e) random sampling from tri-maps generated by erosion and dilation, or (f) by a robot user, i.e. user simulation. A tri-map specifies background, foreground, and mixed areas. Seeds can also be provided by real users via the (g) final seed masks after all interactions on one input image or (h) the ordered iterative scribbles. (i) Questionnaire data from Goals, Operators, Methods, and Selection rules (GO) as well as National Aeronautics and Space Administration Task Load Index (TL) may be retrieved by interviewing users after the segmentation process. Check marks indicate the usage of seeds in the publications listed. Publications with check marks in brackets display these seeds but do not utilize them for evaluation.

Year	Publication	Arbitrary Seeds		Seeds Derived from GT				Multiple User Data based Seeds
		(a)	(b)	(c)	(d)	(e)	(f)	(g)	(h)	(i)
		Manual	Random	Binary Mask	Box	Tri-maps	Robot	Final Seeds	Scribbles	Questionnaire
2019	Amrehn [27]						[9, 28, 29]
2018	Chen [30]	()					[14]		()
	Amrehn [31]
2017	Liew [32]	()	()				[9]
	Wang [33]								()
	Wang [29]						[34]		()
	Amrehn [34]					✓	[29]
	Amrehn [35]
2016	Ramkumar [25]									(GO, TL)
	Ramkumar [26]									(TL)
	Jiang [36]			✓ [37]					()
	Xu [28]	(✓)	(✓)
	Chen [38]
2015	Andrade [39]								✓
	Rupprecht [14]
2014	Bai [40]		✓
2013	Jain [41]
	He [42]
2012	Kohli [9]							()
2011	Zhao [43]
	Top [44]	()							()
	McGuinness [22]	()
2010	Nickisch [17]							()
	Gulshan [45]							()
	Batra [46]
	Ning [47]
	Price [48]				[49]	[50]
	Moschidis [51]
2009	Moschidis [52]
	Singaraju [49]					[50]
2008	Duchenne [53]					[50]
	Levin [54]
	Vicente [55]
2007	Protiere [56]
2006	Boykov [57]
	Grady [58]
2005	Vezhnevets [59]
	Cates,[60]	()							()
2004	Li [61]
	Rother [50]			()	()	✓
	Blake [62]					[37]
2001	Martin [37]					✓