On the Brittleness of Handwritten Digit Recognition Models

Table 1

Dataset independence for pixel-based features, each dataset separately.

Classifier	Trained on	Tested on			Avg. error versus own testset
		MNIST	DIGITS	USPS
IBk1 euclidean	MNIST	3.09	19.21	17.49	5.94x
IBk1 euclidean	DIGITS	36.22	16.59	52.72	2.68x
IBk1 euclidean	USPS	28.41	55.01	5.33	7.83x
IBk1 NCC	MNIST	2.83	17.65	13.70	5.54x
IBk1 NCC	DIGITS	32.42	14.14	44.59	2.72x
IBk1 NCC	USPS	26.06	51.17	4.58	8.43x
IBk1 TD	MNIST	1.51	13.53	5.63	6.34x
IBk1 TD	DIGITS	25.88	10.02	37.77	3.18x
IBk1 TD	USPS	10.51	36.47	3.64	6.45x
SVM linear	MNIST	6.83	34.97	16.24	3.75x
SVM linear	DIGITS	31.57	16.09	45.54	2.40x
SVM linear	USPS	40.64	63.25	6.53	7.95x
SVM polynomial	MNIST	1.27	16.20	11.56	10.93x
SVM polynomial	DIGITS	30.05	11.47	47.68	3.39x
SVM polynomial	USPS	44.78	74.33	4.43	13.44x
SVM RBF	MNIST	4.31	53.34	20.78	8.60x
SVM RBF	DIGITS	51.50	33.74	60.09	1.65x
SVM RBF	USPS	81.05	89.98	7.37	11.60x
convNN	MNIST	0.74	8.24	3.48	7.92x
convNN	DIGITS	21.43	5.73	30.0	4.49x
convNN	USPS	4.25	27.56	3.08	5.16x