On the Brittleness of Handwritten Digit Recognition Models

Table 3

Dataset independence for pixel-based features, two datasets combined.

Classifier	Trained on	Tested on			Error versus avg. of own testsets
		MNIST	DIGITS	USPS
IBk1 eucl.	MNIST-DIGITS	9.23	18.32	25.56	1.86x
IBk1 eucl.	MNIST-USPS	5.97	25.89	5.28	4.60x
IBk1 eucl.	USPS-DIGITS	22.27	22.05	6.98	1.53x
IBk1 NCC	MNIST-DIGITS	7.81	13.64	21.67	2.02x
IBk1 NCC	MNIST-USPS	4.74	24.39	4.53	5.26x
IBk1 NCC	USPS-DIGITS	18.95	15.53	6.98	1.68x
IBk1 TD	MNIST-DIGITS	4.34	11.41	10.81	1.37x
IBk1 TD	MNIST-USPS	2.65	16.09	3.64	5.12x
IBk1 TD	USPS-DIGITS	9.61	14.09	4.53	1.03x
SVM linear	MNIST-DIGITS	10.62	21.10	21.92	1.38x
SVM linear	MNIST-USPS	12.27	43.10	8.27	4.20x
SVM linear	USPS-DIGITS	20.37	23.83	9.67	1.22x
SVM poly.	MNIST-DIGITS	4.96	8.85	16.54	2.40x
SVM poly.	MNIST-USPS	2.66	22.16	3.89	6.77x
SVM poly.	USPS-DIGITS	14.56	9.97	5.23	1.92x
SVM RBF	MNIST-DIGITS	12.60	34.58	31.19	1.32x
SVM RBF	MNIST-USPS	13.60	71.66	5.63	7.45x
SVM RBF	USPS-DIGITS	39.89	47.38	7.03	1.47x
convNN	MNIST-DIGITS	3.21	4.00	6.57	1.82x
convNN	MNIST-USPS	1.25	11.85	2.74	5.94x
convNN	USPS-DIGITS	7.03	5.79	4.88	1.32x