Pipelined Training with Stale Weights in Deep Convolutional Neural Networks

Table 9

LeNet-5, AlexNet, VGG-16, and ResNet-20 memory increase of 4-stage pipelined training.

CNN	Dataset	PPV	Activation memory for minibatch size 1 (MB)	Minibatch size	Total weight memory (MB)	Memory increase % of PipeDream	Memory increase % of this work
LeNet-5	MNIST	(2)	0.06	128	0.24	109	97
AlexNet	CIFAR-10	(3)	0.88	128	88.87	214	37
VGG-16	CIFAR-10	(2)	3.30	128	58.16	124	75
ResNet-20	CIFAR-10	(7)	3.84	128	1.03	61	60
VGG-16	ImageNet	(2)	218.59	32	527.79	105	77