Research Article
Pipelined Training with Stale Weights in Deep Convolutional Neural Networks
Table 9
LeNet-5, AlexNet, VGG-16, and ResNet-20 memory increase of 4-stage pipelined training.
| CNN | Dataset | PPV | Activation memory for minibatch size 1 (MB) | Minibatch size | Total weight memory (MB) | Memory increase % of PipeDream | Memory increase % of this work |
| LeNet-5 | MNIST | (2) | 0.06 | 128 | 0.24 | 109 | 97 | AlexNet | CIFAR-10 | (3) | 0.88 | 128 | 88.87 | 214 | 37 | VGG-16 | CIFAR-10 | (2) | 3.30 | 128 | 58.16 | 124 | 75 | ResNet-20 | CIFAR-10 | (7) | 3.84 | 128 | 1.03 | 61 | 60 | VGG-16 | ImageNet | (2) | 218.59 | 32 | 527.79 | 105 | 77 |
|
|