Deep Neural Networks with Multistate Activation Functions

<div>Restricting running averages to somewhere near 0. The hollow points stand for running averages of the conventional SGD, while the solid points stand for running averages of mean-normalised SGD.</div>

Computational Intelligence and Neuroscience

fig4

Figure 4

Figure 4: Deep Neural Networks with Multistate Activation Functions 