Research on Sparsity of Output Synapses in Echo State Networks

Figure 1

Schemas of the traditional ESN approach and our improved ESN approach. (a) The traditional ESN approach. Input signals , internal neuron activation signals , input weight matrix , feedback matrix , and internal connectivity matrix . , , and are equipped with fixed synaptic connections (solid line). The values of , , and remain constants (sometimes is 0). The output signal is created from through the trained connections . is adjusted by linear regression (bule dotted line). is the teacher sequence observed from the target system. Many applications of ESN aim at minimizing the error . (b) Our improved ESN approach. Fully connected output synapses are only part of the network that needed to be adjusted by the linear regression (bule dotted line). ESN of high energy efficiency with sparse connections matrix has the same reservoir structure as that in (a). Input neurons can be removed from the network. Redundant output connections (red dashed line with the mark “X”) can be removed which will not affect the predictive performance, but will affect the energy consumption of the whole network.

(a)

(b)