On Training Efficiency and Computational Costs of a Feed Forward Neural Network: A Review

Table 1

Analytic AFs.

Ref.	Name	Expression	Notes
[14, 15]	Step-like
	Linear
	Saturated linear		Where to ensure; continuity is α
	Sigmoid
	Hyp. tangent
[16]	Arctangent
[18]	Quadratic sigmoid function
[19, 22]	Logarithmic-Exponential		is responsible for centering the function, distances the sigmoid, and is a coefficient that controls the slope
[19]	Triangular approximation		Where , , , , , and are −0.54324, −0.16957, 1, 0.42654, 1.52, and 2.57, respectively. and are 0.016 and 0.4519, respectively
[21]	Hermite polyn.	,; where; , , with
[22]	Gaussian
[26]	PolyExp		Where , , and are tunable constants
[24, 26]	Wave
[22, 23]	Neural		Where controls the slope while governs the frequency of the resultant function
[23]	Sinusoidal		Where is a parameter related to the frequency of the sine function
[25]	CosGauss		The parameter in the function scales the gradient of the Gaussian function; meanwhile controls the frequency of repetition of the “hills”
[26]	Sinc
[26]	SinCos		Where and are frequencies and and are proportional parameters