Deep Neural Networks with Multistate Activation Functions

<div>A neural network. In this model, <svg height="11.8174pt" id="M51" style="vertical-align:-3.1815pt" version="1.1" viewbox="-0.0498162 -8.6359 8.54313 11.8174" width="8.54313pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g transform="matrix(.013,0,0,-0.013,0,0)"><path d="M244 607C244 633 228 655 200 655C166 655 146 618 146 594C146 564 166 546 191 546C221 546 244 574 244 607ZM222 91L209 114C184 94 148 66 133 66C127 66 124 73 130 96L201 370C213 416 211 448 191 448C162 448 88 407 29 352L42 328C73 354 104 371 114 371C120 371 119 365 115 345L53 92C32 5 45 -12 68 -12C103 -12 186 50 222 91Z" id="g113-106"></path><glyph.data ascent="3473" descent="-2876" horiz-adv-x="273" vert-adv-y="273"></glyph.data></g><g transform="matrix(.0091,0,0,-0.0091,3.484,3.132)"><path d="M389 0V32C297 38 291 46 291 118V635C234 613 175 595 109 583V556L161 554C203 552 207 547 207 497V118C207 46 201 38 110 32V0H389Z" id="g50-50"></path><glyph.data ascent="3443" descent="-2856" horiz-adv-x="487" vert-adv-y="487"></glyph.data></g></svg> and <svg height="11.8174pt" id="M52" style="vertical-align:-3.1815pt" version="1.1" viewbox="-0.0498162 -8.6359 8.54313 11.8174" width="8.54313pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g transform="matrix(.013,0,0,-0.013,0,0)"><path d="M244 607C244 633 228 655 200 655C166 655 146 618 146 594C146 564 166 546 191 546C221 546 244 574 244 607ZM222 91L209 114C184 94 148 66 133 66C127 66 124 73 130 96L201 370C213 416 211 448 191 448C162 448 88 407 29 352L42 328C73 354 104 371 114 371C120 371 119 365 115 345L53 92C32 5 45 -12 68 -12C103 -12 186 50 222 91Z" id="g113-106"></path><glyph.data ascent="3473" descent="-2876" horiz-adv-x="273" vert-adv-y="273"></glyph.data></g><g transform="matrix(.0091,0,0,-0.0091,3.484,3.132)"><path d="M414 144C384 79 371 75 317 75H135L276 221C367 316 408 376 408 465C408 570 327 635 237 635C179 635 131 609 100 575L42 494L67 471C94 510 138 565 205 565C277 565 321 517 321 435C321 348 258 270 195 195C146 137 88 81 33 26V0H411C423 44 433 88 446 135L414 144Z" id="g50-51"></path><glyph.data ascent="3443" descent="-2856" horiz-adv-x="487" vert-adv-y="487"></glyph.data></g></svg> are input units; <svg height="12.4698pt" id="M53" style="vertical-align:-3.18147pt" version="1.1" viewbox="-0.0498162 -9.28833 11.714 12.4698" width="11.714pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g transform="matrix(.013,0,0,-0.013,0,0)"><path d="M499 88L487 113C459 86 422 65 413 65C403 65 403 74 409 98L461 318C487 426 460 448 431 448C408 448 383 441 352 424C305 398 223 344 157 257H155L243 674C249 702 249 712 240 712C227 712 170 680 87 673L82 648H117C155 648 162 644 150 590L23 -4L30 -12C55 -4 81 3 105 8C113 53 131 150 143 187C199 281 323 387 372 387C390 387 393 369 380 311L328 86C313 19 319 -12 347 -12C379 -12 442 24 499 88Z" id="g113-105"></path><glyph.data ascent="3473" descent="-2876" horiz-adv-x="516" vert-adv-y="516"></glyph.data></g><g transform="matrix(.0091,0,0,-0.0091,6.643,3.132)"><path d="M389 0V32C297 38 291 46 291 118V635C234 613 175 595 109 583V556L161 554C203 552 207 547 207 497V118C207 46 201 38 110 32V0H389Z" id="g50-50"></path><glyph.data ascent="3443" descent="-2856" horiz-adv-x="487" vert-adv-y="487"></glyph.data></g></svg> and <svg height="12.4698pt" id="M54" style="vertical-align:-3.18147pt" version="1.1" viewbox="-0.0498162 -9.28833 11.714 12.4698" width="11.714pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g transform="matrix(.013,0,0,-0.013,0,0)"><path d="M499 88L487 113C459 86 422 65 413 65C403 65 403 74 409 98L461 318C487 426 460 448 431 448C408 448 383 441 352 424C305 398 223 344 157 257H155L243 674C249 702 249 712 240 712C227 712 170 680 87 673L82 648H117C155 648 162 644 150 590L23 -4L30 -12C55 -4 81 3 105 8C113 53 131 150 143 187C199 281 323 387 372 387C390 387 393 369 380 311L328 86C313 19 319 -12 347 -12C379 -12 442 24 499 88Z" id="g113-105"></path><glyph.data ascent="3473" descent="-2876" horiz-adv-x="516" vert-adv-y="516"></glyph.data></g><g transform="matrix(.0091,0,0,-0.0091,6.643,3.132)"><path d="M414 144C384 79 371 75 317 75H135L276 221C367 316 408 376 408 465C408 570 327 635 237 635C179 635 131 609 100 575L42 494L67 471C94 510 138 565 205 565C277 565 321 517 321 435C321 348 258 270 195 195C146 137 88 81 33 26V0H411C423 44 433 88 446 135L414 144Z" id="g50-51"></path><glyph.data ascent="3443" descent="-2856" horiz-adv-x="487" vert-adv-y="487"></glyph.data></g></svg> are hidden units; <svg height="6.1673pt" id="M55" style="vertical-align:-0.2063904pt" version="1.1" viewbox="-0.0498162 -5.96091 6.20645 6.1673" width="6.20645pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g transform="matrix(.013,0,0,-0.013,0,0)"><path d="M445 282C445 390 384 448 297 448C266 448 231 437 189 418C96 376 23 279 23 151C23 65 66 -12 169 -12C233 -12 294 18 337 53C411 113 445 201 445 282ZM355 280C355 161 302 62 243 36C231 31 219 30 207 30C147 30 111 78 111 154C111 302 180 380 220 399C231 404 246 407 264 407C311 407 355 367 355 280Z" id="g113-112"></path><glyph.data ascent="3473" descent="-2876" horiz-adv-x="468" vert-adv-y="468"></glyph.data></g></svg> is an output unit; and <svg height="9.25202pt" id="M56" style="vertical-align:-3.29111pt" version="1.1" viewbox="-0.0498162 -5.96091 12.1198 9.25202" width="12.1198pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g transform="matrix(.013,0,0,-0.013,0,0)"><path d="M689 332C689 394 670 448 646 448C620 448 597 421 597 396C597 386 600 381 608 372C619 359 620 334 620 315C620 150 538 45 454 45C414 45 386 67 386 122C386 138 388 158 394 180L457 426L452 432L377 416L315 156C302 100 259 45 216 45C176 45 148 67 148 122C148 133 152 158 156 180C162 212 173 259 194 332C201 357 206 384 206 405C206 430 198 448 174 448C125 448 66 406 23 342L43 319C84 368 110 383 121 383C126 383 128 382 128 377C128 370 127 359 122 343C99 268 84 204 77 156C74 137 70 111 70 104C70 25 125 -12 180 -12C228 -12 276 12 319 50C338 8 378 -12 418 -12C549 -12 689 166 689 332Z" id="g113-120"></path><glyph.data ascent="3473" descent="-2876" horiz-adv-x="712" vert-adv-y="712"></glyph.data></g><g transform="matrix(.0091,0,0,-0.0091,8.931,3.132)"><path d="M250 606C250 634 233 656 203 656C168 656 146 618 146 593C146 564 169 545 192 545C227 545 250 573 250 606ZM227 95L212 119C187 98 152 71 135 71C129 71 128 78 134 102L207 373C219 418 217 451 194 451C165 451 92 411 30 351L44 326C77 353 106 371 114 371C124 371 121 357 117 341L55 97C32 5 46 -12 70 -12C108 -12 191 51 227 95Z" id="g50-106"></path><glyph.data ascent="3443" descent="-2856" horiz-adv-x="280" vert-adv-y="280"></glyph.data></g></svg> and <svg height="12.5794pt" id="M57" style="vertical-align:-3.29107pt" version="1.1" viewbox="-0.0498162 -9.28833 8.16602 12.5794" width="8.16602pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g transform="matrix(.013,0,0,-0.013,0,0)"><path d="M452 333C452 398 425 448 375 448C329 448 235 404 140 292H138L229 683C233 701 234 712 225 712C208 712 149 686 66 677L64 652H97C135 652 140 651 129 598L38 167C23 96 29 57 44 33C60 7 98 -12 132 -12C169 -12 232 3 290 41C383 102 452 223 452 333ZM365 316C365 199 308 87 239 49C221 39 200 35 183 35C137 35 99 70 112 163C116 191 123 215 129 233C190 316 290 392 330 392C348 392 365 372 365 316Z" id="g113-99"></path><glyph.data ascent="3473" descent="-2876" horiz-adv-x="469" vert-adv-y="469"></glyph.data></g><g transform="matrix(.0091,0,0,-0.0091,4.992,3.132)"><path d="M250 606C250 634 233 656 203 656C168 656 146 618 146 593C146 564 169 545 192 545C227 545 250 573 250 606ZM227 95L212 119C187 98 152 71 135 71C129 71 128 78 134 102L207 373C219 418 217 451 194 451C165 451 92 411 30 351L44 326C77 353 106 371 114 371C124 371 121 357 117 341L55 97C32 5 46 -12 70 -12C108 -12 191 51 227 95Z" id="g50-106"></path><glyph.data ascent="3443" descent="-2856" horiz-adv-x="280" vert-adv-y="280"></glyph.data></g></svg> are weights and bias, respectively.</div>

Computational Intelligence and Neuroscience

Deep Neural Networks with Multistate Activation Functions

Figure 6