High Performance Implementation of 3D Convolutional Neural Networks on a GPU

<div>For an input matrix, its size is <span class="nowrap"><svg height="11.5564pt" id="M133" style="vertical-align:-2.26807pt" version="1.1" viewbox="-0.0498162 -9.28833 100.731 11.5564" width="100.731pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g transform="matrix(.013,0,0,-0.013,0,0)"><path d="M300 -147C201 -63 143 98 143 270S200 602 300 686L282 710C136 610 70 450 70 271V270C70 89 136 -72 282 -170L300 -147Z" id="g113-41"></path></g><g transform="matrix(.013,0,0,-0.013,4.498,0)"><path d="M962 650H795L470 145L347 650H176L170 622C268 613 275 606 244 503L190 322C150 188 132 126 118 91C102 50 80 33 18 28L12 0H245L251 28C175 35 170 48 174 93C177 128 191 180 220 284L292 542H294C331 392 383 150 409 4H432L774 555H776L714 137C700 40 694 34 612 28L606 0H868L874 28C793 34 784 37 797 137L849 533C859 612 863 616 956 622L962 650Z" id="g113-78"></path></g><g transform="matrix(.013,0,0,-0.013,17.305,0)"><path d="M95 130C70 130 46 113 46 88C46 72 54 64 59 64C93 55 121 33 121 -3C121 -41 93 -68 44 -88L55 -117C117 -98 186 -56 186 22C186 91 131 130 95 130Z" id="g113-45"></path></g><g transform="matrix(.013,0,0,-0.013,22.448,0)"><path d="M822 650H589L583 622C660 617 677 607 674 561C672 534 664 481 647 390L600 137H596L273 650H126L120 622C176 620 194 615 207 594C221 571 225 557 214 504L161 257C141 166 129 112 121 85C108 42 83 30 29 28L23 0H260L266 28C193 33 173 42 176 89C178 122 186 172 202 255L256 527H259L583 -8H612L690 390C708 481 720 535 728 558C744 603 756 619 816 622L822 650Z" id="g113-79"></path></g><g transform="matrix(.013,0,0,-0.013,36.264,0)"><path d="M471 153C471 170 463 194 452 212C400 220 373 229 322 255C373 281 400 290 452 298C463 316 471 339 471 357C456 366 431 371 410 370C377 329 356 310 308 279C311 336 317 364 336 413C326 432 310 451 294 459C279 451 262 432 252 413C271 364 277 336 280 279C232 310 211 329 178 370C157 371 132 367 117 357C117 340 125 316 136 298C188 290 215 281 266 255C215 229 188 220 136 212C125 194 117 171 117 153C132 144 157 139 178 140C211 181 232 200 280 231C277 174 271 146 252 97C262 78 278 59 294 51C309 59 326 78 336 97C317 146 311 174 308 231C356 200 377 181 410 140C431 139 456 143 471 153Z" id="g113-43"></path></g><g transform="matrix(.013,0,0,-0.013,46.8,0)"><path d="M545 106L524 126C493 85 467 65 455 65C438 65 427 113 405 238C448 295 498 362 543 439L533 448L478 435C453 386 423 331 398 295H395C370 404 347 448 282 448C169 448 23 309 23 153C23 54 65 -12 128 -12C203 -12 283 70 339 155H341C360 29 380 -12 411 -12C444 -12 491 11 545 106ZM333 204C265 95 210 54 169 54C137 54 113 96 113 171C113 302 191 405 252 405C301 405 318 306 333 204Z" id="g113-223"></path></g><g transform="matrix(.013,0,0,-0.013,57.091,0)"><path d="M471 153C471 170 463 194 452 212C400 220 373 229 322 255C373 281 400 290 452 298C463 316 471 339 471 357C456 366 431 371 410 370C377 329 356 310 308 279C311 336 317 364 336 413C326 432 310 451 294 459C279 451 262 432 252 413C271 364 277 336 280 279C232 310 211 329 178 370C157 371 132 367 117 357C117 340 125 316 136 298C188 290 215 281 266 255C215 229 188 220 136 212C125 194 117 171 117 153C132 144 157 139 178 140C211 181 232 200 280 231C277 174 271 146 252 97C262 78 278 59 294 51C309 59 326 78 336 97C317 146 311 174 308 231C356 200 377 181 410 140C431 139 456 143 471 153Z" id="g113-43"></path></g><g transform="matrix(.013,0,0,-0.013,67.628,0)"><path d="M545 106L524 126C493 85 467 65 455 65C438 65 427 113 405 238C448 295 498 362 543 439L533 448L478 435C453 386 423 331 398 295H395C370 404 347 448 282 448C169 448 23 309 23 153C23 54 65 -12 128 -12C203 -12 283 70 339 155H341C360 29 380 -12 411 -12C444 -12 491 11 545 106ZM333 204C265 95 210 54 169 54C137 54 113 96 113 171C113 302 191 405 252 405C301 405 318 306 333 204Z" id="g113-223"></path></g><g transform="matrix(.013,0,0,-0.013,77.918,0)"><path d="M471 153C471 170 463 194 452 212C400 220 373 229 322 255C373 281 400 290 452 298C463 316 471 339 471 357C456 366 431 371 410 370C377 329 356 310 308 279C311 336 317 364 336 413C326 432 310 451 294 459C279 451 262 432 252 413C271 364 277 336 280 279C232 310 211 329 178 370C157 371 132 367 117 357C117 340 125 316 136 298C188 290 215 281 266 255C215 229 188 220 136 212C125 194 117 171 117 153C132 144 157 139 178 140C211 181 232 200 280 231C277 174 271 146 252 97C262 78 278 59 294 51C309 59 326 78 336 97C317 146 311 174 308 231C356 200 377 181 410 140C431 139 456 143 471 153Z" id="g113-43"></path></g><g transform="matrix(.013,0,0,-0.013,88.455,0)"><path d="M545 106L524 126C493 85 467 65 455 65C438 65 427 113 405 238C448 295 498 362 543 439L533 448L478 435C453 386 423 331 398 295H395C370 404 347 448 282 448C169 448 23 309 23 153C23 54 65 -12 128 -12C203 -12 283 70 339 155H341C360 29 380 -12 411 -12C444 -12 491 11 545 106ZM333 204C265 95 210 54 169 54C137 54 113 96 113 171C113 302 191 405 252 405C301 405 318 306 333 204Z" id="g113-223"></path></g><g transform="matrix(.013,0,0,-0.013,95.84,0)"><path d="M275 270C275 450 212 609 64 710L45 686C145 604 203 442 203 270S147 -63 45 -147L64 -170C213 -68 275 89 275 270Z" id="g113-42"></path></g></svg>;</span> <svg height="6.1673pt" id="M134" style="vertical-align:-0.2063904pt" version="1.1" viewbox="-0.0498162 -5.96091 7.51131 6.1673" width="7.51131pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g transform="matrix(.013,0,0,-0.013,0,0)"><path d="M545 106L524 126C493 85 467 65 455 65C438 65 427 113 405 238C448 295 498 362 543 439L533 448L478 435C453 386 423 331 398 295H395C370 404 347 448 282 448C169 448 23 309 23 153C23 54 65 -12 128 -12C203 -12 283 70 339 155H341C360 29 380 -12 411 -12C444 -12 491 11 545 106ZM333 204C265 95 210 54 169 54C137 54 113 96 113 171C113 302 191 405 252 405C301 405 318 306 333 204Z" id="g113-223"></path></g></svg> is the tile size, here equal to 4. After the reshape kernel is applied, lots of small submatrices with new layouts are generated.</div>

Computational Intelligence and Neuroscience

fig2

Figure 2

Figure 2: High Performance Implementation of 3D Convolutional Neural Networks on a GPU 