Kernel Recursive Least-Squares Temporal Difference Algorithms with Sparsification and Regularization

<table class="figure-group"><tr class="fig-image" id="a"><td><object data="https://static.hindawi.com/articles/cin/volume-2016/2305854/figures/2305854.fig.003a.svgz" name="2305854.fig.003a" type="image/svg+xml"></object></td></tr><tr class="fig-caption"><td><b>(a) </b>In the nonnoise chain</td></tr><tr class="fig-image" id="b"><td><object data="https://static.hindawi.com/articles/cin/volume-2016/2305854/figures/2305854.fig.003b.svgz" name="2305854.fig.003b" type="image/svg+xml"></object></td></tr><tr class="fig-caption"><td><b>(b) </b>In the noise chain</td></tr></table>

<div><svg height="15.1149pt" id="M489" style="vertical-align:-2.2681pt" version="1.1" viewbox="-0.0498162 -12.8468 29.3663 15.1149" width="29.3663pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g transform="matrix(.013,0,0,-0.013,.333,-2.897)"><path d="M658 557C549 593 437 629 340 673H318C222 629 109 593 0 557L9 532C116 551 225 577 329 602C435 576 543 551 649 532L658 557Z" id="g119-105"></path><glyph.data ascent="3473" descent="-2876" horiz-adv-x="658" vert-adv-y="658"></glyph.data></g><g transform="matrix(.013,0,0,-0.013,0,0)"><path d="M697 650H468L461 623L492 619C539 613 547 605 518 546C481 471 367 264 278 116H276C239 278 197 500 186 567C180 604 185 613 226 619L252 623L260 650H24L17 623C78 617 92 613 108 533L216 -11H247C365 200 515 462 560 529C616 612 624 615 689 623L697 650Z" id="g113-87"></path><glyph.data ascent="3473" descent="-2876" horiz-adv-x="709" vert-adv-y="709"></glyph.data></g><g transform="matrix(.0091,0,0,-0.0091,9.252,-8.632)"><path d="M583 449L551 460C535 436 529 433 494 433C409 433 311 438 228 438C97 438 53 381 24 341L44 316C85 355 124 372 183 372C161 259 87 53 26 5L34 -12C53 -12 94 -4 118 12C166 92 211 256 236 371L394 369C375 301 352 212 334 121C329 96 326 75 326 58C326 5 349 -12 381 -12C431 -12 482 25 519 66L504 97C480 80 448 64 432 64C419 64 410 80 422 154C432 217 450 303 466 365C499 365 536 366 546 369C557 384 570 409 583 449Z" id="g50-238"></path><glyph.data ascent="3443" descent="-2856" horiz-adv-x="601" vert-adv-y="601"></glyph.data></g><g transform="matrix(.013,0,0,-0.013,15.239,0)"><path d="M300 -147C201 -63 143 98 143 270S200 602 300 686L282 710C136 610 70 450 70 271V270C70 89 136 -72 282 -170L300 -147Z" id="g113-41"></path><glyph.data ascent="3473" descent="-2876" horiz-adv-x="346" vert-adv-y="346"></glyph.data></g><g transform="matrix(.013,0,0,-0.013,19.737,0)"><path d="M338 322C336 342 324 413 314 436C291 446 264 454 218 454C113 453 50 390 50 312C50 226 131 187 178 164C239 135 253 110 253 81C253 51 231 27 199 27C138 27 91 96 69 159L40 156C40 108 49 38 55 22C76 7 128 -12 181 -12C265 -12 360 37 360 136C360 214 306 249 225 284C174 307 151 330 151 361C151 393 176 415 202 415C249 415 281 374 308 313L338 322Z" id="g13-113"></path><glyph.data ascent="1024" descent="-360" horiz-adv-x="383" vert-adv-y="383"></glyph.data></g><g transform="matrix(.013,0,0,-0.013,24.716,0)"><path d="M275 270C275 450 212 609 64 710L45 686C145 604 203 442 203 270S147 -63 45 -147L64 -170C213 -68 275 89 275 270Z" id="g113-42"></path><glyph.data ascent="3473" descent="-2876" horiz-adv-x="346" vert-adv-y="346"></glyph.data></g></svg> approximated by all tested algorithms at the final episode.</div>

Computational Intelligence and Neuroscience

fig3

Figure 3

Figure 3: Kernel Recursive Least-Squares Temporal Difference Algorithms with Sparsification and Regularization 