Development of an AI-Enabled Q-Agent for Making Data Offloading Decisions in a Multi-RAT Wireless Network

<div>The actions learned by the Q-agent, <span class="nowrap"><svg height="12.3706pt" id="M175" style="vertical-align:-2.2681pt" version="1.1" viewbox="-0.0498162 -10.1025 49.3395 12.3706" width="49.3395pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g transform="matrix(.013,0,0,-0.013,0,0)"><path d="M483 97L471 123C436 91 401 65 392 65C388 65 384 74 390 106C414 239 444 378 457 429L455 433C444 433 429 436 416 439C392 444 368 448 344 448C281 448 204 415 152 376C71 315 23 205 23 103C23 21 57 -12 85 -12C114 -12 149 6 185 34C231 70 285 119 329 183H331L309 81C292 0 308 -12 326 -12C350 -12 421 24 483 97ZM374 387C370 363 356 291 345 261C315 193 181 50 139 50C124 50 110 71 110 118C110 224 153 331 218 379C238 394 271 402 301 402C329 402 359 394 374 387Z"></path></g><g transform="matrix(.013,0,0,-0.013,10.212,0)"><path d="M535 323V373H52V323H535ZM535 138V188H52V138H535Z"></path></g><g transform="matrix(.013,0,0,-0.013,21.475,0)"><path d="M574 449L545 460C526 432 516 430 487 430C404 430 311 435 226 435C104 435 56 379 25 341L43 318C81 354 121 372 181 372C161 246 87 53 23 3L30 -12C48 -12 88 -4 113 11C157 75 207 248 232 371L386 367L326 109C321 86 318 66 318 50C318 4 339 -12 366 -12C410 -12 461 21 505 69L492 96C467 79 434 60 418 60C406 60 400 78 411 147C422 217 439 300 457 366C487 366 524 367 536 370C547 385 558 408 574 449Z"></path></g><g transform="matrix(.0091,0,0,-0.0091,29.159,-5.741)"><path d="M486 158C486 177 478 202 466 220C413 228 386 236 336 262C386 288 413 297 466 304C478 323 486 347 485 366C470 376 444 381 422 380C389 338 368 319 321 288C323 345 329 372 349 422C339 442 322 461 305 470C289 461 271 442 262 422C281 372 287 345 290 288C243 319 222 338 189 380C167 381 142 376 125 366C125 347 133 322 145 304C198 296 225 288 275 262C225 236 198 227 145 220C133 201 125 177 126 158C141 148 167 143 189 144C222 186 243 205 290 236C288 179 282 152 262 102C272 82 289 63 306 54C322 63 340 82 350 102C330 152 324 179 321 236C368 205 390 186 422 144C444 143 470 148 486 158Z"></path></g><g transform="matrix(.013,0,0,-0.013,35.238,0)"><path d="M300 -147C201 -63 143 98 143 270S200 602 300 686L282 710C136 610 70 450 70 271V270C70 89 136 -72 282 -170L300 -147Z"></path></g><g transform="matrix(.013,0,0,-0.013,39.736,0)"><path d="M352 391C352 416 319 448 267 448C236 448 173 423 147 400C107 364 96 332 96 304C96 248 143 210 193 181C241 153 258 124 258 100C258 72 232 38 184 38C151 38 107 66 81 108C77 114 64 116 55 111C34 99 23 84 23 65C23 29 81 -12 134 -12C220 -12 325 61 325 141C325 184 297 215 234 256C194 282 161 309 161 346C161 380 188 401 217 401C255 401 279 380 301 353C308 344 313 341 325 347C341 355 352 371 352 391Z"></path></g><g transform="matrix(.013,0,0,-0.013,44.611,0)"><path d="M275 270C275 450 212 609 64 710L45 686C145 604 203 442 203 270S147 -63 45 -147L64 -170C213 -68 275 89 275 270Z"></path></g></svg>,</span> as a function of states <span class="nowrap"><svg height="12.6708pt" id="M176" style="vertical-align:-3.38247pt" version="1.1" viewbox="-0.0498162 -9.28833 97.3519 12.6708" width="97.3519pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g transform="matrix(.013,0,0,-0.013,0,0)"><path d="M352 391C352 416 319 448 267 448C236 448 173 423 147 400C107 364 96 332 96 304C96 248 143 210 193 181C241 153 258 124 258 100C258 72 232 38 184 38C151 38 107 66 81 108C77 114 64 116 55 111C34 99 23 84 23 65C23 29 81 -12 134 -12C220 -12 325 61 325 141C325 184 297 215 234 256C194 282 161 309 161 346C161 380 188 401 217 401C255 401 279 380 301 353C308 344 313 341 325 347C341 355 352 371 352 391Z"></path></g><g transform="matrix(.013,0,0,-0.013,4.875,0)"><path d="M300 -147C201 -63 143 98 143 270S200 602 300 686L282 710C136 610 70 450 70 271V270C70 89 136 -72 282 -170L300 -147Z"></path></g><g transform="matrix(.013,0,0,-0.013,9.373,0)"><path d="M480 416C480 431 465 448 438 448C388 448 312 383 252 330C217 299 188 273 155 237H153L257 680C262 700 263 712 253 712C240 712 183 684 97 674L92 648L126 647C166 646 172 645 163 606L23 -6L29 -12C51 -5 77 2 107 8C115 62 130 128 142 180C153 193 179 220 204 241C231 170 259 106 288 54C317 0 336 -12 358 -12C381 -12 423 2 477 80L460 100C434 74 408 54 398 54C385 54 374 65 351 107C326 154 282 241 263 299C296 332 351 377 403 377C424 377 436 372 445 368C449 366 456 368 462 375C472 386 480 402 480 416Z"></path></g><g transform="matrix(.013,0,0,-0.013,19.544,0)"><path d="M535 323V373H52V323H535ZM535 138V188H52V138H535Z"></path></g><g transform="matrix(.013,0,0,-0.013,30.807,0)"><path d="M300 -147C201 -63 143 98 143 270S200 602 300 686L282 710C136 610 70 450 70 271V270C70 89 136 -72 282 -170L300 -147Z"></path></g><g transform="matrix(.013,0,0,-0.013,35.305,0)"><path d="M383 397C383 424 344 448 299 448C244 448 172 409 132 375C66 319 23 227 23 146C23 42 74 -12 146 -12C208 -12 298 30 359 103L343 124C315 95 248 48 192 48C145 48 111 85 111 163C111 228 129 294 151 330C171 363 201 401 241 401C275 401 302 384 325 356C332 347 339 344 348 348C373 360 383 381 383 397Z"></path></g><g transform="matrix(.0091,0,0,-0.0091,39.478,3.132)"><path d="M481 445L456 468C426 437 403 427 388 427C372 427 356 428 302 438C256 447 221 451 187 451C159 451 130 428 105 384C93 363 80 340 65 305L96 290C122 342 149 381 193 381C216 381 267 377 294 374S346 372 365 376L75 105C37 70 24 35 24 16C24 2 30 -5 35 -8C42 -8 46 -6 48 -2C58 14 75 35 90 41C102 46 116 46 147 32C200 8 267 -21 298 -21C347 -21 390 12 448 126L420 151C369 77 338 56 311 56C287 56 249 68 212 81C180 92 149 100 127 98C212 172 353 308 413 372L481 445Z"></path></g><g transform="matrix(.013,0,0,-0.013,44.589,0)"><path d="M95 130C70 130 46 113 46 88C46 72 54 64 59 64C93 55 121 33 121 -3C121 -41 93 -68 44 -88L55 -117C117 -98 186 -56 186 22C186 91 131 130 95 130Z"></path></g><g transform="matrix(.013,0,0,-0.013,49.732,0)"><path d="M689 332C689 394 670 448 646 448C620 448 597 421 597 396C597 386 600 381 608 372C619 359 620 334 620 315C620 150 538 45 454 45C414 45 386 67 386 122C386 138 388 158 394 180L457 426L452 432L377 416L315 156C302 100 259 45 216 45C176 45 148 67 148 122C148 133 152 158 156 180C162 212 173 259 194 332C201 357 206 384 206 405C206 430 198 448 174 448C125 448 66 406 23 342L43 319C84 368 110 383 121 383C126 383 128 382 128 377C128 370 127 359 122 343C99 268 84 204 77 156C74 137 70 111 70 104C70 25 125 -12 180 -12C228 -12 276 12 319 50C338 8 378 -12 418 -12C549 -12 689 166 689 332Z"></path></g><g transform="matrix(.0091,0,0,-0.0091,58.663,3.132)"><path d="M481 445L456 468C426 437 403 427 388 427C372 427 356 428 302 438C256 447 221 451 187 451C159 451 130 428 105 384C93 363 80 340 65 305L96 290C122 342 149 381 193 381C216 381 267 377 294 374S346 372 365 376L75 105C37 70 24 35 24 16C24 2 30 -5 35 -8C42 -8 46 -6 48 -2C58 14 75 35 90 41C102 46 116 46 147 32C200 8 267 -21 298 -21C347 -21 390 12 448 126L420 151C369 77 338 56 311 56C287 56 249 68 212 81C180 92 149 100 127 98C212 172 353 308 413 372L481 445Z"></path></g><g transform="matrix(.013,0,0,-0.013,63.774,0)"><path d="M275 270C275 450 212 609 64 710L45 686C145 604 203 442 203 270S147 -63 45 -147L64 -170C213 -68 275 89 275 270Z"></path></g><g transform="matrix(.013,0,0,-0.013,68.272,0)"><path d="M95 130C70 130 46 113 46 88C46 72 54 64 59 64C93 55 121 33 121 -3C121 -41 93 -68 44 -88L55 -117C117 -98 186 -56 186 22C186 91 131 130 95 130Z"></path></g><g transform="matrix(.013,0,0,-0.013,73.415,0)"><path d="M499 88L487 113C459 86 422 65 413 65C403 65 403 74 409 98L461 318C487 426 460 448 431 448C408 448 383 441 352 424C305 398 223 344 157 257H155L243 674C249 702 249 712 240 712C227 712 170 680 87 673L82 648H117C155 648 162 644 150 590L23 -4L30 -12C55 -4 81 3 105 8C113 53 131 150 143 187C199 281 323 387 372 387C390 387 393 369 380 311L328 86C313 19 319 -12 347 -12C379 -12 442 24 499 88Z"></path></g><g transform="matrix(.013,0,0,-0.013,80.123,0)"><path d="M95 130C70 130 46 113 46 88C46 72 54 64 59 64C93 55 121 33 121 -3C121 -41 93 -68 44 -88L55 -117C117 -98 186 -56 186 22C186 91 131 130 95 130Z"></path></g><g transform="matrix(.013,0,0,-0.013,85.266,0)"><path d="M530 686C535 705 530 712 521 712C504 712 448 684 359 674L358 648H393C437 648 439 646 429 593L400 435C372 447 345 448 332 448C286 448 194 414 144 373C68 311 23 203 23 111C23 26 57 -12 91 -12C120 -12 147 3 188 29C227 54 290 102 341 170H343L322 71C308 6 320 -12 341 -12C373 -12 442 27 501 96L485 120C455 91 422 67 408 67C401 67 401 76 404 91C440 294 479 473 530 686ZM387 375L355 241C326 187 200 53 142 53C126 53 109 73 109 130C109 217 154 337 218 381C240 396 265 404 297 404S372 390 387 375Z"></path></g><g transform="matrix(.013,0,0,-0.013,92.482,0)"><path d="M275 270C275 450 212 609 64 710L45 686C145 604 203 442 203 270S147 -63 45 -147L64 -170C213 -68 275 89 275 270Z"></path></g></svg>.</span> Here, <object data="https://static.hindawi.com/articles/jcnc/volume-2024/9571987/figures/9571987.fig-inline.001.svgz" name="9571987.fig-inline.001" type="image/svg+xml"></object> denotes remain idle, <svg height="9.69047pt" id="M177" style="vertical-align:-0.3499298pt" version="1.1" viewbox="-0.0498162 -9.34054 10.8779 9.69047" width="10.8779pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g transform="matrix(.013,0,0,-0.013,0,0)"><path d="M799 12L418 655H408L27 12L34 0H792L799 12ZM711 50H115L413 553L711 50Z"></path></g></svg> denotes download data through cellular RAT, and <svg height="6.01072pt" id="M178" style="vertical-align:-0.04980993pt" version="1.1" viewbox="-0.0498162 -5.96091 7.75925 6.01072" width="7.75925pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g transform="matrix(.013,0,0,-0.013,0,0)"><path d="M471 153C471 170 463 194 452 212C400 220 373 229 322 255C373 281 400 290 452 298C463 316 471 339 471 357C456 366 431 371 410 370C377 329 356 310 308 279C311 336 317 364 336 413C326 432 310 451 294 459C279 451 262 432 252 413C271 364 277 336 280 279C232 310 211 329 178 370C157 371 132 367 117 357C117 340 125 316 136 298C188 290 215 281 266 255C215 229 188 220 136 212C125 194 117 171 117 153C132 144 157 139 178 140C211 181 232 200 280 231C277 174 271 146 252 97C262 78 278 59 294 51C309 59 326 78 336 97C317 146 311 174 308 231C356 200 377 181 410 140C431 139 456 143 471 153Z"></path></g></svg> denotes download data through Wi-Fi RAT.</div>

Journal of Computer Networks and Communications

fig3

Figure 3

Figure 3: Development of an AI-Enabled Q-Agent for Making Data Offloading Decisions in a Multi-RAT Wireless Network 