How Can Brain Learn to Control a Nonholonomic System?

Abstract

Humans can often conduct both linear and nonlinear control tasks after a sufficient number of trials, even if they initially do not have sufficient knowledge about the system's dynamics and the way to control it. Theoretically, it is well known that some nonlinear systems cannot be stabilized asymptotically by any linear controllers and we have reported by an f-MRI experiment that different types of information may be involved in linear and nonlinear control tasks, respectively, from a brain function mapping point of view. In this paper, from a controllability analysis, we still show a possibility that human may use a linear control scheme for such nonlinear control tasks by switching the linear controllers with a virtual constraint. It is suggested that the proposed virtual constraint can play an important role to overcome a limitation of the linear controllers and to mimic human control behavior.

1. Introduction

In the fields of control engineering, system engineering, and brain sciences, the excellent human’s control abilities have been widely studied [1–6]. Among such studies, Wolpert and Kawato [7] have proposed a new model of human control mechanism called Modular Selection And Identification Control (MOSAIC) in which many elemental prediction and control system modules are combined in order to conduct the desired motion. Imamizu et al. [8] have evaluated the MOSAIC model using linear control tasks. They used a functional MRI (f-MRI) scanner to observe brain activities during the control tasks and reported plausibility of the model.

However, human ability is not limited to the linear control case. In fact, humans can often achieve difficult control tasks after a sufficient number of trials, even if they initially do not have sufficient knowledge about the system’s dynamics and the way to control it. For example, Goto et al. [9] have reported manual control ability to operate a 2-link planar under actuated manipulator (2PUAM). The 2PUAM is a nonlinear system with a nonholonomic constraint and cannot be stabilized asymptotically by any linear controller. In this case, to control such nonholonomic manipulator, the operators have to plan the object’s trajectory first and then control the system to follow the trajectory. To do so, the operator must use shape information of the manipulator while the operator does not need such shape information to control linear manipulators.

To verify this difference between linear and nonlinear control tasks, we have conducted another f-MRI experiment using the 2PUAM [10]. The results suggest that the difference can be observed through the brain activities. In fact, a specific area involving the shape information processing was activated with a significant difference compared to the linear task only when subjects control the 2PUAM.

In this paper, to further clarify the experimental results, a new model-based analysis is conducted. The model is a multiple model-based reinforcement learning (MMRL) [11] for nonlinear controls but it is a linear controller. Using the MMRL, we attempt to explain the f-MRI results of the human information processing for the 2PUAM control task from a control theory point of view.

2. 2-Link Planer Under Actuated Manipulator (2PUAM)

In this study, we use the 2PUAM shown in Figure 1 as the control object. The 2PUAM moves in a horizontal plane. Thus, the 2PUAM is free from gravity, and arbitrary positions are equilibrium points. Motor is mounted on the first joint. Then the dynamic equations of the 2PUAM are𝑀₁₁𝜃̈₁+𝑀₁₂𝜃̈₂+𝐶₁=𝑇₁+𝑇_𝑓𝑟1,𝑀₂₁𝜃̈₁+𝑀₂₂𝜃̈₂+𝐶₂=𝑇₂+𝑇_𝑓𝑟2, (1) where𝑀₁₁=(𝑚₁+𝑚₂)𝑙²₂+𝑚₂𝑙²₂+2𝑚₂𝑙₁𝑙₂cos𝜃₂,𝑀₁₂=𝑀₂₁=𝑚₂𝑙²₂+𝑚₂𝑙₁𝑙₂cos𝜃₂,𝑀₂₂=𝑚₂𝑙²₂,𝐶₁=−𝑚₂𝑙₁𝑙₂𝜃̇₁(2𝜃̇₁+𝜃̇₂)sin𝜃₂+𝑐₁̇𝐶𝜃,₂=𝑚₂𝑙₁𝑙₂𝜃̇²₁sin𝜃₂+𝑐₂𝜃̇₂. (2) Here, as shown in Figure 1, 𝑚₁ and 𝑚₂ are the mass of the first and second links, 𝑙₁ and 𝑙₂ are the length, 𝑐₁ and 𝑐₂ are the resistance coefficient, respectively, 𝜃₁  and 𝜃₂  are angles, 𝜃̇₁  and  𝜃̇₂  are angular velocities, and 𝜃̈₁  and 𝜃̈₂ are angular accelerations of joints, respectively. The right-hand sides of (1) are the input and friction torques. In this study, we simplify the model as follows:𝑚₁=𝑚₂=𝑚,𝑙₁=𝑙₂𝑐=𝑙,₁=𝑐₂=0,𝑇₂=𝑇_𝑓𝑟1=𝑇_𝑓𝑟2=0. (3)

Figure 1 

2-link planner under actuated manipulator (2PUAM).

It is known that the 2PUAM is not controllable by the standard continuous feedback. This implies that reaching the 2PUAM and stopping it at some point is a difficult task. Although some indirect methods have been proposed to control the 2PUAM from the control engineering point of views [9], it is still unclear how human can control the nonlinear object such as the 2PUAM.

3. Summary of f-MRI Experiments

3.1. Control Tasks and Training

Here, human operator is required to feed an input torque 𝜏∈[−1,1] so that the end effector of the arm will be driven to the goal point and kept at the point. Since the position of the first joint is fixed, there are at least two objective positions of the second joint as shown in Figure 2: the upper position of the second joint 𝑝₁ and the lower 𝑝₂.

Figure 2 

Positioning task.

The manual control experiment was conducted by 6 neurologically normal subjects (19–24 years of age; six males) participated in the experiments. All subjects were right-handed. Informed written consent was obtained from each participant. They had no knowledge about the dynamic response of the system before the experiment. They observed the virtual manipulator’s states (positions, angles, and velocities of the 2PUAM) through visual data displayed on an LCD monitor and fed the torque by using a joystick. The time limit was 30 seconds for each trial, and 300 training trials have been conducted outside of the MRI scanner before the following scanning sessions. The duration of 300 training trials depended on the subject’s willingness and tiredness, but all the subjects completed them for 2-3 days.

For each trial, the performance index defined by the following equation was recorded:𝐽=^𝑇_𝑘=0(𝑥(𝑘)−𝑥_𝐺)²+(𝑦(𝑘)−𝑦_𝐺)². (4) Here (𝑥(𝑘),𝑦(𝑘)) denotes the position of the end effector at time 𝑘Δ𝑡(𝑘=0,1,2,…,𝑇), Δt is the interval of time step, (𝑥_𝐺,𝑦_𝐺) is the position of the goal point, and 𝑇 is the maximum steps of each trial. The performance index 𝐽 has been displayed on the monitor to guide subjects’ criterion.

3.2. MRI Scanning Sessions

In scanning sessions, the trained subjects control two kinds of virtual manipulators whose shapes are projected on a screen in the MRI scanner as shown in Figure 3, by using an optical (magnet-free) joystick [10]. Both manipulators are the same in shape, but the first manipulator is the 2PUAM and the other is a manipulator which has a linear input and output relation (subjects can control it like a PC mouse). Controlling the 2PUAM is the main task while the linear control task is a comparative one, and thus the former is called test trial and the latter is baseline trial, and their duration are called test and baseline periods, respectively. In other words, subjects can directly operate the coordinate of the end effector of the manipulator by the joystick in baseline trial. On the other hand, subjects operate only the torque of the first joint of the manipulator by the joystick and indirectly control the coordinate of the end effector in test trial. In each trial, subjects try to move the end effector of the manipulator to the goal point and keep it at the goal. The coordinates of the goal points are chosen randomly and displayed on the screen when trial starts.

Figure 3 

Experimental system.

MRI scanner (1.5T SIEMENS: Symphony 1.5T) was used to obtain blood oxygen level-dependant contrast functional images. Images weighted with the apparent transverse relaxation time were obtained with an echo-planner imaging sequence (repetition time: 3.0 s; echo time: 50 ms; flip angle: 90; field of view: 256×256 mm). High-resolution anatomical images of all subjects were also acquired with a T1-weighted sequence.

3.3. Results

Figure 4 indicates regions significantly more activated during test periods than baseline periods (𝑡>5.89, 𝑃<.001). Activities in the primary motor cortex, the somatosensory cortex, the somatosensory association cortex, the prefrontal cortex, the inferior temporal gyrus, and the fusiform gyrus were observed [10].

(a)

(b)

Figure 4 

Significant activated areas of test period compared to baseline period (for more details, see [10]).

In the nonlinear control task, significant activities of the inferior temporal gyrus and the fusiform gyrus were observed. These areas are known to have an intimate involvement in recognizing the characteristic (color and shape) of the object. On the other hand, the prefrontal cortex is known as a region that receives the information that is necessary for action planning from both the temporal association area and the occipital association area and assembles complicated action plan. These suggest that in operating 2PUAM, subjects use the information about shape or position of 2PUAM and make trajectory planning of the positioning task based on that information. From a viewpoint of control theory, it may be worth to mention that such information is not needed for the linear control, but it is necessary for the nonholonomic systems control.

4. Controllability Analysis

The difference between linear and nonlinear control tasks observed through the significant brain activities can be a reflection of the human control mechanism that can cognize the target nonlinear dynamics and use an appropriate piece of information. However, this is not sufficient to conclude that human does not use any linear control scheme. Instead, the hypothesis proposed in this paper is that human can use a linear control scheme by switching linear controllers responsible for specific regions where the linear approximation can work well for the target nonlinear task. The following linear model can then be employed to verify the hypothesis.

4.1. Multiple Model-Based Reinforcement Learning (MMRL)

MMRL has multiple modules that are pairs of prediction model which predicts future state of the controlled objects and reinforcement learning controller which learns the control output. “Responsibility signals” are calculated from the softmax function of the prediction errors. The prediction model which outputs the more accurate prediction has the larger responsibility signal. By weighting control signal and learning of the each module with responsibility signals, these modules are adapted to the corresponding specific situations, respectively.

In this study, we use a multiple linear quadratic controller (MLQC) by using multiple linear prediction and quadratic reward models as an efficient implementation of the MMRL [11]. Change in the state vector of the target system, 𝑥∈𝑅⁴, is given by𝑥̇(𝑡)=𝑓(𝑥(𝑡),𝑢(𝑡)), (5) where 𝑥 is the state vector of the system, and 𝑢 is the control output. Each variable of the vector for the 2PUAM can be defined as𝑥₁=𝜃₁,𝑥₂=𝜃̇₁,𝑥₃=𝜃₂,𝑥₄=𝜃̇₂. (6)

Linear prediction models of the MMRL can be represented as follows:𝑥̇_𝑖(𝑡)=𝑓_𝑖(𝑥(𝑡),𝑢(𝑡))=𝐴_𝑖𝑥(𝑡)+𝐵_𝑖𝑢(𝑡),𝑖=1,2,…,𝑛. (7) where 𝑛 denotes the number of modules. State prediction 𝑥̇ is given by a weighting sum of prediction models with responsibility signals 𝜆_𝑖 [11]:𝑥̇(𝑡)=^𝑛_𝑖=1𝜆_𝑖(𝐸_𝑖(𝑡))𝑥̇_𝑖(𝑡)=^𝑛_𝑖=1𝜆_𝑖(𝐸_𝑖(𝑡))(𝐴_𝑖𝑥(𝑡)+𝐵_𝑖𝑢(𝑡)), (8) where 𝐸_𝑖(𝑡) is a short-time average of the prediction error 𝑥̇(𝑡)−𝑥̇(𝑡). Learning of each prediction model in (7), 𝑖∈{1,2,…,𝑛}, is conducted by changing its parameter vector 𝑤_𝑖 consisting of all the elements of the matrices 𝐴_𝑖 and 𝐵_𝑖 as follows: 𝑤̇_𝑖(𝑡)=𝜂_𝑤𝜆_𝑖(𝐸_𝑖(𝑡))(𝜕𝑓_𝑖𝜕𝑤_𝑖)^𝑇(𝑥̇(𝑡)−𝑥̇(𝑡)), (9) where 𝜂_𝑤 (>0) is an update coefficient. Schematic diagram of the multiple predictor-controller pair architecture is shown in Figure 5.

Figure 5 

Schematic diagram of multiple predictor-controller pair architecture [11].

4.2. Controllability of 2PUAM

We obtain the following state equation of the 2PUAM from (1) and (6) under the condition in (3):𝑥̇₁=𝑥₂,𝑥̇₂=12−cos²𝑥₃{(3+cos𝑥₃)𝑥²₂sin𝑥₃+𝑥₂𝑥₄sin𝑥₃+𝑢/𝑚𝑙²𝑥̇},₃=𝑥₄,𝑥̇₄=−(1+cos𝑥₃)𝑥̇₂−𝑥²₂sin𝑥₃. (10)

Equation (10) can be described using the vector form as𝑥̇₁𝑥̇₂𝑥̇₃𝑥̇₄𝑥=𝑓(¹𝑥₂𝑥₃𝑥₄,𝑢). (11) Here, we define the microscopic fluctuation at some positions 𝑥_𝑎 in phase space as 𝛿𝑥=𝑥−𝑥_𝑎, 𝛿𝑢=𝑢−𝑢_𝑎. Calculating the Taylor series of (11) and ignoring the higher-order terms from the second order, we get𝛿𝑥̇=𝐴𝛿𝑥+𝐵𝛿𝑢. (12) Here matrices A and B are𝐴=𝜕𝑓=0𝜕𝑥̇𝜕𝑥0100₂𝜕𝑥₂𝜕𝑥̇₂𝜕𝑥₃𝜕𝑥̇₂𝜕𝑥₄0𝜕𝑥̇0001₄𝜕𝑥₂𝜕𝑥̇₄𝜕𝑥₃𝜕𝑥̇₄𝜕𝑥₄,𝐵=𝜕𝑓=1𝜕𝑢𝑎𝑚𝑙²010−(1+cos𝑥₃), (13) where𝜕𝑥̇₂𝜕𝑥₂=(2𝑥₂(3+cos𝑥₃)+𝑥₄)sin𝑥₃𝑎,𝜕𝑥̇₂𝜕𝑥₃=(6𝑥₂+𝑥₄)𝑥₂cos²𝑥₃−2𝑥²₂𝑎+𝑢sin2𝑥₃𝑚𝑙²𝑎,𝜕𝑥̇₂𝜕𝑥₄=𝑥₂sin𝑥₃𝑎,𝜕𝑥̇₄𝜕𝑥₂=−(1+cos𝑥₃)𝜕𝑥̇₂𝜕𝑥₂−2𝑥₂sin𝑥₃,𝜕𝑥̇₄𝜕𝑥₃=𝑥̇₂sin𝑥₃−(1+cos𝑥₃)𝜕𝑥̇₂𝜕𝑥₃−𝑥²₂cos𝑥₃,𝜕𝑥̇₄𝜕𝑥₄=−(1+cos𝑥₃)𝑥₂sin𝑥₃𝑎,𝑎=2−cos²𝑥₃. (14)

Since the MMRL is a linear controller, it cannot control the 2PUAM. Indeed, the rank of the controllability matrix 𝑈_𝑐=[𝐵𝐴𝐵𝐴²𝐵𝐴³𝐵] at any equilibrium position 𝑥₀ becomes 2, which is not the full rank, 4. That is, let us denote an equilibrium point by using arbitrary values 𝜃₁₀ and 𝜃₂₀𝑥₀=𝜃₁₀0𝜃₂₀0^𝑇 (15) with no input u = 0. Then, from (14), 𝐴=𝜕𝑓|𝜕𝑥^𝑥=𝑥0_𝑢=0=,0100000000010000𝐵=𝜕𝑓|𝜕𝑥^𝑥=𝑥0_𝑢=0=1𝑎₀𝑚𝑙²010−(1+cos𝜃₂₀), (16) where 𝑎₀=2−cos²𝜃₂₀. In this case, the rank of the 𝑈_𝑐 is not equal to the full rank, 4, but is 2 as follows:rank[𝑈_𝑐]=rank[𝐵𝐴𝐵𝐴²𝐵𝐴³𝐵]=2. (17)

4.3. Possible Control Strategy

The rank of the controllability matrix can, however, be the full rank, 4, at some positions 𝑥_𝑠 in the phase space where angular velocity of the first joint is not exactly zero, 𝜃̇₁≠0, even if the 𝜃̇₁ is very close to zero, as shown in Figure 6.

Figure 6 

Rank of the controllability matrix $U_c$ at some positions $x_s({\dot{\theta }}_2=0,u=0)$.

To verify the controllability, let us denote positions 𝑥_𝑠 in the phase space by using arbitrary values 𝜃_1s, 𝜃_2s, and 𝜃̇_2s:𝑥_𝑠=𝜃_1𝑠𝜃̇_1𝑠𝜃_2𝑠𝜃̇_2𝑠^𝑇,𝜃̇_1𝑠≠0. (18) Then, from (14),𝐴_𝑠=𝜕𝑓|𝜕𝑥^{𝑥=𝑥𝑠}_{𝑢=𝑢𝑠}=0𝜕𝑥̇0100₂𝜕𝑥₂𝜕𝑥̇₂𝜕𝑥₃𝜕𝑥̇₂𝜕𝑥₄0𝜕𝑥̇0001₄𝜕𝑥₂𝜕𝑥̇₄𝜕𝑥₃𝜕𝑥̇₄𝜕𝑥₄,𝐵_𝑠=𝜕𝑓|𝜕𝑥^{𝑥=𝑥𝑠}_{𝑢=𝑢𝑠}=1𝑎_𝑠𝑚𝑙²010−(1+cos𝜃_2𝑠), (19) where 𝑎_𝑠=2−cos²𝜃_2𝑠. In this case, the rank of the controllability matrix is 4, the full rank:rank[𝑈_𝑐𝑠]=rank𝐵_𝑠𝐴_𝑠𝐵_𝑠𝐴²_𝑠𝐵_𝑠𝐴³_𝑠𝐵_𝑠=4. (20) Thus, it is confirmed that 2PUAM cannot be stabilized at any equilibrium point, but if 𝜃̇_1s≠0, the end effector of the 2PUAM can approach any geometrical point with any slow speed.

4.4. Discussions

The slow speed approach verified above might be an interesting result because, nevertheless human subjects can control the 2PUAM very well, it is often very hard even for human subjects to stop the 2PUAM completely in the manual control tasks [9, 10]. The linear controller can be responsible only for a small region in which the linear approximation can work well. Thus, by switching multiple linear models, there is a possibility to control the 2PUAM with the MMRL.

Different from (16), we assumed the condition in (19) where 𝜕𝑥̇_𝑖/𝜕𝑥_𝑗≠0, 𝑖=2,4,and𝑗=2,3,4 to make the rank of controllability matrix be the full rank. The condition implies a virtual constraint of the manipulator’s shape (joint movement) because, for example, 𝜕𝑥̇₂/𝜕𝑥₃≠0 implies the virtual existence of relation between 𝜃̈₁ and 𝜃₂ that makes a constraint on the angular acceleration depended on the angle of the manipulator shape. Interestingly, if human subjects could feel and realize such virtual constraint on the 2PUAM control (joint movement), the control task can sometimes be achieved relatively easier [9].

According to the f-MRI results, subjects may use the shape and position information of the 2PUAM. The virtual constraint discussed above can further be created in order to make the control easier. Unfortunately, the controllability analysis does not prove this hypothesis, but there is a possibility that in operating the 2PUAM, subjects use the shape information to switch the controllers [10] and create the virtual constraint to make the control easier [9]. In this sense, MMRL can be regarded as a linear model for controlling the 2PUAM. Then, if human’s superior learning ability to control the complex nonlinear system could be based on such linear control schemes, its implementation on a robot system might be easier than we expected.

5. Conclusions

In this paper, by using a controllability analysis, we have revised the previous f-MRI experimental results that reveal significant activation areas for the nonlinear control task compared to the linear one. Even the useful pieces of information for the linear task may be different from nonlinear ones, the analysis suggests some possibility to control the 2PUAM with a set of linear control models in a similar way by which human subjects can control it. In fact, to stop the 2PUAM at an equilibrium point completely seems very hard or almost impossible, but to approach there with any arbitrary slow speed seems an easier task. The additional information of shape and position of the 2PUAM can then be used for switching the linear controllers. Although the internal relation between the virtual constraint and the controllability of the nonlinear task is still unclear and should be clarified further from both computational and brain sciences point of views, the hypothesis proposed in this paper implies that it could be possible to design or realize robots with the human-level learning ability in an easier way.