Research Article | Open Access

Ariel Masuri, Oded Medina, Shlomi Hacohen, Nir Shvalb, "Gait and Trajectory Optimization by Self-Learning for Quadrupedal Robots with an Active Back Joint", *Journal of Robotics*, vol. 2020, Article ID 8051510, 7 pages, 2020. https://doi.org/10.1155/2020/8051510

# Gait and Trajectory Optimization by Self-Learning for Quadrupedal Robots with an Active Back Joint

**Academic Editor:**L. Fortuna

#### Abstract

This paper presents an efficient technique for a self-learning dynamic walk for a quadrupedal robot. The cost function for such a task is typically complicated, and the number of parameters to be optimized is high. Therefore, a simple technique for optimization is of importance. We apply a genetic algorithm (GA) which uses real experimental data rather than simulations to evaluate the fitness of a tested gait. The algorithm actively optimizes 12 of the robot’s dynamic walking parameters. These include the step length and duration and the bending of an active back. For this end, a simple quadrupedal robot was designed and fabricated in a structure inspired by small animals. The fitness function was then computed based on experimental data collected from a camera located above the scene coupled with data collected from the actuators’ sensors. The experimental results demonstrate how walking abilities are improved in the course of learning, while including an active back should be considered to improve walking performances.

#### 1. Introduction

The main advantage of legged robots over their wheeled counterparts is their ability to overcome obstacles such as stairs and hard outdoor terrains. For example, in [1], the authors presented a continuous free gait generation method for quadrupedal robots when walking on rough terrains based on the CoG trajectory planning method. Upright robots can manipulate objects and interact in a human environment using their hands [2]. Nevertheless, dynamic walking gaits are harder to achieve for these robots compared to quadrupedal (see [3, 4] for a survey on robot learning from demonstrations). Also, note that there is a clear distinction between bipedal and quadrupedal mechanisms in the literature. The term gait, which is often used in characterization of legged robot walking patterns, refers to the patterns of the limbs during locomotion on a solid surface.

Running and walking gaits differ by the interaction time with the ground. For walking, the interaction period with the ground is more than 50% of the entire gait cycle [5]. Quadruped and hexapod robots have stability and speed advantages over bipedal robots (see, for example, [6], where Juarez-Campos et al. implemented a Peaucellier–Lipkin mechanism for a hexapedal robot). Moreover, trafficability of these robots increases with their degrees of freedom [7–9].

An important approach in designing biped walking gait is the zero moment point (ZMP) [10]. It is defined as a point on the floor where equilibrium is established, while the horizontal portion of the reaction-moments vanishes. The main idea is that when the ZMP is within the convex hull of the contact points between the feet and the floor (support polygon), stable walking may be maintained. For example, see [11] which used particle swarm optimization (PSO) in order to maintain stable walking and [12] which used PSO for quadruped gait adaptation using terrain classification and gait optimization.

The reader is referred to [13] for some additional stabilization methods. Controlling a real-life walking robot according to its dynamic model alone is difficult (see [14]). The associated dynamics is complicated with multidimensional vector-state which is basically nonlinear and time-variant. Moreover, uncertainties in the robot’s model and state add complexity (see [15]).

A considerable portion of research in the field aims to ease calculations and enhance the robustness of walking. Explicitly, improvement is aimed at [16] (1) trajectory planning; (2) achieving ZMP optimal walking gait; (3) calculating the ZMP’s position feedback-force system; and (4) planning reference trajectories of the center of gravity (CoG) of the body. Many researchers apply reinforcement learning to calculate the CoG of the robot such as [17] which used reinforcement learning for posture stabilizing enhancement by exerting random disturbances. In [18], the researchers implemented a hybrid adaptive fuzzy dynamic evolutionary neural network technique. Their performances are demonstrated in simulations.

Genetic algorithm is yet another approach for optimizing a set of parameters according to an objective function. For biped motion control, Taherkhorsandi et al. [19] presented a sliding control which was used to optimize an adaptive robust hybrid PID controller while a GA was applied to select the controller’s coefficients (from the Pareto front). Kato et al. [20] presented a research study where they used a GA in a simulation for gait optimization and implemented their results in a real model. Though not all gaits performed well, some gaits generated by a GA on a simulated environment did. Therefore, it is preferable to apply a GA on real-world experiments. This is the main aim of this paper.

In real implementations, it is mostly common to close the loop using sensors. Loffler et al. [21] used IMU sensors and encoders for every motor shaft and also applied a 6-axis force-torque sensor placed in the robot’s foot. As expected, they reported an overall improvement in walking and jogging. The same strategy is used for recent quadrupedal robots such as HyQ2Max and HyQ2Centaur [22] or Cheetah 2 [23] or the well-known BigDog [24], LittleDog [25], and WildCat [26]. These robots also map out their environment by using infrared cameras, retroreflective markers, and range sensors. The usage of cameras may be implemented for position and velocity sensing. A camera may be placed on the robot, or alternatively, as an external sensor, and data on the robot’s performance are gathered during its motion. Obviously, using cameras as sensors in the gait optimizing procedure requires real-time techniques for analyzing the video stream (see [27, 28]).

The back joint (spine) was investigated in [29], where the research studies presented simulations showing that an active back is a key driving factor for the improvement of speed and cost-of-transport in quadrupeds. In [30], Khoramshahi et al. presented a simplified (wheeled) locomotion system which is behaviorally and structurally similar to a galloping quadruped and showed that fast locomotion requires a flexible spine.

The examples presented above present sophisticated robots in terms of their sensory equipment and mechanical structure (at least 3 degrees of freedom per each leg), which lead to impressive trafficability and load-bearing abilities. In this work, we use a machine learning approach based on a genetic algorithm (GA) for improving a four-legged robot’s walking abilities. Such an approach was introduced in [31] for quadrupedal robots with a rigid back, where the researchers optimized the forward speed as their objective function. Here, we extend their work to the case where an active back is present by optimizing “straightness” of movement coupled with the power efficiency. After optimizing these gaits, the robot will have the ability to track on a predefined path such as in [32].

This paper is organized as follows: in Section 2, we introduce the mechanical concept of our quadrupedal robot. In Section 3, we specify the initial gaits for the first generation and explain the genetic algorithm which we use. Section 4 provides a description of the experimental setup and the optimization results. We conclude the paper in Section 5.

#### 2. Mechanical Design

Nine servo motors were incorporated into the robot’s structure (Figure 1), two MG90S microservos per leg and one VEX-EDR-393 for the back joint (see Figure 2(b)). The robot’s body was designed and built as a two-dimensional platform of approximately 200 mm length made of 4 mm thick Perspex. Each leg was connected to the robot body by a shaft with torsion springs (see Figure 2(a)). These are used for maintaining the elasticity during the robot’s movement. The robot’s body consists of two parts connected by a shaft actuated by a servo motor that enables the back curvature (see Figure 3).

**(a)**

**(b)**

**(a)**

**(b)**

The leg was designed as a five-bar mechanism (see Figure 2(b))—a two-DOF leg. The advantage of such a structure is that the motors are mounted on the robot’s skeleton rather than on its joints (see also [33]). This design reduces the leg’s physical volume and mass. In addition, such a design enables scaling the motors without changing the leg’s structure and its moment of inertia [34]. To enable a proper foot path, , where (Figure 2(b)), one solves the inverse-kinematics problem. This yields the motors’ angles for a single cycle duration.

#### 3. Genetic Algorithm for Self-Learning

The set of parameters controlling the robot’s motion is

The parameter is the time duration of a single cycle. The rest of the parameters are dimensionless and are given with respect to . So, indicates the -th leg time phase from the cycle’s beginning, and indicates the time duration of the -th leg to complete its individual cycle. indicates the time phase of the back joint and its time duration, respectively (see Figure 4).

We initiate the algorithm by selecting the gait’s parameter vector (1). To do so, we manually generated the following gaits: walk, rack, amble, canter, trot, and gallop. Their values were extracted from [35]; see Figure 4. Additionally, 4 initial gaits were randomly chosen.

In general, the GA searches for an optimal gait by applying genetic operators to a population of gaits [36]. Gaits that perform well are rewarded and proliferate through the population, whereas gaits that perform poorly are removed. In our GA, from the second generation onward, in order to avoid *s* local stationary solution, we used a random operator function to generate random new gaits. Our GA maintains a population of the 6 best gaits and uses mutation and pairing to manipulate the gaits in the population. Each generation of tests consists of a random function (20%), mutation (40%), and pairing (40%).

The random operator randomly generates a new gait with UB and LB—the upper and lower parameter bounds, respectively. The quadrupedal feet are impacted by the floor during the gait. So, when is too short, the robot’s limbs jerk, which may harm the mechanism. On the contrary, long time periods are not desirable as well (see, for example, Figure 4 for gallop gait). These lower and upper constraints are used in order to reduce the search area and to avoid such undesirable gaits.

The mutation operator acts on the gait by randomly altering its parameters with some predefined probability :

Here, we used .

The pairing operator generates a new gait from two “parent” gaits by performing multipoint recombination of their set of parameters. The gaits chosen for “reproduction” were (1) the gait which gained the best score, paired with (3), and a random gait chosen from the top eight, equation (3). We define , the prioritizing weight. The offspring gait is then

##### 3.1. The GA Weight Function

The weight function (also known as the evaluation or cost function) evaluates how close a given solution is to the optimum solution of the desired problem. Each solution is given a score, specifying its fitness to the desired solution. We are interested in a smooth trajectory that requires minimal power consumption while maximizing the resulting velocity. The efficiency is the ratio between the input power and the power consumption (i.e., the ratio between the average kinetic energy and the input power , compare with [29]). We also include an additional dimensionless ratio which measures the “straightness” of the path (compare with [37]). The fitness function used for scoring the robot’s gait is then

Here, are the average velocity, distance, and overall trajectory length, respectively. To comply with the dimensions indicated above, we set , though these may be chosen differently for other purposes.

The power consumption of the motors is calculated by summing the electrical current sensor’s value in every constant time interval. Since the voltage is maintained constant, the current summation will do. The power consumption was extracted from an INA219 high side DC current sensor placed on the robot that sampled the current consumption of the motors during walking. In addition, an IR LED was mounted to the robot’s back, faced upward. A webcam with an IR filter was positioned above the experimental area in order to detect the robot’s location. The images received from the camera underwent image processing for identifying the locomotion of the robot on the experimental surface.

#### 4. Experimentation and Results

The experimental setup (see a short movie in [38]) included a horizontal surface (Figure 5). In order to avoid skidding during the experiment, the surface and the robot’s foot were covered with fabric. The computation lasted for fifteen generations in which the algorithm tested a total of 146 different gaits and converged to a solution. It began with 10 initial gaits followed with 14 generations having 10 gaits each. During the experiments, the trajectories’ length and smoothness were improved as the number of generations increased (Figures 6 and 7). Each experiment begins by placing the robot in a predefined starting point. The experiment ends after 7 time periods or alternatively when a nonmotion scenario occurs. Each gait was repeated 3 times, and results were averaged.

**(a)**

**(b)**

**(c)**

**(d)**

The results show that both the mutation and the random functions helped the algorithm to converge. The authors believe that the random procedure was required in order to move “freely” in the solution space at the first stages of the algorithm, preventing convergences into local minima solutions (compare with a simulation annealing approach). The pairing function had little effect on finding the optimal gait. These insights were manually examined in the course of the experimentation by tracking the convergence rate.

The back joint was found to be significant. We performed two sets of experiments:(1)The back joint was activated in accordance with the parameters and indicating the time phase of the back joint and its time duration, respectively(2)The back joint was fixed to a flat angle during the optimization

Figure 8 depicts the corresponding results of these two setups. At the first stages of the optimization, the back-joint activation did not play a significant role. A possible explanation for this is that, in these early stages, robot locomotion was far from optimal, and thus, improvements due to any of the parameters were of the same importance. Nevertheless, in generations 7 to 10, where fine tuning of the parameters took place, having an addition joint that completely changes the robot’s dynamics, such as the back joint, is expected to be of importance and indeed it was.

The final walking gaits were found to be which include the back parameters. The final walking gaits which ignore the back parameters were found to be .

#### 5. Conclusions

We introduced a real-world set of experiments for optimizing the robot’s walking gait using a genetic algorithm. In order to demonstrate our solution, a low-cost mechanical model of a quadrupedal robot was designed and fabricated. Our results show that, after 15 generations, the robot’s trajectory improved significantly relative to the first generation gaits. The improvement was shown in terms of the robot’s velocity, trajectory length, and smoothness, as well as power consumption. The authors believe that the concept described in this paper may be implemented in a totally autonomous experimental system, which releases the need to lift the robot to the start point at each test and enables a larger number of generations. In addition, as shown in the literature [39], the back structure has a significant function in the gait quality. Here, we implemented an active back to examine these advantages.

Future work will include a mechanism having the ability to perform a large number of experiments without a human in the loop. Moreover, we shall examine, in addition to the active back, the qualities of an active tail.

#### Data Availability

The authors confirm that the data supporting the findings of this study are available within the article.

#### Conflicts of Interest

No potential conflicts of interest were reported by the authors.

#### References

- S. Zhang, M. Fan, Y. Li, X. Rong, and M. Liu, “Generation of a continuous free gait for quadruped robot over rough terrains,”
*Advanced Robotics*, vol. 33, no. 2, pp. 74–89, 2019. View at: Publisher Site | Google Scholar - M. Lee, K.-S. Kim, and S. Kim, “Design of robot hand for bipedal/quadrupedal transformable locomotive robot, in safety, security, and rescue robotics (SSRR),” in
*Proceedings of the 2016 IEEE International Symposium on IEEE*, pp. 160–165, IEEE, Lausanne, Switzerland, October 2016. View at: Publisher Site | Google Scholar - Q. Wu, C. Liu, J. Zhang, and Q. Chen, “Survey of locomotion control of legged robots inspired by biological concept,”
*Science in China Series F: Information Sciences*, vol. 52, no. 10, pp. 1715–1729, 2009. View at: Publisher Site | Google Scholar - B. D. Argall, S. Chernova, M. Veloso, and B. Browning, “A survey of robot learning from demonstration,”
*Robotics and Autonomous Systems*, vol. 57, no. 5, pp. 469–483, 2009. View at: Publisher Site | Google Scholar - J. Ashar, D. Claridge, B. Hall et al., “Robocup standard platform league-runswift 2010,” in
*Proceedings of the Australasian Conference on Robotics and Automation*, University of New South Wales (UNSW), Brisbane, Australia, December 2010. View at: Google Scholar - I. Juárez-Campos, D. A. Núñez-Altamirano, L. Márquez-Pérez, L. Romero-Muñoz, and B. Juárez-Campos, “Hexapod with legs based on peaucellier–lipkin mechanisms: a mathematical structure used in reconfiguration for path planning,”
*International Journal of Advanced Robotic Systems*, vol. 15, no. 4, 2018. View at: Publisher Site | Google Scholar - E. Kagan, N. Shvalb, and I. Ben-Gal,
*Autonomous Mobile Robots and Multi-Robot Systems Motion Planning Communication and Swarming*, John Wiley & Sons, Hoboken, NJ, USA, 2019. - N. Shvalb, B. B. Moshe, and O. Medina, “A real-time motion planning algorithm for a hyper-redundant set of mechanisms,”
*Robotica*, vol. 31, no. 8, pp. 1327–1335, 2013. View at: Publisher Site | Google Scholar - O. Medina, A. Shapiro, and N. Shvalb, “Motion planning for an actuated flexible polyhedron manifold,”
*Advanced Robotics*, vol. 29, no. 18, pp. 1195–1203, 2015. View at: Publisher Site | Google Scholar - S. Park and J. Oh, “Real-time continuous ZMP pattern generation of a humanoid robot using an analytic method based on capture point,”
*Advanced Robotics*, vol. 33, no. 1, pp. 33–48, 2019. View at: Publisher Site | Google Scholar - F. A. Raheem and M. K. Flayyih, “Quadruped robot creeping gait stability analysis and optimization using PSO,” in
*Proceedings of the 2017 Second Al-Sadiq International Conference on Multidisciplinary in IT and Communication Science and Applications (AIC-MITCSA)*, pp. 79–84, IEEE, Baghdad, Iraq, December 2017. View at: Publisher Site | Google Scholar - J.-J. Kim and J.-J. Lee, “Adaptation of quadruped gaits using surface classification and gait optimization,” in
*Proceedings of the 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems*, pp. 716–721, IEEE, Tokyo, Japan, Noveber 2013. View at: Publisher Site | Google Scholar - M. Dekker,
*Zero-Moment Point Method for Stable Biped Walking*, Eindhoven University of Technology, Eindhoven, Netherlands, 2009. - M. Bucolo, A. Buscarino, C. Famoso, L. Fortuna, and M. Frasca, “Control of imperfect dynamical systems,”
*Nonlinear Dynamics*, vol. 98, no. 4, pp. 2989–2999, 2019. View at: Publisher Site | Google Scholar - S. Hacohen, S. Shoval, and N. Shvalb, “Probability navigation function for stochastic static environments,”
*International Journal of Control, Automation and Systems*, vol. 17, no. 8, pp. 2097–2113, 2019. View at: Publisher Site | Google Scholar - D. Katic and M. Vukobratovic, “Survey of intelligent control techniques for humanoid robots,”
*Journal of Intelligent and Robotic Systems*, vol. 37, no. 2, pp. 117–141, 2003. View at: Google Scholar - W. Wu and L. Gao, “Posture self-stabilizer of a biped robot based on training platform and reinforcement learning,”
*Robotics and Autonomous Systems*, vol. 98, pp. 42–55, 2017. View at: Publisher Site | Google Scholar - T. T. Huan, P. D. Huynh, C. Van Kien, and H. P. H. Anh, “Implementation of hybrid adaptive fuzzy sliding model control and evolutionary neural observer for biped robot systems,” in
*Proceedings of the International Conference on System Science and Engineering (ICSSE)*, pp. 77–82, IEEE, Ho Chi Minh City, Vietnam, July 2017. View at: Publisher Site | Google Scholar - M. Taherkhorsandi, M. J. Mahmoodabadi, M. Talebipour, and K. K. Castillo-Villar, “Pareto design of an adaptive robust hybrid of pid and sliding control for a biped robot via genetic algorithm optimization,”
*Nonlinear Dynamics*, vol. 79, no. 1, pp. 251–263, 2015. View at: Publisher Site | Google Scholar - T. Kato, K. Shiromi, M. Nagata, H. Nakashima, and K. Matsuo, “Gait pattern acquisition for four-legged mobile robot by genetic algorithm,” in
*Proceedings of the 41st Annual Conference of the IEEE Industrial Electronics Society*, IEEE, Yokohama, Japan, November 2015. View at: Publisher Site | Google Scholar - K. Loffler, M. Gienger, and F. Pfeiffer, “Sensor and control design of a dynamically stable biped robot, in robotics and automation,” in
*Proceedings of the IEEE International Conference on Robotics and Automation*, vol. 1, pp. 484–490, IEEE, Taipei, Taiwan, September 2003. View at: Google Scholar - Y. Yang, C. Semini, N. G. Tsagarakis, E. Guglielmino, and D. G. Caldwell, “Leg mechanisms for hydraulically actuated robots, in intelligent robots and systems,” in
*Proceedings of the IEEE/RSJ International Conference on. IEEE*, pp. 4669–4675, IEEE, St. Louis, MO, USA, October 2009. View at: Publisher Site | Google Scholar - H.-W. Park, S. Park, and S. Kim, “Variable-speed quadrupedal bounding using impulse planning: untethered high-speed 3D running of MIT cheetah 2,” in
*Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)*, pp. 5163–5170, IEEE, Seattle, WA, USA, May 2015. View at: Publisher Site | Google Scholar - M. Raibert, K. Blankespoor, G. Nelson, and R. Playter, “Bigdog, the rough-terrain quadruped robot,”
*IFAC Proceedings Volumes*, vol. 41, no. 2, pp. 10 822–10 825, 2008. View at: Publisher Site | Google Scholar - M. P. Murphy, A. Saunders, C. Moreira, A. A. Rizzi, and M. Raibert, “The littledog robot,”
*The International Journal of Robotics Research*, vol. 30, no. 2, pp. 145–149, 2011. View at: Publisher Site | Google Scholar - J. R. Rebula, P. D. Neuhaus, B. V. Bonnlander, M. J. Johnson, and J. E. Pratt, “A controller for the littledog quadruped walking on rough terrain, in robotics and automation,” in
*Proceedings of the 2007 IEEE International Conference on Robotics and Automation*, pp. 1467–1473, IEEE, Roma, Italy, May 2007. View at: Publisher Site | Google Scholar - F. Sapuppo, M. Bucolo, M. Intaglietta, L. Fortuna, and P. Arena, “Cellular non-linear networks for microcirculation applications,”
*International Journal of Circuit Theory and Applications*, vol. 34, no. 4, pp. 471–488, 2006. View at: Publisher Site | Google Scholar - F. Sapuppo, M. Bucolo, M. Intaglietta, L. Fortuna, and P. Arena, “A cellular nonlinear network: real-time technology for the analysis of microfluidic phenomena in blood vessels,”
*Nanotechnology*, vol. 17, no. 4, pp. S54–S63, 2006. View at: Publisher Site | Google Scholar - S. Bhattacharya, A. Singla, D. Dholakiya et al., “Learning active spine behaviors for dynamic and efficient locomotion in quadruped robots,” in
*Proceedings of the 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)*, IEEE, New Delhi, India, October 2019. View at: Publisher Site | Google Scholar - M. Khoramshahi, H. Jalaly Bidgoly, S. Shafiee, A. Asaei, A. J. Ijspeert, and M. Nili Ahmadabadi, “Piecewise linear spine for speed-energy efficiency trade-off in quadruped robots,”
*Robotics and Autonomous Systems*, vol. 61, no. 12, pp. 1350–1359, 2013. View at: Publisher Site | Google Scholar - N. Kohl and P. Stone, “Machine learning for fast quadrupedal locomotion,” in
*Proceedings of the Nineteenth National Conference on Artificial Intelligence (AAAI)*, vol. 4, pp. 611–616, San Francisco, CA, USA, July 2004. View at: Google Scholar - S. Hacohen, S. Shoval, and N. Shvalb, “Applying probability navigation function in dynamic uncertain environments,”
*Robotics and Autonomous Systems*, vol. 87, pp. 237–246, 2017. View at: Publisher Site | Google Scholar - C. Zhou, B. Wang, Q. Zhu, and J. Wu, “An online gait generator for quadruped walking using motor primitives,”
*International Journal of Advanced Robotic Systems*, vol. 13, no. 6, 2016. View at: Publisher Site | Google Scholar - F. Luo, G. Xie, Q. Wang, and L. Wang, “Development and gait analysis of five-bar mechanism implemented quadruped amphibious robot,” in
*Proceedings of the 2010 IEEE/ASME International Conference on Advanced Intelligent Mechatronics*, pp. 633–638, IEEE, Montreal, Canada, July 2010. View at: Publisher Site | Google Scholar - T.-C. Huang, Y.-J. Huang, and W.-C. Lin, “Real-time horse gait synthesis,”
*Computer Animation and Virtual Worlds*, vol. 24, no. 2, pp. 87–95, 2013. View at: Publisher Site | Google Scholar - M. Gen and R. Cheng,
*Genetic Algorithms and Engineering Optimization*, vol. 7, John Wiley & Sons, Hoboken, NJ, USA, 2000. - M. Azarkaman, M. Aghaabbasloo, and M. E. Salehi, “Evaluating GA and PSO evolutionary algorithms for humanoid walk pattern planning,” in
*Proceedings of the 22nd Iranian Conference on Electrical Engineering (ICEE)*, pp. 868–873, IEEE, Tehran, Iran, May 2014. View at: Publisher Site | Google Scholar - A. Masuri, O. Medina, S. Hacohen, and N. Shvalb, “Genetic algorithm for quadrupedal robot walking/running gait,” 2017, https://www.youtube.com/watch?v=sLo64NP0dsA. View at: Google Scholar
- C. Semini, V. Barasuol, J. Goldsmith et al., “Design of the hydraulically actuated, torque-controlled quadruped robot HyQ2Max,”
*IEEE/ASME Transactions on Mechatronics*, vol. 22, no. 2, pp. 635–646, 2017. View at: Publisher Site | Google Scholar

#### Copyright

Copyright © 2020 Ariel Masuri et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.