Abstract
Introduction. Development of a robotic arm that can be operated using an exoskeletal position sensing harness as well as a dry electrode brain-computer interface headset. Design priorities comprise an intuitive and immersive user interface, fast and smooth movement, portability, and cost minimization. Materials and Methods. A robotic arm prototype capable of moving along 6 degrees of freedom has been developed, along with an exoskeletal position sensing harness which was used to control it. Commercially available dry electrode BCI headsets were evaluated. A particular headset model has been selected and is currently being integrated into the hybrid system. Results and Discussion. The combined arm-harness system has been successfully tested and met its design targets for speed, smooth movement, and immersive control. Initial tests verify that an operator using the system can perform pick and place tasks following a rather short learning curve. Further evaluation experiments are planned for the integrated BCI-harness hybrid setup. Conclusions. It is possible to design a portable robotic arm interface comparable in size, dexterity, speed, and fluidity to the human arm at relatively low cost. The combined system achieved its design goals for intuitive and immersive robotic control and is currently being further developed into a hybrid BCI system for comparative experiments.
1. Introduction
Brain-computer interfaces (BCIs) are interactive systems that aim at providing users with an alternative way of translating their volition into control of external devices. Their most popular applications lie within the scope of rehabilitation and motor restoration for patients with severe neurological impairment [1]. Although BCI research is currently undergoing a transitional stage of exploratory efforts [2], commercial applications of BCIs are beginning to emerge [3].
The use of brainwaves to control robotic devices has produced promising clinical results in terms of feasibility [4]. Restoration of a certain degree of motor functions [5, 6] and high accuracy control of robotic prosthetic arms using invasive BCIs has already been demonstrated [7]. Nevertheless, in order for such BCI-controlled robotic applications to achieve end-user maturity, the use of noninvasive, portable, and relatively low-cost systems is considered a required development.
Given these recent technological advances, we have focused our research efforts in noninvasive, minimally intrusive, and low-cost BCI. We have designed, partly implemented, and tested an electromechanical robotic system to investigate the capabilities and limitations in combining these technologies for biomedical applications [8]. All components used for the developed system presented in this paper have been designed, implemented, and tested by our research and development team.
Design requirements included fast robotic movement that approximates the natural movement of a human operator’s arm, an intuitive and immersive interface, portability, potential for further development, scalability, and relatively low cost (i.e., less than $3000). We targeted investigative comparative neurophysiological scenarios in which an operator remotely controls a 6-degree-of-freedom (DOF) robotic arm using their arm movement, their brainwaves, or both.
2. Materials and Methods
The hybrid system under development comprises two input devices and a robotic arm. The input devices are an exoskeletal position sensing harness (EPSH) and a commercially available dry electrode BCI headset. The robotic arm is the actuator device in our system and was also developed entirely by the authors. There is no feedback to the human operator in the proof-of-concept prototype described in this system.
The human arm, excluding the hand, possesses the ability to move along 7 DOF: pitch, yaw, and roll at the shoulder joint, pitch and roll at the elbow, pitch and yaw at the wrist. Only three DOF are needed to move the hand at a particular point in space, while the remaining 4 DOF permit humans to approach and grasp objects from different angles.
In order to simplify the design of both the EPSH and the robotic arm, we made the choice to omit wrist yaw, since it does not significantly affect the ability of the robot to manipulate objects. Still, a 7th DOF was added in order to operate the gripper.
All parts were designed using computer-aided design (CAD) software, manufactured using computer numerical control (CNC) and laser cutting machines, and assembled and tested by our team.
2.1. The Exoskeletal Position Sensing Harness
An EPSH was developed as a form of minimally intrusive, intuitive, and immersive interface for the robotic arm. It copies the operator’s arm movement with measurable and repeatable accuracy (Figure 1). Apart from providing research data directly on immersive human computer interfaces, it will also form the basis for future comparative neurophysiological experiments in which a dry electrode BCI headset is evaluated against other forms of immersive robotic remote control.
The harness sensors and accompanying electronics provide real-time data on the position of the human operator’s joints from wrist to shoulder. Hand and finger position is not sensed; however, a finger-operated switch allows the user to operate the robotic arm’s gripper in order to pick up and release items.
Copying of the human operator’s movements is achieved through the EPSH which is worn around their arm. The harness measures the angles between the different parts of the arm and hand. Electronic output originating from the sensing harness is passed on to the robotic arm’s control circuit, which also takes into account feedback output from the robotic arm itself. During the design phase, we, therefore, faced a classic automation closed-loop control challenge: the input is provided by the EPSH worn by the operator, the robotic arm is the recipient of the control output, and a feedback loop takes into account the actual position of the arm.
2.2. The Robotic Arm
The robotic arm is capable of the 6 following types of movement.(i)2 DOF for the “shoulder” joint (“right–left” and “up–down”).(ii)1 DOF for the “elbow” joint.(iii)1 DOF for the “wrist” joint.(iv)2 DOF for rotation between the “shoulder–elbow” and “elbow–wrist” parts.The robotic arm (Figure 2) is also equipped with an electromechanical gripper, which is operated by two servomotors controlled through a separate 7th communication channel.
2.3. The Dry Electrode BCI Headset
The selection of a commercially available BCI headset depended on the number of sensing channels, signal quality, price, and ease of use [9]. The ability of electrodes to perform dry was set as an important requirement. The inconvenience caused by the application of conductive gel to the scalp and the time-consuming preparation routine were considered decisive factors in limiting our selection to dry electrode headsets. While gel-contact electrodes provide better contact and measurement accuracy, the goals of the planned MERCURY comparative experiments are better served by a less precise, less expensive wearable headset.
Furthermore, the ideal BCI headset would need to meet the requirements of multiple data acquisition channels, low weight, and low cost. Frequency-based automatic classification of mental states by the hardware device and the ability to export the raw EEG signal were considered, the former being a strong factor for preference, the latter a decisive requirement. The maximum number of automatically detected mental states was not considered a priority, since this feature can be provided by the signal processing capabilities of our experimental setup. Still, the capacity of a commercial system to automatically detect multiple mental states was considered an indirect indication of the quality and breadth of its sensing capabilities. Thus, it was deemed a desirable—albeit less significant—advantage.
For this reason we examined two low-cost, commercially available headsets (Figure 3), the Emotiv EPOC [3] and NeuroSky MindWave [9].
(a)
(b)
Both devices export raw EEG as well as processed, automatically classified mental state data. Our choice between them depended on their sensing capabilities where the NeuroSky MindWave uses one sensor that can provide only three values: attention, meditation, and eye blinking. The Emotiv EPOC uses a series of 16 sensors, which are capable of detecting specific conscious thoughts, levels of attention, facial expressions, and head movements (the latter using the embedded gyroscope). The sampling frequency of the Emotiv EPOC is 4 times greater than the NeuroSky MindWave making it comparable to more complex and expensive virtual rehabilitation EEG devices [10].
The drawback of both devices is the occasional unreliability of signal quality, primarily associated with the use of dry electrodes. For this reason, the designers of the Emotiv headset suggest that users further improve skin conductance by the moistening of the sensors using a saline solution. While this procedure is not ideal for our purposes, it was considered the least inconvenient among commercially available low-cost solutions.
Despite this drawback, we selected the Emotiv EPOC for use in our hybrid system design, since it integrates the largest number of sensors at the highest sampling rate among all portable low-cost BCI headsets available in the market at the time this paper was submitted.
In the MERCURY hybrid setup, frequency-based EEG data classification is performed both inside the Emotiv EPOC device [11] as well as on the PC accepting raw data, depending on the mode of operation. The PC supporting the MERCURY hybrid system performs frequency-based analysis on selected channels and communicates results to the microcontroller operating the robotic arm, through a digital wired connection. The microcontroller can subsequently choose to control the arm based on incoming data from the EPOC or the exoskeletal sensing harness (Figure 4) or redirect both to a fusion algorithm that produces movement instructions (hybrid interface mode).
3. Results and Discussion
Development of the first two components of the hybrid system, the robotic arm and EPSH, has recently been completed. Validation and characterization tests have been performed in order to measure response times, angular velocity and acceleration, maximum payload, and power consumption (Figure 5).
3.1. Response Time
The average response time of M2, the motor operating within the shoulder joint of the robotic arm, was measured to be 120 ms ± 10. This was measured using an oscilloscope, measuring (10 repetitions average) the drop in current consumption by the motor as soon as the rotor started rolling. This motor is the slowest in the robot, so this measurement is used to formulate worst-case scenario comparisons. The aforementioned value was obtained using an oscilloscope to measure the initial setup time before the motor starts accelerating continuously. Anecdotal evidence from initial tests indicates that this delay is hardly noticeable by human operators.
3.2. Angular Velocity and Acceleration
A typical range of values for the average angular velocity of a human arm has been reported in the literature [12] to be 23°–50°/sec for relaxed, 36°–87°/sec for regular, and 122°–251°/sec for strained quick movement. The equivalent ranges for average angular acceleration were 29°–41°/sec2, 72°–135°/sec2, and 721°–1151°/sec2, respectively. These average values were extracted from a series of experiments involving multiple subjects moving a horizontally rotating handle, a task which combined shoulder, elbow, and wrist movements.
We experimentally measured the average angular velocity of motors M2 (shoulder) and M4 (elbow) of our robotic arm throughout their full range of motion, 180° and 150°, respectively [12]. These two motors were selected since they are known to be the slowest in the robotic arm, carry the most weight, and consume the largest share of power.
The experiments were set up so that the robotic arm be in an upwards movement (impeded by gravity) and were performed twice, with and without an additional 50 gr load. The robot input was to move as fast as possible, and multiple measurements were made in order to obtain average values. Results are summarised in Table 1.
Even though the experimental results are not directly comparable, focusing on the slowest response times of the robot—a deliberately pessimistic scenario—leads to some useful conclusions. The robotic arm (i)is capable of combining speed and acceleration that exceed the typical range of regular human arm movements [13], even when impeded by gravity and carrying a small load,(ii)is capable of accelerating faster than the human arm,(iii)has an average speed reduction of less than 6% when carrying a small 50 gr load (averaged across the full range for motion for any joint),(iv)has a qualitative attribute, not quantified yet, informally indicated by testing engineers: motion is smooth and the control is immersive and “feels natural”;(v)combined with the EPSH and the BCI headset, the entire integrated device is portable and suitable for carriage by a single person.Complete qualitative assessment studies are planned in the immediate future, prior to the comparative experiments mentioned in following sections.
3.3. Dimensions, Payload, and Power Consumption
The EPSH measures 21 × 21.4 × 69 cm and weighs 3 kg. Including its wooden base, the robotic arm weighs 5.9 kg. When extended vertically, the robotic arm measures 25 × 73.9 × 30.2 cm. The moving part of the arm measures 25 × 12.5 × 46 cm. Its working space is approximately a hemisphere with a radius equal to its reach (46 cm), approximately 60% the reach of an adult human arm [14].
The robotic arm prototype has a rated maximum payload of 300 gr. The maximum payload is 750 gr. These figures are a direct consequence of striving to maintain low development cost and can be dramatically improved in future prototypes. No tests were performed with loads greater than 750 gr in order to prevent damage to the prototype.
The robotic arm is powered by a 24 V DC power supply. The peak current is 3.5 A ± 0.1, and the average power consumption is 25.3 W ± 0.1. All measurements were made with gravity impeding movement without additional load.
4. Future Work
With respect to electromechanics and electronics, the next development steps involve integrating the selected dry electrode headset with the EPSH and robotic arm into a hybrid device. A PC will be capable of recording experimental sessions in which the operator uses the headset, the harness, or both to control the robotic arm. Once experiments are concluded with the hybrid setup, there are further plans to introduce a feedback loop so that the operator gains a tactile feeling of resistance when the robotic arm touches an object or obstacle.
With respect to software development, MATLAB-based (the MathWorks Inc., Natick, Massachusetts, USA) software code will process the output of the BCI headset, evaluate it against precise motion data captured by the harness and arm, and draw comparative conclusions. Our team is interested in comparing the BCI output to that of a conventional 10/20 EEG data acquisition system as well as to the output of the EPSH.
4.1. Comparative Experiments
The novelty of our experimental setup is the ability to capture all 6 degrees of freedom of the human arm’s physical movement electromechanically, at low cost, with relative accuracy, and in real time. Furthermore, we plan to proceed with comparing our results to those acquired through the use of both a commercial BCI headset and a more conventional multichannel EEG-BCI paradigm, while the subject actually performs motor execution tasks. It is also among our research plans to comparatively investigate the role of virtual reality (VR) in controlling a virtual prosthetic arm versus controlling an actual robotic arm.
A series of neurophysiological experiments involve comparison of brain activation results between scenarios of motor execution (ME) versus motor imagery (MI) [15] and an unrelated control scenario. We plan to compare brain activation maps during ME tasks performed with the EPSH of the robotic arm, as well as the dry electrode headset BCI versus similar maps acquired during MI tasks performed with the BCI system.
The role of functional connectivity (FCN) of the brain in the fluid motions of the arm is also within our scope of experimentation. We wish to explore whether graph analysis of FCN during ME and MI of the arm can contribute towards the goal of making BCI systems more intuitive, easy to learn and easy to use.
A hybrid man-machine interface (MMI) and brain-computer interface (BCI) systems offer numerous investigative research advantages. Fluid and intuitive control of a prosthetic robotic arm using BCI is yet to be demonstrated, due to limitations of current BCI data acquisition and classification technologies. Those limitations could be addressed using the added benefit the MMI-control parameter for the robotic arm and artificial intelligence classifiers. The main question to be answered, as far as this hybrid system is concerned, can be identified in whether the unintuitive, hard-to-adjust-to and limited in functionality BCI systems which are currently commercially available can benefit from this approach. Are BCIs after all destined to be exclusively used for research and clinical purposes or could they evolve into a mature mainstream technology?
5. Conclusions
We have developed an intuitively controlled 6 DOF robotic arm and accompanying operator’s sensing harness, satisfying the design requirements for an immersive, hybrid robotic control system. The first proof-of-concept prototype has been developed, evaluated, and deemed adequate for the next development step: integration with a commercial dry-electrode BCI headset. The intended research objectives for this system include BCI optimization through comparative experiments, using the motion and position sensing harness, the dry electrode BCI headset, and a combination of both. The ultimate research goals are to better understand the function of the motor cortex, improve neurofeedback training for people suffering from neurological disorders, and optimize robotic prosthetics.
Acknowledgments
The research leading to these results has received funding from the European Union’s Seventh Framework Programme (FP7/2007-2013) under grant agreement no 288532. For more details, please see http://www.usefil.eu/.