From mind to movement

A robot arm moves back and forth in a hangar at the Faculty of Engineering. It passes a cup of orange juice to a test subject, directed by the power of thought alone. Behind the manoeuvre - top-level research from the BrainLinks-BrainTools Excellence Cluster which could make life much easier for people with physical disabilities.

Test participant Nicolas Riesterer takes a sip of orange juice from a cup and smiles. He has good reason to grin - the experiment has been a complete success. He didn't have to lift a finger to enjoy the drink. Riesterer's only effort was in thinking. After the electrical pulses in his brain were read and transformed by a computer interface into machine commands, robots looked for the orange juice independently, poured it into a cup, and lifted the cup to his mouth. All this while, he sat still in his chair.

The experiments are conducted in a hangar belonging to the Faculty of Engineering; decades ago aircraft were stored here. The scene could have been lifted from a science fiction movie. Banks of computers, laser cameras, enormous robot arms, strange-looking vehicles. It is all equipment used in various robotics research projects. One of those projects is NeuroBots from the University of Freiburg's BrainLinks-BrainTools Excellence Cluster, headed by robotics specialist, Professor Wolfram Burgard. NeuroBots brings together experts from the fields of medicine, biology, and informatics to develop machines which can act autonomously and be steered by human thought. "Our goal is to develop an assistance system for paralyzed and otherwise disabled people, which will still be robust under various forms of daily use, says Burgard. Unlike protheses, which are guided by motor signals, the Freiburg team uses an extended procedure based on human imagination.

Who's been drinking from my cup? Using mindpower, Nico Riesterer guides the robot arm which is lifting a cup to his mouth.
Photo: Mathilde Bessert-Nettelbeck

Put simply, the current procedure goes like this: mental images of the movement of a finger or hand are recorded using an electroencephalogram; they are then decoded by special software and "translated" into language the computer can understand. The test subject first decides which object he or she is interested in. Then he or she gives the order to "approach," "grasp," and "bring." A planning module breaks the command down into a logical progression of single steps which can be physically taken and relays them to the machines. Within seconds, the mobile robot omniRob goes into action, fetching the drink. It knows where it has to go, finds its way around its vicinity without help, recognizes the bottle, and takes it to its "colleague," the robot arm LBR iiwa - the type of robot we see on production lines. The robot arm grasps the bottle, pours the drink into a cup, and holds it out for the test person to drink. The interaction between the machines, their surroundings, and the human test subject appears delicate, sometimes even artistic. Riesterer is not the only one smiling. The researchers involved in the experiment also look happy. "Once the setup has been installed, the demonstration can go ahead more or less smoothly. But you must remember that we were years preparing for this day," says computer scientist Dr. Joschka Bödecker, who coordinated the experiment jointly with the neurologist Dr. Tonio Ball.

Drinking with the help of a robot arm – this video shows how the experiment works:

f90729ac4a1ba1307377caf8244e70d5

The eleven-person team is working on a variety of different problems, covering a number of core issues and processes in artificial intelligence. For instance, the electroencephalogram provides a flood of information from the test subject's brain; the researchers have to find a way of filtering out precisely the information which corresponds to the right commands. One of the biggest challenges is enabling omniRob to manage unfamiliar objects in unfamiliar surroundings. The robot arm, too, is complicated. The researchers have to use an algorithm to train it to make human-like movements so that the juice is poured evenly and without spilling out of the bottle and into the cup. And it has to make especially gently movements when it puts the cup to the test subject's mouth.

Despite a number of successful experimental setups, NeuroBots are still taking baby steps. "Not until the system is able to deal with the unpredictability of real life can we use it with real patients," says Burgard.

The project comes to a close at the end of 2018; if it gets new funding from the excellence initiative, it could continue until 2025. But it is not likely that Riesterer will still be a test subject in the experiments by then. Following completion of his Master's degree in Cognitive Science, he will soon begin his doctoral studies at the Freiburg Institute of Informatics - and may soon be one of the NeuroBots research team.

Levin Sottru