Teaching robots to improvise (with tools)

Humans are very good at using tools in devious ways. Don't have a spoon? Use a pen to stir your coffee. Missing a bolt to hang a lamp? A rubber band will temporarily do the trick. Need a tray? A book or a digital tablet will do the trick. This ability to improvise is not born "ex nihilo", it's the result of cognitive abilities that enable us to make links between objects, the tools at our disposal, their uses...

Ganesh Gowrishankar, University of Montpellier

The use of tools in robots has hitherto been seen as a problem of exploration and learning: a robot needs to discover how a tool can be used, either by trying out various strategies, or by observing and imitating other humans or robots. In our study published in Nature Machine Intelligence, we showed that robots can be taught to think more creatively, "outside the box".

To enable robots to use tools intuitively like humans, we first examined how we humans are able to do so.

Towards "tool cognition" in robots

A robot needs to pick up a bucket, but the path is blocked by certain obstacles. This robot can't move the obstacles or jump over them. But it was able to look for - and find - a cleaning stick and use it as a tool to pick up the bucket.

This simple use of a tool may seem obvious to a human, but it's a complex challenge for a robot.

Indeed, the use of tools first requires the robot to understand that it cannot perform a task without a tool (1). It then needs to find an object in its environment that it can use as a tool to perform the required task (2). Then he has to figure out how to use this tool, i.e. determine the actions to be performed (3). And finally, of course, to carry out these actions... and thus complete the task (4).

The second and third challenges are fundamental cognitive ones, which humans seem to be very good at... and robots, much less so.

Our new algorithm for "tool cognition" in robots takes a major step forward in meeting these challenges.

How do humans recognize that an object is sufficient to serve as a tool for a given task?

In our work, we have categorized tools according to the way we humans use them. This categorization shows that humans can in fact only intuitively recognize a very specific category of tools.

These "category 1" tools enable us to perform tasks that are already part of the human repertoire. For example, pliers help us grasp objects, but we can already do this with our fingers; a hammer helps us strike objects, which we can already do with our fist.

Using these "category 1" tools requires the same gesture as we would make without tools: pliers need you to pinch, which is similar to what you would do with your fingers; a hammer requires you to make oscillating striking movements, and you would make the same movement without a hammer.

Category 1 tools are probably the most common in human and animal life. The first tools used by humans, stones for striking and breaking fruit and cutting, were Category 1 tools, because humans could perform these tasks, with similar actions, with their fist and fingernails respectively. And indeed, the tools used by animals can be classified as Category 1.

We call "category 2" tools those that enable us to perform tasks that humans can do without tools, but with very different gestures (such as using a car jack to lift a car).

Category 3 tools are used for tasks outside the human repertoire (such as a vacuum cleaner or chainsaw).

These category 2 and 3 tools cannot be used intuitively by humans: you have to read instructions, imitate another user, or explore the tool to find out how to use it.

Tools that humans use intuitively

It is believed in psychology that one of the reasons humans are good at using tools is that tool use leads to an "embodiment" of the tools, i.e. the brain ends up seeing the tools as an "extension" of our body.

In our work on the "embodiment" of the tool and the limb by man, we have observed that the similarity of an object's characteristics with the functional characteristics of the limb is important for its embodiment.

This observation, along with our characterization of tools suggests that to identify Category 1 tools, humans may be using their limbs as a reference: a hammer is recognized as a tool for striking objects because it closely resembles a fist and arm, pliers are recognized as a tool for pinching because they resemble pure fingers, and similarly, a plate resembles a palm and a bowl resembles cupped hands. By identifying and comparing the posture of our limbs when we perform a task, we can identify the tools that can be used for the task performed with the same posture.

Using similarity with its members is the key idea used to program our robot.

Teach robots to recognize tools that resemble what they know

Our algorithm uses this idea to enable robots to recognize objects (even those they are seeing for the first time) as tools for every task they are capable of performing "without tools" (i.e. with their limbs, without additional objects).

Once a robot has learned to perform a task with its limbs, our algorithm enables it to use its limbs as a reference to recognize the tools that help it perform the same task. What's more, by definition, since Category 1 tools require the same action as without tools, the robot can use the same skill ("controller") to use the tool by simply increasing its body kinematics to match the new tool.

The robot doesn't need to have used a single tool before, or even to have observed others using them.

In our proposal, we also provide a grasp planning algorithm for grasping and using the identified tools. Unlike algorithms previously described in the scientific literature, ours is the first to enable robots to use tools without having to learn first with the same or other tools.

However, for the time being, our algorithm only allows robots to use category 1 tools, i.e. tools for tasks that the robot is capable of performing without tools. For category 2 and 3 tools, learning or observation (followed by imitation) is required, as with previous tool use algorithms.

In addition, we can extend the algorithm to the use of tools similar to those already known to the robot: basically, we replace "similarity to a member" with "similarity to a known tool", as suggested by the algorithms previously described.

But as we have seen, Category 1 tools seem to be the most basic tools in humans and animals, and we therefore believe that our algorithm represents a significant step towards tool cognition in robots.

Ganesh Gowrishankar, Researcher at the Montpellier Laboratory of Computer Science, Robotics and Microelectronics, University of Montpellier

This article is republished from The Conversation under a Creative Commons license. Read theoriginal article.