Adaptive Behavior, 5 (3/4)

Instrumental (or operant) conditioning, a form of animal learning, is similar to reinforcement learning (Watkins, 1989) in that it allows an agent to adapt its actions to gain maximally from the environment while being rewarded only for correct performance. However, animals learn much more complicated behaviors through instrumental conditioning than robots presently acquire through reinforcement learning. We describe a new computational model of the conditioning process that attempts to capture some of the aspects that are missing from simple reinforcement learning: conditioned reinforcers, shifting reinforcement contingencies, explicit action sequencing, and state space refinement. We apply our model to a task commonly used to study working memory in rats and monkeys--the delayed match-to-sample task. Animals learn this task in stages. In simulation, our model also acquires the task in stages, in a similar manner. We have used the model to train an RWI B21 robot.

Key Words

operant conditioning; instrumental learning; shaping; chaining; learning robots

Pages 249-280

Transfer of Elementary Skills via Human-Robot Interaction

By Michael Kaiser

Abstract

Transferring elementary skills to robots by means of demonstrations is a very intuitive approach to robot programming. Within this article, an analysis of the process of skill transfer and skill acquisition is provided that explicitly considers the nature of elementary skills and the role of the human user, who acts as a teacher. A process model for skill acquisition is developed, and methods and algorithms supporting the several phases of this process are presented. The whole approach is exemplified by means of manipulation- and navigation-related skills. Finally, the strengths and limitations of the demonstration-based approach in general are discussed.

Key Words

skill acquisition; robot learning; human-robot interaction; machine learning; robotics

Pages 281-315

Anytime Learning and Adaptation of Structured Fuzzy Behaviors

By Andrea Bonarini

Abstract

We present an approach to support effective learning and adaptation of behaviors for autonomous agents with reinforcement learning algorithms. These methods can identify control systems that optimize a reinforcement program, which is, usually, a straightforward representation of the designer's goals. Reinforcement learning algorithms usually are too slow to be applied in real time on embodied agents, although they provide a suitable way to represent the desired behavior. We have tackled three aspects of this problem: the speed of the algorithm, the learning procedure, and the control system architecture. The learning algorithm we have developed includes features to speed up learning, such as niche-based learning, and a representation of the control modules in terms of fuzzy rules that reduces the search space and improves robustness to noisy data. Our learning procedure exploits methodologies such as learning from easy missions and transfer of policy from simpler environments to the more complex. The architecture of our control system is layered and modular, so that each module has a low complexity and can be learned in a short time. The composition of the actions proposed by the modules is either learned or predefined. Finally, we adopt an anytime learning approach to improve the quality of the control system on-line and to adapt it to dynamic environments.

The experiments we present in this article concern learning to reach another moving agent in a real, dynamic environment that includes nontrivial situations such as that in which the moving target is faster than the agent and that in which the target is hidden by obstacles.

Key Words

reinforcement learning, autonomous agents; fuzzy control; hierarchical behaviors; anytime learning; anytime algorithms

Pages 317-342

Incremental Evolution of Complex General Behavior

By Faustino Gomez and Risto Miikkulainen

Abstract

Several researchers have demonstrated how complex action sequences can be learned through neuroevolution (i.e., evolving neural networks with genetic algorithms). However, complex general behavior such as evading predators or avoiding obstacles, which is not tied to specific environments, turns out to be very difficult to evolve. Often the system discovers mechanical strategies, such as moving back and forth, that help the agent cope but are not very effective, do not appear believable, and do not generalize to new environments. The problem is that a general strategy is too difficult for the evolution system to discover directly. This article proposes an approach wherein such complex general behavior is learned incrementally, by starting with simpler behavior and gradually making the task more challenging and general. The task transitions are implemented through successive stages of Delta coding (i.e., evolving modifications), which allows even converged populations to adapt to the new task. The method is tested in the stochastic, dynamic task of prey capture and is compared with direct evolution. The incremental approach evolves more effective and more general behavior and should also scale up to harder tasks.

Key Words

neuroevolution; incremental evolution; shaping; knowledge transfer; pursuit and evasion; stochastic environments

Pages 343-363

Using Emergent Modularity to Develop Control Systems for Mobile Robots

By Stefano Nolfi

Abstract

A new way of building control systems, known as behavior-based robotics, has recently been proposed to overcome the difficulties of the traditional artificial intelligence approach to robotics. This new approach is based on the idea of providing the robot with a range of simple behaviors and letting the environment determine which behavior should have control at any given time. We will present a set of experiments in which neural networks with different architectures have been trained to control a mobile robot designed to keep an arena clear by picking up trash objects and releasing them outside the arena. Controller weights are selected using a form of genetic algorithm and do not change during the lifetime (i.e., no learning occurs). We will compare, in simulation and on a real robot, five different network architectures and will show that a network that allows for fine-grained modularity achieves significantly better performance. By comparing the functionality of each network module and its interaction with a description of the simple behavior components, we will show that it is not possible to find simple correlations; rather, module switching and interaction are correlated with low-level sensorimotor mappings. This implies that the engineering-oriented approach to behavior-based robotics might have serious limitations because it is difficult to know in advance the appropriate mappings between behavior components and sensorimotor activity for complex tasks.

Key Words

autonomous robots; behavior-based robotics; modularity; neural networks; genetic algorithms

Pages 365-390

Measuring the Effectiveness of Reinforcement Learning for Behavior-Based Robots

By John Shackleton and Maria Gini

Abstract

We explore the use of behavior-based architectures within the context of reinforcement learning and examine the effects of using different behavior-based architectures on the ability to learn correctly and efficiently the task at hand. In particular, we study the task of learning to push boxes in a simulated two-dimensional environment originally proposed by Mahadevan and Connell (1992). We examine issues such as effectiveness of learning, flexibility of the learning method to adapt to new environments, and effect of the behavior architecture on the ability to learn, and we report results obtained on a large number of simulation runs.

Key Words

reinforcement learning; behavior-based architectures; robot learning

Pages 391-405

Précis of Robot Shaping: An Experiment in Behavior Engineering

By Marco Dorigo and Marco Colombetti

Pages 407-420

Engineering Adaptive Behavior

By Dario Floreano

Pages 417-420

Reply to Dario Floreano's "Engineering Adaptive Behavior"

By Marco Dorigo and Marco Colombetti

Pages 421-423

Author Index to Volume 5

Pages 425-429

Key Word Index to Volume 5

back to TOC, back to top

Adaptive Behavior, 5 (3/4)

Adaptive Behavior

Volume 5, Number 3-4

Winter/Spring 1997

Table of Contents

Maja Mataric´

Introduction to the Special Issue: Environment, Structure, and Behavior

David S. Touretzky and Lisa M. Saksida

Operant Conditioning in Skinnerbots