Inter-layer Learning Towards Emergent Cooperative Behavior

Authors: Shawn Arseneau, Wei Sun, Changpeng Zhao, Jeremy R. Cooperstock
Institution: Centre for Intelligent Machines, McGill University
Published: AAAI-00 Proceedings, 2000

---

1. Research Background and Problem

• Complexity of Environments: Real-world dynamic environments (such as the RoboCup soccer simulator) are extremely complex; it is no longer feasible to rely on clever heuristics or hand-tuned behaviors.
• Limitations of Traditional Approaches: Multi-agent approaches (e.g., role-based decision trees, subsumption architectures) depend on hard-coded rules and struggle with uncertainty, oscillatory behavior, and incomplete information.
• Lack of Learning: Existing hierarchical architectures can coordinate agents but lack machine-learning mechanisms and cannot automatically learn inter-layer decision policies.

---

2. Proposed Core Architecture: Inter-layer Learning

The architecture is a hierarchical tree structure built bottom-up, with each successive layer operating at a higher level of abstraction and control:

• Layer 1: Basic Skills – Individually learned low-level actions.
• Layer 2: Individual Strategy – Decides which basic skill to execute at any moment.
• Layer 3: Team Strategy – Coordinates multiple agents’ individual strategies to produce emergent cooperative behavior.

Key Principles

• Strictly Bottom-up: Each layer treats the output of the layer below as its input.
• Minimalism Principle: Uses only the minimal perception information relevant to that layer to avoid information overload.
• Extensibility: Supports supervised learning, reinforcement learning (Q-TD), neural networks, etc., and is extensible to additional layers or sub-teams.

---

3. Layer Implementation Details (RoboCup Soccer Testbed)

Basic Skills Layer

• Techniques used:
  • BPNN (Back-propagation neural networks): For ball interception, shooting, and passing.
  • Q-TD (Reinforcement learning): For dribbling and anticipating passes.
  • Analytical methods: For skills such as marking and clearing.
• Results: Dribbling converged after 400 epochs; passing success rate improved from 77.2% (random) to 83.9%.

Individual Strategy Layer

• Logic: Divides strategies by ball possession (aggressive/auxiliary) and field position (offensive/defensive).
• Implementation: BPNN learns “dribble vs. pass” decisions (converged after 15,228 iterations).

Team Strategy Layer

• Logic: Defines three formations (half/half, offensive, defensive) coordinated via reinforcement learning.
• Evaluation: Uses discretized field representation + pseudo-variance metric to evaluate team distribution.

---

4. Experiments and Results

• Platform: RoboCup soccer simulator (Noda et al., 1998) with limited vision, stamina, oral communication, and sensor noise.
• Competition: Full system integrated and entered the 1999 RoboCup competition (McGill University team).
• Outcome: Achieved a 12:0 victory against opponents.
• Finding: Demonstrated that inter-layer learning produces genuine emergent cooperative behavior rather than purely hand-designed tactics.

---

5. Conclusions and Future Work

• Core Advantage: The architecture supplies an explicit, learnable decision model for both individual and cooperative tactics. It extends machine learning from “learning single tasks” to “learning when to select each task,” greatly improving adaptability.
• Limitation: Sub-task decomposition and reward functions still require human design.
• Future Directions: * Adding more layers.
  • Developing sub-team cooperation.
  • Exploring unsupervised learning.

Summary: The paper shows that inter-layer learning is both feasible and effective for complex multi-agent tasks and represents an early landmark in combining hierarchical architectures with machine learning.