Tony H. Zhang , Connor Mattson , Daniel S. Brown
University of Utah
Demonstrations are an effective way to accelerate single-agent reinforcement learning (RL), but extending this idea to multi-agent reinforcement learning (MARL) is non-trivial as it is difficult and often impossible for a single person to teleoperate multiple agents simultaneously. Round-Robin Behavior Cloning (R2BC) addresses this by letting a single demonstrator control one agent at a time while the others execute their current policies. However, the resulting imitation policies are limited by the quality of the demonstrations.
We propose Imitation Bootstrapped Multi-Agent Reinforcement Learning (IBMARL), a framework that extends Imitation Bootstrapped Reinforcement Learning (IBRL) to cooperative teams and uses R2BC to train the imitation policies. At each time step, every agent receives one action proposal from its imitation policy and one from its RL policy, and an ensemble of centralized critics selects among the per-agent combinations of the two. A gated imitation penalty, a soft Q-filtered behavior cloning loss, regularizes each RL policy toward its imitation policy only where the critics value the imitation action more, so that after training each agent executes its RL policy alone, without the centralized critics.
On four cooperative multi-agent RL benchmarks, IBMARL surpasses its imitation policies 2.6–4.2× faster than demonstration-based baselines on two tasks, is the only method to surpass it on the other two, and exceeds the demonstrator itself on two of the four. With real human demonstrations IBMARL still surpasses imitation learning alone, and does so more than twice as fast as the best baseline.
The paper’s Fig. 1, step by step: teach once, then act and learn in a loop.
Swipe sideways to see the whole figure.
The task. Three robots must push a heavy package onto the goal; one robot alone moves it too slowly. Step through the figure, or press Play. 1. Teach. A human teleoperates one robot at a time while the other robots run their current IL policies (Round-Robin Behavior Cloning). The IL policies are retrained after every round; the result is a frozen IL policy for each robot. 1 Propose. Each robot’s IL policy and RL policy each propose an action from that robot’s own observation. From here on only the RL policies and the critics are trained. 2 Choose. Centralized critic ensembles score every per-robot mix of IL and RL proposals (2N joint candidates) and choose, per robot, which one the team executes. Here robots 1 and 3 follow their IL policy and robot 2 its RL policy. 3 Store. The executed step (observations, actions, rewards) goes into a replay buffer that was pre-loaded with the demonstrations from Teach (blue band). 4 Update critics. The critics are trained on replayed experience, bootstrapping through the same IL-or-RL choice at the next step, so they learn the value of the better of the two policies. 5 Update RL policies. Each RL policy climbs the critics’ value. A gate adds an imitation penalty that pulls it toward its IL action only where the critics prefer IL; where they prefer the RL action, the penalty switches off. Deploy. At deployment the critics, the IL policies and the replay buffer are dropped: each robot runs its RL policy alone on its own observation, with no communication.
IBMARL generalizes IBRL to cooperative teams under centralized training with decentralized execution.
Swipe sideways to see the whole figure.
Imitation Bootstrapped RL (IBRL, Hu et al.) is a single-agent actor-critic method that keeps a behavior-cloned IL policy μψ in the loop. When acting, both policies propose an action and the target critic picks the better one, and the same choice is made at the next observation inside the TD target, so the critic estimates the value of the better of the two policies:
The IL policy only supplies an action for the critic to pick or reject; it never enters the actor’s loss. And because the propose-and-select step is also used at test time, IBRL needs the critic and the IL policy at deployment, which for a team would mean centralized execution.
Round-Robin Behavior Cloning (R2BC) learns a cooperative policy for N agents from one demonstrator who controls a single agent at a time. In each round the demonstrator drives one agent while every other agent executes its current learned policy, and control then passes round-robin to the next. Because teammates run their current policies, the demonstrator reacts to the team as it actually behaves, which is what lets single-agent demonstrations produce cooperative behavior. R2BC inherits the limits of behavior cloning, chiefly a ceiling at the demonstrator’s performance.
Each agent forms two noised candidate actions, one from its RL actor and one from its IL policy. The arbiter chooses among the product of the per-agent choices, so it can follow the IL policy for one agent and the RL policy for another:
An ensemble of centralized critics scores each of the 2N candidates; following REDQ, a random subset K of the target critics is used, and each term takes the minimum over it:
The team executes a joint action sampled from a Boltzmann distribution over the candidates (evaluation takes the argmax). Mixed candidates let the team keep a reliable IL action for an agent whose RL actor is still poor while another agent, whose actor has improved, acts on its own. They also give the critics counterfactual information: candidates differ in a single agent’s action, so their scores isolate each agent’s contribution at the current state. The critics are trained with a TD(0) target that bootstraps through the same arbiter:
The actors follow the deterministic policy gradient through the centralized critics. That gradient only climbs the critics locally around the actors’ own actions: when the arbiter executes IL actions, their successes raise the critics’ value of those actions, but nothing moves the actors toward them, so the RL actors lag behind the centralized policy that is actually executed. IBMARL adds an imitation term to the actor loss, scaled by a gate that is the critics’ own preference for the IL policy:
Here Q̂ is the detached minibatch mean of |Qtot|, so α is dimensionless and transfers across tasks with different return scales. The term clones the IL action only where the critics still prefer it and switches itself off, state by state, as the actor overtakes the IL policy. It needs no annealing schedule, and it acts on the states the learner visits rather than on the demonstration set. It is a soft version of the Q-filter of Nair et al., recovering the hard filter as βg → 0. The gate uses the same target critics and scoring as the arbiter, so the actors are trained to reproduce the actions the arbiter would execute, and after training each agent can run its actor alone.
Four cooperative tasks from the Vectorized Multi-Agent Simulator (VMAS), with sparse rewards.
Navigation N = 3 agents
Agents and their assigned goals spawn at random in a bounded 2-D workspace. Each agent observes its own state, its goal’s relative position, and a LiDAR scan of its teammates. Sparse reward: −1 per step spent outside the goal region.
Buzzwire N = 2 agents
Two agents connected to a ball must thread it through a narrow corridor to a goal without touching the walls. Each observes its own position, velocity and relative position to the goal. Sparse reward on the ball’s distance to the goal, shared by both agents.
Balance N = 3 agents
Three agents support a rigid beam to lift a spherical package resting on top to a goal, without the package or beam touching the floor. One agent cannot lift the beam. Binary reward: 1 on the step the package reaches the goal, which ends the episode.
Transport N = 3 agents
Three agents collaborate to push a package onto a goal. One agent alone moves it too slowly to reach the goal within the horizon. Keeps VMAS’s native reward, the per-step decrease of the package’s distance to the goal: a hard exploration signal.
Agents act through continuous velocity commands over a fixed horizon. The native VMAS rewards of Navigation, Balance and Buzzwire are dense, which poorly reflects real-world reward signals, so the paper replaces them with sparse variants. On each task the IL policy is the decentralized policy of one R2BC session (one small MLP per agent), trained from demonstrations given by a synthetic demonstrator: a PPO policy on Navigation, a MAPPO policy on Balance, and scripted controllers on Buzzwire and Transport. The experts are far from perfect: the Transport expert delivers the package in 28% of its episodes and the Balance expert lifts it to the goal in 37%.
Synthetic demonstrations, five seeds per method. Every method is evaluated with its RL actors alone.
2.6–4.2×
fewer environment steps than RLfD and RFT to surpass the IL policy on Navigation (47k against 122k and 198k)
2.8×
fewer steps on Buzzwire: 104k against 296k for both RLfD and RFT
2 of 4
tasks where IBMARL’s actors end above the demonstrator: Buzzwire (1.02) and Balance (1.14), in normalized return
Only one
to surpass the IL policy on Balance and Transport; the three baselines never do within the budget
Swipe sideways to see the whole figure.
| Method | Navigation | Buzzwire | Balance | Transport | ||||
|---|---|---|---|---|---|---|---|---|
| steps | final | steps | final | steps | final | steps | final | |
| R2BC (IL policy) | – | −48.7 ± 1.2 | – | −137.5 ± 1.8 | – | 0.287 ± 0.048 | – | 15.7 ± 1.0 |
| Expert (demonstrator) | – | −17.9 ± 0.4 | – | −71.2 ± 1.4 | – | 0.372 ± 0.009 | – | 37.2 ± 0.4 |
| MADDPG | 3/5 | −43.9 ± 6.7 | never | −194.6 ± 2.5 | never | 0.001 | never | −0.05 ± 0.01 |
| RLfD | 122k | −19.6 ± 0.5 | 296k | −85.9 ± 8.1 | never | 0.000 | never | −0.06 ± 0.10 |
| RFT | 198k | −35.0 ± 4.9 | 296k | −111.2 ± 4.9 | never | 0.014 ± 0.014 | never | 0.35 ± 0.09 |
| IBMARL | 47k | −20.9 ± 0.8 | 104k | −68.9 ± 0.7 | 2.33M | 0.423 ± 0.017 | 2.63M | 21.3 ± 0.3 |
| IBMARL (centralized) | – | −21.4 ± 0.9 | – | −68.8 ± 1.2 | – | 0.425 ± 0.012 | – | 22.7 ± 0.6 |
Table 1. Environment steps until the RL actors surpass the IL policy (median over 5 seeds; a fraction gives the seeds that ever surpass it) and final return (mean ± s.e.m. over 5 seeds of each seed’s last 10 evaluations). Returns are in each task’s own units: higher is better, and Navigation and Buzzwire returns are negative.
Swipe sideways to see the whole figure.
| Final return | Navigation | Buzzwire | Balance | Transport |
|---|---|---|---|---|
| IBMARL | −20.9 ± 0.8 | −68.9 ± 0.7 | 0.423 ± 0.017 | 21.3 ± 0.3 |
| centralized | −21.4 ± 0.9 | −68.8 ± 1.2 | 0.425 ± 0.012 | 22.7 ± 0.6 |
| No penalty | −19.4 ± 0.5 | −111.6 ± 24.0 | 0.333 ± 0.015 | 17.3 ± 1.0 |
| centralized | −19.8 ± 0.3 | −81.6 ± 6.2 | 0.347 ± 0.012 | 19.2 ± 1.4 |
| No mixing | −21.5 ± 0.8 | −71.2 ± 3.0 | 0.422 ± 0.022 | 15.3 ± 0.6 |
Table 2. Ablations: final return (mean ± s.e.m., 5 seeds) of the RL actors and, where it differs, of the centralized policy.
Navigation is the easiest task: goal regions are large enough that undirected exploration eventually finds them. IBMARL’s actors surpass the IL policy after a median of 47k steps, against 122k for RLfD and 198k for RFT, while MADDPG reaches it in only 3 of 5 seeds. Given the full budget, RLfD ends slightly above IBMARL (−19.6 ± 0.5 against −20.9 ± 0.8), both within 0.04 of the expert in normalized return.
Buzzwire is considerably harder: random joint exploration essentially never observes a reward, and MADDPG never reaches the goal in any seed. IBMARL’s actors surpass the IL policy after 104k steps and finish at −68.9 ± 0.7, against −85.9 ± 8.1 for RLfD and −111.2 ± 4.9 for RFT. IBMARL is also the first to reach the expert’s level, after 264k steps in every seed. RLfD takes 552k steps in 4 of 5 seeds and RFT reaches it in 1 of 5.
Balance: none of the baselines learns the task within 6.75M steps. A team that cannot already lift together drops the package within a few steps and never sees a reward. IBMARL’s actors surpass the IL policy (0.287) after 2.33M steps, reach the expert’s level (0.372) after 3.19M steps in every seed, and end at 0.423 ± 0.017, a success rate of 42%.
Transport: within 4M steps none of the baselines improves beyond its initial performance. IBMARL’s actors surpass the IL policy (15.7) after 2.63M steps and end at 21.3 ± 0.3. Transport is the only task with a large remaining gap to the expert (normalized return 0.57).
The gated imitation term. On Navigation it is neutral. On the three harder tasks it is decisive, in two different ways. On Buzzwire it closes the actor lag: without it the centralized policy still crosses the IL level (ending at −81.6 ± 6.2), but the RL actors alone cross in only 4 of 5 seeds and end at −111.6 ± 24.0, below RLfD. With the term the actor and centralized curves coincide, so the critics and the IL policy can be discarded at deployment. On Balance and Transport the term improves the centralized policy itself, not only the actors.
Per-agent mixing is neutral on Navigation, Buzzwire and Balance and decisive on Transport, where No mixing ends at 15.3 ± 0.6, the IL policy’s own level, against 21.3 ± 0.3 for IBMARL. On Transport the IL policy is often right for some agents and wrong for others at the same state, so the arbiter gains from keeping the useful parts of the IL action. On Balance the IL policy tends to be right or wrong for the whole team at once, so the pure candidates suffice.
Every clip is a trained policy playing five start states, the same five for every policy of a task, with the return each episode earned.
The R2BC imitation policy everything starts from, three baselines, and IBMARL. The RL methods act with their RL actors alone.
Task
Show
Navigation. The return is −1 for every step an agent is outside its goal region, averaged over the three agents, so a higher return means reaching the goals sooner. The worst possible is −100, the length of an episode.
R2BC
Behavior-cloned from the round-robin demonstrations. Needs no RL.
Returns in this clip: −39.5, −75.9, −31.7, −70.4, −33.1
Mean over 100 start states: −50.0
MADDPG
RL only. Uses no demonstrations.
Returns in this clip: −48.0, −56.4, −46.4, −45.2, −74.0
Mean over 100 start states: −54.3
RLfD
Demonstrations in a separate replay buffer, mixed in on an annealed schedule.
Returns in this clip: −24.7, −27.1, −15.8, −17.6, −20.3
Mean over 100 start states: −20.9
RFT
Behavior-cloning regularizer on demonstration states, annealed.
Returns in this clip: −45.6, −69.3, −16.5, −28.9, −25.6
Mean over 100 start states: −37.5
IBMARL
Critics pick between IL and RL proposals; gated imitation term. RL actors only.
Returns in this clip: −24.5, −38.9, −16.0, −17.9, −21.1
Mean over 100 start states: −24.4
Buzzwire. The return is −1 for every step the ball is outside the goal region, so a higher return means threading the corridor to the goal sooner. The worst possible is −200, the length of an episode.
R2BC
Behavior-cloned from the round-robin demonstrations. Needs no RL.
Returns in this clip: −158.0, −94.0, −119.9, −169.0, −146.0
Mean over 100 start states: −137.2
MADDPG
RL only. Uses no demonstrations.
Returns in this clip: −184.0, −200.0, −200.0, −200.0, −200.0
Mean over 100 start states: −195.5
RLfD
Demonstrations in a separate replay buffer, mixed in on an annealed schedule.
Returns in this clip: −22.9, −83.0, −103.9, −72.9, −60.9
Mean over 100 start states: −73.7
RFT
Behavior-cloning regularizer on demonstration states, annealed.
Returns in this clip: −28.9, −96.9, −200.0, −73.9, −61.9
Mean over 100 start states: −92.3
IBMARL
Critics pick between IL and RL proposals; gated imitation term. RL actors only.
Returns in this clip: −21.0, −83.0, −102.9, −74.0, −61.0
Mean over 100 start states: −67.9
Balance. The return is 1 if the package is lifted to the goal and 0 otherwise, so the mean return is the fraction of start states solved.
R2BC
Behavior-cloned from the round-robin demonstrations. Needs no RL.
Returns in this clip: 0, 0, 0, 1, 1
Mean over 100 start states: 0.31
MADDPG
RL only. Uses no demonstrations.
Returns in this clip: 0, 0, 0, 0, 0
Mean over 100 start states: 0.00
RLfD
Demonstrations in a separate replay buffer, mixed in on an annealed schedule.
Returns in this clip: 0, 0, 0, 0, 0
Mean over 100 start states: 0.00
RFT
Behavior-cloning regularizer on demonstration states, annealed.
Returns in this clip: 0, 0, 0, 0, 0
Mean over 100 start states: 0.00
IBMARL
Critics pick between IL and RL proposals; gated imitation term. RL actors only.
Returns in this clip: 0, 0, 0, 1, 1
Mean over 100 start states: 0.42
Transport. The return is the package’s total progress toward the goal (VMAS’s native reward) over an episode of 500 steps. Higher is better.
R2BC
Behavior-cloned from the round-robin demonstrations. Needs no RL.
Returns in this clip: −18.5, 33.3, 40.4, 18.1, 4.9
Mean over 100 start states: 15.5
MADDPG
RL only. Uses no demonstrations.
Returns in this clip: 0.0, 0.0, 0.0, 0.0, 0.0
Mean over 100 start states: 0.1
RLfD
Demonstrations in a separate replay buffer, mixed in on an annealed schedule.
Returns in this clip: −0.7, 0.0, −0.3, 0.0, 0.0
Mean over 100 start states: −0.1
RFT
Behavior-cloning regularizer on demonstration states, annealed.
Returns in this clip: 0.3, 1.1, −0.5, 0.0, 0.0
Mean over 100 start states: 0.2
IBMARL
Critics pick between IL and RL proposals; gated imitation term. RL actors only.
Returns in this clip: 29.9, 5.0, 43.8, 12.4, 17.4
Mean over 100 start states: 22.0
No policy is selected.
IBMARL with one ingredient removed, and IBMARL’s centralized policy. In the centralized clip a ring shows who acts: orange for the RL actor, blue for the IL policy.
Task
Show
Navigation. The return is −1 for every step an agent is outside its goal region, averaged over the three agents, so a higher return means reaching the goals sooner. The worst possible is −100, the length of an episode.
IBMARL
Critics pick between IL and RL proposals; gated imitation term. RL actors only.
Returns in this clip: −24.5, −38.9, −16.0, −17.9, −21.1
Mean over 100 start states: −24.4
No penalty
IBMARL without the gated imitation term (IBRL’s objective).
Returns in this clip: −24.2, −24.6, −16.0, −17.4, −20.1
Mean over 100 start states: −19.7
No mixing
IBRL’s all-IL or all-RL arbiter applied to the team.
Returns in this clip: −23.4, −24.1, −20.1, −18.7, −20.0
Mean over 100 start states: −21.3
IBMARL (centralized)
IBMARL’s arbitrated policy; needs the critics, so training-time only.
Returns in this clip: −24.8, −41.4, −15.8, −20.3, −21.1
Mean over 100 start states: −24.3
Buzzwire. The return is −1 for every step the ball is outside the goal region, so a higher return means threading the corridor to the goal sooner. The worst possible is −200, the length of an episode.
IBMARL
Critics pick between IL and RL proposals; gated imitation term. RL actors only.
Returns in this clip: −21.0, −83.0, −102.9, −74.0, −61.0
Mean over 100 start states: −67.9
No penalty
IBMARL without the gated imitation term (IBRL’s objective).
Returns in this clip: −19.0, −82.0, −200.0, −72.0, −60.0
Mean over 100 start states: −81.7
No mixing
IBRL’s all-IL or all-RL arbiter applied to the team.
Returns in this clip: −21.0, −83.0, −104.0, −74.0, −61.0
Mean over 100 start states: −69.2
IBMARL (centralized)
IBMARL’s arbitrated policy; needs the critics, so training-time only.
Returns in this clip: −21.0, −83.0, −102.9, −74.0, −61.0
Mean over 100 start states: −68.2
Balance. The return is 1 if the package is lifted to the goal and 0 otherwise, so the mean return is the fraction of start states solved. This task has no centralized clip because its runs did not keep critic checkpoints.
IBMARL
Critics pick between IL and RL proposals; gated imitation term. RL actors only.
Returns in this clip: 0, 0, 0, 1, 1
Mean over 100 start states: 0.42
No penalty
IBMARL without the gated imitation term (IBRL’s objective).
Returns in this clip: 0, 0, 0, 1, 1
Mean over 100 start states: 0.44
No mixing
IBRL’s all-IL or all-RL arbiter applied to the team.
Returns in this clip: 0, 0, 0, 1, 1
Mean over 100 start states: 0.44
Transport. The return is the package’s total progress toward the goal (VMAS’s native reward) over an episode of 500 steps. Higher is better.
IBMARL
Critics pick between IL and RL proposals; gated imitation term. RL actors only.
Returns in this clip: 29.9, 5.0, 43.8, 12.4, 17.4
Mean over 100 start states: 22.0
No penalty
IBMARL without the gated imitation term (IBRL’s objective).
Returns in this clip: 16.1, 19.6, 16.2, 12.2, 30.2
Mean over 100 start states: 18.9
No mixing
IBRL’s all-IL or all-RL arbiter applied to the team.
Returns in this clip: −5.9, 19.5, 15.8, 20.4, 33.4
Mean over 100 start states: 16.6
IBMARL (centralized)
IBMARL’s arbitrated policy; needs the critics, so training-time only.
Returns in this clip: 29.9, 20.9, 42.9, 14.1, 16.3
Mean over 100 start states: 23.7
No policy is selected.
Each task has a fixed pool of 100 start states (environment seeds 20000–20099). Every policy was run on all 100, deterministically and with no exploration noise, and each clip shows five of them: the same five for every policy of that task, chosen so that each policy’s mean return over the five matches its mean return over all 100, and so that the ranking between policies is the same. The “mean over 100 start states” under each clip is the full-pool number. Nothing was dropped or altered.
The clips show returns, not successes or failures: an episode’s return is exactly what the policy earned, and no threshold is needed to read it. Each RL policy is the final checkpoint of its median training seed out of five; the imitation policy is the single R2BC session. These numbers can differ noticeably from the paper’s tables, which average each seed’s last ten evaluations over all five seeds (for example, on Buzzwire RFT is −92 here against −111 in Table 1, and RLfD −74 against −86). Five episodes are also a small sample, so read the clips as a picture of the behaviour and the paper’s tables as the results.
The same Buzzwire experiment, with demonstrations from a real person.
123k ± 26k
steps for IBMARL’s actors to surpass the human-trained IL policy, in every seed (117k ± 8k with the scripted expert)
2.3–2.4×
faster than the baselines: 286k ± 94k for RLfD, which fails in one seed, and 296k ± 34k for RFT
−87.7 ± 3.2
final return, ahead of RFT (−109.4 ± 6.8) and RLfD (−132.9 ± 26.6)
A single demonstrator, one of the authors, commands the controlled agent’s velocity with the analog stick of an Xbox gamepad, and control passes to the other agent in round-robin order. Between rounds the R2BC policies are updated on the data so far, so later demonstrations are given alongside a progressively better teammate. The budget is 24 episodes (4,800 transitions), of which only 13 reach the goal. The resulting IL policy attains −143.4 ± 0.8, slightly below the expert-trained IL policy’s −137.5.
Human demonstrations cost every method some final return, but they cost the baselines more: RLfD, which replays the inconsistent demonstration transitions, falls from −85.9 to −132.9, close to the IL policy itself, while IBMARL falls from −68.9 to −87.7 and keeps its lead in both speed and final return. The difference is the gate: inconsistent demonstrations are followed where the critics still rate them above the actors and ignored elsewhere.
At deployment the team needs only the per-agent actors. IBMARL requires no communication, no centralized critic and no copy of the IL policy. The demonstrations cost only 12 to 500 episodes of single-agent teleoperation, compared with hundreds of thousands to millions of steps for RL.
The actors are also not bound by the demonstrator: on Buzzwire and Balance they end up outperforming it, so the ceiling that limits R2BC is not inherited. And the imitation policy is more useful as a participant than as a data generator: on Balance and Transport, keeping it acting in the training loop is the difference between learning and not learning within the budget.
Scaling. The arbiter scores all 2N joint candidates. This takes one batched forward pass for N ≤ 3 but may not scale to large teams; a greedy linear sweep over agents or a random subset of candidates could replace full enumeration.
Simulation only. All results are in simulation, with synthetic demonstrators plus one human operator on Buzzwire.
Known reward. The team reward is assumed known. A reward learned from human preferences could replace it, so a single person could supply both the demonstrations and the reward.
Frozen IL policy. The IL policy is frozen once RL begins. The demonstrator could keep teaching during RL and use the learned critics to find where the team still needs help.
The code reruns every experiment in the paper (IBMARL, its ablations and the MADDPG, RLfD and RFT
baselines on the four tasks) and regenerates the figures and tables from the recorded learning curves of the paper's runs. Its
README lists the exact configuration of each run and the commands to repeat them. The
demonstration data is a 20 MB archive that scripts/get_data.py unpacks and verifies.
To generate demonstrators of your own, use R2BC, which produced the imitation policies and demonstrations here: the simulation code (scripted and learned demonstrators, gamepad teleoperation), the paper and its project page. DATA.md describes the file format that IBMARL's loader expects.