arxiv:2611.17167

Actor, Judge, or Teacher? Understanding Decision Models in Long-Horizon Agents

Published on Nov 12

· Submitted by

wangzixuan on Nov 13

Zhejiang University

Upvote

Authors:

Zixuan Wang ,

Abstract

A decision model that acts poorly can still teach reinforcement learning policies effectively by judging complete action trajectories, improving success on ALFWorld and WebShop.

Generated by deepseek-ai/DeepSeek-V4.1-Flash

Decision models answer a question with a typed output, such as a choice among options with a probability for each, so one model can choose an agent's actions, evaluate actions already taken, or score every action to teach a policy with reinforcement learning. Prior work has studied them mainly as actors, judging their usefulness by how well they act. We ask whether a decision model that acts poorly can still teach well. We compare three decision models and three general LLMs as actors and judges on ALFWorld, and use JEV as a teacher on ALFWorld and WebShop. As actors, all six models are pulled away from a subgoal that must come first by an easier one. As judges that see the whole trajectory, the same models credit the steps toward that subgoal, and asking whether each action helped, rather than whether the task will succeed, separates useful from useless actions. As a teacher, JEV raises a Qwen2.5-1.5B policy over GRPO from 72.8% to 82.9% success on ALFWorld and from 56.8% to 71.4% on WebShop. These results suggest that a poor actor can still teach well, because judging an action after the episode ends reveals the consequences that acting must anticipate. Code is available at https://github.com/ZJU-REAL/decision-agents.

View arXiv page View PDF Project page GitHub 44 Add to collection

Community

wangzx1210

Paper author Paper submitter 2 days ago

Decision models output typed decisions (e.g., a probability over candidate actions), so the same model can act as an Actor (choose actions), a Judge (evaluate actions already taken), or a Teacher (score every action to provide step-level rewards for RL). Prior work mostly evaluates them as actors. We ask: can a model that acts poorly still teach well?

🔍 Key findings

On ALFWorld, we compare 3 decision models and 3 general LLMs. As actors, all six get pulled toward an easier subgoal and skip a prerequisite one.
As judges with the full trajectory, the same models correctly credit the steps that advance that prerequisite.
What separates useful from useless actions is judging whether each step helped, not predicting whether the task will succeed.

⚡ As a teacher, our JEV gives step-level signals to a Qwen2.5-1.5B policy trained with GRPO:

ALFWorld: 72.8% → 82.9%
WebShop: 56.8% → 71.4%

Our takeaway: judging an action after the episode reveals consequences that acting can only anticipate, so a weak actor can still be a strong teacher.

🌐 Project page: https://zjureal.com/decision-agents/
💻 Code: https://github.com/ZJU-REAL/decision-agents

polyarti

1 day ago

Asking whether each action helped, rather than whether the task will succeed, is such a clean way to tell useful steps from useless ones. I like that a model that acts badly can still teach well. Did the judge ever credit steps toward the easier subgoal that made the actor stray?

Jethro37

1 day ago

Thank you for sharing your work. Your actor–judge–teacher comparison shows that acting performance need not predict teaching value. On ALFWorld and WebShop, hindsight-based action feedback from JEV improves the learner over GRPO, demonstrating how a model that struggles as an actor can still provide useful supervision.

We discuss your work in Section 8.5 (“Active coaching and physical recursive self-improvement”), pp. 79–80 of the updated manuscript of our survey, In-Context Learning for Robots: Methods and Applications. We use these agent-benchmark findings to motivate the active-coaching question of which teacher and feedback signal best improve the learner’s subsequent decisions, connecting it to RoboCoach and robot self-improvement. Read the updated discussion.

If you find our discussion of active coaching and robot self-improvement useful for your own work, we would be grateful if you would consider citing the survey.