HyGAE: Hybrid Advantage Estimation with Unified Critic

Method Overview

Hybrid Advantage and Unified Critic

Turn-wise Advantage

Hybrid Advantage

Token-wise Advantage

Turn-wise Value

Unified Value

Token-wise Value

$A_1$ $A_1$ $A_1$ $A_1$

$+$$+$ $+$$+$

$A_1^1$ $A_1^2$ $A_1^3$ $A_1^4$

Turn 1

$\sum\gamma^k r_k$$+V_1$ $\sum\gamma^k r_k$$+V_1$ $\sum\gamma^k r_k$$+V_1$ $V_1$

$+$$+$ $+$$+$

$V_1^1$ $V_1^2$ $V_1^3$ $V_1^4$

$A_2$ $A_2$ $A_2$ $A_2$

$+$$+$ $+$$+$

$A_2^1$ $A_2^2$ $A_2^3$ $A_2^4$

Turn 2

$\sum\gamma^k r_k$$+V_2$ $\sum\gamma^k r_k$$+V_2$ $\sum\gamma^k r_k$$+V_2$ $V_2$

$+$$+$ $+$$+$

$V_2^1$ $V_2^2$ $V_2^3$ $V_2^4$

$A_3$ $A_3$ $A_3$ $A_3$

$+$$+$ $+$$+$

$A_3^1$ $A_3^2$ $A_3^3$ $A_3^4$

Turn 3

$\sum\gamma^k r_k$$+V_3$ $\sum\gamma^k r_k$$+V_3$ $\sum\gamma^k r_k$$+V_3$ $V_3$

$+$$+$ $+$$+$

$V_3^1$ $V_3^2$ $V_3^3$ $V_3^4$

HyGAE first computes token- and turn-wise advantages and values together. It then combines the two advantages at every token. Finally, each turn-end value is broadcast across its action tokens before turn- and token-wise values are combined for one unified critic.

Overview

Abstract

Large Vision-Language Models are increasingly used as agents in interactive environments, where success depends on decisions made across multiple turns. Existing training methods mainly optimize either concatenated token trajectories or turn-level actions with uniform within-turn credit. We establish a unified formulation for both views and derive a hybrid advantage that serves both objectives. With compatible discount factors and learning targets, one critic can estimate values at both scales. HyGAE jointly optimizes token- and turn-level objectives and reaches an average success rate of 91% across five multi-turn decision-making environments, improving over prior methods by 10 percentage points.

Background

From atomic actions to generated action strings

Classic RL predicts one environment action. VLM-for-RL first generates a structured response, then parses its action tokens into an executable command.

Classic RL predicts one action, while VLM for RL generates reasoning and action tokens that are parsed into an executable action. — Classic RL acts directly from a state. A conventional VLM generates tokens from visual and textual context. VLM-for-RL connects those generated action tokens back to an interactive environment through a parser and reward loop.

Formulation

One trajectory, two views

A VLM turn is both a macro-action and an autoregressive sequence of token actions. Writing both POMDPs explicitly reveals that their policy objectives share the same optimization structure.

Turn-wise POMDP One advantage is shared by every token in a turn

State: $\boldsymbol{\tau}_t=(\boldsymbol{o}_0,\boldsymbol{a}_0,\boldsymbol{r}_0,\ldots,\boldsymbol{o}_t)$, the interaction history available at turn $t$.
Action: $\boldsymbol{a}_t=(a_t^1,\ldots,a_t^{I_t})$, the entire generated response treated as one macro-action.
Reward: $\boldsymbol{r}_t$, the environment reward returned after executing the turn action.

$$L^{\mathrm{turn}}=\mathbb{E}\!\left[\sum_t\sum_i \log \pi_\theta(a_t^i\mid\tau_t^i)\,\boldsymbol{A}^{\pi}(\boldsymbol{\tau}_t,\boldsymbol{a}_t)\right].$$

Token-wise POMDP Every generated token receives its own advantage

State: $\tau_t^i=(\boldsymbol{\tau}_t,a_t^1,\ldots,a_t^{i-1})$, the interaction history plus the token prefix generated before position $i$.
Action: $a_t^i$, the $i$-th generated reasoning or executable-action token.
Reward: $r_t^i=\boldsymbol{r}_t$ when $i=I_t$, and $r_t^i=0$ for intermediate tokens.

$$L^{\mathrm{token}}=\mathbb{E}\!\left[\sum_t\sum_i \log \pi_\theta(a_t^i\mid\tau_t^i)\,A^{\pi}(\tau_t^i,a_t^i)\right].$$

Key finding

The objectives differ only in how credit is assigned

Because a turn action is generated autoregressively, its log-probability decomposes into the same token log-probabilities used by the token-wise formulation:

$$\log \pi_\theta(\boldsymbol{a}_t\mid\boldsymbol{\tau}_t)=\sum_i\log \pi_\theta(a_t^i\mid\tau_t^i).$$

Algorithm

Hybrid estimation, shared value learning

The actor receives a hybrid advantage at every token, while the critic learns a precise hybrid return from a single value model.

Theorem 1

Policy-gradient equivalence

If $\gamma^{\mathrm{turn}}_t=(\gamma^{\mathrm{token}})^{I_t}$, then every $\alpha\in[0,1]$ preserves the original policy gradient.

$$\nabla J(\theta)=\nabla L^{\mathrm{hybrid}}(\theta)=\nabla L^{\mathrm{turn}}(\theta)=\nabla L^{\mathrm{token}}(\theta).$$

Why it matters. Token- and turn-wise advantages can be mixed directly to optimize both surrogate functions without introducing hierarchical modeling.

Theorem 2

Unified value consistency

Under the same discount compatibility, the turn-wise value equals the token-wise value at the last token of every turn.

$$\boldsymbol{V}^{\pi}(\boldsymbol{\tau}_t)=V^{\pi}(\tau_t^{I_t}),\qquad \forall t.$$

Why it matters. HyGAE needs only one critic: it supplies turn-wise values at turn boundaries and token-wise values at intermediate positions, avoiding the memory and computation of two separate value models.

Theorem 3

Bias–variance tradeoff

Turn-wise estimation has the tighter bias bound, while token-wise estimation has the tighter variance bound.

$$b_{\mathrm{turn}}\le b_{\mathrm{token}},\qquad v_{\mathrm{turn}}>v_{\mathrm{token}}.$$

Why it matters. The two estimators are complementary rather than redundant. Their convex combination lets HyGAE retain the lower-bias turn signal while benefiting from the lower-variance token signal.

HyGAE estimation

One mixed target for the actor and one for the critic

After computing turn-wise and token-wise GAE backward through the trajectory, HyGAE forms the final training signals at every token position.

Actor target

Hybrid advantage

Combine the shared turn advantage with the token-specific advantage:

$$\widehat{\mathsf{A}}_t^i=\alpha\widehat{\boldsymbol{A}}_t+(1-\alpha)\widehat{A}_t^i,\qquad \alpha\in[0,1].$$

The resulting $\widehat{\mathsf{A}}_t^i$ is used by the clipped PPO policy objective.

Critic target

Hybrid return

First compute the token return and broadcast the turn return to position $i$:

$$\widehat{G}_t^i=\widehat{A}_t^i+\widehat{V}_\psi(\tau_t^i),$$

$$\widehat{\boldsymbol{G}}_t^i=\widehat{\boldsymbol{A}}_t+\sum_{k=i+1}^{I_t}(\gamma^{\mathrm{token}})^{k-i}r_k+\widehat{V}_\psi(\boldsymbol{\tau}_t).$$

Then combine both returns into the unified critic target:

$$\widehat{\mathsf{G}}_t^i=\alpha\widehat{\boldsymbol{G}}_t^i+(1-\alpha)\widehat{G}_t^i.$$

The policy $\pi_\theta$ is updated with $\widehat{\mathsf{A}}_t^i$, while the single value model $\widehat{V}_\psi$ is trained by TD regression to $\widehat{\mathsf{G}}_t^i$.

Experiments

State-of-the-art across five environments

Values are environment success rates over complete test episodes: 128 each for Sokoban and FrozenLake, 128 per Navigation and Primitive Skill task, and 32 ID plus 18 OOD episodes for VIRL. The overall average weights all ten task columns equally.

0.91Overall average

+0.10Over Token-PPO

0.86VIRL average

1.00Primitive Skill average

Comparison on five multi-turn decision-making benchmarks
Model / Method	Sokoban	F.Lake	Navigation		Primitive Skill				VIRL		Avg.
Model / Method	Sokoban	F.Lake	Base	CS	Place	Stack	Draw	Align	ID	OOD	Avg.
Proprietary Models
GPT-5	0.70	0.77	0.75	0.81	1.00	0.63	0.00	1.00	0.06	0.56	0.63
Gemini 2.5 Pro	0.58	0.78	0.63	0.63	0.63	0.63	0.00	0.75	0.16	0.39	0.52
Claude 4.5 Sonnet	0.31	0.80	0.67	0.67	0.63	0.50	0.00	1.00	0.09	0.44	0.51
Frozen Open-Source Models
Qwen2.5-VL-3B	0.18	0.13	0.34	0.28	0.00	0.00	0.00	0.00	0.00	0.00	0.09
Gemma-3-4B	0.07	0.05	0.03	0.02	0.00	0.00	0.00	0.00	0.00	0.00	0.02
VLM-R1-3B	0.16	0.10	0.34	0.15	0.00	0.00	0.00	0.00	0.03	0.00	0.08
Cosmos-R1-7B	0.14	0.23	0.01	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.04
Qwen3-VL-4B	0.27	0.08	0.01	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.04
Reinforcement Learning Methods
Token-PPO	0.59	0.72	0.90	0.84	1.00	1.00	0.99	0.95	0.16	0.94	0.81
Turn-PPO	0.38	0.70	0.78	0.84	0.00	0.00	0.00	1.00	0.09	0.94	0.47
GRPO	0.20	0.57	0.88	0.81	0.00	0.00	0.00	1.00	0.03	0.28	0.38
RL4VLM	0.20	0.35	0.86	0.78	0.00	0.95	0.00	0.98	0.38	1.00	0.55
VAGEN-Base	0.61	0.71	0.78	0.80	1.00	0.88	0.88	0.88	0.06	0.83	0.74
HyGAE Ours	0.83	0.80	0.90	0.86	1.00	1.00	1.00	0.99	0.72	1.00	0.91

Training curves for value targets, reward aggregation, and value selection. — Precise hybrid returns prevent collapse; small token rewards need not be aggregated; turn values provide a robust broadcast baseline.

Qualitative comparison

Observation Reasoning Prediction Action

Baseline

Frozen Qwen2.5-VL

Paper trace

Turn 1

<think><observation>The box is to the left of the player and the target is directly above the player.</observation><reasoning>The player needs to push the box to the target, which is directly above the player.</reasoning><prediction>The player will be above the box and target, and the target and box will be at the same row.</prediction></think><answer>Down.</answer>

Turn 2

<think><observation>The box is to the left of the player and the target is directly above the player.</observation><reasoning>The player needs to push the box to the target, which is directly above the player.</reasoning><prediction>The player will be above the box and target, and the target and box will be at the same row.</prediction></think><answer>Down.</answer>

Turn 3

<think><observation>The box is to the left of the player and the target is directly above the player.</observation><reasoning>The player needs to push the box to the target, which is directly above the player.</reasoning><prediction>The player will be above the box and target, and the target and box will be at the same row.</prediction></think><answer>Down.</answer>

Ours

HyGAE

Paper trace

Turn 1

<think><observation>The box is to the left of the player and the target is to the left of the player.</observation><reasoning>The box needs to be moved to the target. The player can move right, up, or down to reach the box.</reasoning><prediction>The player will move right to reach the box.</prediction></think><answer>Right.</answer>

Turn 2

<think><observation>The box is to the left of the player and the target is to the left of the player.</observation><reasoning>The box needs to be moved to the target. The player can move left, up, or down to reach the box.</reasoning><prediction>The player will move left to reach the box.</prediction></think><answer>Left.</answer>

Turn 3

<think><observation>The box is to the left of the player and the target is to the left of the player.</observation><reasoning>The box needs to be moved to the target. The player can move left or down to reach the box.</reasoning><prediction>The player will move left to reach the box.</prediction></think><answer>Left.</answer>

Verified, unpaired W&B rollouts. Both traces use the same FrozenLake configuration but are separate sampled episodes. Episode rewards and outputs are taken directly from the artifacts.

Baseline

RL4VLM

Run 1pl7j2yu · episode reward 0

Turn 1

<think><observation>There is a hole at the top left, a hole at the bottom left, a hole at the bottom right, a hole at the right, a hole at the top right, and a hole at the top center. The player is at the bottom center, and there is a gift box at the top center.</observation>
reasoning>The player is already at the same row as the gift box, and the hole at the top center is also the same row. I should move down to reach the gift box.</reasoning>
<prediction>Down</prediction>
</think><answer>Down</answer>

Turn 2

<think><observation>There is a hole at the top left, a hole at the bottom left, a hole at the bottom right, a hole at the right, a hole at the top right, and a hole at the top center. The player is at the bottom center, and there is a gift box at the top center.</observation>
reasoning>The player is already at the same row as the gift box, and the hole at the top center is also the same row. I should move down to reach the gift box.</reasoning>
<prediction>Down</prediction>
</think><answer>Down</answer>

Turn 3

<think><observation>There is a hole at the top left, a hole at the bottom left, a hole at the bottom right, a hole at the right, a hole at the top right, and a hole at the top center. The player is at the bottom center, and there is a gift box at the top center.</observation>
reasoning>Since I'm already at the same row as the gift box and the hole at the top center is the same row, I should move down to reach the gift box.</reasoning>
<prediction>Down</prediction>
</think><answer>Down</answer>

Ours

HyGAE

Run z54nt8qs · episode reward 10.5

Turn 1

<think><observation>The player is above the gift box and to the left of the hole. There is a hole below the player.</observation><reasoning>Since the player is close to the gift box, the most logical move would be to move down to reach the gift box. The presence of a hole below the player makes it necessary to move down.</reasoning><prediction>The player will be in the same row as the gift box and to the right of the hole.</prediction></think><answer>Down,Down</answer>

Verified, unpaired W&B rollouts. The baseline is a base-split cup task; the HyGAE trace is a common-sense-split lamp task. They are independent examples and are not presented as a matched episode.

Baseline

Token-PPO

Run z3qglf6x · episode reward 2

Turn 1

<think><observation>There is a cup on the counter next to the sink. To reach it, I need to move forward and then left.</observation><reasoning>I should move forward to get closer to the counter and then move left to reach the cup.</reasoning><prediction>I will be closer to the cup.</prediction></think><answer>moveahead, moveleft</answer>

Turn 2

<think><observation>There is a cup on the counter next to the sink. To reach it, I need to move forward and then left.</observation><reasoning>I should move forward to get closer to the counter and then move left to reach the cup.</reasoning><prediction>I will be closer to the cup.</prediction></think><answer>moveahead, moveleft</answer>

Turn 3

<think><observation>There is a cup on the counter next to the sink. To reach it, I need to move forward and then left.</observation><reasoning>I should move forward to get closer to the counter and then move left to reach the cup.</reasoning><prediction>I will be closer to the cup.</prediction></think><answer>moveahead, moveleft</answer>

Turn 4

<think><observation>There is a cup on the counter next to the sink. To reach it, I need to move forward and then left.</observation><reasoning>I should move forward to get closer to the counter and then move left to reach the cup.</reasoning><prediction>I will be closer to the cup.</prediction></think><answer>moveahead, moveleft</answer>

Ours

HyGAE

Run flghi2qf · episode reward 11

Turn 1

<think><observation>There is a book on a desk, a lamp on the desk, and a picture on a table. The lamp is the target object.</observation><reasoning>To reach the lamp, I should move forward and then rotate left to face the desk.</reasoning><prediction>I will be facing the desk with the lamp.</prediction></think><answer>moveahead, moveleft</answer>

Turn 2

<think><observation>There is a lamp on the desk, a book on the desk, and a picture on the wall. The lamp is the target object.</observation><reasoning>To reach the lamp, I should move forward to the desk and then rotate left to face the lamp.</reasoning><prediction>I will be facing the lamp on the desk.</prediction></think><answer>moveahead, moveleft</answer>

Final recorded state for the HyGAE Navigation rollout. — Final recorded state; no additional model response was logged.

Verified, unpaired W&B rollouts. Both traces are StackThreeCube episodes. The selected default-HyGAE trace has an episode reward of 0.9 and is only partially complete; the page does not label it as a success.

Baseline

Token-PPO

Run 97l8ltny · episode reward 0

Turn 1

<think>Observation: The robot is positioned above the green cube and the red cube. The purple cube is in the background on the right side of the screen.
Reasoning: I need to pick up the red cube and place it on top of the green cube. Then, I need to pick up the purple cube and place it on top of the red cube. The coordinates provided are in millimeters and within the workspace limits.
Prediction: After executing these actions, the red cube will be on top of the green cube, and the purple cube will be on top of the red cube.
</think><answer>pick(63,109,20)|place(76,-183,20)|pick(-69,68,20)|place(76,-183,50)</answer>

Turn 2

<think>Observation: The red cube is on top of the green cube, as predicted. The purple cube is still in the background.
Reasoning: I need to pick up the purple cube and place it on top of the red cube. The coordinates provided are in millimeters and within the workspace limits.
Prediction: After executing these actions, the purple cube will be on top of the red cube.
</think><answer>pick(-69,68,20)|place(76,-183,50)</answer>

Turn 3

<think>Observation: The red cube is on top of the green cube, as predicted. The purple cube is still in the background.
Reasoning: I need to pick up the purple cube and place it on top of the red cube. The coordinates provided are in millimeters and within the workspace limits.
Prediction: After executing these actions, the purple cube will be on top of the red cube.
</think><answer>pick(-69,68,20)|place(76,-183,50)</answer>

Ours

HyGAE, $\alpha=0.5$

Run f36mqad1 · episode reward 0.9

Turn 1

<think><observation>From the image, I see a red cube at (-1, 165, 20) and a green cube at (-23, -138, 20).</observation><reasoning>I need to stack the red cube on top of the green cube first. The green cube is closer to the robot's gripper, so it should be the first object I pick up and place.</reasoning><prediction>After executing this action, the red cube will be on top of the green cube.</prediction></think><answer>pick(-23, -138, 20)|place(-1, 165, 20)</answer>

Turn 2

<think><observation>From the image, I see a red cube at (51, 175, 20) and a green cube at (-7, 194, 20).</observation><reasoning>I need to stack the red cube on top of the green cube first. The green cube is closer to the robot's gripper, so it should be the first object I pick up and place.</reasoning><prediction>After executing this action, the red cube will be on top of the green cube.</prediction></think><answer>pick(-7, 194, 20)|place(51, 175, 20)</answer>

Turn 3

<think><observation>From the image, I see a red cube at (63, 208, 20) and a purple cube at (-21, 203, 20).</observation><reasoning>I need to stack the purple cube on top of the red cube. The red cube is already on the green cube, so I can move to pick up the purple cube and place it on top of the red cube.</reasoning><prediction>After executing this action, the purple cube will be on top of the red cube.</prediction></think><answer>pick(-21, 203, 20)|place(63, 208, 50)</answer>

Final recorded state for the partial HyGAE Primitive Skill rollout. — Final recorded state: partial stack, not a completed episode.

Paper example. The output content below follows the readable template reported in the paper. The frozen model repeats a static plan, while HyGAE tracks orientation and instruction progress.

Baseline

Frozen Qwen2.5-VL

Paper trace

Turn 1

[Observation] The scene contains buildings with glass facades, a blue sign, trees, and a sign for "The Langham, Melbourne."
[Reasoning] The instruction requires facing northeast, moving forward until the next intersection, and then turning left to face northeast.
[Prediction] Continue forward to the next intersection.
[Answer] forward.

Turn 2

[Observation] The scene again shows the same buildings, trees, and road context.
[Reasoning] The model repeats that it should face northeast, move forward until the next intersection, and then turn left.
[Prediction] Continue forward to the next intersection.
[Answer] forward.

Turn 3

[Observation] The scene still contains the same road and building context.
[Reasoning] The model repeats the previous plan to turn left, move forward until reaching the next intersection, and then continue.
[Prediction] I should continue forward to the next intersection.
[Answer] forward().

Ours

HyGAE

Paper trace

Turn 1

[Observation] I am facing east.
[Reasoning] First instruction is to turn right to face east.
[Prediction] I will turn right.
[Answer] turn_direction(east), forward().

Turn 2

[Observation] I am facing east.
[Reasoning] I have turned right, now I need to move forward.
[Prediction] I will move forward.
[Answer] forward(), turn_direction(north), forward().

Turn 3

[Observation] I am facing north.
[Reasoning] I have turned right, now I need to move forward.
[Prediction] I will move forward.
[Answer] forward().

Trace provenance: Sokoban and VIRL reproduce the response content reported in the paper; Sokoban is presented in the common tagged template for consistency. FrozenLake, Navigation, and Primitive Skill are verified against W&B table artifacts by run ID, image hash, response, and episode reward.

Reference

Citation

European Conference on Computer Vision (ECCV), 2026

@inproceedings{zhang2026hygae,
  title={Hybrid Advantage Estimation with Unified Critic for VLM Agentic Reinforcement Learning},
  author={Zhang, Wenxuan and Wang, Yuhui and Jia, Donggang and Shen, Xiaoqian and Ding, Jian and Viola, Ivan and Schmidhuber, J\"urgen and Elhoseiny, Mohamed},
  booktitle={European Conference on Computer Vision (ECCV)},
  year={2026}
}

HyGAE: Hybrid Advantage Estimation with Unified Critic for VLM Agentic Reinforcement Learning

Hybrid Advantage and Unified Critic

Abstract

From atomic actions to generated action strings

One trajectory, two views

The objectives differ only in how credit is assigned

Hybrid estimation, shared value learning

Policy-gradient equivalence

Unified value consistency

Bias–variance tradeoff

One mixed target for the actor and one for the critic

Hybrid advantage

Hybrid return

State-of-the-art across five environments

Frozen Qwen2.5-VL

HyGAE

RL4VLM

HyGAE

Token-PPO

HyGAE

Token-PPO

HyGAE, \(\alpha=0.5\)

Frozen Qwen2.5-VL

HyGAE

Citation