Turn-wise Advantage
Hybrid Advantage
Token-wise Advantage
Turn-wise Value
Unified Value
Token-wise Value
\(A_1\)
\(A_1\)
\(A_1\)
\(A_1\)
\(+\)\(+\)
\(+\)\(+\)
\(A_1^1\)
\(A_1^2\)
\(A_1^3\)
\(A_1^4\)
Turn 1
\(\sum\gamma^k r_k\)\(+V_1\)
\(\sum\gamma^k r_k\)\(+V_1\)
\(\sum\gamma^k r_k\)\(+V_1\)
\(V_1\)
\(+\)\(+\)
\(+\)\(+\)
\(V_1^1\)
\(V_1^2\)
\(V_1^3\)
\(V_1^4\)
\(A_2\)
\(A_2\)
\(A_2\)
\(A_2\)
\(+\)\(+\)
\(+\)\(+\)
\(A_2^1\)
\(A_2^2\)
\(A_2^3\)
\(A_2^4\)
Turn 2
\(\sum\gamma^k r_k\)\(+V_2\)
\(\sum\gamma^k r_k\)\(+V_2\)
\(\sum\gamma^k r_k\)\(+V_2\)
\(V_2\)
\(+\)\(+\)
\(+\)\(+\)
\(V_2^1\)
\(V_2^2\)
\(V_2^3\)
\(V_2^4\)
\(A_3\)
\(A_3\)
\(A_3\)
\(A_3\)
\(+\)\(+\)
\(+\)\(+\)
\(A_3^1\)
\(A_3^2\)
\(A_3^3\)
\(A_3^4\)
Turn 3
\(\sum\gamma^k r_k\)\(+V_3\)
\(\sum\gamma^k r_k\)\(+V_3\)
\(\sum\gamma^k r_k\)\(+V_3\)
\(V_3\)
\(+\)\(+\)
\(+\)\(+\)
\(V_3^1\)
\(V_3^2\)
\(V_3^3\)
\(V_3^4\)
HyGAE first computes token- and turn-wise advantages and values together. It then combines the two advantages at every token. Finally, each turn-end value is broadcast across its action tokens before turn- and token-wise values are combined for one unified critic.