MorphAct

Robot Policies Morphing
with the Worlds They Act In

Every interaction brings new demands. MorphAct reshapes a shared policy’s perception and action weights to meet them.

Ziyan Liu^*,1,2
Jiting Liu^*,2
Weihong Zhong^*,2
Wenda Yu⁵
Yeqiu Chen¹

Jason Li⁴
Xiaojiang Peng⁶
Shuyuan Hu²
Kai Wang^3,†
Bo Zhao^2,†

¹University of Science and Technology of China
²School of Artificial Intelligence, Shanghai Jiao Tong University
³National University of Singapore
⁴Nanyang Technological University
⁵Tongji University
⁶Shenzhen Technology University

^*Equal contribution^†Corresponding authors

bo.zhao@sjtu.edu.cn kai.wang960112@gmail.com

liuziyan@mail.ustc.edu.cn liujiting616@gmail.com

arXiv Code

MorphAct in the real world · π_0.5 with context-dependent weights

A changing world calls for a changing policy

Scaling robot learning means spanning more environments, operations, and interactions. A shared policy must accommodate the different perceptual and control demands that come with this diversity. These demands keep changing as a robot acts.

“Put the object in the tray.” A robot must choose a grasp, make contact, carry the object, and place it precisely. One instruction takes it through changing visual cues and control demands, while a conventional policy applies the same fixed weights throughout. Different operations can pull those weights in competing directions.

MorphAct gives a shared policy the ability to reshape itself for the interaction in front of it. The scene, instruction, and robot state drive a learned mechanism that generates updates for both perception and action. Shared generators, context-dependent weights.

{\hat{Y}}_{t} = π_{θ + Δ θ (x_{t})} (x_{t})

x_t = (o_t, ℓ, s_t) is the current context; Δθ(x_t) is its generated update; Ŷ_t is the predicted action chunk.

Paper teaser: (a) a block-placement task progresses through free-space motion, contact, transport, and precise placement; (b) static adaptation applies one fixed update across changing contexts; (c) MorphAct uses a shared adaptation rule to generate context-dependent parameter updates. — **One shared mechanism, a policy shaped by each interaction.** From approach and contact to transport and precise placement, MorphAct lets changing context reshape the parameters used to see and act.

Different skills pull a shared policy in different directions

Different skills compete inside a fixed policy. We put a shared π_0.5 checkpoint under the microscope using 65 atomic manipulation tasks from RoboCasa365, examining where their learning signals agree, where they collide, and how an update transfers to other tasks.

The conflict reaches both sides of the policy: the vision–language module that interprets the scene and the action module that turns that interpretation into movement. Across 16 sampling rounds, almost half of cross-task gradient comparisons point in competing directions.

260 improving updates. Every one raises loss on other tasks. Each update reduces its source-batch loss, yet increases loss on batches from 22–46 of the other 64 tasks, with a median of 32. MorphAct answers this tension by sharing how to adapt and letting the current interaction shape the update.

At a fixed checkpoint, within-task versus cross-task gradient-conflict rates are 38.94 versus 47.80 percent for VLM and 42.69 versus 48.28 for Action. All 260 source-improving updates increase loss on some other sampled tasks. — **Skills pull the same weights in competing directions.** Cross-task gradients clash more often than within-task gradients; source-improving updates interfere with other tasks; and opposing parameter-group contributions cancel within both modules.

Gradient alignment

\begin{array}{l} g_{i}^{(m)} = \nabla_{θ_{m}} ℒ (U_{i}; θ) \\ c_{i j}^{(m)} = \frac{⟨ g_{i}^{(m)}, g_{j}^{(m)} ⟩}{{‖ g_{i}^{(m)} ‖}_{2} {‖ g_{j}^{(m)} ‖}_{2}} \end{array}

For module m, negative cosine reveals two tasks asking the same parameters to change in competing directions.

One-step transfer

\begin{array}{l} θ_{i}^{+} = θ - η \nabla_{θ} ℒ (U_{i}; θ) \\ Z_{i \to j} = 1 - \frac{ℒ (V_{j}; θ_{i}^{+})}{ℒ (V_{j}; θ)} \end{array}

The transfer score tracks the effect of a source update on target loss: positive scores improve it, and negative scores increase it.

Negative contributions concentrate in a few groups

A few action groups carry most of the negative contribution. The groups outside Transformer blocks, block 00, and block 11 together account for 78.88% of the action module’s pooled negative contribution.

Paper chart of pooled negative cross-task contribution shares by VLM and action parameter group, including the five largest groups and the remaining groups combined. — Parameter-group contributions reveal where competing updates concentrate inside each module.

Conflict spans the depth of each Transformer

Conflict extends across vision, language, and action layers. Its strength varies with depth, exposing different adaptation pressures throughout the policy.

Six paper plots compare cross-task and within-task negative-cosine frequency and mean negative part across the vision, language, and action Transformer blocks. — Negative-cosine frequency and mean negative part trace conflict across the vision, language, and action Transformers.

Morph from the environment, then act

Morph first. Then act. At each new observation, a vision–language generator reshapes the policy’s perception weights. The adapted model builds a representation of the scene, and an action generator uses that representation together with robot state to reshape the action weights.

Perception and action meet through the adapted representation. The action generator takes its cue from the policy’s adapted understanding of the scene, tying the two updates to the same interaction. Joint training with the native action objective learns how to generate them together.

Paper method figure: the VLM generator adapts multimodal features; adapted features and projected robot state condition the action generator. The lower panels show joint learning through two differentiable routes and reuse of generated updates throughout an action chunk. — **Adapt perception, generate action weights from what the policy now perceives, then act.** Two generators connect through the adapted vision–language representation and learn together from action supervision.

Shared generators, changing parameters

{Δ θ}_{v, t} = G_{v} (e_{t})

{Δ θ}_{a, t} = G_{a} (C_{a} (h_{t}, s_{t}))

The generators share their learned weights across interactions; the updates they produce follow the current context. G_v reads observation and instruction embeddings e_t. Its adapted features h_t combine with robot state s_t to condition G_a, while the same adapted pass supplies native action conditioning.

Technical details: factor synthesis, joint learning, and execution

Coupled parameter generation

\begin{array}{l} {Δ θ}_{v, t} = G_{v} (e_{t}; φ_{v}) \\ (h_{t}, C_{t}) = F_{v} (e_{t}; θ_{v} + {Δ θ}_{v, t}) \end{array}

\begin{array}{l} u_{a, t} = C_{a} (h_{t}, s_{t}; ρ_{a}) \\ {Δ θ}_{a, t} = G_{a} (u_{a, t}; φ_{a}) \end{array}

Both h_t and C_t come from the adapted VLM pass: h_t conditions the action generator, while C_t follows the native action-conditioning path.

Structured low-rank synthesis

Each generator uses HY-WU’s structured low-rank synthesis. Learned queries target layers and projection/rank blocks; attention to the condition tokens produces the A/B factors that adapt each selected projection.

{Δ W}_{j, t} = \frac{α_{j}}{r_{j}} A_{j, t} B_{j, t}

For block j, A: d_in × r and B: r × d_out. Frozen base weights provide the shared foundation; each context supplies its own low-rank factors.

One action loss, two differentiable routes

Action supervision shapes both the scene representation and the action weights generated from it.

\nabla_{φ_{v}} ℓ_{t} = J_{r_{t}, φ_{v}}^{⊤} (g_{cond} + g_{param})

g_{cond} = \frac{\partial ℓ_{t}}{\partial r_{t}}

Action conditioning

g_{param} = J_{{Δ θ}_{a, t}, r_{t}}^{⊤} \frac{\partial ℓ_{t}}{\partial {Δ θ}_{a, t}}

Action-side parameter generation

Here r_t = (h_t, C_t), ℓ_t is the per-sample native loss, and J_y,z = ∂y/∂z. Both routes carry action supervision back to the vision–language generator.

Apply the factors directly

y = u W_{base} + \frac{α_{j}}{r_{j}} (u A_{j, t}) B_{j, t}

The row-vector projection applies the low-rank residual as two factored operations, uA followed by B. Q/K/V projections use output-head blocks; attention-output projections sum input-head contributions; feed-forward projections use matrix-level factors. Each generated block has its own rank, with layouts following the backbone’s actual widths, depths, and head counts.

Retain the native action objective

min_{Φ} 𝔼_{(x_{t}, Y_{t}) \sim 𝒟, ξ} [ℒ_{native} ({\hat{y}}_{Φ} (x_{t}, Y_{t}; ξ), Y_{t}; ξ)]

Φ includes both generators, the state projection, and projectors retained as trainable by the adaptation configuration. ξ is the randomness used by the native objective.

Backbone interfaces and chunk-level reuse

Generate once for the current observation, reuse throughout the next action chunk, and refresh when the robot observes the world again. The generated updates stay shared across the chunk’s internal prediction steps.

For π₀ and π_0.5, training samples a time τ and predicts the velocity on the action–noise path:

\begin{array}{l} z_{τ} = τ ε + (1 - τ) Y_{t} \\ v_{target} = ε - Y_{t} \end{array}

Adapted multimodal tokens and robot state condition the action generator. Layer-wise prefix key–value states feed the native action path, which retains the backbone’s action, state, and solver-time interfaces.

Stronger robot policies across tasks and environments

One adaptation framework lifts three robot-policy backbones. MorphAct improves average success on both LIBERO and Meta-World for X-VLA, π₀, and π_0.5. It adds up to 16.30 percentage points on Meta-World and takes π_0.5 to 98.60% mean success on LIBERO.

Meta-World

50 tasks · 4 difficulty groups · 1,000 rollouts per seed

BaselineBaseline + MorphAct

π₀ nearly triples success on Very Hard. It jumps from 20.00% to 58.00%, a gain of 38.00 percentage points. π_0.5 raises Hard success from 38.30% to 66.39%, adding 28.09 points.

All suite scores & evaluation protocol

Scroll to see every column

Success rate (%) · complete breakdown
Backbone	Method	Easy	Medium	Hard	Very Hard	Avg.	Var. (pp²)
X-VLA	Baseline	72.50	46.36	48.30	55.00	55.54	—
X-VLA	MorphAct	83.99	58.18	55.28	55.00	63.11	1.8400
π₀	Baseline	77.90	51.80	53.30	20.00	50.75	—
π₀	MorphAct	82.44	71.36	56.39	58.00	67.05	0.6342
π_0.5	Baseline	70.50	45.00	38.30	41.00	48.70	—
π_0.5	MorphAct	85.77	59.85	66.39	45.67	64.42	0.3970

Twenty rollouts per task. Avg. gives equal weight to Easy (28 tasks), Medium (11), Hard (6), and Very Hard (5). MorphAct scores average three random seeds. Var. is the sample variance of their overall scores (pp²). Bold marks the best score within each backbone, including ties.

LIBERO

40 tasks · 4 suites · 2,000 rollouts per seed

BaselineBaseline + MorphAct

Long-horizon manipulation benefits the most on every backbone. Long gains 9.00 percentage points on X-VLA, 7.20 on π₀, and 5.13 on π_0.5.

All suite scores & evaluation protocol

Scroll to see every column

Success rate (%) · complete breakdown
Backbone	Method	Spatial	Object	Goal	Long	Mean	Var. (pp²)
X-VLA	Baseline	95.40	96.60	96.00	84.20	93.05	—
X-VLA	MorphAct	98.67	99.87	97.07	93.20	97.20	0.0300
π₀	Baseline	96.80	98.80	95.80	85.20	94.15	—
π₀	MorphAct	98.00	99.60	96.60	92.40	96.65	0.0517
π_0.5	Baseline	98.80	98.20	98.00	92.40	96.85	—
π_0.5	MorphAct	98.33	99.67	98.87	97.53	98.60	0.0525

Each suite contains 10 tasks, evaluated with 50 rollouts per task and execution horizon H10. Mean averages the four suites. MorphAct scores average three random seeds. Var. is the sample variance of their overall scores (pp²). Bold marks the best score within each backbone, including ties.

Adaptability comes from context and coordination

The gains come from responding to the current interaction and keeping perception and action connected. X-VLA ablations on LIBERO isolate the value of context, module coverage, and the feature link between the generators.

Mean over all suites

Long-horizon tasks

MorphActAblated variant

Generate from the current context. Task identity takes mean success to 88.25%; current-context generation takes it to 97.20%. MorphAct gains 8.95 pp over Task-ID and 9.70 pp over constant generation.

Adapt perception as well as action. Action-only reaches 81.00% mean success; VLM-only reaches 43.50%. Giving the action predictor an adapted perception of the scene lifts mean success to 86.00%. Coupling adds the next leap.

Generate action parameters from adapted perception. Coupling lifts mean success from 86.00% to 97.20%, adding 11.20 pp. On Long, it adds 21.60 pp, taking success from 71.60% to 93.20%.

All ablation scores & controlled changes

Scroll to see every column ↔

X-VLA ablations on LIBERO · success rate (%)
Variant / controlled change	Spatial	Object	Goal	Long	Mean
MorphActCurrent context; both modules; adapted features	98.67	99.87	97.07	93.20	97.20
ConstantOne fixed generation condition across all tasks	93.40	90.00	89.00	77.60	87.50
Task-IDOne fixed generation condition per task	90.00	90.80	91.20	81.00	88.25
VLM-onlyGenerated updates in the VLM only	40.00	59.00	42.00	33.00	43.50
Action-onlyGenerated updates in the action module only	88.80	92.00	73.80	69.40	81.00
UnadaptedPre-adaptation features condition the action generator	91.00	97.80	83.60	71.60	86.00

Constant uses one fixed generation condition across tasks; Task-ID uses one per task. Both retain the policy’s image, language, and robot-state inputs. VLM-only and Action-only adapt their named module. Unadapted adapts both modules and conditions action prediction on adapted features, while its action generator uses pre-adaptation features. Mean averages the four suites.

Adaptation that carries into the real world

The same idea carries onto the robot. Across arranging letters, sorting hardware, delivering water, and storing objects in a cabinet, MorphAct improves success on every physical task. It adds 8–14 percentage points over π_0.5 across 50 rollouts per task and method.