Model Architecture
WholeBodyWAM combines scalable motion pretraining with a unified Video–Motion–Action world-action architecture. Stage I learns task-conditioned future body dynamics from language–motion pairs, and Stage II grounds this pretrained motion prior into target-robot control through layer-wise Mixture-of-Transformers attention.
(a) Attention Mask
Asymmetric Video–Motion–Action Attention. During Stage II training, Motion and Action streams can attend to the current visual tokens and to each other, but cannot access future visual tokens. Future visual latents therefore provide predictive supervision without leaking privileged information. At inference, future visual tokens are omitted entirely while current visual observations continue to condition motion and action generation.
(b) Stage I: Motion Pre-training
Learning Task-Conditioned Future Body Dynamics. A 30-block Motion Expert is pretrained on UniMotion-4K using continuous flow matching. Given 16 frames of motion history and a language instruction, it predicts the next 32 frames in the unified 63D motion space. This stage uses only language–motion pairs and requires neither target-robot actions nor target-robot demonstrations.
(c) Stage II: Robot Post-training
Grounding Motion Priors into Robot Control. The Video, Motion, and Action Experts are jointly post-trained on target-robot demonstrations to predict future visual latents, future whole-body motion, and robot action chunks. Their modality-specific streams exchange information through layer-wise joint attention while retaining separate expert parameters. The predicted actions directly control the arms, hands, head, and waist, while high-level leg commands are executed by an RL-based whole-body controller for locomotion and balance.
At deployment, WholeBodyWAM uses the current onboard image, the latest 16 motion states, and a language instruction to jointly predict future whole-body motion and an executable action chunk. It replans in closed loop without generating or decoding future video, preserving predictive foresight while keeping inference efficient.
Motion Prediction
WholeBodyWAM conditions on language and motion history to predict future whole-body motion, learning scalable body-dynamics priors before robot-specific training.
Prompt
Kneel
A person kneels facing forward.
Prompt
Pick
A person bends down to pick up an object with their right hand and then rises back up.
Prompt
Run
A person jogs forward at a steady pace.
Prompt
Walk
A person is walking forward at a normal pace.
Real-world Demos
TianGong 3.0
Toy Pickup
Bend down, pick up the toys, and place them in the basket.
TianGong 3.0
Laundry Loading
Pick up the clothes and put them in the washing machine.
TianGong 3.0
Pillow Transfer
Pick up the cushion, walk to the sofa, and place it on the seat.
TianGong 3.0
Kneeling Toy Storage
Kneel down, pick up the toy, and place it in the box.
TianGong 3.0
Toy Transfer
Pick up the toy, walk to the box, and place it inside.
TianGong 3.0
Box Transfer
Lift the box, carry it to the side table, and place it on the table.
Experiment Results
Real-World Tiangong Task Performance
Table I. Real-World Humanoid Whole-Body Control. WholeBodyWAM achieves the highest average normalized task score of 72.2% across six real-world TianGong 3.0 tasks, outperforming the strongest baseline, GR00T N1.7, by 11.4 percentage points. Removing Stage-I motion pretraining reduces the average score to 59.1%, while blocking Motion-to-Action attention further lowers it to 46.3%. Integrating the pretrained Motion Expert into τ0-WM improves its score from 39.4% to 62.7%, demonstrating that the learned motion prior transfers beyond the native WholeBodyWAM architecture.
Scaling Motion Pretraining
(a) Motion Prediction Scaling. Increasing Stage-I motion pretraining from 0 to 4K+ hours consistently improves future whole-body prediction across Toy Pickup, Laundry Loading, and Pillow Transfer. Mean MPJRE decreases from 1.127° to 0.817°, a 27.5% reduction.
(b) Downstream Control Scaling. Larger motion-pretraining datasets yield corresponding gains in real-world control. With the same target-robot demonstrations and post-training setup, the average task score rises from 57.1% to 67.6% as pretraining scales from 0 to 4K+ hours.
(c) Target-Robot Data Efficiency. Motion pretraining improves average task scores by 4.2, 11.2, and 10.5 percentage points when using 20%, 50%, and 100% of the target-robot demonstrations. With only 50% of the demonstrations, the 4K+ hour model reaches 46.9%, surpassing FastWAM trained with the full dataset (42.9%).
Analysis Experiments
Generalization to Unseen Task Variations
Robustness to Distribution Shifts. WholeBodyWAM generalizes better than GR00T N1.7 to both spatial and object-level variations. It achieves 55.0% versus 46.3% when the Toy Pickup basket is shifted by 15 cm, and 75.0% versus 63.3% on unseen toy colors and shapes in Kneeling Toy Storage.
Inference Efficiency
Efficient Predictive Modeling. On a single NVIDIA A100 GPU, WholeBodyWAM runs at 363 ms end-to-end latency—19.5× faster than DreamZero, 14.5× faster than LingBot-VA, and 1.45× faster than FastWAM—while remaining close to τ0-WM at 313 ms.