Figure 1. Overview of SKIP. Our framework predicts event-preserving sparse keyframes from an initial observation and a language instruction, then recovers the dense rollout via learned gap prediction and action-conditioned interpolation.
A short narrated walkthrough of SKIP.
Long-horizon manipulation videos waste compute on redundant motion. SKIP decomposes video generation into three stages: SKIP-Selector picks event-preserving keyframes via multimodal fusion, SKIP-Generator diffuses only these sparse anchors, and SKIP-Reconstructor recovers the dense rollout with a gap predictor plus an action-conditioned interpolator. On LIBERO, SKIP runs 4.16× faster than dense generation, cuts FVD by 89.0%, and—remarkably—when its videos fully replace real demonstrations for π0.5 training, success drops only 1.3 pp in simulation and 6.7 pp on a four-task Franka robot, while dense generation collapses by 48–58 pp.
Embodied world models have emerged as a promising paradigm in robotics by predicting how robot actions affect the surrounding scene. However, rollout inference remains computationally expensive in pixel space, as long-horizon manipulation videos typically have to be generated frame by frame. This cost cannot be easily reduced by indiscriminately dropping frames, since downstream policies rely on complete preservation of sparse task-relevant events such as approach, contact, grasp, and release.
To address this challenge, we propose the Sparse Keyframe Interpolation Paradigm (SKIP), an event-preserving sparse-to-dense framework that avoids dense frame-by-frame generation. SKIP first identifies task-relevant keyframes by leveraging robot-aware multimodal features. It then synthesizes only these keyframes with a sparse video diffusion model. A learned gap predictor and an action-conditioned interpolator subsequently reconstruct the missing intervals according to the robot actions.
On LIBERO, SKIP generates dense rollouts 4.16× faster than a dense baseline while improving visual fidelity and reducing aggregate FVD by 89.0%. Importantly, SKIP-generated videos are effective policy-training data. Even when they fully replace real demonstrations, π0.5 success drops only 1.3 pp in LIBERO simulation and 6.7 pp on the real robot, whereas fully dense frame-by-frame generation collapses by 48 to 58 pp.
Figure 2. SKIP architecture. Three stages: SKIP-Selector performs event-preserving keyframe selection through multimodal fusion of visual flow, semantic change, and proprioceptive features, with explicit gripper-event protection. SKIP-Generator is a sparse video diffusion model (Wan 2.2-TI2V-5B fine-tuned) that synthesizes only the selected keyframes from an initial frame and a language prompt. SKIP-Reconstructor uses a learned gap predictor and an action-conditioned interpolator to recursively recover the missing intervals from robot actions.
Event-preserving keyframe selection via multimodal fusion of visual flow, semantic change (DINOv3), and proprioceptive features — with explicit gripper-event protection.
A sparse video diffusion model (Wan 2.2-TI2V-5B fine-tuned) synthesizes only the keyframes from an initial frame and a language prompt.
A learned gap predictor sets the per-segment frame count; an action-conditioned interpolator recursively recovers the missing intervals from robot actions.
| Method | PSNR ↑ | SSIM ↑ | LPIPS ↓ | FVD ↓ |
|---|---|---|---|---|
| Wan 2.2 (chunked, dense baseline) | 21.423 | 0.832 | 0.162 | 4177 |
| Uniform + SKIP-Generator/Reconstructor | 21.345 | 0.847 | 0.130 | 716 |
| RDP + SKIP-Generator/Reconstructor | 21.076 | 0.843 | 0.137 | 724 |
| TriPSS + SKIP-Generator/Reconstructor | 21.033 | 0.827 | 0.157 | 640 |
| SKIP (ours) | 21.635 | 0.856 | 0.119 | 458 |
Averaged across four LIBERO test suites. Best per column in bold. SKIP cuts aggregate FVD from 4177 to 458 (−89.0%) over the dense baseline.
| Suite | SKIP (ours) seconds/traj | Wan 2.2 chunked ↓ | Speedup ↑ | |||
|---|---|---|---|---|---|---|
| Keyframe gen. | Gap pred. | Interp. | Total ↓ | |||
| LIBERO-10 | 34.78 | 0.23 | 4.78 | 39.79 | 243.86 | 6.13× |
| LIBERO-Goal | 35.24 | 0.02 | 2.47 | 37.73 | 144.66 | 3.83× |
| LIBERO-Object | 35.45 | 0.02 | 2.28 | 37.75 | 137.92 | 3.65× |
| LIBERO-Spatial | 34.68 | 0.02 | 1.94 | 36.64 | 124.74 | 3.40× |
| Weighted avg. | 35.06 | 0.06 | 2.74 | 37.86 | 157.37 | 4.16× |
Per trajectory on one NVIDIA H20, 50 denoising steps, bf16.
| Training data | LIBERO-10 | LIBERO-Goal | LIBERO-Object | LIBERO-Spatial | Avg. |
|---|---|---|---|---|---|
| Real demos (all-real reference) | 90.1 | 96.5 | 96.3 | 97.9 | 95.2 |
| SKIP-Mix70 (30% real + 70% SKIP) | 89.6 | 96.6 | 96.0 | 97.5 | 94.9 |
| SKIP-Mix100 (100% SKIP) | 88.8 | 96.0 | 94.2 | 96.4 | 93.9 |
| Dense-Mix70 (30% real + 70% dense Wan) | 68.5 | 67.7 | 60.6 | 59.4 | 64.1 |
| Dense-Mix100 (100% dense Wan) | 35.5 | 38.6 | 37.2 | 36.6 | 37.0 |
Each condition uses 1503 training trajectories. SKIP-Mix70 matches all-real training within 0.3 pp on average; dense generation collapses by 30–58 pp.
Figure 3. π0.5 success rate across five training mixes. Left: screenshots of the four Franka Panda real-robot tasks T1–T4 with per-task success rate over 30 rollouts. Right: average success rate on LIBERO simulation and the Franka real robot.
| Keyframe selection streams | Recovery | Avg. SR (%) | ||||
|---|---|---|---|---|---|---|
| Visual | Semantic | Proprio. | Gripper | Action | Sim | Real |
| ✓ | — | — | — | ✓ | 71.5 | 43.3 |
| — | ✓ | — | — | ✓ | 68.7 | 36.7 |
| — | — | ✓ | — | ✓ | 73.6 | 50.0 |
| ✓ | ✓ | — | — | ✓ | 81.2 | 53.3 |
| ✓ | ✓ | ✓ | — | ✓ | 91.4 | 66.7 |
| ✓ | ✓ | ✓ | ✓ | — | 90.4 | 66.7 |
| ✓ | ✓ | ✓ | ✓ | ✓ | 94.9 | 73.3 |
No single feature stream suffices; multimodal fusion plus gripper-event protection plus action conditioning is required for the best sim/real performance.
Figure 4. Held-out LIBERO trajectories bucketed by gripper-event count (0–1, 2, 3, 4, 5+). SKIP-Selector's margin over Uniform, RDP, and TriPSS widens as manipulation complexity grows, with the largest gains on the 5+-event bucket.
Figure 5. A LIBERO trajectory with four ground-truth gripper events. SKIP-Selector (bottom row) hits all four exactly, while Uniform, RDP, and TriPSS each miss multiple events.
Figure 6. Event coverage and downstream video quality saturate near $K_{kf}=41$. Higher budgets give smaller marginal gains while increasing inference cost.
Figure 7. The Operation-Aware Score (OAS) of keyframe quality strongly tracks downstream π0.5 success rate across selection ablations (circles) and cross-method points (triangles).
Figure 8. On a long two-object LIBERO trajectory, event-aware keyframes help preserve object-in-gripper states that uniform or geometry-only anchors can miss.
Figure 9. Sampled frames from SKIP's reconstructed rollouts (Gen) against ground-truth episodes (GT). Failures concentrate in compound multi-stage tasks where errors near phase switches accumulate.
Figure 10. The four Franka Panda real-robot tasks (T1–T4) used for downstream π0.5 evaluation.
All clips loop automatically (LIBERO at 256×256, 16 fps). We first show SKIP across all four LIBERO suites, then zoom into single episodes to swap keyframe selectors and inspect the sparse keyframes themselves. Every keyframe‑based method shares the same generator and action‑conditioned interpolator; only the keyframe selector changes.
LIBERO-10 · long-horizon — “put both the alphabet soup and the cream cheese box in the basket”
LIBERO-Goal · “put the bowl on the stove”
LIBERO-Object · “pick up the alphabet soup and place it in the basket”
LIBERO-Spatial · “pick up the black bowl from table center and place it on the plate”
Across every suite, SKIP tracks the ground-truth episode while running 4.16× faster than the dense per-frame baseline and cutting aggregate FVD by 89.0%.
KTS-VSP fusion
Equally spaced
Geometry-based
Saliency-based
Same generator and action-conditioned interpolator throughout — only the keyframe selector changes. Uniform, RDP, and TriPSS miss task-relevant events, matching the FVD and π0.5 gaps in the tables above.
SKIP-Generator output (diffused anchors only)
After gap prediction + action-conditioned interpolation
The diffusion model synthesizes only the 41 event-preserving keyframes (left); SKIP-Reconstructor expands them back into the full dense video (right).
T1 · “put both the red and blue mug in the basket”
T2 · “put the red bowl in the drawer and close it”
T3 · “put the white mug on the plate”
T4 · “stack the red bowl on the green bowl”
π0.5 trained on SKIP-generated data, deployed on a Franka Panda — two successful rollouts per task.
@misc{he2026skipsparsekeyframeinterpolation,
title={SKIP: Sparse Keyframe Interpolation Paradigm for Efficient Embodied World Models},
author={Ziheng He and Yixiang Chen and Ning Yang and Zhanqian Wu and Qisen Ma and Yuan Xu and Jiabing Yang and Peiyan Li and Xiangnan Wu and Xiaofeng Wang and Zheng Zhu and Jing Liu and Nianfeng Liu and Yan Huang},
year={2026},
eprint={2606.00664},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2606.00664}
}