SkillMemo

Compositional embodied manipulation

SkillMemo

Expert-guided Skill Memory Framework for Compositional Embodied Manipulation

Changyuan Wang1 Chubin Zhang1 Zhenyu Wu2 Runhao Li3 Angyuan Ma2 Ke Chao4
Yinan Liang2 Xiuwei Xu2 Ziwei Wang3 Yansong Tang1,* Jiwen Lu2
1 Shenzhen International Graduate School, Tsinghua University 2 Department of Automation, Tsinghua University 3 Nanyang Technological University 4 Beijing Normal University

* Corresponding author.

Paper Coming Soon Model Demo
Explore

Demo

SkillMemo Demonstration

SkillMemo demo Watch on YouTube →
TL;DR

SkillMemo uses MoE routing to implicitly decompose trajectories into latent atomic skills, stores compact skill representations and complete gating profiles as retrievable key-value pairs, and fuses relevant skills with the current gating distribution during inference.

Paper

Abstract

Embodied visuomotor models, including Diffusion Policy (DP) and Vision-Language-Action (VLA) models, have demonstrated promising performance on robotic manipulation benchmarks. However, their potential remains fundamentally constrained by the scarcity of large-scale embodied trajectory datasets, leading to insufficient compositional generalization in out-of-distribution (OOD) scenarios with limited capability to capture reusable skill structures. To address this limitation, we propose Skill-Based Memory (SkillMemo) framework that implicitly decomposes long-horizon demonstrations into latent atomic skills and integrates skill-level features into a dynamic episodic memory bank for solving compositional tasks. Specifically, we first introduce an expert-guided trajectory segmentation module built upon a Mixture-of-Experts (MoE) architecture, which implicitly partitions trajectories into distinct skill primitives represented by learned gating coefficients. We further design a skill-level episodic memory architecture that stores compact skill representations as retrievable key-value pairs. During inference, the memory bank retrieves the most relevant skill primitives which are subsequently fused with the model’s current gating distribution, providing a robust contextual prior to refine action predictions. Extensive experiments on the simulation benchmark and real-world manipulation tasks demonstrate that SkillMemo consistently enhances both DP and VLA backbones, achieving state-of-the-art performance and outperforming π0.5, while exhibiting strong compositional generalization to unseen task configurations.

Method

SkillMemo Framework

SkillMemo couples MoE-based skill discovery with a dynamic episodic memory bank, enabling structured storage and compositional retrieval of learned skills.

Overall pipeline Open full size ↗
SkillMemo pipeline with expert-guided trajectory segmentation, skill-level memory storage, retrieval, fusion, and dynamic memory update
Figure 2. Overview of the SkillMemo framework. The Expert-Guided Trajectory Segmentation module implicitly extracts latent atomic skills from demonstrations via MoE routing. The Skill-Level Memory Architecture structurally stores and dynamically prunes these learned gating profiles, enabling context-aware retrieval for robust compositional generalization on unseen tasks.
01

Expert-Guided Trajectory Segmentation

Each expert captures a recurring local motion primitive, while the gating network implicitly defines temporal segmentation boundaries by modulating expert activations over time. A synergy-based information loss, expert load balancing, and top-k routing encourage complementary specialization and prevent expert collapse.

02

Skill-Level Memory Architecture

For each atomic skill segment, temporal aggregation produces a compact centroid of latent trajectory features as the memory key. The memory value stores the complete sequence of gating weights across all timesteps, preserving detailed temporal dynamics.

03

Memory Retrieval and Update

The current latent feature retrieves the top-N memory entries using cosine similarity. Reliable expert activation profiles are fused with the current gating distribution, while outdated, unused, or unreliable entries are dynamically pruned from the finite-capacity memory bank.

Experiments

Visualization Analysis

The gating coefficients dynamically shift and exhibit pronounced peaks during critical action phases such as grasping a bowl or turning on a stove.

Expert activation weights over time Open PDF →

Open Visualization.pdf
Figure 3. Visualization of expert activation weights over time. The gating coefficients dynamically shift and exhibit pronounced peaks during critical action phases such as grasping a bowl or turning on a stove, demonstrating that the EGTS module successfully drives individual experts to implicitly specialize in distinct, semantically meaningful motion primitives.