HSR: Hierarchical Skill Retrieval

Abstract

While Vision-Language-Action (VLA) models pretrained on large-scale robot datasets provide a strong foundation for robot manipulation, their performance can degrade when adapted to new tasks with limited task-specific demonstrations. Retrieval offers a practical way to reuse existing demonstrations for data-efficient adaptation, but existing methods often rely on visual similarity, state-action representations, or task-level language matching. These approaches may overlook the hierarchical structure of long-horizon manipulation tasks, where complete task matches are rare but reusable skills are often abundant. To address this challenge, we propose Hierarchical Skill Retrieval (HSR), a retrieval framework for data-efficient VLA adaptation. Specifically, HSR first decomposes a target task into candidate skill sequences. It evaluates each plan based on both semantic plausibility and skill reliability estimated from the prior dataset. The selected decomposition is then used for hybrid retrieval. This combines subtask-level language retrieval with behavior-feature reranking to identify demonstrations that are both semantically relevant and compatible with the target task. Finally, we adapt the policy through a two-stage pretraining and finetuning pipeline, which separates general skill acquisition from task-specific adaptation. Experiments on the LIBERO benchmark and several real-world robot manipulation tasks show that HSR improves the average success rate by 10.3% and 21.3% over the strongest baseline, respectively. These results demonstrate the effectiveness of structured skill-level retrieval for data-efficient VLA adaptation.

Overview

LLM-based long-horizon task decomposition enables structured demonstration retrieval, which effectively leverages existing skills from prior datasets for data-efficient VLA adaptation.

Contributions

We propose a hierarchical retrieval framework for long-horizon manipulation, which decomposes tasks into skill sequences and retrieves demonstrations at the skill level based on task structure and semantics.
We introduce a two-stage adaptation strategy. It first uses retrieved demonstrations to acquire transferable skills, and then finetunes on target data together with feature-reranked retrieved samples.
We validate the proposed framework on both the LIBERO benchmark and real-world long-horizon manipulation tasks, showing improved success rates and data efficiency over strong retrieval-based baselines.

Method

HSR identifies existing skills in the prior dataset based on task-instruction clustering. Given a long-horizon target task,
1. HSR first decomposes the task into several subtasks, evaluates and selects the plan with the highest score;
2. For each subtask, HSR retrieves data based on the subtask instruction, and further reranks using representative demonstrations from the target task to filter out low-consistency data;
3. HSR first pretrains the policy on retrieved data to learn general skills, and then finetunes it on task-related data to adapt to the target task.

Evaluation & Results

Real-World

Close the drawer

Behavior Retrieval

HSR (Ours)

Pick up the crumpled paper and throw it in the trashcan

Behavior Retrieval

HSR (Ours)

Put the cup in the drawer and close the drawer

Importance Weighted Retrieval

HSR (Ours)

Put the tea bag next to the cup and pour water into the cup

Importance Weighted Retrieval

HSR (Ours)

Simulation: LIBERO-10

Put both the alphabet soup and the tomato sauce in the basket

BC

BR

IWR

HSR (Ours)

Key Results

Improved the average success rate by 10.3% on LIBERO benchmark.
Improved success rate by 21.3% on real-world long-horizon manipulation tasks.

Citation

@misc{hao2026hierarchicalskillretrievaldataefficient,
      title={Hierarchical Skill Retrieval for Data-Efficient Adaptation of Vision-Language-Action Models}, 
      author={Haoran Hao and Shahram Najam Syed and Jeff Schneider and Jeffrey Ichnowski},
      year={2026},
      eprint={2608.24042},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2608.24042}, 
}