SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation

We sincerely thank Reviewer auqZ for recognizing the contributions of our work. We are pleased the reviewer found novelty in our method, noting its ~5% improvement in success rate and state-of-the-art performance on benchmarks, as well as our memory components reducing failures by ~80% on memory-intensive tasks. We also appreciate that reviewer acknowledged our comprehensive experiments and thorough evaluation.

Despite this, the reviewer raised several concerns and suggestions, which we address below:

The baselines are a little lacking.

We appreciate the reviewer's feedback on the baselines. Our evaluation spans three benchmarks: RLBench (18 tasks), The Colosseum, and MemoryBench. We achieve state-of-the-art results on RLBench and The Colosseum. For MemoryBench, we tested OpenVLA, which obtained a 0% success rate across all tasks (All baselines on our anonymous website). OpenVLA demonstrated basic interactions (e.g., approaching objects) but failed key actions like closing drawers or grasping, due to visual ambiguities—identical scenes requiring different actions (e.g., opening vs. closing). Conventional VLAs struggle as they rely solely on visual and language cues without temporal context. Our method integrates timestep information into proprioceptive inputs, resolving these ambiguities effectively. Consequently, even RVT-2 and SAM2Act (without explicit memory) succeed in such tasks, though occasionally erring on subtasks requiring actual memory. Extensive baseline comparisons and ablations confirm our method’s robustness on MemoryBench, highlighting areas for VLA improvement. Full results tables will appear in the final manuscript.

Clarification on how the higher-resolution embeddings from the SAM2 encoder are passed to the multi-resolution resampling methods in Stage 2?

We apologize for the confusion and clarify briefly: Multi-resolution embeddings from the SAM2 encoder integrate into Stage 2 as in Stage 1 (see simplified Figure 4). Lower-resolution (res 16) embeddings pass through MVT and Memory Attention (conditioned on Memory Bank embeddings) before multi-resolution upsampling. Higher-resolution embeddings skip MVT and Memory Attention, entering upsampling directly. The resulting action heatmap is encoded into memory, indirectly involving all resolutions in memory processing.

What are the cases where SAM2Act+ still fails on MemoryBench?

Our analysis of the reopen_drawer task indicates that failures primarily stem from memory issues rather than execution errors. Specifically, SAM2Act+ often reopens the wrong drawer, showing reliable command execution but difficulty recalling the correct drawer. This aligns with our benchmark's focus on isolating memory-related errors. We found that shadows in the simulation occasionally caused ambiguous visual cues, complicating memory recall. To test this, we regenerated the dataset without shadows and, after retraining, SAM2Act+ achieved 100% success. This confirms that visual ambiguity from shadows was the primary cause of memory failures and demonstrates our approach effectively resolves these memory challenges.

Can you speculate about why the robustness is so improved in Section 5.3? Is it the pre-trained knowledge from the SAM2 image encoder that helps distinguish semantic meaning despite visual perturbations?

Similar response to Question 2 from Reviewer 24ak; please kindly refer to that for more details.

For SAM2Act+ why does the fine branch not have the memory component?

Our design separates coarse and fine branches based on information density. The coarse branch uses scene-level point clouds to generate virtual images and employs a memory module for spatially consistent heatmaps, critical for memory-dependent tasks. The fine branch processes localized, rapidly changing views focused on immediate interactions, where incorporating memory would disrupt spatial consistency.

Suggestion to refine our figures and clarify our writing and highlighting key results and impacts

We thank the reviewer for their detailed suggestions and fully agree that their proposed improvements to the figures and manuscript will strengthen our paper. Due to ICML’s no-revision policy during rebuttal, we will incorporate these enhancements in future versions.

Why make the views independent in the memory architecture?

We treat views independently in the memory architecture for two reasons. First, the MVT effectively integrates multi-view cues by processing each view both independently and dependently through attention layers. Second, consistent with SAM2’s design, avoiding cross-view dependencies prevents memory complications due to significant camera angle changes. This approach simplifies memory integration and enhances robustness and reliability.

We hope our responses have adequately addressed all of the reviewers' concerns. If so, we would greatly appreciate your consideration in raising your ratings.

We sincerely appreciate the Reviewer 24ak for recognizing the novelty of our approach, specifically in leveraging multi-resolution features from SAM2, and our innovative up-sampling scheme designed to facilitate high-precision manipulation policy learning. Additionally, we are grateful that the reviewer acknowledged our effective use of SAM2's memory mechanism to address complex, memory-based tasks, as well as our carefully designed MemoryBench, explicitly created for evaluating spatial memory capabilities. We are pleased that the reviewer found our claims well-supported by clear and convincing evidence (agreed by Reviewer yA68 and auqZ). Moreover, we thank the reviewer for emphasizing the timeliness and relevance of our research within robotic manipulation(similar to Reviewer yA68 and auqZ), and for highlighting the novel insights our study provides towards training generalist robotics agents(agreed by Reviewer auqZ).

Despite this, the reviewer raised several concerns and suggestions, which we address below:

How does the proposed architecture differ from world models? Can SAM2Act+ be adapted to serve as a backbone for training world models?

World models (e.g., DINO-WM, TD-MPC) differ significantly from SAM2Act in both objectives and training paradigms. While world models are trained to predict future observations and latent dynamics using various objective functions, SAM2Act is trained via behavior cloning. Although both incorporate memory and visual features, SAM2Act targets more realistic scenarios that violate the Markov assumption, unlike world models as mentioned by the reviewer. Moreover, world models typically decouple visual encoding and memory during training, whereas SAM2Act+ tightly integrates visual embeddings with spatial memory in an end-to-end fashion. While SAM2Act+ could potentially serve as a backbone for world modeling—by replacing the behavior cloning head with dynamics prediction and incorporating self-supervised objectives—that direction is beyond the scope of this work. Due to ICML's no-revision policy during rebuttal, we'll discuss this in a future version.

Is SAM2 the best choice of a vision foundation model used as backbone for robotic manipulation as compared to other visual encoders?

Our choice of SAM2 as the vision backbone is strongly justified by both prior work and our own ablation studies. In the SAM-E paper, a thorough evaluation compared various visual encoders—including CLIP, DINO, and the robotics-specific R3M—and found that the SAM image encoder outperformed the others when paired with an RVT-based action sequence prediction module. Since SAM2 builds on SAM by introducing multi-resolution features, it is expected—and indeed confirmed—to deliver superior performance and also supported from in-depth studies from the original SAM2 paper and SAM2-Adapter. To further validate this, we replaced SAM2 with alternative encoders suggested by the reviewer. Our experiments show that when using original upsampling (a fair comparison, as only SAM2 provides multi-resolution embeddings), DINOv2 achieves 82.2 ± 0.5 and DepthAnythingV2 achieves 81.1 ± 1.2 on the RLBench 18 tasks. Even the ablated version of SAM2Act, which uses the SAM2 encoder without multi-resolution output, already outperforms both alternatives with a success rate of 84.2 ± 0.9. Moreover, when leveraging SAM2’s multi-resolution outputs, we anticipate an even larger performance gap in its favor. Thus, the evidence strongly supports SAM2 as the best choice for our application, as its multi-resolution capabilities are uniquely advantageous for the robot manipulation tasks addressed in our work.

Clarification on whether the reopen-drawer and put_block_back tasks effectively evaluate different aspects of spatial memory (namely, 3D versus 2D spatial information) as claimed?

We realize our description of spatial memory evaluation may have caused confusion. Our intention with the two tasks is not to distinguish 3D from 2D spatial memory, but rather to isolate distinct components of spatial memory. Specifically:

• Reopen_Drawer: This task is designed to assess whether the agent can retain and recall information along the vertical (z-axis) dimension. It tests the agent’s ability to remember which specific drawer was interacted with previously.
• Put_Block_Back: This task evaluates the agent’s memory of the horizontal layout (x and y axes), ensuring it can accurately reposition objects within the overall spatial configuration.

Both tasks involve processing 2D pixel input but focus on different spatial dimensions within a 3D environment. By decoupling these aspects, our benchmark evaluates how effectively an agent integrates spatial cues across different axes. We hope this clarifies the confusion regarding the spatial memory tasks.

We hope our responses have addressed all of the reviewers' concerns. If so, we would appreciate your consideration in raising your ratings.

We sincerely thank Reviewer yA68 for the detailed and insightful feedback. We appreciate the recognition of our innovative use of SAM2 for generalization, the intuitive memory module design, and our comprehensive benchmarking. We're especially grateful for the positive remarks on our real-world evaluations, also echoed by Reviewer 24ak and auqZ. We also value the reviewer’s recognition of MemoryBench as a meaningful contribution to addressing critical research questions, as well as the acknowledgment that our claims are well-supported by the evidence—an assessment shared across reviewers.

Despite this, the reviewer raised several concerns and suggestions, which we address below:

The model architecture appears to have limited originality, as the SAM2Act module and coarse-to-fine pipeline closely follow RVT-2, and the memory module in SAM2Act+ strongly resembles SAM2’s tracking module.

We acknowledge that our work naturally builds upon prior research, embracing the principle of 'standing on the shoulders of giants,' much like RVT-2 evolved from RVT through targeted architectural changes, or SAM-E integrated elements from RVT with SAM's image encoder and action-chunking module. Our approach similarly leverages existing architectures but introduces critical, innovative adaptations that significantly enhance performance. In particular, our novel contribution is in effectively addressing the challenging problem of memory, representing one of the first systematic attempts in this area. Our key adaptations that make SAM2Act and SAM2Act+ effective are: (1) Multi-Resolution Upsampling, which leverages SAM2’s multi-scale image embeddings to boost RLBench performance and generalization (Tables 2 & 7), and (2) Memory Task Adaptation, where we extend SAM2’s tracking module to multi-view, multi-step settings by treating action heatmaps as object masks and integrating MVT embeddings into memory. This extension, which required careful design and extensive experimentation, is novel and essential—naively using the base models without these changes fails, as shown in our ablations, Table 7.

Given the performance drop when replacing SAM2 with SAM—falling below even the RVT-2 baseline on Colosseum and RLBench (Tables 2 & 7)—to what extent does the method's effectiveness rely on SAM2 rather than the proposed contributions?

We appreciate the reviewer’s observation and agree that the SAM2 encoder contributes significantly to performance. The drop observed when replacing SAM2 with SAM aligns with findings from the SAM2 and SAM2-Adapter papers (to be cited in future version), which show that SAM2’s lightweight Hiera encoder yields stronger embeddings for segmentation and downstream tasks. However, our ablations also demonstrate that the strong performance of SAM2Act arises from the combination of SAM2 with the other novel contributions from our proposed method and not solely from the encoder. Our primary contributions include innovations like multi-resolution upsampling, which adeptly leverages multi-resolution embeddings. As shown in Table 2, the improved generalization on Colosseum is primarily driven by multi-resolution upsampling. Without multi-resolution embeddings, performance matches the SAM-based variant, highlighting our architectural contributions as the main driver of generalization. Moreover, when comparing the SAM-based variant to RVT-2, their overlapping performance intervals (80.8 ± 1.9 vs. 81.4 ± 3.1) indicate no statistically significant differences. In summary, our approach combines several novel contributions that together enhance both performance and practicality.

Could the authors clarify the distinction between general long-horizon tasks and the specific tasks in MemoryBench, particularly highlighting the unique challenges that make a dedicated memory module necessary?

Robotic manipulation tasks typically follow the Markov assumption, where the optimal action depends solely on the current observation. Even in long-horizon tasks key information is directly observable and might not need memory. In contrast, MemoryBench is explicitly designed to violate this assumption—tasks are ambiguous, and visually identical states may require different actions based on prior interactions. We appreciate the reviewer’s suggestion of LIBERO, it emphasizes action-based models, which differ from our keyframe-based approach in SAM2Act. To investigate the relation between long-horizon and memory-based tasks, we curated four cube-stacking tasks with increasing complexity and keyframe length. We observed consistent performance degradation for both SAM2Act and SAM2Act+ as horizon length increased, suggesting that memory-based challenges extend beyond task length alone. (Results on our anonymous website).

We sincerely hope our responses have adequately addressed all of the reviewers' concerns. If so, we would greatly appreciate your consideration in raising your ratings.