Dense Representation Learning for a Joint-Embedding Predictive Architecture

Dear Reviewer KGbU,

Thank you once again for your valuable efforts and constructive feedback on our paper. As the discussion period comes to a close, we trust that your concerns have been thoroughly addressed in our response. We dedicated significant effort to this, incorporating several new experiments, visualizations, and discussions. We sincerely hope you will consider our reply in your assessment. We look forward to hearing from you, and if the opportunity arises, we can further address unclear explanations and remaining concerns if any.

Regards,

Authors

We deeply appreciate your thorough and insightful reviews. We trust that our response adequately addresses any concerns you may have.

Q1. Clarity for Equations and Figure 2

We acknowledge your suggestion to improve the clarity of symbols and equations in the paper, particularly around equations 2-5. To address this, we updated Figure 2 to include symbols of features, enhancing readability and understanding. Additionally, we provided a more detailed structure of local aggregation head, clearly indicating the Q, K, V of features $h_\theta$ and $h_{\hat{\theta}}$. This should make the methodology quicker and easier to grasp for readers.

Q2. Clarity for Equation 4

Your observation about the confusing subscript in equation 4 is valid. The subscript t and i in $s^{LAT}_i = h_{\hat{\theta}}(x_{j_{j\in P_i}}, x_t)$ indeed require clarification. We will correct and clarify this in the final draft. The subscript t refers to the target patch, while i refers to the index of the patch in the local aggregation process. We will ensure this distinction is more clearly articulated in the final draft.

Q3. Clarity for Local Aggregation Target

We understand your request for a more detailed explanation of the LAT module's motivation and its contribution to performance improvement. The LAT module is designed based on the premise that semantically similar patches should have similar targets. By incorporating LAT, our model can achieve better semantics and discriminative capabilities, as not every patch is treated with the same target. We will expand on this explanation in the manuscript, providing more analysis and insights to help the community understand the benefits of this approach.

Q4. Clarity for Local Aggregation Head structure

Regarding the structure of $h_\theta$ and the apparent confusion between $s_x$ and $x_c$, we will make sure to clarify these in our revised paper. $s_x$ and $x_c$ indeed refer to different embeddings. $s_x$ denotes the context embeddings, while $x_c$ represents the averaged embeddings from unmasked patches in the context encoder. This distinction is crucial for understanding the operation of our model, and we will provide a clearer explanation to eliminate any confusion in the final draft.

We deeply appreciate your thorough and insightful reviews. We trust that our response adequately addresses any concerns you may have.

Q1. Clarity for Section 3.3

You pointed out a need for clarity in explaining the cross-attention operations in the local aggregation layer. In the revised manuscript, we will include a more detailed explanation of these computations. Specifically, the cross-attention mechanism works by using the average patch representation of the target $x_t$ (or context $x_c$) as the query, and the semantically similar patches {$x_j$}$_{j\in P_i}$ (or the entire context $s_x$) as the key/value. This process allows for generating distinct representations for each spatial location by dynamically adjusting the focus of the attention mechanism on different parts of the input based on the query. Key is each query patch has distinct neighborhood patches, and objective derived from distinct neighborhood will ensure their distinct target objectives. We will revise to ensure this process is thoroughly explained and illustrated for better understanding in the final draft.

Q2. Generalizablity of Dense-JEPA

Regarding the generalizability of our findings, it's important to note that while Dense-JEPA might not directly extend to masked-image auto-encoders (MAE), it is indeed applicable to masked latent reconstruction models. Our approach is designed to be flexible and adaptable, allowing for potential application in various learning frameworks including masked reconstruction on latent space.

Moreover, we have tested Dense-JEPA with different model scales, including ViT-S, ViT-B, and ViT-L. These tests indicate that our method can effectively scale to larger models, maintaining its performance benefits. We will add a section in our manuscript discussing these aspects, highlighting the versatility and scalability of Dense-JEPA in different contexts.

Dear Reviewer RoAg,

Thank you once again for your valuable efforts and constructive feedback on our paper. As the discussion period comes to a close, we trust that your concerns have been thoroughly addressed in our response. We dedicated significant effort to this, incorporating several new experiments, visualizations, and discussions. We sincerely hope you will consider our reply in your assessment. We look forward to hearing from you, and if the opportunity arises, we can further address unclear explanations and remaining concerns if any.

Regards,

Authors

We deeply appreciate your thorough and insightful reviews. We trust that our response adequately addresses any concerns you may have.

Q1. Our novelty and discussion to dense SSL literature

We acknowledge your suggestion to include references such as DenseCL, VICRegL, and ODIN in our literature review. While these works are indeed relevant, we emphasize that Dense-JEPA's core novelty lies in enhancing the latent JEPA architecture for learning local/dense representations, a different focus from the mentioned works. Our approach is not merely about dense representation learning but about innovatively applying it to the existing latent JEPA framework. For example, our idea is independent to hand-crafted augmentations and we believe that ours can be incorporated with further augmentation-based dense SSL ideas. Nevertheless, we will add regarding discussion in final draft to position our work appropriately within the field.

Regarding why our work is important and have significance, we belive that mask reconstruction on latent space will have gain much attention on its efficacy and efficiency, so study of training objective on latent space is crucial. Therefore, our investigation of the latent objective is invaluable and could inspire future research direction to explore better SSL objective, unlike the previous stereotype of masked reconstruction objective on pixel space.

Dear Reviewer wqKm,

Thank you once again for your valuable efforts and constructive feedback on our paper. As the discussion period comes to a close, we trust that your concerns have been thoroughly addressed in our response. We dedicated significant effort to this, incorporating several new experiments, visualizations, and discussions. We sincerely hope you will consider our reply in your assessment. We look forward to hearing from you, and if the opportunity arises, we can further address unclear explanations and remaining concerns if any.

Regards,

Authors

Q4. Clarify of Figure 2

We acknowledge the need to improve Figure 2 for better clarity and differentiation from I-JEPA. The revised figure includes a clearer distinction and detailed explanation between “Local Aggregation Head” and “Masked Semantic Neighboring” components, emphasizing the distinctions from I-JEPA.

Q5. Qualitative anlaysis on Dense-JEPA

In order to qualitatively demonstrate the effectiveness of the proposed Dense-JEPA, we further compared learned attention maps from the target encoder using pretrained I-JEPA and our Dense-JEPA, and attention maps from the cross-attention layer in the proposed Local Aggregation Target (LAT) modules in Appendix section C in our revised manuscript. Our Dense-JEPA achieves much better attention maps, compared to I-JEPA. Furthermore, we provide more examples on learned cosine similarity map to further demonstrate that our Dense-JEPA performs effectively by encouraging the model to learn dense representations. Regarding Figure 1, we will enhance the visualization in the final draft to more convincingly demonstrate how Dense-JEPA improves upon I-JEPA's semantic information capture, especially in the context of neighboring patches.

Q6. Role of cross-attention in Local aggregation head

The choice of cross-attention for aggregation in Dense-JEPA is deliberate. We experimented with other pooling methods, such as average pooling and max pooling, but found them less effective. Cross-attention allows for more nuanced aggregation of features, which is crucial for our model's performance. We included additional results in Table B.1 in the Appendix to support this claim.

Q7. Rationale of Dense-JEPA

Your question about the role of transformer blocks in our encoder is insightful. While these blocks do perform self-attention among token pairs, our method enhances this process by focusing on semantically relevant patches. This targeted approach is theoretically significant as it guides the transformer to learn more meaningful local semantics. We provided additional evidence in the revised manuscript to demonstrate the superiority of our targeted patch representations over generic self-attention mechanisms, as shown in Figure C1-C4 in the Appendix.

Q2. Comparision with DINO family methods

Thank you for highlighting the comparison between our Dense-JEPA method and the DINO family of methods. We acknowledge the impressive achievements of DINO and DINOv2 in the domain of self-supervised learning. However, we would like to emphasize a few key aspects of our work to clarify our contributions and the significance of our results.

1. Contributions on Numbers

Our results shown in the Table above, particularly in segmentation and detection tasks using a ViT-B/16 backbone, demonstrate significant improvements over DINO. While DINO's performance is commendable, our method shows a marked advantage in these specific areas. This is a crucial aspect of our contribution, as it highlights the effectiveness of Dense-JEPA in handling diverse and complex visual tasks. In the context of DAVIS-2017, where linear evaluation on frozen tasks is critical, our method also shows promising results. While DINO reports higher J&M scores with a smaller ViT-S/16 backbone, our Dense-JEPA method achieves competitive scores with a larger ViT-L backbone in Table 2, suggesting potential for further optimization and tuning to enhance performance.

2. Comparison Fairness with DINOv2

It is important to note that DINOv2 was trained on a significantly larger dataset compared to the dataset used for training Dense-JEPA. This difference in training conditions can substantially affect the performance outcomes. Therefore, a direct comparison may not entirely reflect the intrinsic strengths of our method. Our approach still shows competitive performance under more constrained data conditions, which is a testament to its robustness and efficiency. Compared to DINOv2 using ViT-L/16 using different pre-training resolutions in Figure 6 in the original paper, our Dense-JEPA with ViT-B/16 backbone achieves competitive results (49.0 mIoU vs 46.2 mIoU). However, it should be noted that DINOv2 performed linear evaluation settings on the ADE20K dataset by freezing representations at different resolutions. Although the comparison is not totally fair, these results further demonstrate the competitive fine-tuning performance on segmentation by learning local semantics during self-supervised pre-training. We believe that our new framework can be scaled up to more pre-training data to achieve better downstream performances.

3. Fine-Tuning Comparison with DINO and MAE

In terms of full fine-tuning capabilities, particularly using a ViT-B/16 model, our method exhibits superior performance compared to both DINO and MAE. This is a crucial metric, as fine-tuning is often essential in real-world applications of self-supervised learning models. Our method's adaptability and effectiveness in fine-tuning scenarios underscore its practical utility beyond standard benchmark tasks.

Overall, while we recognize the excellence of the DINO family in certain areas, our Dense-JEPA method demonstrates its unique strengths and contributions, especially in segmentation, detection, and fine-tuning contexts. We believe these aspects significantly contribute to the field of self-supervised learning and merit recognition.

Q3. Computational complexity of Dense-JEPA

We understand your concern about the computational complexity of Dense-JEPA compared to I-JEPA. In our revised manuscript, we include a detailed analysis of total pre-training epochs, max memory usage, and running time per training step in Section B, demonstrating that Dense-JEPA maintains similar efficiency to I-JEPA while achieving superior performance. Besides, our method is much efficient and achieves short training time than MAE.

Dear Reviewer NVsT,

Thank you once again for your valuable efforts and constructive feedback on our paper. As the discussion period comes to a close, we trust that your concerns have been thoroughly addressed in our response. We dedicated significant effort to this, incorporating several new experiments, visualizations, and discussions. We sincerely hope you will consider our reply in your assessment. We look forward to hearing from you, and if the opportunity arises, we can further address unclear explanations and remaining concerns if any.

Regards,

Authors

We deeply appreciate your thorough and insightful reviews. We trust that our response adequately addresses any concerns you may have.

Q1. Our novelty and Clarify for Figure 2

You pointed out that Dense-JEPA's novelty is questionable, particularly in reference to Figure 2's similarity to I-JEPA's Figure 2. We acknowledge this observation and updated Figure 2 to better illustrate the distinctions between Dense-JEPA and I-JEPA.

Regarding why our work is important and have significance, we believe that mask reconstruction on latent space will have gain much attention on its efficacy and efficiency, so study of training objective on latent space is crucial. Therefore, our investigation of the latent objective is invaluable and could inspire future research direction to explore better SSL objective, unlike the previous stereotype of masked reconstruction objective on pixel space. To clarify, Dense-JEPA's novel contribution lies in its unique approach to enhancing target representations. Unlike I-JEPA, Dense-JEPA employs a more nuanced strategy in selecting neighboring patches based on semantic similarity. This approach leads to a more accurate and granular understanding of local semantics, significantly contributing to the field of dense representation learning.

Q2. Clartiy for MSN and LAT

We agree that the descriptions of the MSN (Masked Semantic Neighboring) and LAT (Local Aggregation Target) modules could be more precise. MSN is indeed preparatory for LAT, but it is crucial for establishing fine-grained semantic relations among patches. On the other hand, LAT focuses on a coarser level of aggregation.

In the revised paper, we provided a clearer distinction between these two modules in Figure 2. MSN is not just a preparatory step but a critical component in refining the target representation by identifying the most semantically relevant neighbors. LAT then aggregates these for a comprehensive representation. We will revise further for better clarity in the final draft.

Regarding your question about the role of MSN without LAT (Table 5, second row), MSN independently contributes to the training procedure by identifying the most semantically relevant patches. This process is crucial for Dense-JEPA's effectiveness, as it ensures that even in the absence of LAT, the model doesn't revert to a coarse, generalized representation but maintains a degree of fine-grained semantic understanding (e.g., 53.7->55.2 scores in Table 5). Adding MSN to the vanilla baseline highly increases the results of the video segmentation performance regarding all metrics, which validates the benefit of masked semantic neighboring in finding semantically similar neighboring patches for masked patches (e.g., 53.7->55.2 scores in Table 5).

Without MSN, Dense-JEPA would lack the nuanced understanding of local semantics, leading to a more generalized and potentially less accurate representation (e.g., 57.1->54.6 scores in Table 5).