InfoAug: Mutual Information Informed Augmentation for Representation Learning

First of all, thank you very much for your acknowledgement of the article, including the novelty of our ideas and the validity of the method. We also thank you for your time in reviewing the article. In response to your questions, we provide the following answers.

1. First, for the first question, using only neighboring patches as positive samples, although simple in idea, is not rigorous in principle, and will greatly reduce the performance of the model. For example, imagine a dog standing on the edge of a flowerpot, due to their physical proximity, using neighboring patches as positive samples would force the model to push the representation of the dog and the flowerpot closer to each other, however in reality they are not objects of the same class, thus posing significant damage to the model.

2. For the second question, your suggestion is exactly one of our considerations in this article, we confirmed the effectiveness of InfoAug with great certainty through a large number of comparative and ablation experiments, and we would prefer that subsequent work continues to explore how to manipulate the large in-the-wild dataset to achieve an organic fusion with InfoAug, and to gain the ability to learn from the video dataset with a large amount of unsupervised learning capability from video datasets.

3. For the third problem, first, we give the following specific data as a reference, for a video data of 80 frames, we only need one second to complete the tracking of nearly 100 points on the whole video. For mutual information estimation, even with the most common algorithm, we only need 0.0025 seconds to complete the mutual information estimation for a pair of random variables with nearly 50 observations. Further, in order to demonstrate the effectiveness of InfoAug, we conducted comparison experiments using randomly selected positive samples, and found that merely increasing the number of iterations (amount of data) does not lead to effective performance improvement, thus illustrating the absolute superiority of mutual information as a selection metric.

4. Finally, on the question of how to determine the patch size, we are not that clear on what you mean by "main dataset". In any case, patch size is an hyperparameter that should be determined by roughly estimating the size of individual objects contained in the video dataset. Although we found in our experiments that the choice of this hyperparameter does not bring much effect variation, it is still a case-by-case (dataset) choice, and the ultimate goal is to expect the patch size to match the majority of the objects in the scene to get the best possible cover effect.

Finally, thank you for your questions, especially the first one which is intriguing. We hope our answers have solved your confusion.

First of all, we sincerely appreciate the valuable time of the reviewer and thank you for your questions and corrections. We also appreciate your recognition of the novelty and the acknowledgment of the use of mutual information and dual branch formulation in our article.

Now, I will respond to your corrections and questions in a sequential order:

1. Regarding the first concern, we appreciate that you noticed our argument about the size of the pretraining dataset. Since the existing InfoAug is well-suited for learning from small to medium-sized datasets by imitating the learning process of humans through mutual information estimation, we considered that using a larger model such as ResNet-50 for training would lead to overfitting, as ResNet-18 is already a large model for the existing dataset scale. This way, we could not effectively showcase the performance of InfoAug itself.

2. For the second concern, we agree with your suggestion, and thus, we examined prominent state-of-the-art models such as SimCLR, BYOL, MoCoV2, which indeed employ a two-layer fully connected head as the projection head. Consequently, we conducted experiments based on this suggestion, and the results are put into Appendix A.3. The results still, under standard 2-layer projection head, show a consistent improvement over all the SOTA models. As for your concern about "random-twin-patch," I believe there might be a misunderstanding regarding the purpose of that experiment. We intentionally used randomly selected patches to illustrate the importance of mutual information as the metric for selection. Randomly selecting patches as positive samples without considering mutual information would not yield good results. This experiment was intentionally done to provide a comprehensive demonstration of the significance of mutual information in InfoAug, as a comparative study to only adopt random selection.

3. Regarding the third concern, we chose 100 epochs as the main experiment, as we found that it almost reached convergence within that timeframe. The reason behind this is that we used patches as training units instead of whole images, which increased the number of iterations per epoch by a factor of N (where N is the number of patches in an image).

Regarding your questions:

1. Certainly, we would be pleased to explain the specific process of patch selection. Each image is divided into N non-overlapping patches. Although using overlapping patches may further improve performance, we intentionally avoided it to demonstrate the true effectiveness of InfoAug. Overlapping patches may contain the same object, which would not clearly illustrate the necessity of mutual information guidance. Additionally, the patches are uniformly divided without any artificial design, intended to not hurt its generalizability.

2. For the second question, yes, the model is first pretrained on a video dataset without labels and then fine-tuned and tested on an image dataset. This aligns with the designed workflow of InfoAug and resembles the way humans learn.

We hope our additional experiment and explanation shall address your concern, and could be seen by more, allowing for further consideration of integrating mutual information into contrastive learning.

First of all, thank you very much for your time devoted in giving us the valuable feedback and your acknowledgement on the novelty of our method and design of loss function.

With respect to the questions you raised, we would like to provide the following responses:

1. Regarding the doubts about the first, second, and third points, we offer the following explanation: We aim to provide a more rigorous method for defining positive samples, inspired by human learning. In a given environment, we observe various objects and obtain a rough estimate of the mutual information between objects. One implicit requirement of this method is that we have a sufficient number of observations of objects in an environment. This requirement is naturally fulfilled in human learning, but it is severely violated in large-scale in-the-wild video datasets. These datasets often suffer from significant camera movements and occasional scene changes. Sometimes, an object only appeared for first frame and quickly left the camera in subsequent frames. This setup is not consistent with our human learning perception and does not clearly demonstrate the intended concept of InfoAug, which aims to judge positive and negative samples based on the moderate perception of information sharing between objects. Although we have conducted corresponding experiments on large-scale datasets, even achieving results comparable to traditional SOTA models, we did not focus solely on this aspect. It is unrelated to the experimental results but only mismatched with the method we wanted to convey. Therefore, we explicitly explained this mismatch with large in-the-wild datasets in our paper, hoping that future work can further expand this method and find unrestricted learning approaches for InfoAug on large-scale datasets. Nevertheless, we believe this should not hinder the demonstration of InfoAug's powerful features. Through a combination of 21 models and datasets, 3 comparative experiments, and ablative experiments across 5 scenarios, we demonstrated the improvement of InfoAug over the existing SOTA baseline, proving the interesting connection between mutual information and positive/negative samples. We hope this method of defining positive and negative samples can be further popularised.

2. Regarding your fourth point, we sincerely appreciate you pointing out the issue, and we have corrected the errors in the paper. To present more clearly the improvement of InfoAug over the existing SOTA and provide more detailed usage instructions, we have supplemented the appendix with experiments and calculations of the computational cost of mutual information on different projection heads. Additionally, we have made modifications to two figures in the paper to enhance readers' understanding of the natural workings of InfoAug. As for your fifth suggestion, we greatly appreciate your proposal and assistance. However, since CLIP is a multimodal model for language and image embedding, and our method does not involve the language modality, we would appreciate it if you could provide further clarification and explanation.

Thanks!

Thank you for your response. However, I think the author's response does not clarify any of my concerns, especially on the comparison against existing methods. It is hard to understand the rebuttal despite the simple question of why the baseline method's performance does not match the reported numbers (I believe reviewer LYQ4 asks similar questions). I'll stick to my current score.

Thank you very much for your time in reviewing our work, giving valuable feedback. We also thank you for giving credit that it serves as novel for contrastive learning in video dataset.

Now, we will address your concern down below in a sequential order:

1. Regarding your concern 1 and 2, yes, we use a small-scale dataset for pretraining, which leads to a downscaled accuracy on the downstream task. The reason behind that lies in what we would like to contribute with this paper: To demonstrate and elicit a natural definition of “positive samples” in contrastive learning. The inspiration lies in how we human recognizes positive samples: we learn in an environment with necessary amount of observation (you cannot see two cats walking together for one millisecond and learning they are of same species). The problem with current large scale video dataset is that, the scene is dynamically moving, leading to extremely insufficient observation for almost any object, which greatly differs from human-learning settings. So, although some of the result looks competitive with the traditional contrastive learning paradigm on those in-the-wild datasets, which exhibits high camera dynamic, the result is a compromised one in that the advantage of InfoAug is offset by the divergence of the intentional learning setting and that of large video dataset shoot casually, which might be misleading to the question of “what is the strength and how to apply it”. Therefore, we make clear our intention and proposed learning setting explicitly in the paper to avoid the misunderstanding of the suitable application of InfoAug, and test its performance on video dataset that do offers similar level of observation for a group of objects. The main results on three major downstream dataset, ablation study on loss weight and training epoch, all consistently shows the benefit of InfoAug as a valuable approach for defining positive samples. We notice the fact that the improvement ranges from 1% to 4% and we hope the improvement will be larger if the scale can be further expanded by future work. To this extent, we also examined if InfoAug really bring significant result by doing various comparison experiment to ablate out all the possible contributing factor, including the increased training sample, dual-branch formulation. and it shows that all the consistent contribution vanish, without using InfoAug to provide mutual-information awareness, demonstrating the absolute value in defining “positive sample” by InfoAug.

2. For your third concern, we attach the relevant diagram at the end of appendix for your valid concern. For a robot to determine the positive samples in a video, where we assume it focus simultaneously on as much as 40 objects, for an interpolation of 50 frames, it only took 2 seconds on a single CPU core. If, as we did by multi-thread it with 16 working thread, it is reduced to less that 0.2 second. For the most updated MI estimation technique, it could be further reduced to 0.1 second without any multi-threading, without GPU-acceleration.

In summary, we show by multiple comparative experiment (please see more in section 4.1, 4.2 and 4.3)that the improvement is significant and consistent over any SOTA model-dataset combination, that the way of defining positive samples by mutual information is both natural and of great potential, we hope such an idea could be seen by others to together expand it towards to more unified contrastive learning approach.