Data-Evolution Learning

Regarding the label is actually a feature map of projection head and predictor (line 918(This paper essentially rebrands several established concepts from self-supervised learning (SSL) approaches and can be seen as a minor modification of the original SSL framework, MoCo [1], which was one of the first SSL methods. The primary difference lies in how the feature map is generated. Specifically:

In MoCo:

$y = \left(\sum_{n=1}^N (1 - \lambda)^n f_n \right)(x)$

where $f_n$ denotes the feature map generated by the $n$-th iterate of the student model (or a “momentum-updated” encoder). Here, the summation is applied to the model function first, and the combined model output is then applied to the input $x$ .

In DELA:

$y = \sum_{n=1}^N (1 - \lambda)^n f_n(x)$

where each feature map $f_n(x)$ is generated by applying the $n$-th iterate model directly to the input $x$ , and then the outputs are summed.

This difference in parentheses placement results in effectively the same computation process. Thus, DELA lacks substantial novelty, as it simply repackages MoCo’s concept by slightly modifying the feature generation approach.

Also, in line 918, algorithm section, it says the update loss is just contrastive loss calculating the similarity. This indicates that every structructure is same except the order of parenthesis

Furthermore, this dynamic updating of feature maps could be inefficient from a memory perspective, as it requires continuously storing feature maps on the GPU in addition to the existing MoCo algorithm. In this case, the number of floating-point values would be $O(N \times d)$, where N is the number of data points, and d is the size of each feature map. This approach offers no computational advantages over MoCo in terms of efficiency.

[1] He, Kaiming, et al. "Momentum contrast for unsupervised visual representation learning." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

1. There is a significant performance variation depending on the choice of the prior model. A method for either reducing performance discrepancies or pre-selecting prior models capable of achieving high performance is necessary.
2. In Table 2, results are reported for experiments using only 20/40/60% of the total data samples, but the intention behind these experiments is unclear. Additionally, on what criteria was the data selected?
3. While Section 4.2 emphasizes training efficiency, based on the caption in Table 1, this appears to be calculated simply by training steps (if otherwise, please provide the specific criteria used to calculate the training budget). Since DELA requires additional computation costs to evolve data labels, this may not be a fair comparison. A more reliable efficiency metric, such as wall clock time for total training duration, would be beneficial.
4. Given the nature of the methodology, where the model undergoes iterative self-learning, it seems likely that overfitting could be an issue. An analysis of overfitting would be appreciated.

• The largest concern is that it is unclear why the proposed method provides improved generalization performance. The paper claims that the proposed method optimizes the dataset, but in fact, it does not define an objective function for optimizing the data; rather, it simply updates it with existing labels and pseudo-labels represented by the EMA of the model prediction. Why does this achieve the desirable properties described in Definition 1? The paper attempts to prove that the model and dataset converge at the optimal solution by Theorem 1, but the proof in Appendix A uses a different definition of the pseudo-labels and does not appear to prove the property claimed in Theorem 1. The paper should clarify why the generalization performance could be improved by the proposed method more solid evidence..
• The proposed method's design is unconvincing. It restricts the loss function to cosine similarity in updating the model in Eq. (3), but there is no sufficient explanation for this.
• The problem setting is ambiguous. In the experiments, the paper uses self-supervised learning (SSL) and weakly supervised learning (WSL) as baselines for comparison. However, the proposed method and these paradigms are orthogonal and should not be compared to each other. Therefore, comparisons with SSL and WSL are not meaningful in asserting the validity of the method.
• The paper does not compare to a simple but important baseline; the blending of labels by Eq. (4) is similar to knowledge distillation because knowledge distillation also uses the prediction of a teacher model for training student models. In this sense, the paper should be the most naive baseline for the dataset created by Eq. (4) using the trained teacher model with the original datasets.
• The proposed method is not a perfect data-centric approach. Although the paper introduces the proposed method as a data-centric approach, experimental results such as Table 1 confirm that there are model-dependent differences in accuracy. The data-centric approach [a] is a general term for methods that fix the model and improve the data. In this sense, the proposed method is a model-centric approach rather than data-centric.
• Figure 1 is misleading because it draws the input sample $x$ as if it is being optimized, but in fact, just data augmentation is applied.
• The paper emphasizes the proposed method's efficiency but ignores the cost of updating datasets. Table 1 shows that the optimal prior model for each target dataset and architecture is different, so the proposed method just adds a new hyperparameter, which does not solve the efficiency issue.
• The experimental protocol is unclear and unreliable. For example, the SSL experiment in Sec. 4.5 claims to use an unlabeled dataset. However, the DeLA results in Table 4 are the same as the DeLA* results in Table 2, and thus, it is possible that they are comparing results trained on a supervised dataset with the results on the unsupervised dataset.

[a] Zha, Daochen, et al. "Data-centric artificial intelligence: A survey." arXiv preprint arXiv:2303.10158 (2023).

This method partially relies on leveraging the pre-trained models, which could be restrictive for the model training without relevant prior models/datasets.

The tuning of hyper-parameters seems critical for achieving good results, which might be challenges for reproducing the results on different model structures.

There has no comparison results with other methods on the table.