Collaborative Data Optimization

Complexity in Alignment Strategy: While the target alignment strategy effectively addresses the heterogeneity issue, the approach could become computationally expensive, particularly when aligning multiple participants with highly diverse prior models. The paper could benefit from discussing the potential trade-offs in more detail, including the computational cost of performing alignment versus the gains from collaborative optimization and versus the SSL methods conducted by a single participant.

Limited exploration of scalability: Although COOPT is designed to handle large-scale datasets, the experiments are primarily conducted on datasets like CIFAR and Tiny-ImageNet. It would strengthen the paper if larger datasets such as full ImageNet were used to demonstrate scalability more convincingly.

weak points.

• w1. the method is very simple.
• w2. the description of the method is overly complex.
• w3. the rationale for combining data augmentation and data subsetting techniques is not really clear.
• w4. the experiments are not fully clear.
• w5. the formalization of the problem is wrong.

review.

Researching the combination of different data augmentation techniques is interesting. But the methods proposed by the paper seems very simple, overly complex and the results unclear. In more detail:

w1. the method is very simple. w2. the description of the method is overly complex.

• can you describe your method in pseudocode?
• the most specific step, to align the pseudo targets of the different component methods, is only forward referenced in sec. 3.3, but it is not clear how it affects the overall method.

w3. the rationale for combining data augmentation and data subsetting techniques is not really clear.

• what evidence is there that combining both might be promising?
• how is data subsetting being used in your method? does it select subsets only from its own training data partition?

w4. the experiments are not fully clear.

• can you compare your approach to published results of an existing approach (referencing their tables) and clearly say which information the different approaches used?

w5. the formalization of the problem is wrong.

• a. def. 1 has many errors and loopholes:
  • do you only want to compare training losses? and why? usually one would be interested in validation losses here.
  • you likely mean that T' is given? for a general T' it cannot work, you could just choose T' := 0 or 1.
  • \Phi_{\theta} on the left and the right side of the equation denotes different models. The notation should make this clear.
• b. prop. 1 is not a proposition, it does not state a fact, but it defines what you later call "target distribution inconsistency".
  • this mismatch between the different output dimensionalities of the different component methods should be better introduced.
• c. def. 2 likely has a typo:
  • what is meant by "where (T^(i) and T^(j)) \in [0,1]" ? just a typo and "D_TV" is missing?
  • you later on never measure this quantity G.
  • to what extent is your re-alignment method guaranteeing to re-establish target distribution consistency? it seems just to reduce it somewhat.
• line 206: what does "O(|D|^2/K)^2" mean? (the last square sits outside of "O(...)".)

1. The novelty is limited. The collaborative data optimization method with multiple participants makes sense and has potential practical value, but I don’t think it is a novel method.

2. The writing and organization is good. But the technique soundness is low, lacking of significant and in-depth technical contribution.

3. The theoretical analysis is missing. And many technical details are not well explained. For example, why choosing the uniform value loss for selecting prior model? It is better to add theoretical analysis about the target alignment since the alignment is very important in the proposed collaborative data optimization framework.

4. Many experimental details are not introduced. How many participants are used? How to split the unlabeled data for data optimization? How to process the scenarios where the input datasets are totally different from the prior datasets?

1. Lack of comparative analysis of alignment strategies: Although the paper introduces a target alignment strategy, it lacks a thorough comparison with alternative alignment or normalization techniques that could handle target inconsistencies. For example, domain adaptation approaches (such as source-free unsupervised domain adaptation by Tian et al., 2024) could potentially address similar issues, and comparing CoOPT's alignment strategy to these might reveal its unique advantages or limitations.

2. Conceptual clarity: While the paper is generally well-organized, with clearly defined sections for motivation, methodology, and experiments, the visual elements provided require further improvement. For instance, figure 1 could be enhanced in quality, figure 4d should show both the training and test accuracies, and figure 3c shows over 90% accuracy without any indication of why there is a sudden gap from figure 3b. Additionally, in Definition 2, the variable "I" is introduced without a prior definition, making the intended meaning unclear.

3. Scalability concerns: The computational and storage costs associated with the different stages of CoOPT are not thoroughly discussed. The authors should provide more insights into the computational complexity of the target alignment process relative to participant count and dataset size.

4. Theoretical justification: The approach's reliance on uniform value as a quality metric for prior models is supported mainly by limited empirical evidence (e.g., Figure 3c), yet the theoretical justifications of this metric in the context of data optimization remain vague. A more thorough theoretical discussion or derivation of why uniform value correlates with target quality would add depth to the method's rigour.

5. Experimental details: The paper would benefit from a clearer description of hyperparameters, hardware specifications (types and counts of GPUs/CPUs), experimental settings (e.g., participant counts in each experiment), and any additional configurations for implementing CoOPT. Moreover, some of the experimental results reported in the paper are unclear. For example, Table 4 shows comparisons across datasets rather than model architectures, and Figure 4d would be more informative if it included both training and test accuracies. The authors should revise the experimental results to improve clarity and correct errors, such as Table 2’s caption reading "four" instead of "three" datasets, an incomplete sentence in the experiments on shared data sizes ("However, increasing from 0.05% to 0.8%"), and Table 1’s subjective assessments, which would benefit from clearer criteria or more objective metrics.

6. Privacy concerns: The paper does not explicitly address or discuss privacy concerns, which could be a potential weakness in collaborative or distributed contexts where data privacy is crucial.

• Lack of Comparative Results: While Table 1 summarizes several alternative data optimization methods, such as KD and DD, the experimental section lacks comparisons with these approaches. Additionally, the authors claim that the heavy costs associated with KD and DD stem from task-specific models. However, could using a pre-trained prior model on a larger dataset (like the ImageNet-1K mentioned in Table 3) effectively reduce data optimization costs?