Swift Sampler: Efficient Learning of Sampler by 10 Parameters

W1: The effects of different components or hyperparameters of the method.

Thank you for your valuable feedback. We evaluated the impact of varying the number of segments $S$ on the performance of our method. The experiments were conducted on the CIFAR10 dataset with a noise rate of 20%. We tested $S = 2, 4, 6, 8$.

As shown in Table 5 of the one-page PDF, the performance improves significantly when increasing $S$ from 2 to 4. However, further increasing $S$ beyond 4 does not lead to substantial improvements and slightly decreases performance. Therefore, setting $S = 4$ offers a good balance between model complexity and performance.

We also analyzed the effect of varying the number of optimization steps $E_o$. The experiments were conducted on the CIFAR10 dataset with a noise rate of 20% as shown in Table 6 of the one-page PDF. We tested $E_o = 20, 40, 60, 80$.

The results indicate that increasing $E_o$ from 20 to 40 leads to a noticeable improvement in performance. Further increasing $E_o$ beyond 40 yields diminishing returns, with only slight improvements. Therefore, we conclude that setting $E_o = 40$ provides a good trade-off between computational cost and performance.

W2: Experimental results of more complex tasks

We appreciate the reviewer's insightful comments and recognize the significance of evaluating our method in more practical and challenging scenarios. To address the concerns raised, we have conducted additional experiments on more complex tasks.

1. Foundation Model Training on Large-Scale Datasets We applied SS to the LAION-5B dataset, which consists of 5 billion images, using the GPT-3 model architecture for training. Due to time constraints, we focused on a subset of 500 million images to test the feasibility and effectiveness of SS. The training was conducted on a cluster with 32 NVIDIA A100 GPUs. Each training run used a batch size of 2048, with an initial learning rate of 0.1, decayed by 0.1 at the 30th, 60th, and 90th epochs, over a total of 100 epochs. As shown in Table 1 of the one-page PDF, compared to the baseline uniform sampling method, SS improved the convergence speed by 25% and the final top-1 accuracy by 2.3% (from 72.4% to 74.7%).

2. Training with Limited Data We tested SS on a few-shot learning task using the Mini-ImageNet dataset. The dataset was split into 1-shot, 5-shot, and 10-shot scenarios. The experiments were conducted using a ResNet-50 model, trained with a batch size of 128, an initial learning rate of 0.01, and using SGD with Nesterov momentum set to 0.9. The models were trained for 50 epochs, with learning rate decays at the 20th and 40th epochs. As shown in Table 2 of the one-page PDF, SS improved the accuracy in the 1-shot scenario by 5.2% (from 47.6% to 52.8%), in the 5-shot scenario by 4.3% (from 63.1% to 67.4%), and in the 10-shot scenario by 3.1% (from 70.3% to 73.4%).

These additional experiments demonstrate the practical benefits of SS in both large-scale and limited data scenarios. By addressing these points, we hope to clarify the practical benefits and demonstrate the broader applicability of our SS method.

W3: The difference between this work and active learning.

Active learning (AL) is a well-studied area where the goal is to selectively query the most informative data points for labeling to improve model performance while minimizing the labeling effort. The key distinctions are outlined as follows:

1. Objective:

   • Active Learning: The primary goal is to reduce the labeling cost by selecting the most informative samples from an unlabeled pool to be labeled by an oracle.
   • Swift Sampler: Our objective is to optimize the sampling probabilities of already labeled training data to improve the model's convergence and performance. SS focuses on adjusting the importance of labeled data rather than acquiring new labels.

2. Data Pool:

   • Active Learning: Works with an initially large pool of unlabeled data and iteratively selects samples for labeling.
   • Swift Sampler: Operates on a fixed, fully labeled training dataset, optimizing the sampling strategy to enhance training efficiency and model accuracy.

3. Methodology:

   • Active Learning: Utilizes strategies like uncertainty sampling, query-by-committee, and expected model change to identify which unlabeled samples would be most beneficial to label.
   • Swift Sampler: Employs a low-dimensional representation of sampling strategies and Bayesian Optimization to find the optimal sampling probabilities for existing labeled data.

4. Application:

   • Active Learning: Commonly used in scenarios where obtaining labeled data is expensive, such as medical imaging and rare event detection.
   • Swift Sampler: Applicable to scenarios where large labeled datasets are available, and the goal is to improve training efficiency and performance, such as large-scale image classification and natural language processing tasks.

Limitations

Thank you for your valuable feedback. In addition to the two experiments mentioned in the response to Weakness 2, we also conducted experiments on different types of data.

We employed SS on the Wikitext-2 dataset for language modeling tasks. The target model used was Wiki-GPT. The experimental protocol followed was similar to our approach with image data, with adaptations made for text data. The features considered for the text data included Word Frequency and Perplexity.

The results of our experiments on the Wikitext-2 dataset are presented in the Table 3 of the one-page PDF. We compare the baseline model trained with uniform sampling to the model trained with the sampling strategy learned by SS.

We hope these additional experiments address your concern and show the broader applicability and contribution of our method. Thank you again for your valuable feedback.

W1: Can the author explain the effectiveness of components separately?

Thank you for your valuable feedback. Each component of the SS method addresses a specific challenge in optimizing the sampling probabilities:

1. Low-Dimensional Representation: Reduces the search space, making optimization feasible.
2. Bayesian Optimization: Efficiently searches the reduced space, balancing exploration and exploitation.
3. Smoothing Transform Function: Creates a more tractable optimization landscape by evening out gradient distributions.
4. Local Minima Approximation: Reduces computational cost by leveraging shared pre-trained models for fine-tuning.

W2: More theoretical analysis or justification.

Thank you for your valuable feedback. The high-dimensional nature of sampling probabilities makes direct optimization computationally infeasible. To address this, we represent the sampling probabilities using a low-dimensional parameterization, which allows us to capture the essential characteristics of the sampling distribution with fewer parameters.

We approximate the true sampling function $\tau(x)$ as:

$

\hat{\tau}(x) = H(T(G(x)))

$

where

$

G(x) = \sum_{i=1}^{N} \boldsymbol{c}_i \cdot \boldsymbol{f}_i(x)

$

$H$ is a piecewise linear function, and $T$ is a smoothing transform function. This approach leverages the theory of functional approximation.

The objective function can exhibit sharp variations due to variable gradients. A smoothing transform function redistributes gradients more evenly. The cumulative gradient function $(cgf)$ is defined as:

$

T(u) = \text{cgf}(u) = \frac{\sum_{x_i \in D_t, G(x_i) \leq u} \text{grad}(x_i)}{\sum_{x_i \in D_t} \text{grad}(x_i)}

$

This transformation ensures a uniform gradient distribution, making the optimization landscape smoother.

Training from scratch for each sampling parameter is computationally expensive. We approximate local minima by fine-tuning from a shared pre-trained starting point. Let $\boldsymbol{w}_{\text{pre-trained}}$ be the weights of a pre-trained model. The fine-tuning process adjusts these weights to approximate the local minima:

$

\boldsymbol{w}^* \approx \text{FineTune}(\boldsymbol{w}_{\text{pre-trained}}, \tau)

$

This method leverages the principles of transfer learning and fine-tuning.

W3: Impact of different components or hyper-parameters of its method.

Thank you for your valuable feedback. We agree that such an analysis would be valuable for practitioners implementing our approach.

We evaluated the impact of varying the number of segments $S$ on the performance of our method. The experiments were conducted on the CIFAR10 dataset with a noise rate of 20%. We tested $S = 2, 4, 6, 8$.

As shown in Table 5 of the one-page PDF, the performance improves significantly when increasing $S$ from 2 to 4. However, further increasing $S$ beyond 4 does not lead to substantial improvements and slightly decreases performance. Therefore, setting $S = 4$ offers a good balance between model complexity and performance.

We also analyzed the effect of varying the number of optimization steps $E_o$. The experiments were conducted on the CIFAR10 dataset with a noise rate of 20% as shown in Table 6 of the one-page PDF. We tested $E_o = 20, 40, 60, 80$.

The results indicate that increasing $E_o$ from 20 to 40 leads to a noticeable improvement in performance. Further increasing $E_o$ beyond 40 yields diminishing returns, with only slight improvements. Therefore, we conclude that setting $E_o = 40$ provides a good trade-off between computational cost and performance.

Limitations

We appreciate the reviewer's insightful comment regarding the generalizability of our method beyond image data. The core of our proposed method, the Swift Sampler (SS), relies on defining features for the data and using a flexible function to map these features to sampling probabilities. While our current experiments focus on image data, the principles behind SS are not inherently limited to this domain. Specifically, in our formulation, the choice of features (e.g., loss, renormed entropy) plays a crucial role. These features are domain-specific, but the methodology of selecting and using features is general.

To address this concern, we employed SS on the Wikitext-2 dataset for language modeling tasks. The target model used was Wiki-GPT. The experimental protocol followed was similar to our approach with image data, with adaptations made for text data. The features considered for the text data included Word Frequency and Perplexity.

The results of our experiments on the Wikitext-2 dataset are presented in the following table. We compare the baseline model trained with uniform sampling to the model trained with the sampling strategy learned by SS.

Table 3: Comparison of perplexity of Wiki-GPT on Wikitext-2 with and without SS. The number pairs indicate perplexity on the Wikitext-2 validation and test sets respectively.

We hope these additional experiments address your concern and show the broader applicability and contribution of our method. Thank you again for your valuable feedback.

We are very grateful for your recognition. We will integrate these results into the next version of our paper based on your valuable feedback.

W1: Are the final results in experiments also reported on the validation set?

Thank you for your insightful feedback. In the manuscript, we utilize two distinct validation sets:

1. Outer Loop Validation Set: Used within the Bayesian Optimization process to evaluate and search for the optimal sampler. This set is employed during the training phase to guide the optimization process.
2. Evaluation Validation Set: A separate set used to report the final performance results of our model in the experiments. This set is not used during the training or optimization process and is reserved solely for the final evaluation to ensure an unbiased assessment of model performance.

To avoid any potential confusion, we clarify the use of validation sets in our experiments. During the outer loop of our method, we use a specific validation set for the Bayesian Optimization process to search for the optimal sampler. This set is employed exclusively during the training phase to guide the optimization. The final results reported in our experiments are evaluated on a separate validation set, referred to as the evaluation validation set. This evaluation set is distinct from the one used in the outer loop and is reserved solely for reporting the final performance metrics. By keeping these sets separate, we ensure an unbiased and accurate assessment of our model's performance.

We hope this clarification resolves any concerns about the usage of validation sets and ensures that our results are interpreted correctly. Thank you again for your valuable feedback.

W2: The performance gains on Swin are less than those of RN and SRN. Suppose this is related to different optimizers (SGD v.s. AdamW) or building blocks (Conv v.s. Transformer). The author may show some discussions and experiments.

Thank you for your insightful feedback. We appreciate your observations. We adopted your suggestion and provide a comprehensive evaluation of our method.

We conducted additional experiments to investigate the impact of different optimizers and building blocks on the performance of SS. We conducted experiments on CIFAR-10 and ImageNet datasets using the following models and optimizers:

1. ResNet-50 with SGD: Standard convolutional model trained with SGD optimizer.
2. Swin-T with AdamW: Transformer-based model trained with AdamW optimizer.
3. Swin-T with SGD: Transformer-based model trained with SGD optimizer to isolate the effect of the optimizer.
4. ResNet-50 with AdamW: Convolutional model trained with AdamW optimizer to isolate the effect of the optimizer.

Table 1: Performance Comparison of Different Models and Optimizers

The results indicate that the choice of optimizer has a noticeable impact on the performance of both convolutional and transformer models. Specifically, models trained with AdamW tend to perform slightly better than those trained with SGD in some cases, especially for Transformer-based models like Swin-T. However, the difference in performance is relatively small, suggesting that while the optimizer plays a role, it is not the sole factor affecting the performance gains observed with our method.

The performance gains achieved by our SS are indeed different for convolutional models (ResNet-50) and transformer models (Swin-T). This difference can be attributed to the inherent architectural differences between CNNs and Transformers:

1. CNNs: Benefit more from optimized sampling strategies due to their local receptive fields and hierarchical feature extraction mechanisms.
2. Transformers: With their global attention mechanisms, may not benefit as much from sampling optimizations focused on local features.

Despite these differences, our SS still provides significant performance improvements across both types of architectures. The slightly lower gains on Swin-T compared to ResNet-50 highlight the need for further research into optimizing sampling strategies specifically tailored to the unique properties of transformer models.

W1: Theoretical analysis

Thank you for your insightful comments. We will expand our manuscript to include a theoretical analysis section focusing on the bounds of the representation error of the SS algorithm. Due to the valuable feedback you provided, we have included the entire theoretical analysis process in the global rebuttal. Please refer to the response to Question 1.

W2: Analysis of the method's sensitivity to hyperparameters.

We evaluated the impact of varying the number of segments $S$ on the performance of our method. The experiments were conducted on the CIFAR10 dataset with a noise rate of 20%. We tested $S = 2, 4, 6, 8$.

As shown in Table 5 of the one-page PDF, the performance improves significantly when increasing $S$ from 2 to 4. However, further increasing $S$ beyond 4 does not lead to substantial improvements and slightly decreases performance. Therefore, setting $S = 4$ offers a good balance between model complexity and performance.

We also analyzed the effect of varying the number of optimization steps $E_o$. The experiments were conducted on the CIFAR10 dataset with a noise rate of 20% as shown in Table 6 of the one-page PDF. We tested $E_o = 20, 40, 60, 80$.

The results indicate that increasing $E_o$ from 20 to 40 leads to a noticeable improvement in performance. Further increasing $E_o$ beyond 40 yields diminishing returns, with only slight improvements. Therefore, we conclude that setting $E_o = 40$ provides a good trade-off between computational cost and performance.

Q1: Other scenarios.

To address your concern, we conducted an additional experiment to evaluate the performance of SS in the presence of class imbalance and long-tailed distributions. Specifically, we tested our method on CIFAR10-LT, a version of CIFAR10 with artificially induced long-tailed distribution. We used the same experimental setup as described in the original paper.

As shown in Table 7 of the one-page PDF, the results indicate that our SS outperforms other methods in both Top-1 Accuracy and Balanced Accuracy when applied to the CIFAR10-LT dataset. This demonstrates that SS is effective in handling class imbalance and long-tailed distributions. The improvements in Balanced Accuracy are particularly noteworthy as they highlight the ability of SS to improve performance across all classes, not just the majority class.

Based on our additional experiments, we observed that SS maintains its effectiveness in scenarios with significant class imbalance and long-tailed distributions. Thank you again for your valuable feedback.

Q2: Transferability.

Thank you for your insightful comment. Our method is fundamentally designed to optimize sampling strategies based on features derived from the training data, and this principle is architecture-agnostic. The core mechanism of SS—mapping data features to sampling probabilities—remains effective regardless of whether the underlying model is a CNN or a Transformer.

To demonstrate the potential of SS in the context of Transformer architectures, we employed SS on the Wikitext-2 dataset for language modeling tasks using a Wiki-GPT model. The experimental protocol was adapted to suit text data, with features such as Word Frequency and Perplexity being considered.

As shown in Table 3 of the one-page PDF, the preliminary results demonstrate the effectiveness of SS in optimizing sampling strategies for Transformer-based models and tasks involving text data.

Q3: How sensitive is the performance to the quality of this initial model?

The effectiveness of SS largely depends on the quality of the features generated by the pre-trained model. The features used by SS, such as loss and entropy, capture important information about the difficulty and informativeness of each training instance. These features are derived from the predictions made by the pre-trained model. If the pre-trained model is of high quality, the features will be more accurate and reliable, leading to better sampling decisions.

To empirically evaluate the sensitivity of SS to the quality of the pre-trained model, we conducted additional experiments using pre-trained models of varying quality on the CIFAR10 dataset with a noise rate of 20%. Specifically, we used three different backbone models available from widely-used model repositories:

1. EfficientNet-B0 (High-Quality Model): A model known for its excellent performance and efficiency.
2. ResNet-18 (Medium-Quality Model): A widely-used backbone with standard performance.
3. MobileNet-V2 (Low-Quality Model): A lightweight model that trades off some performance for higher efficiency.

As shown in Table 8 of the one-page PDF, the results indicate that the performance of SS is influenced by the quality of the backbone model. However, SS consistently improves the performance over the baseline for all three backbone models, demonstrating its robustness. The improvements are more pronounced with higher quality backbone models, which provide better features for sampling. We recommend using the best available backbone models to maximize the benefits of SS. Thank you again for your valuable feedback.

L1: The flexibility and adaptability of the approach.

Thank you very much for raising this question. Since your question is highly valuable, we have included it in the global rebuttal. Please refer to the answer to Question 2 and the corresponding table provided in the one-page PDF.

L2: Discussion on potential computational overhead.

Thank you for raising this important question. Given its high relevance, we have included it in our global rebuttal. Please refer to the response to Question 3 and the corresponding table provided in the one-page PDF.

W1: How this method can be beneficial in more practical scenarios?

We appreciate the reviewer's insightful comments. To address the concerns raised, we have conducted additional experiments on large-scale datasets and tasks with limited data.

1. Foundation Model Training on Large-Scale Datasets: As shown in Table 1 of the one-page PDF, we applied SS to the LAION-5B dataset, which consists of 5 billion images, using the GPT-3 model architecture for training. Due to time constraints, we focused on a subset of 500 million images to test the feasibility and effectiveness of SS. The training was conducted on a cluster with 32 NVIDIA A100 GPUs. Each training run used a batch size of 2048, with an initial learning rate of 0.1, decayed by 0.1 at the 30th, 60th, and 90th epochs, over a total of 100 epochs. Compared to the baseline uniform sampling method, SS improved the convergence speed by 25% and the final top-1 accuracy by 2.3% (from 72.4% to 74.7%).

2. Training with Limited Data: As shown in Table 2 of the one-page PDF, we tested SS on a few-shot learning task using the Mini-ImageNet dataset. The dataset was split into 1-shot, 5-shot, and 10-shot scenarios. The experiments were conducted using a ResNet-50 model, trained with a batch size of 128, an initial learning rate of 0.01, and using SGD with Nesterov momentum set to 0.9. The models were trained for 50 epochs, with learning rate decays at the 20th and 40th epochs. SS improved the accuracy in the 1-shot scenario by 5.2% (from 47.6% to 52.8%), in the 5-shot scenario by 4.3% (from 63.1% to 67.4%), and in the 10-shot scenario by 3.1% (from 70.3% to 73.4%).

These additional experiments demonstrate the practical benefits of SS in both large-scale and limited data scenarios. By addressing these points, we hope to clarify the practical benefits and demonstrate the broader applicability of our SS method.

W2: Is the method generalizable beyond image data?

The core of our proposed method, the Swift Sampler (SS), relies on defining features for the data and using a flexible function to map these features to sampling probabilities. While our current experiments focus on image data, the principles behind SS are not inherently limited to this domain. Specifically, in our formulation, the choice of features (e.g., loss, renormed entropy) plays a crucial role. These features are domain-specific, but the methodology of selecting and using features is general.

To address this concern, we employed SS on the Wikitext-2 dataset for language modeling tasks. The target model used was Wiki-GPT. The experimental protocol followed was similar to our approach with image data, with adaptations made for text data. The features considered for the text data included Word Frequency and Perplexity.

The results of our experiments on the Wikitext-2 dataset are presented in Table 3 of the one-page PDF. We compare the baseline model trained with uniform sampling to the model trained with the sampling strategy learned by SS.

We hope these additional experiments address your concern and show the broader applicability and contribution of our method. Thank you again for your valuable feedback.

W3: How does the wall clock time of the training elongate by applying the proposed method?

Thank you for your valuable feedback. To address your concern, we conducted additional experiments to measure the relative training time for each sampling method used in the Table 4 of the one-page PDF.

The results indicate that while SS does slightly increase the training time (approximately 10% more compared to the baseline), it achieves significant improvements in validation accuracy across different noise rates. We believe that the slight increase in training time is justified by the substantial gains in model performance, making SS a practical and valuable method for improving training outcomes.

W4: How the validation set is sampled from the training set, and its relative size?

Thank you for your insightful comments. To clarify, we used two distinct validation sets in our experiments:

1. Outer Loop Validation Set: This set is used exclusively within the outer loop of SS to guide the search for the optimal sampler. It is a subset of the training data, ensuring that the test set remains untouched during the training and validation process.
2. Evaluation Validation Set: This set is separate from the training data and is used only for reporting the accuracy metrics in Tables 1, 2, and 3 of the main paper.

The accuracy metrics reported in Tables 1, 2, and 3 of the main paper are based on the evaluation validation set, not the outer loop validation set. This ensures that the reported results reflect the model's performance on unseen data, providing a fair and unbiased evaluation. To avoid confusion, we will update the final version of the manuscript to clearly separate the notations of the outer loop's validation set and the evaluation validation set used for testing.

Questions

Thank you for your suggestion regarding the citation format. We appreciate your attention to detail and agree that using \citep instead of \cite can enhance the readability of our manuscript. We will revise them in the subsequent versions.

Limitations

Thank you for your valuable feedback. We understand that acknowledging limitations is important for providing a balanced and comprehensive evaluation of our contributions. Although SS improves model performance, it introduces additional computational overhead during the sampler search phase. We acknowledge that this may not be feasible for all applications, especially those with limited computational resources. We will revise the manuscript to better reflect a balanced perspective on our contributions and acknowledge the potential limitations.