Align-VL: Can Being Modest Help in the Alignment of Vision-Language Models?

W1: In lines 080 to 082, the claim lacks convincing support. The ALIGN model [1] has shown through training on billions of noisy image-text pairs that dataset scale can compensate for noise and yield state-of-the-art representations, which contradicts the author’s claim. Additional references or empirical studies would strengthen this argument.

ALIGN demonstrates that large-scale datasets can yield strong representations despite noise, which we discuss further below:

1. ALIGN achieves strong performance with billions of image-text pairs but at high training costs, limiting its applicability in resource-constrained scenarios. Align-VL addresses this by enhancing noise robustness and reducing data and computational demands, enabling broader applicability.

2. ALIGN no longer achieves state-of-the-art performance*, as shown in the table. Align-VL enhances computational and data efficiency for modal alignment in resource-efficient settings, outperforming state-of-the-art methods on MSCOCO with significant improvements in R@1 scores. Using just 3.5M data, Align-VL surpasses ALIGN’s performance on MS COCO, achieving this with 300 times less data than ALIGN.

3. ALIGN relies on large-scale noisy datasets for performance but is heavily impacted in resource-constrained scenarios. Align-VL mitigates noise effects by simulating uncertainty in embedding spaces and addressing overconfidence from weakly correlated data, acting as a feature enhancement for alignment. Incorporating Align-VL’s strategies into ALIGN could further boost its performance.

4. We will add references [1-2] to support the claim that limited data amplifies the impact of low-quality samples, leading to model overconfidence and incorrect alignment with mismatched pairs.

We will include additional references and experiments in the revision as you suggested.

[1] Wang, Y., Hong, J., Cheraghian, A., Rahman, S., Ahmedt-Aristizabal, D., Petersson, L., & Harandi, M. (2024). Continual test-time domain adaptation via dynamic sample selection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 1701-1710).

[2] Xu, Y., Ding, J., Zhang, L., & Zhou, S. (2021). Dp-ssl: Towards robust semi-supervised learning with a few labeled samples. Advances in Neural Information Processing Systems, 34, 15895-15907.

W2: The proposed methods, Random Perturbation and Embedding Smoothing are existing methods and are often used as training engineering techniques, which limits their technical novelty.

We further elaborate on the unique design and contributions of Random Perturbation and Embedding Smoothing to multimodal alignment tasks: We reject this biased judgment, as our contributions are clearly outlined below:

1. Previous methods apply random perturbation by adding Gaussian noise at the input level (e.g., image), while Align-VL introduces it in the embedding space to simulate uncertainty in image-text embeddings, addressing overconfidence from weakly correlated data. Embedding smoothing, rarely mentioned in prior literature, has been occasionally applied in knowledge graphs, but it differs significantly from its use there, where it enforces semantic smoothness [3]. A similar technique to embedding smoothing is label smoothing. However, unlike label smoothing (unimodal classification), embedding smoothing in Align-VL adjusts the embedding space distribution to alleviate over-reliance on positive and negative pairs, smoothing their similarity distribution and reducing overconfidence, which is crucial for image-text alignment tasks.
2. In Align-VL, Random Perturbation and Embedding Smoothing address weakly correlated samples and overconfidence by introducing controlled uncertainty and smoothing in the embedding space, tailored for multimodal alignment. Unlike traditional unimodal methods, these innovations optimize alignment with limited data and resources, enhancing robustness and performance. This unique combination gives Align-VL a significant advantage in resource-constrained scenarios.

[3] Guo, S., Wang, Q., Wang, B., Wang, L., & Guo, L. (2015, July). Semantically smooth knowledge graph embedding. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (pp. 84-94).

W1: The proposed method introduces new parameters, i.e. sigma and alpha. It is needed to select proper parameters.

Thank you for your attention to parameter selection! The roles of parameters $\sigma$ and $\alpha$:

1. • $\sigma$: Regulates random perturbation intensity, introducing noise to simulate uncertainty in embeddings and reduce overconfidence from weakly correlated data. Smaller $\sigma$ ensures stability, while larger $\sigma$ risks excessive uncertainty.

• $\alpha$: Controls embedding smoothing to reduce overconfidence when handling weakly correlated positive samples. It smooths the embedding space, making similarity predictions for image-text pairs more robust. An appropriate $\alpha$ enhances generalization while minimizing reliance on perfectly matched positive samples.

2. Parameter Selection: During experiments, we determined the optimal $\sigma$ and $\alpha$ values through grid search on the test set to ensure robust alignment. Results showed that $\sigma$ (0.01) and $\alpha$ (0.1) achieved the best performance (Dynamic $\alpha$ refers to a strategy where the value of $\alpha$ decreases dynamically as training progresses).

W2: It seems that how performances are effected with varying sigma and alpha.

Thank you for your attention to the impact of parameters $\sigma$ and $\alpha$ on model performance! Below, we provide further clarification on their roles and effects on performance:

1. Impact of parameter $\sigma$: As the control parameter for random perturbation intensity, $\sigma$ influences the model’s robustness in the embedding space. Experimental results indicate:

• Smaller $\sigma$ values (e.g., 0.005) maintain embedding stability but may be insufficient for handling weakly correlated samples.
• Larger $\sigma$ values (e.g., 0.05 or higher) enhance the model’s adaptability to noise but may overly disturb the feature space, negatively impacting alignment accuracy.

2. Impact of parameter $\alpha$: As the control parameter for embedding smoothing, $\alpha$ influences the distribution smoothness between positive and negative pairs. An appropriate $\alpha$ value effectively mitigates overconfidence issues and enhances generalization:

• Lower $\alpha$ values (e.g., 0.05) may fail to adequately reduce the model’s overconfidence.
• Higher $\alpha$ values (e.g., 0.3) may make the model less confident even for fully matched positive samples, causing interference with positive pair alignment.

3. Parameter search experiments: To quantify the specific impact of $\sigma$ and $\alpha$ on performance, we conducted experiments to systematically test how different $\sigma$ and $\alpha$ values affect the model’s performance.

Thank you for the reviewer’s suggestion! We will include the relevant experimental results in the final version to help readers better understand the impact of parameters on model performance.

Q1: Are the proposed methods generally effective across more benchmarks, such as the zero-shot image classification datasets mentioned above, as well as additional image-text retrieval datasets?

Thank you for highlighting the broad applicability of our method! Regarding Align-VL’s performance on additional benchmarks, we have the following responses and plans:

1. In our experiments, we used multiple training datasets including COCO, SBU, and CC3M to validate Align-VL’s performance. We chose the Flickr dataset for testing because of its frequent use in image-text retrieval tasks, facilitating standardized comparisons with other models. Additionally, we tested on the MS-COCO dataset (25K) to assess Align-VL’s generalization across different scales, demonstrating its superior performance across various datasets.

2. Align-VL aims to optimize multimodal alignment while minimizing data and resource demands. Evaluating zero-shot classification is essential to further validate its generality.

• To evaluate Align-VL’s out-of-distribution performance on tasks like zero-shot classification and understand its cross-task applicability, we analyzed its performance on diverse medical datasets (CheXpert, Shenzhen, IDRiD, Brain Tumor). Specifically, 50 images from different categories were selected as subsets. As shown in the table, Align-VL achieved a 77.21% accuracy on the CheXpert, significantly outperforming CLIP’s 53.33% accuracy. On other datasets, Align-VL’s results were close to CLIP’s classification performance.
• To validate Align-VL’s accuracy on natural images, we tested it on Kaggle’s open-source D&C dataset. Results showed that CLIP achieved 100.00% accuracy, significantly outperforming Align-VL’s 67.31%. These results highlight CLIP’s strength in its advantage of pretraining on a 400M dataset. However, Align-VL demonstrated effectiveness across multiple datasets in zero-shot classification tasks, underscoring its generalization capabilities.

With only 3.5M data, Align-VL achieves performance comparable to or even surpassing CLIP on certain datasets, while using 100 times less data and over 600 times less training time than CLIP, which requires 3000 GPU days and 400M training data. This highlights the importance of Align-VL’s applicability in resource-constrained environments.

W1: The experiments presented in the paper are not sufficient and would benefit from the inclusion of additional evaluation datasets and tasks. For instance, incorporating zero-shot image classification results could enhance the robustness of the findings.

Thank you for the valuable suggestions on the experimental section! Regarding the advice to expand the evaluation datasets and tasks, we have the following responses:

1. In our experiments, we used multiple training datasets including COCO, SBU, and CC3M to validate Align-VL’s performance. We chose the Flickr dataset for testing because of its frequent use in image-text retrieval tasks, facilitating standardized comparisons with other models. Additionally, we tested on the MS-COCO dataset to assess Align-VL’s generalization across different scales, demonstrating its superior performance across various datasets.

2. Align-VL aims to optimize multimodal alignment while minimizing data and resource demands. Evaluating zero-shot classification is essential to further validate its generality.

• To evaluate Align-VL’s out-of-distribution performance on tasks like zero-shot classification and understand its cross-task applicability, we analyzed its performance on diverse medical datasets (CheXpert, Shenzhen, IDRiD, Brain Tumor). Specifically, 50 images from different categories were selected as subsets. As shown in the table, Align-VL achieved a 77.21% accuracy on the CheXpert, significantly outperforming CLIP’s 53.33% accuracy. On other datasets, Align-VL’s results were close to CLIP’s classification performance.
• To validate Align-VL’s accuracy on natural images, we tested it on Kaggle’s open-source D&C dataset. Results showed that CLIP achieved 100.00% accuracy, significantly outperforming Align-VL’s 67.31%. These results highlight CLIP’s strength in its advantage of pretraining on a 400M dataset. However, Align-VL demonstrated effectiveness across multiple datasets in zero-shot classification tasks, underscoring its generalization capabilities.

With only 3.5M data, Align-VL achieves performance comparable to or even surpassing CLIP on certain datasets, while using 100 times less data and over 600 times less training time than CLIP, which requires 3000 GPU days and 400M training data. This highlights the importance of Align-VL’s applicability in resource-constrained environments.

Thank you to the reviewer for suggesting additional experiments. We believe these extra tests will further enhance the demonstration of Align-VL’s practicality and robustness.

Q1: The method has not been tested on extremely large datasets. Could the authors address potential challenges in scaling Align-VL to such datasets? Additionally, Random Perturbation resembles regularization techniques like dropout or mixup. How does Align-VL compare to these methods, and would substituting Random Perturbation with another regularization approach yield similar results?

1. Align-VL faces challenges in scalability on larger datasets, including

• Despite using lightweight V-L Adapters and frozen unimodal encoders to reduce computational demands, Align-VL still faces significant increases in computing and storage needs when processing datasets with hundreds of millions of data points, potentially requiring thousands of GPU days. Furthermore, its “embedding smoothing” technique, which adjusts the probability distribution for each batch, incurs overhead on large-scale datasets.
• On very large datasets, the variability in data quality leads to increased noise and mismatched samples, amplifying model overconfidence. This requires fine-tuning “embedding smoothing” and “random perturbation” parameters to enhance the model’s generalization and stability.

However, the current design of Align-VL also reduces computational needs and improves robustness to such data challenges. Additionally, it is worth noting that in tests conducted on the 25K MS-COCO retrieval dataset, Align-VL surpassed the retrieval performance of much larger datasets, as illustrated in the table.

2. Comparison of random perturbation with mixup regularization techniques: • In Align-VL, “random perturbation” is designed not merely for adding noise but to simulate uncertainty in image-text embedding spaces, addressing overconfidence from weakly correlated data. This is distinct from dropout or mixup, which aim primarily to prevent model overfitting. • Compared to dropout and mixup: Dropout is already integrated into both baseline and Align-VL training. Although mixup was tested, it proved less effective than “random perturbation” for multimodal alignment tasks (In Table, Text->Image R@1). Align-VL’s perturbation strategy specifically targets minor mismatches between images and texts, unlike mixup, which increases data diversity but does not reduce model overconfidence.

Q2: I would like to hear the authors' opinion on whether the proposed method would be effective for training a multimodal model from scratch rather than leveraging pre-trained unimodal encoders.

Thank you for the valuable questions raised by the reviewer! Align-VL is designed to use pre-trained unimodal encoders for efficient multimodal alignment. For training from scratch, adjustments and optimizations may be necessary. Regarding the issue of training multimodal models from scratch, we have the following perspectives:

1. The core design of Align-VL involves adding lightweight adapters to pre-trained unimodal encoders, enabling efficient multimodal alignment with reduced computational and data needs. This approach leverages existing unimodal models to minimize the costs of training from scratch, making Align-VL ideal for resource-limited settings while still delivering robust performance.

2. Challenges of training from scratch: If Align-VL were trained entirely from scratch, its design might need adjustments. It relies on strong unimodal representations from pre-trained encoders, which ground-up models might initially lack, especially for complex image-text alignments. Thus, Align-VL’s “random perturbation” and “embedding smoothing” strategies may not perform as effectively in early training stages.

3. Adaptability and exploration: While Align-VL currently utilizes pre-trained unimodal encoders, its core strategies—optimizing alignment and reducing overconfidence via perturbations and embedding smoothing—are potentially adaptable to training from scratch. Our preliminary results, as detailed in the table, suggest that Align-VL remains effective even when built from the ground up.

W1: A key limitation is the lack of evaluation on larger-scale datasets comparable to those used by models like CLIP (e.g., 400 million data pairs). The paper acknowledges this but does not include any further experiments or discussions on large-scale datasets to substantiate the scalability of Align-VL. There are publicly available large-scale text-image datasets, such as the LAION dataset, which could have been used to provide additional insights. Including a discussion on potential challenges or a small-scale simulated analysis could strengthen this point.

Thank you for the reviewer’s suggestions! While we may not have sufficient time to complete experiments on large-scale datasets, we conducted a small-scale analysis on the LAION-2M (including COCO), increasing data pairs to assess performance and training time. Results showed a significant increase in training days and demonstrated that Align-VL consistently exceeds the baseline Fusemix across various data volumes under resource constraints.

Our proposed Align-VL aims to enhance computational and data efficiency for modal alignment in resource-efficient settings. It has outperformed state-of-the-art methods on public datasets, notably improving the R@1 score in retrieval tasks. With only 3.5M data, Align-VL exceeds CLIP’s performance on the MS COCO dataset and does so with 100 times less data and over 600 times less training time than CLIP, which requires 3000 GPU days and 400M training data.

We were unable to complete experiments on large-scale datasets for two reasons. However, we will include the results from the large-scale datasets in our submission as soon as they are completed.

1. Powerful models require extensive datasets and significant computational resources, making them costly for practical applications. Recent successes in multimodal alignment rely on intensive GPU use and large datasets, such as CLIP’s 3000 GPU Days, which are impractical in resource-limited settings. Thus, creating cost-effective frameworks for multimodal alignment is crucial.

2. Align-VL faces challenges in scalability on larger datasets, including 1) increased computational demands (potentially thousands of GPU days for datasets with hundreds of millions of data points), 2) scarcity of high-quality multimodal pairs, and large proportions of noisy data. However, its design reduces computational needs and improves robustness to such data challenges.