Exploring the Effectiveness of Object-Centric Representations in Visual Question Answering: Comparative Insights with Foundation Models

Thank you for your review and your positive assessment of our work. We answer your questions and concerns below:

Weaknesses and Questions

As stated in the paper, the datasets would have included more real-world data.

We kindly refer you to our general response to all reviewers, where we explain that we are incorporating the GQA dataset as an additional real-world dataset into our study and will share the results as soon as they are ready.

Some analysis of the application of the findings is lacking.

Could you give some more insights on the application of the findings in the paper?

We would like to highlight that the role of OC models in downstream tasks has been partially underexplored and our empirical study tries to cover this gap in the literature. We have summarised some of the takeaways of our study and their contribution to the community below:

• We’ve empirically clarified the trade-offs of foundation models vs OC models and found out that foundation models will perform the same or better than standard OC models on complex or real-world data without any training or hyperparameter tuning. However, they require much more downstream compute and they have less explicit representations compared to OC models. However, applying the OC bias on top of a foundation model (DINOSAUR) will benefit us with the best of both worlds, leveraging the general vision capabilities of foundation models and thus, showing strong performance on hard synthetic and real-world datasets while having explicit object representations. The only downside is that the OC bias should be trained on the downstream dataset. Therefore, depending on the settings at hand, one can select the proper type of model based on the insights of our study.

• For evaluation, the focus of the OC community has been mainly on unsupervised object discovery tasks and segmentation metrics while downstream evaluation of representations has often been overlooked. We’ve empirically shown that representations that perform well in downstream tasks do not necessarily excel in segmentation tasks, and that segmentation accuracy (ARI) or reconstruction error (MSE) are not reliable predictors of downstream performance. We hope this highlights the importance of incorporating downstream evaluation as a key aspect of model assessment in the OC community.

I'm wondering if the conclusions would be applied in other domains out of VQA.

We would like to emphasize that VQA involves a diverse range of questions that require understanding and reasoning about the visual scene. Demonstrating that the methods work effectively on this task suggests that the results can be extended to other tasks that require similar levels of understanding and reasoning.

What makes me curious is what the performance would be after fine-tuning the foundation models on the specific VQA tasks, since other models have been fitted on the data. I suppose this would make it more fair for the comparison of different models and may inspire more insights.

That’s a good point. In our initial set of experiments, we analyzed the effect of fine-tuning DINOv2 on the VQA task, and as expected, we observed that the downstream performance improved but the gain was not significant and came with a substantial computational burden. This made the comparison with OC models less fair in terms of computational requirements. Thus, we decided not to pursue fine-tuning further.

I thank the authors for their efforts and their response has addressed my concerns to some extent. Hence, I decide to keep my original score and am leaning to accept this paper.

Thank you for your review and your positive assessment of our work. We answer your questions and concerns below:

Weaknesses

Some findings align with existing intuitions. For example, it is not suprising that large foundation models perform comparably to specialized OC models, and that their combination yields better performance. While valuable, these conclusion is intuitive and somehow obvious to the maching learning field.

Thank you for your comment. We would like to highlight the following points:

1. We appreciate the reviewer’s observation that some findings align with existing intuitions. However, we emphasize that scientific progress relies on rigorous validation, even for results that might seem obvious. While it may seem intuitive that large foundation models perform comparably to specialized OC models and that their combination further improves the performance, such outcomes have not been systematically shown on downstream reasoning tasks. Additionally, not everyone in the field shares the same intuition—some may expect the combination to perform worse and view the OC inductive bias as a potential bottleneck. By empirically testing these assumptions, we believe our work provides a strong basis for future research and offers valuable insights for the community.
2. We would like to highlight that despite their improvement in recent years, OC models still lag behind other types of models and foundation models in several tasks such as unsupervised segmentation in real-world settings [1]. Furthermore, the comparison between OC representations and other types of representations is still partially underexplored. The closest work that addresses this is [2] which evaluates OC representations in specific RL tasks that are significantly simpler than our VQA tasks. Therefore, it’s hard to predict how OC representations perform on harder tasks that require reasoning such as VQA without conducting a proper study to analyze it.
3. We would like to clarify that our focus is not solely on performance, but also on other important aspects such as the explicitness of representations and computational requirements. While performance is a key consideration, it is not the only relevant metric, and its significance depends on the specific setting and objectives at hand. Measuring other factors alongside performance is crucial for gaining a more comprehensive understanding of model behavior and evaluating their practical applicability in different contexts.

The correlation between property/attribute accuracy and overall VQA performance may be due to the characteristics or preference of the selected 4 datasets, rather than a fundamental relationship. This correlation might not generalize well to relational or reasoning or logical questions, potentially make the conclusion less convincing.

We agree that there is a possibility that this correlation is due to the properties of the datasets. However, we would like to clarify that many generated questions for synthetic datasets do in fact involve relational reasoning. For example:

• “Is the shape of the small object that is on the right side of the medium cube the same as the large thing that is behind the medium block?”
• “How many other things are there of the same shape as the large cyan thing?”

While there are many real-world QA datasets like VCR, GQA, DAQUAR which would capture more general data intrinsics of QA question, only one real-world dataset (VQA-v2) was used in this work. The three synthetic datasets is quantitatively extensive, but might exhibit construction biases or preferences and not capture the full complexity of real-world scenarios. Focusing on more diverse real-world datasets would strengthen the conclusions

We kindly refer you to our general response to all reviewers, where we explain that we are incorporating the GQA dataset as an additional real-world dataset into our study and will share the results as soon as they are ready.

Thank you for your review of our work and the detailed feedback. We answer your questions and concerns below:

Weaknesses

The question that this paper wants to answer is: whether OC, fixed-region or global representation is better for VQA. To answer this question, the authors need to provide a model with the same architecture, model size and pre-training data. The only variant in the model is the feature representation way, i.e., OC, fixed-region or global. However, I didn't find this experiment in the paper. Maybe I missed it. Please point it out and highlight it in the rebuttal.

Thank you for pointing this out. We completely agree that the fairest way to compare these different representations would be to maintain the same factors of variation such as model architecture, size, and pre-training data while varying only the representation type (OC, fixed-region, or global). However, implementing such a controlled experiment would require substantial engineering effort and computational resources to ensure all factors except the representation type remain consistent, and everything works as expected. In our study, we instead took a best-effort approach by utilizing existing models with minimal modifications, which allowed us to assess the effectiveness of different representations without setting up the entire experiment from scratch. While we acknowledge this as a limitation, we have made efforts to minimize the impact of other factors, particularly by comparing a foundation model with and without the OC inductive bias (DINOv2 and DINOSAURv2), ensuring as fair a comparison as possible within our available resources.

Synthetic data such as CLEVR and CLEVRTex are not challenging enough since the objects can be easily detected (and even hard-coded).

We agree that solely using these synthetic datasets might not be enough for our study and therefore, we have included VQA-v2 to confirm our results in real-world settings. We have selected these datasets mainly because they are some of the traditional synthetic datasets that have been popular in the OC community in the past years. Furthermore, we kindly refer you to our general response to all reviewers, where we explain that we are incorporating the GQA dataset as an additional real-world dataset into our study and will share the results as soon as they are ready.

Many symbolic methods have solved these datasets with 100% accuracy.

Can you please elaborate on which methods have achieved the perfect performance, and on which tasks?

Nonetheless, we would like to highlight that this is not directly relevant to our study. The fact that some models can achieve perfect accuracy indicates that those models are effective at solving all the specific challenges presented by the dataset. However, for our study, the objective is not to solely push the boundaries of performance but to investigate how different model representations, including OC representations, behave on the VQA tasks defined on these datasets. In fact, our goal is not to solve the VQA task and achieve state-of-the-art results but to analyze the benefits and trade-offs of OC representations compared to other types of representations, with VQA serving as the downstream task for this evaluation. Therefore, as long as we observe a difference in the performance of models on these datasets and not all of the models achieve perfect accuracy, they would be suitable datasets for this goal.

Thank you for your review of our work and the detailed feedback. We answer your questions and concerns below:

Weaknesses and Questions

While the study includes VQA-v2, it primarily relies on synthetic datasets. The evaluation would be more compelling if it included additional established real-world VQA datasets such as GQA or CRIC. Each image in the GQA and CRIC datasets is associated with a scene graph describing the image's objects, attributes, and relations. Moreover, questions in these datasets involve multiple reasoning skills, spatial understanding, and multi-step inference, making them ideal for analyzing object-centric models.

The majority of claims and findings are based on synthetic datasets, which feature simpler scenes with clear object-background separation. This raises concerns about the generalizability of the findings to more complex real-world scenarios. It would be great if the author can provide additional analysis on the challenging datasets such as GQA and CRIC

How would the proposed approach perform on more challenging real-world datasets like GQA or CRIC?

How would the findings about the benefits of object-centric representations translate to more challenging scenarios in real datasets?

We kindly refer you to our general response to all reviewers, where we explain that we are incorporating the GQA dataset as an additional real-world dataset into our study and will share the results as soon as they are ready.

The downstream architecture is limited to BERT-style transformer encoders. The authors should explore decoder-based transformer architectures, which have become increasingly popular and achieved more favorable results than BERT-style encoders on VQA tasks in recent research.

Why did the authors choose to focus on transformer encoder architectures for downstream models? Would incorporating decoder-based architectures potentially lead to different conclusions?

In our work, we frame VQA as a classification task and adopt an encoder-decoder architecture with a transformer encoder to process the input and a classification head applied to the [CLS] token. A transformer encoder is well-suited for this setup because its bidirectional attention mechanism enables it to integrate and contextualize information from both the vision and language representations into the [CLS] token, which is specifically designed to represent the aggregated information required for classification tasks. However, if we were to shift to a token generation approach for the VQA task, using a transformer decoder could indeed improve the performance. On the other hand, handling VQA as a classification task helps avoid the potential influence of language variability, open-ended outputs, and other confounding factors. Despite its simplicity, this approach still aligns with our goal and effectively captures what we aim to measure: the models’ understanding of the visual scene and their relational reasoning abilities, rather than merely solving the VQA task. Moreover, it avoids the added complexity of language generation, ensuring that the results remain clearer and easier to interpret.

While the authors claim they evaluated 640 downstream models, the paper lacks sufficient detail and analysis regarding these experiments. They should provide comprehensive information about these models in the paper

Could the authors provide more detailed analysis of experiments with 640 downstream models?

We apologize for the confusion. This number corresponds to the total number of VQA downstream models we trained for the study. Given the number of datasets (4) and the number of different split sizes of Multi-dSprites (4; 40k, 80k, 160k, and 320k images. So 7 different datasets in total), number of models (15), number of seeds (1 for pre-trained models, 3 for other models), and the number of downstream model sizes (3; T-2, T-5, and T-15), the total number of downstream models can be calculated as:

• For Foundation models, we will have: 6 * 7 * 1 * 3 (#foundation models * #datasets * #seeds * #downstream_models) = 126
• For other models on all datasets excluding VQA-v2: 9 * 6 * 3 * 3 (#models * #datasets excluding VQA-v2 * #seeds * #downstream_models) = 486
• And for other models on VQA-v2: 3 * 1 * 3 * 3 (#models * #datasets * #seeds * #downstream_models) = 27

Additionally, for each dataset, we have one baseline with only the information about the questions, which will add 7 more downstream models to our study. Therefore, in total, we’ve trained 646 downstream models.

Regarding the request for a “more detailed analysis of experiments,” we’d be happy to provide this and will address any specific aspects you’d like clarified. As long as it does not involve new experiments with significant runtime, we should be able to include the results before the discussion deadline.

We are happy to answer any further questions you may have.

Based on the reviewers’ concerns about the need for experiments on additional real-world datasets, we have updated our submission by adding the results on the GQA dataset (highlighted in blue color). The updates in the submission are as follows:

• Added three new plots for the results on GQA: two in Figure 4 and one in Figure 14. Additionally, the “Real-world Data” paragraph in Section 4.1 is updated accordingly.
  • Figure 4 (left) shows the overall accuracies of different models w.r.t. downstream GFLOPS on GQA.
  • Figure 4 (middle) shows the average accuracies of different models on GQA, with T-15 as the downstream model.
  • Figure 14 (right) shows the average accuracies of different models w.r.t. downstream model sizes on GQA.
• Added the table with the average accuracies of models on GQA, with T-15 as the downstream model (Table 14 in Appendix D).
• Updated “Datasets” section (Section 3.3) with a brief description of GQA, and a more detailed description is added to Appendix B.
• Added upstream and downstream training details of GQA to Appendix A.
• Adjusted several minor parts of the main text and the appendix to include the addition of GQA.

We provide a summary of the updates below:

On GQA, we repeat the VQA experiments using the same pipeline on a subset of best-performing OC models and all foundation models. We use the balanced version of the dataset, with the “train” split used for training and “testdev” split used for evaluation. The new results align perfectly with our main findings:

• Performance Comparison: Large foundation models, together with DINOSAURv2, outperform other models while other OC models do not perform well on GQA (see Figure 4, middle). This trend matches the results observed for VQA-v2 (see Figure 4, right).
• Effect of OC Bias: We observe in Figure 14 (right) and Figure 4 (left) that DINOSAURv2 outperforms DINOv2 on T-2. On T-5, DINOv2 starts catching up and on T-15, it performs on par with DINOSAURv2. Furthermore, for all downstream model sizes, DINOSAURv2 requires significantly less downstream compute than DINOv2 (see Figure 4, left).

These findings on GQA are consistent with those from other datasets, confirming our main claims (Section 4.1): Applying the OC inductive bias on top of a foundation model provides a compelling approach to get the best of both worlds. This approach achieves comparable or superior results to foundation models while significantly reducing compute requirements and providing the advantage of explicit object representations.