Improving Viewpoint-Independent Object-Centric Representations through Active Viewpoint Selection

We sincerely thank you for your valuable feedback and constructive comments. We have carefully considered each point raised and provide our responses below.

1. Presentation

Thank you for your feedback on the presentation. We will revise Section 3.1.2 to provide more details and clarity regarding the notation used in the pseudocode. These variables are defined as follows: $h_m$ represents the image features extracted by DINO, and $v$ represents the value function of the attention mechanism. We agree that larger figures and plots would enhance readability, and will enlarge key figures and plots in the main paper, moving additional results to the appendix to maintain focus.

2. Dataset Selection and Re-Rendering

• Our choice to re-render datasets rather than using the OCLOC datasets was primarily due to the need for specific viewpoint parameters ($\rho$, $\phi$, $\theta$) that were not available in the OCLOC datasets. These parameters were initially intended to be used in our method, but we later shifted to using timesteps for simplicity.
• For the datasets, we aimed to maintain consistency with the configuration and rendering methods used in OCLOC. However, we chose the CLEVRTEX dataset over the simpler CLEVR and SHOP datasets used by OCLOC to provide more complex scenes with richer textures and variability, which better test the model's capabilities.

3. Real-World Data Evaluation

We understand the importance of assessing the model's generalization to real-world data. However, there is currently a lack of suitable multi-viewpoint real-world datasets for comprehensive evaluation of our model and similar methods. This poses a challenge in demonstrating real-world applicability. Nonetheless, the objects in the GSO and ShapeNet datasets closely resemble real-world objects, providing a realistic basis for testing our model's generalization beyond purely synthetic environments. These datasets effectively bridge the gap between synthetic and real-world scenes.

4. Evaluation Settings

We appreciate the reviewer's feedback regarding the clarity of the evaluation settings. We apologize for any confusion and provide the following clarifications:

• The unsupervised segmentation results are derived from the attention masks computed within the multi-viewpoint slot attention module.
• The temporal timesteps are encoded using the viewpoint encoder. These encoded timesteps are then concatenated with the viewpoint-independent object-centric representations to serve as conditioning input for the diffusion model, guiding the image generation process.

5. Discrepancy Between LPIPS and FID Rankings

Thank you for pointing out the discrepancy between LPIPS and FID rankings in Tables 1 and 2. This discrepancy may arise from the different aspects of image quality they measure. LPIPS evaluates perceptual similarity at a fine detail level, while FID assesses global distributional similarity. The models being evaluated might perform differently in preserving local details versus maintaining overall image realism and distribution, leading to variations in how each metric ranks the results.

6. Discrepancy Between Fig. 3 and Tab. 5 → ARI-O == ARI-FG?

Thank you for pointing out this discrepancy. We apologize for any confusion caused. The notation ARI-O and ARI-FG should indeed be consistent across Figure 3 and Table 5.

We sincerely thank you for your valuable feedback and constructive comments. We have carefully considered each point raised and provide our responses below.

1. Computational Complexity

We acknowledge that our method may be more time-consuming due to the presence of two loops in the selection strategy. However, it is important to highlight that this increased computational complexity is offset by our method's unique capability to predict unknown viewpoints, a feature not available in alternative methods. Additionally, our approach demonstrates superior performance in scene segmentation compared to other methods.

2. Benefit of the Viewpoint Selection Strategy

We appreciate your observation regarding the role of the viewpoint selection strategy in predicting images from unknown viewpoints.

• The viewpoint selection strategy is integral to enhancing the quality of viewpoint-independent object-centric representations. It ensures that the most informative viewpoints are selected for observation, thereby reducing redundancy and improving prediction accuracy. Accurately predicting images from unknown viewpoints is both a necessary condition for the effectiveness of our viewpoint selection strategy.
• $S^{view}$ can be obtained in image prediction tasks by inputting the target viewpoint timestep into the viewpoint encoder. The model uses $S^{view}$ to guide the prediction of the image from target viewpoint.

3. Necessity of the Prediction Set

We apologize if Figure 2 is not sufficiently clear. In our framework, the images from the Unknown Set are not directly accessible to the model. Only the viewpoint timesteps of the Unknown Set are available. The Prediction Set consists of images predicted from these viewpoint timesteps. Generating the Prediction Set is a necessary step to estimate the information increment of the unknown viewpoints, which plays a crucial role in our viewpoint selection strategy.

We sincerely thank you for your valuable feedback and constructive comments. We have carefully considered each point raised and provide our responses below.

1. Computational Complexity

We acknowledge that our method may be more time-consuming due to the presence of two loops in the selection strategy. However, it is important to highlight that this increased computational complexity is offset by our method's unique capability to predict unknown viewpoints, a feature not available in alternative methods. Additionally, our approach demonstrates superior performance in scene segmentation compared to other methods.

2. Viewpoint Continuity

We appreciate your concern regarding the reliance on viewpoint continuity. For scenes involving discontinuous viewpoints, we can use specific viewpoint parameters (denoted as $\rho$, $\phi$, and $\theta$ and defined in Appendix A) instead of relying on viewpoint timesteps. This approach allows for more precise control over the viewpoint and improves the model's ability to handle novel view synthesis.

3. Generalization and Evaluation on Real-World Data

We understand the importance of assessing the model's generalization to real-world data. However, there is currently a lack of suitable multi-viewpoint real-world datasets for comprehensive evaluation of our model and similar methods. This poses a challenge in demonstrating real-world applicability. Nonetheless, the objects in the GSO and ShapeNet datasets closely resemble real-world objects, providing a realistic basis for testing our model's generalization beyond purely synthetic environments. These datasets effectively bridge the gap between synthetic and real-world scenes.

4. Open Access to Code and Data

We are preparing our code and datasets for public release and plan to make them available on Github upon acceptance of the paper. This will ensure reproducibility and facilitate further research by the community.