Foundation Vision Models are Unsupervised Image Canonicalizers

Thank you for the review. We answer your questions/concerns below:

“The method is dependent on human-designed transformations; may have limited potential to generalize real-world transformations.”:

This understanding is incorrect: our method can be used with any parameterizable transformation, e.g., the latent space of a generative model. It does not have to be a human-designed transformation. We chose intuitive transformations as these are already popular and easy to understand/debug, but in principle, this could be data-driven. Importantly, this generality is the key strength of our method compared to classical methods like equivariant nets!

Our viewpoint transformation example shows this comparative advantage. Equivariant nets cannot be used for viewpoint transformations on images. However, viewpoint transforms can be learned by a generative model (like Zero123). Our method can leverage this generative model to deliver equivariance/invariance. In future work, we will apply this technique to more complex and realistic transformations.

We also argue that even the transformations shown in this paper are helpful – e.g., viewpoint changes often appear in robotics settings. This drastically increases the number of demonstrations required to act as data augmentation in applications such as home robotics [1]. This is because there are currently no reliable methods for viewpoint invariance. Our work represents a useful step towards solving this long-standing problem.

Overall, we agree that it is important to model complex transformations in the real world for real-world robustness. If that is the goal, our method is more well-suited for complex transformations than classical invariance approaches and is a useful step toward real-world robustness.

“Optimization is slow and sample-inefficient.”:

Our goal is generality: The main goal of our work was to show that our canonicalization method is a general and training-free method for invariance. This enables us to generalize to a wide range of transformations (as described above) much better than classical invariance methods. As a result, we focused on generality rather than speed in this work. To our knowledge, methods that use only 1 sample do not generalize across tasks, models, and transformations.

Multiple samples are necessary for generality: Please also note that there are no other guaranteed invariance methods that are as general and still fast. It is currently unknown if that goal is even possible.

To our knowledge, existing single-sample solutions for invariance/equivariance require an equivariant/invariant network in the pipeline. This is a consequence of Equation 2 in [3]: Either the canonicalizer or the classifier has to be equivariant. This can be achieved either with an equivariant network (which enables you to sample 1 point but is limited to simple human-designed transforms) or by using multiple samples (which we do). If generality is required, then the only known method so far is optimization.

Please let us know if you disagree and have an example of a method that satisfies the three criteria of generality, invariance, and speed.

Sample Efficiency: Even so, for settings requiring many samples (e.g., color shift), Bayesian optimization (BO) is incredibly sample-efficient. For example, in our color shift experiments, 10-30 samples with BO achieve similar accuracy as an exhaustive 21x21 grid (i.e., 441 samples on the grid). This is a nearly 20x speedup.

Additionally, state-of-the-art “few-shot” Bayesian optimization methods can be even more sample-efficient by learning the statistics of the energy functions [2]. For example, Figure 2 of [2] shows their method achieving the same result in 4-5 samples as naive BO in 20-30 steps. Thus, there is great potential for speedups in future work.

For clarity and experimental rigor, we will add a plot of samples vs accuracy for the camera-ready version.

During this rebuttal process, we have:

1. Added a subsection to the paper discussing foundation model choice.
2. Provided sample complexity results for the color setting, demonstrating a significant advantage compared to brute-force.
3. Provided ablations for the energy functions on segmentation.
4. Shown preliminary results on 3 different realistic transformations: CO3D viewpoint shifts, day-night, and embodied vision in a virtual environment. Our method demonstrates promising canonicalization performance in these experiments, and we plan to do large-scale testing for the camera-ready draft.

We would like to repeat our main point that our approach is significantly more general than contemporary work (e.g., equivariant nets), which shows results on group transformations like 2D image rotations. Our work not only beats existing baselines (e.g., PRLC) published in similar conferences (NeurIPS 2023) on their own benchmarks but also generalizes to entirely new domains (e.g., Objaverse in the main draft, plus 3 new settings as shown in the rebuttal).

We hope these results help convey the potential of our approach as a step towards more general invariance methods.

We have further updates/results to share: new applications with real-world transforms and ablations.

As promised, here are some energy function ablation results (for segmentation results specifically). We will do further ablations for the camera-ready draft.

Day-Night transformation: Since you mentioned wanting to see realistic transformations, we tried day-night transformations recently modeled by a state-of-the-art relighting paper (https://github.com/zhihao-lin/urbanir). We took their trained model checkpoint and applied it to 2000 images from their provided dataset, creating 2000 day-night pairs. Our energy function picks day images 91% of the time, showing its strong canonicalization ability.

We further plotted the energy function while interpolating between the day and night images in Stable Diffusion 2’s latent space and found that the model energy is consistently lower for day images than night images. Finally, we ran optimization in this restricted subspace of the diffusion latent space and found that our method is indeed able to canonicalize these images. See results here: https://imgur.com/a/mhCjfnb

Active embodied vision with 6-degree camera movement (x,y,z,phi,theta,roll): As another example of more realistic transformations, we applied our energy function to optimizing camera parameters in a virtual environment (modeled by a Gaussian splat, using https://github.com/nerfstudio-project/gsplat). We believe this setting represents a more realistic version of our 3D viewpoint experiments and has useful implications for embodied AI.

Here, a camera moves around a virtual scene, searching for the view that optimizes the energy function. We find that this leads the camera to focus on salient objects in the scene and replicate typical angles (like in our previous experiments). Here are some results: https://imgur.com/a/cnizqMx.

While these results are still preliminary, we hope they show the strength of our approach in dealing with many different kinds of transformations, both analytical and realistic.

We also note that contemporary invariance works only show results on simple transformations like 2D image rotation, so we have already gone above and beyond the standards of this subfield.

If you have any other concerns, please let us know.

Thanks for the review! We are glad that you find our approach to be novel, well-motivated, and convincingly evaluated across different settings. We hope this approach is a useful step towards general canonicalization-based invariance.

As for your questions:

Fig 6 (a) and (c): In case you are referring to the roughness of those curves, that is an artifact of the random sampling used to compute the accuracy statistics. To fix this, we will add error bars and average over more samples for the camera-ready version. Please let us know if we misinterpreted your comment.

Zero123 + Objaverse: You are correct in pointing out that Zero123 is trained on Objaverse. Since our main contribution is the energy function for canonicalization, Zero123 is just another augmenter from our perspective (like the 2D rotation or color transform functions). We used Zero123 here, but in principle, our FMC framework could use any 3D viewpoint generation method.

Better augmenters: Replacing Zero123 with augmenters like One-2-3-45 would likely lead to much better results if the generation quality is improved. We simulated what the “ideal” generator could achieve by using the ground-truth Objaverse renders. In this case, the improvement is significant as shown in the following figure (with updated data processing; see “updated 3D results” below for details):

https://imgur.com/a/AUDGaXl

CO3D: Figure 6 (a-b) shows that our energy function can rank CO3D frames well. In practice, we found it difficult to generate high-quality CO3D 3D generations with Zero123. We were held back by practical issues like unreliable foreground-background separation, frames where the object is not fully in the frame, frames where multiple objects are in the scene, and aspect ratio.

We show an example below. The original image (top) does not contain the entire couch, and the background removed version (bottom) segmented the incorrect object. This limits our ability to use Zero123.

https://imgur.com/a/sKkKjrz

Solving these practical issues is possible but outside the scope of this paper. In future work, we plan to focus more on the 3D setting and try better generative models with real-world datasets, especially for robotics applications like home robotics [1].

Updated 3D Results: We noticed that Objaverse had poor label quality. The LVIS labels that came with the Objaverse-LVIS subset have many similar labels which confused the models (e.g., orange vs. mandarin orange vs. tangerine, ring vs. wedding ring, etc.). This was impacting both the ground-truth baseline curve and our curve. By simply filtering the data, we achieved much better results.

We only kept objects with (1) more than 10% of renders classified correctly and (2) a clear winner class (i.e., the frequency of the most common class should be at least 33% more than 2nd most common class). This selects roughly 30% of the objects. Please note that these are preliminary results, and we will add error bars and more points in the plot for our camera-ready draft.

See below for the updated plot, with the new lines in solid and the previous lines in dashed lines:

https://imgur.com/a/99SY0A8

Please let us know if there is anything else we can provide to help increase your rating/confidence.

[1] https://arxiv.org/abs/2311.16098

Hyperparameter tuning: We argue that hyperparameter tuning is not comparable to training neural networks. (1) The hyperparameter tuning cost is comparatively negligible because hyperparameters can be tuned on just a few dozen images. There are also very few numbers that need to be selected. (2) Energy hyperparameters don’t suffer the same limitations as trained neural nets since hyperparameters transfer well across different datasets and downstream networks. We will include hyperparameter transfer and sensitivity analyses in the appendix for the camera-ready draft.

Typical number of transformations required: For C4 and C8 experiments, we chose 4 and 8 samples, respectively. For viewpoint, we use 60. For our most sample-intensive setting, i.e., color shift experiments with CIFAR100, an exhaustive search requires 21x21=441 samples. We use 10-30 samples in the Bayesian optimization (BO). Doubling (i.e., 20-60) increases the accuracy by less than half a percent. Similarly, halving (i.e., 5-15 samples) decreases the accuracy by less than a percent.

Since our goal was not computational efficiency, we did not try to decrease the number of transforms in any of these settings. However, in our experience, Bayesian optimization is quite sample-efficient.

Additionally, state-of-the-art “few-shot” Bayesian optimization methods can be even more sample-efficient by learning the statistics of the energy functions [7]. For example, Figure 2 of [7] shows their method achieving the same result in 4-5 samples as naive BO in 20-30 steps. Thus, there is excellent potential for speedups in future work.

We will also add a graph of samples vs accuracy for each of the three transformation settings.

Other Speedup methods:

Two-stage pipeline: We have preliminary results for an approach that may significantly reduce amortized computational complexity by skipping the canonicalization when unnecessary.

For 2D rotations, we can skip the canonicalization if the input image’s energy is significantly lower than nearby poses (+90 and -90 degree rotations). This can detect upright vs. not upright with 95% accuracy.

In summary, with only 3 CLIP inferences, we can detect whether to do canonicalization with 95% accuracy. If the extreme rotations are rare (as in the real world), this can bring significant computational savings and, on average, be even cheaper than TTA.

This approach is similar to the “mental rotation” [8] phenomenon in humans, where we classify familiar poses quickly but go through a slow mental uprighting process to classify unfamiliar poses.

We did not include these results initially since our focus is not computational efficiency, but we hope this alleviates some concerns about the cost of this approach.

Fewer diffusion steps: In practice, it is sufficient to use only 10 diffusion steps for evaluation (500th, 550th, 600th, ..., 950th). We use this in our experiments to keep inference costs down. This significantly reduces the overall cost of energy computation. It is also possible to use only one step (500), as shown in [9], but this decreases the accuracy too much. We can include a graph of accuracy vs steps as well.

Distilling FMC energy into a cheaper (or one-shot) model: Another computational cost reduction might come from distilling the pose knowledge in these foundation models into a single-shot model, like how [8] distills depth knowledge from diffusion models.

In summary, we will add the following to the camera-ready draft: (1) TTA baseline for every experiment, (2) Downstream classifier energy baseline, (3) computational cost (in FLOPs) and runtime for each experiment (in seconds), (4) Hyperparameter transfer/sensitivity analysis, (5) samples vs. accuracy graphs for bayesian optimization and diffusion steps.

To our knowledge, FMC is the only way to get reliable invariance/equivariance that generalizes across a wide range of settings - outperforming TTA (as shown above). Thus, we argue that this contribution is valuable even if it is slower than prior approaches since a slow solution is better than no solution.

Please let us know if you have any questions or if there is something you would like to see to change your rating.

[1] https://arxiv.org/pdf/2206.00051

[2] https://arxiv.org/pdf/2010.11882

[3] https://arxiv.org/pdf/2202.10638

[4] https://arxiv.org/pdf/2309.16672

[5] https://arxiv.org/pdf/2211.06489

[6] https://arxiv.org/abs/2202.00665

[7] https://proceedings.mlr.press/v151/tighineanu22a/tighineanu22a.pdf

[8] https://psycnet.apa.org/record/1971-28060-001

[9] https://arxiv.org/pdf/2303.16203

Thanks for your review. We are glad you appreciate the simplicity and novelty of our approach.

In our opinion, this method can achieve guaranteed invariance/equivariance in much more general settings, allowing us to solve problems that previous approaches like equivariant nets could not. Rather than training new canonicalizers for every new setting, FMC simply extracts the information already encoded in foundation models.

Accuracy comparison to test-time-augmentation (TTA): We agree that comparisons against TTA are helpful. Here are the results for CLIP with C8 rotations. We will also add TTA as a baseline for every experiment in the paper for the camera-ready draft.

Result: We achieve significantly higher accuracy than TTA for the same evaluated points. Intuitively, this happens because we choose the best point rather than rely on group averages (which could be dragged down by bad samples). While TTA works well for commonly used small ranges (e.g., 10-degree rotations), it degrades when averaging over larger ranges (i.e., full 360-degree circle) necessary for full invariance.

Advantage against downstream model energy: Using the energy function from the downstream model is a special case of FMC if the downstream model is a large model like CLIP.

If the downstream is a smaller model like ResNet50, then there are two main drawbacks: (1) R50’s energy function may not work well outside its training setting (e.g., new datasets), (2) it may not be as accurate as FMC due to seeing less data during training.

To show the difference, we used PRLC’s trained ResNet50 classifier and compared it against our approach.

Result: Even in the best-case scenario where the classifier is trained for the specific dataset, its energy function is worse. If the test dataset were different, the gap would likely be even larger (similar to PRLC canonicalizers failing on new datasets in Figure 7).

The lower accuracy combined with the fact that this approach might not work outside its training setting are significant challenges for this approach.

For improved experimental rigor, we will add another table to the appendix with comparisons between FMC and classifier energies.

Computational expense: Speed/cost is not our priority since we focused on solving a problem that previous approaches could not solve: a general and training-free method for invariance. With FMC, we propose a method that enables a single canonicalization system to generalize across tasks, models, and transformations.

You are correct that the main paper does not discuss the computational expense. We only briefly mention it in the limitations.

Our cost is (# of transforms evaluated) X (Cost of transforming + Cost of evaluating [CLIP + SAM + Diffusion model] + Cost of inference)

Relative to TTA, this cost is 1 + (Cost of evaluating [CLIP + SAM + Diffusion model])/(Cost of transforming + Cost of inference)

In practice, when using CLIP as the downstream model and 10 diffusion steps (see “Speedup methods” for details), TTA is roughly an order of magnitude more expensive than simple inference, and our method can be roughly an order of magnitude more expensive than TTA.

This difference is much smaller in 3D settings because the generation cost (from Zero123) dominates energy function costs.

We will add a table detailing the computational cost (in FLOPs) and runtime for each experiment (in seconds) for the camera-ready appendix.

Thank you for taking the time to review our paper. However, we see critical factual misunderstandings in your assessment. We point out the critical oversights below. We hope you reconsider your review with an open mind.

Experiments only on 2D rotation: Your understanding is incorrect. Figures 5 and 6 show color shift and viewpoint results, respectively. The corresponding experiment settings are described in Section 5.3. Earlier figures/tables focus on 2D rotation because the main baseline (PRLC) evaluates on those, but we do consider viewpoint and lighting (illumination color specifically) as well. We can make these results more prominent in the camera-ready draft.

Evaluated only on small datasets like CIFAR10/STL: This understanding is also incorrect. We also evaluate on COCO (Table 2), ImageNet (Takeaway #2 in Section 5.1; Table 3 in Appendix), and subsets of CO3D and Objaverse (Figure 6).

Please also note that popular papers in the invariance learning subfield, e.g., Augerino, Prior-Regularized Learned Canonicalization, LILA [1, 2, 3] in similar conferences (NeurIPS, ICML) have used datasets like CIFAR10/STL rather than larger datasets such as ImageNet. We still went beyond the standard practices of this subfield to show our claims more rigorously. We hope you consider this factor in your rating.

We can also make these results more prominent in the camera-ready draft.

2D plane rotation is too simple: While it is easy to train a 2D rotation estimator for specialized settings (like our baseline PRLC), our paper addresses a more general approach that works across many settings. One example of why this matters is mobile agents; to enable them to work in the real world, we need a system that can adapt to various transformations and settings.

Specialist approaches like PRLC (our baseline) are insufficient for a generally robust system. PRLC trains specialized networks to estimate rotations on a dataset and uses it to upright the image. However, we find that such approaches (1) fail to generalize to new datasets (Figure 7); (2) don’t work for new transformations by design (e.g., viewpoint, lighting color); and (3) degrade on new downstream models (Figure 9). Put simply, such approaches do not work outside of their training setting.

Our work aims to overcome these fundamental limitations. Notably, we beat these specialized models in all relevant settings without any training. This aspect is crucial because it allows us to create one single approach that works across many settings. Thus, our work is an important step towards a generally robust system.

Please let us know if we missed something. We would also appreciate knowing specifically what we can do to change your rating of the paper. Thank you for your questions.

See https://anonymous.4open.science/api/repo/ICLR_25_Rebuttal-D1DD/file/index.html?v=fc7f36c7 for a summary of the new results, which we reference in the appropriate sections.

Real-World Results:

Updated 3D Results (Reviewer CgAq): We provided updated 3D results on Objaverse-LVIS. We found that Objaverse-LVIS contained many examples of similar labels which confused the models (e.g., orange vs. mandarin orange vs. tangerine, ring vs. wedding ring, etc.). These label quality issues affected both the ground truth and our curve. We filtered out such objects, and our results improved (Fig. 1a in the above link).

Since then, we additionally noticed that due to our data-rendering process, some viewpoints on exact side angles (0, 90, 180, etc.) are aligned front-to-back with the camera, hiding large parts of the object (also called accidental viewpoints) (Fig. 1b in the above link). We changed the input views to the algorithm to be offset by 10 degrees (i.e., choosing non-accidental viewpoints) to avoid this issue and found our results to be further improved (Fig. 1c in the above link).

Preliminary CO3D example (Reviewer CgAq and wapX): We also added a preliminary example on CO3D showing a TV video where our approach achieved a 12.3% improvement compared to a non-invariant baseline. We used SAM2 as the background remover. Critically, we achieved this without any change in our energy function.

New Day-Night Transformation Results (Reviewer wapX): To further evaluate on real-world transformations, we applied FMC on day-night transformations modeled by a SOTA relighting paper (https://github.com/zhihao-lin/urbanir). Our energy function picks day images 91% of the time, and we can canonicalize these images by searching in the restricted diffusion latent space (Fig. 2 in the above link). New Active Vision Result (Reviewer wapX): As a “real-world” embodied AI application, we evaluate our approach on an active vision setting to optimize camera parameters (x,y,z,phi,theta,roll) in a virtual environment. We find that our energy function can select camera parameters that focus it on salient objects in the scene at typical angles (Fig. 3 and 4 in the above link).

Runtime / Cost Concerns:

New Sample Complexity Analysis (Reviewer dVF9 and wapX): We showed that our Bayesian Optimization approach uses significantly fewer samples than brute force on color (Fig. 5 in the above link).

New TTA Comparisons (Reviewer dVF9): Our approach outperforms TTA when evaluated on the same points (+10.9%, +14.5%, +0.9% on CIFAR10, CIFAR100, and STL10, respectively).

Runtime Cost vs TTA (Reviewer dVF9): We compared the cost of our approach vs. TTA. In practice, TTA is about an order of magnitude greater than plain inference, and our approach is about an order of magnitude greater than TTA.

Ablations:

New Energy Function Ablations (Reviewer wapX): We ran ablations on our energy functions for segmentation, showing the benefit of using each foundation model.

New Downstream Model Energy Comparison (Reviewer dVF9): We find that our approach with CLIP outperforms using ResNet50 energy by 4.6%, even on a ResNet50 downstream model.

Writing:

New Discussion on Foundation Model Choices (Reviewer wapX): We have added a discussion to the draft on the conditions in which foundation models would be well-suited to use with our approach.