Leveraging Color Channel Independence for Improved Unsupervised Object Detection

We thank the reviewer for the valuable feedback and hope we can address the questions below.

• "Would it be possible to test on more challenging datasets? Do you still expect a benefit there?"

  We do not expect a large benefit from more challenging datasets, as the models already struggle with the complexity of Movi-C. However, we argue that our experiments are on the same level of complexity as related works [1,2] that do not rely on additional data (such as self-supervised signals). We argue in Appendix, Section D.6, that research on color channels remains important, even though not yet applicable to real-world datasets.

• "There are many color spaces that provide decorrelated channels. A simplest one is to use a linear rotation of the space spanned by the channels R,G and B by doing PCA on it."

  As the design space of composite color spaces is combinatorial, we could not include an extensive evaluation of all color spaces in our analysis. However, we note that the decorrelation of color channels alone is insufficient to improve OCRL - for example, the LAB space is not correlated but does not improve over RGB representations. We agree that using a PCA-decorrelated RGB space is an interesting opportunity for future research, and we added it to our future work section.

[1] O. Biza, S. V. Steenkiste, M. S. M. Sajjadi, G. F. Elsayed, A. Mahendran, and T. Kipf. 2023. Invariant slot attention: object discovery with slot-centric reference frames. In Proceedings of the 40th International Conference on Machine Learning (ICML'23), Vol. 202. JMLR.org, Article 106, 2507–2527.

[2] G. Singh, Y. Kim and S. Ahn. 2022. Neural Systematic Binder. International Conference on Learning Representations.

We thank the reviewer for the insightful feedback and hope our answer addresses the concerns and questions raised.

"The authors only discuss grayscale for the single-channel approach. Considering the high correlation between RGB channels, what would the impact for OCRL be if using just one of the RGB channels as the target?"

We implemented experiments to investigate this effect and added training on the "R" channel of RGB to our analysis in Section 3. We observe moderately substantial differences between training on the single R channel and RGB. The single R channel suffices for the Clevr dataset but lacks performance on the more challenging Clevrtex dataset.

"Why does introducing composite color spaces improve object-centric representation? Could you provide a more detailed explanation from a feature perspective?"

We argued in Section 3 about the drawbacks of the RGB space. For example, there is a high correlation between lightness and only subtle differences between the color channels representing hue and saturation. Regarding the effect of composite color spaces on the observed object-centric representation improvements, we could not identify a single detailed reason - We investigated the effect and potential reasons in more detail in Appendix D4, where we explore the impact of composite color spaces for scenes under different lighting conditions. We observed that slots obtained from RGB-S targets tend to be more robust against lighting changes, which can be one effect. We furthermore highlight the attention maps of slots in Appendix D5. Slots trained on the RGB space tend to occupy fixed spatial areas instead of objects. We suppose this happens because spatial areas often share similar lighting conditions, which are overrepresented in the RGB space - the hue and saturation channels of the HSV space are robust to lighting, helping slots in discriminating objects from the background.

"How does the model behave for different values of the number of slots? Is there an ablation showing the effect of that parameter?"

We added an ablation for the Clevrtex dataset in the Appendix, Section D.7. Although models trained on RGB do not degenerate as often with fewer slots, the RGB-S models still outperform them consistently - the only exception is for a very low number of slots (5).

"Although the authors state that their work focuses on color spaces, the essence of their approach lies in enhancing the model's perceptual abilities by adjusting target objectives. Therefore, using only an RGB-RGB baseline may not be reliable. It would be more robust to include comparisons with methods that use different representations as targets, such as (Seitzer et al., 2023)."

We agree with the Reviewer that a comparison to other target signals, e.g., self-supervised objectives, is interesting. We thus added a comparison to the Dinosaur [1] method in the Appendix, Section D.6. Our preliminary results show that Dinosaur shows superior performance on Movi-C but lacks performance on Clevrtex. Furthermore, we argue in the appendix that self-supervised signals show a massive improvement for real-world datasets, but the corresponding self-supervised signal also limits their performance. Contrarily, the composite RGB-S spaces contain all information and might thus be combined with self-supervised signals. For example, one could predict the composite color spaces as additional targets to self-supervised signals.

"While the authors' idea is interesting, they do not provide a reliable analysis from a representation perspective on the inclusion of different color space targets. For instance, in Figure 2 of the main paper, there is a noticeable similarity between saturation and the mask in highlighting foreground objects; however, the mask is discrete and contains categorical information, while saturation focuses solely on object shape. There is insufficient technical analysis explaining the impact of using saturation."

We ask the Reviewer to clarify this point. We do not claim that saturation acts similarly to the object masks. However, we claim that the saturation contains complementary features that might be used to discriminate between foreground and background objects, which are not well represented in the RGB space.

[1] M. Seitzer, M. Horn, A. Zadaianchuk, D. Zietlow, T. Xiao, C.-J. Simon-Gabriel, T. He, Z. Zhang, B. Schölkopf, T. Brox, and F. Locatello. Bridging the Gap to Real-World Object-Centric Learning, March 2023. URL http://arxiv.org/abs/2209.14860. arXiv:2209.14860 [cs].

We thank the reviewer for their comments and helpful suggestions. In the following section, we will address the concerns and questions raised.

• "The paper is not easy to read as the cross-referencing of figures and tables is disordered."

  We changed the placement of figures in the main text to appear next to their reference in the text. Specifically, we adjusted the positions of Figure 1, Figure 2, and Figure 4. The tables are positioned next to their reference in the main text.

• "It claims that "We challenge the common assumption that RGB images are the optimal target for unsupervised learning in computer vision and demonstrate this for OCRL". It's better to give a reference which shows the RGB image is optimal."

  We agree with the reviewer and changed the sentence to "We challenge the common assumption that RGB images are the optimal color space for unsupervised learning in computer vision and demonstrate this for OCRL." Furthermore, we added references to SOTA unsupervised computer vision networks, such as CLIP, DINO, and Slot Attention.

• "Table 2 is not too strong to support that RGB-S is good enough. Mostly, it's not the best on different datasets, for example, RGB-SV is better than RGB-S on Clevr, MultishapesNet-4, and MultishapesNet-24; HSV is the best for Clevrtex."

  We agree with the reviewer that the RGB-S space is not the best-performing color space for all considered datasets, as stated. We toned down potentially ambiguous claims in the main text in lines 334 and 538 to clarify that other composite color spaces next to RGB-S can be similarly helpful and that the RGB-S and RGB-SV space consistently improve the performance over the current color space used for OCLR (RGB).

• "The experimental analysis is not convincing, for example in Line 457, which one is most important for representation, the color space or a powerful decoder?"

  We thank the reviewer for pointing out this unclarity in our writing. Table 2 shows that the composite color spaces improve the performance of OCLR networks independent of the model's complexity. We changed our main text (line 415) for clarity.

• "How does the model guarantee universality when it is trained on the combined color space? I still think RGB is the most universal space even though the performance is slightly worse than others in Table 2."

  There might be a misunderstanding. We do not aim to claim that the model guarantees universality. We argue that models trained on combined color spaces are less dependent on inherent biases of the color spaces and evaluate this effect statistically. We added this to our main text in Section 3.3 to avoid confusion. For example, differences in hue and saturation are subtle in the RGB space (due to the high correlation with the lightness). We countersteer this effect by combining multiple color spaces - for example, adding the hue and saturation to the RGB color space pronounces those attributes more. Regarding the claim that RGB is still the most universal color space, we find such a claim difficult, also based on the physics behind color itself (light spectrum) and the somewhat arbitrary, biologically-induced choice for three channels. Also, such a claim would then transfer to some other color spaces as well, such as HSV and LAB, as they can be losslessly induced from the RGB space, then claimed to be as universal - but we still see different statically evaluation results for them. Nevertheless, we agree with the Reviewer that RGB is the most popular color space in computer vision and widely used.

• "The paper is too technical and lacks novelty as the combination of color space is often used as a trick in computer vision and image processing tasks."

  We agree with the Reviewer that changing the color space is a common trick in computer vision and is already well explored and tested. Nevertheless, we contribute two novelties, going beyond the tricks and computer vision contributions referenced in the related work section: Firstly, we combine color spaces and theoretically and empirically reason about its advantages. Secondly, unlike related work, we utilize these color spaces in unsupervised OCLR as a target instead of the input, showing improvements above the SOTA. We state this more explicitly in our contribution section. Those two contributions set apart our work from previous research.