Learning Color Equivariant Representations

What is the connection/difference of the proposed setting with Domain Generalization?

The goal of domain generalization is to learn representations that are invariant to distribution shift [1,2,3]. This is in contrast to the related but distinct goal of equivariant representation learning which is to learn representations that are equivariant to (i.e., transform predictably in response to) specific transformations of the inputs. While not specifically designed for generalization, equivariant neural networks have been shown to improve model generalization performance (both empirically and theoretically [4,5,6,7]) when the specific transformations are symmetries (or nuisance transformations) of the underlying task. We provide mathematical definitions of invariance and equivariance in Section 3: Equivariance.

[1] Gulrajani, Ishaan, and David Lopez-Paz. "In search of lost domain generalization." arXiv preprint arXiv:2007.01434 (2020).
[2] Blanchard, Gilles, Gyemin Lee, and Clayton Scott. "Generalizing from several related classification tasks to a new unlabeled sample." Advances in neural information processing systems 24 (2011).
[3] Muandet, Krikamol, David Balduzzi, and Bernhard Schölkopf. "Domain generalization via invariant feature representation." International conference on machine learning. PMLR, 2013.
[4] Behboodi, Arash, Gabriele Cesa, and Taco S. Cohen. "A pac-bayesian generalization bound for equivariant networks." Advances in Neural Information Processing Systems 35 (2022): 5654-5668.
[5] Elesedy, Bryn. "Group symmetry in pac learning." ICLR 2022 workshop on geometrical and topological representation learning. 2022.
[6] Zhu, Sicheng, Bang An, and Furong Huang. "Understanding the generalization benefit of model invariance from a data perspective." Advances in Neural Information Processing Systems 34 (2021): 4328-4341.
[7] Brehmer, Johann, et al. "Does equivariance matter at scale?." arXiv preprint arXiv:2410.23179 (2024).

Missing baselines to support the argument in L44-50. converting input image to grayscale data augmentation

We ran additional experiments to compare the performance of our color equivariant approach to baselines that use data augmentation or grayscale input images and found that our approach performed on-par or better than these color invariant approaches. We have included these results in Table 4. We’d like to point out that even though all methods perform similarly on classification, our color equivariant architecture is more interpretable by construction, has higher sample efficiency (which others have shown both theoretically and empirically [4,5,6,7], and we’ve shown Figure 4), and naturally allows for downstream tasks such as color based sorting (see Figure 7) and color alignment.

What is the “Model sample efficiency”? The advantages of the proposed methods are evident only when a small proportion of the training example is used. What does this mean?

We use the definition of sample efficiency given in [8]: “Sample efficiency refers to the ability of a statistical method or algorithm to make accurate inferences or predictions using a minimum amount of sample data.” In Figure 4, we empirically show that our models have greater sample efficiency than conventional CNNs by plotting the performance of each model when trained on 100%, 10%, 1%, and 0.1% percent of the examples in the training dataset.

Our models outperform the conventional CNN when less than 10% of the training examples are used. This happens because our network architecture is constrained to be equivariant to hue variation. By constraining our network we achieve strong generalization performance even when only a small number of samples are used (i.e., high sample efficiency). The generalizability of representations learned in equivariant neural networks has been observed empirically and proved theoretically in [4,5,6,7].

[8] “Sample Efficiency - (Sampling Surveys) - Vocab, Definition, Explanations | Fiveable.” Fiveable.me, 2024, fiveable.me/key-terms/sampling-surveys/sample-efficiency. Accessed 20 Nov. 2024.

How can the error improvement be computed?

We compute the error improvement as the difference between the classification error of Hue-N (ours) and the classification error of the conventional CNN. We chose to show the error improvement rather than the absolute error for clarity of the visual presentation.

I appreciate the authors' efforts for their responses to my questions. However, I still have remaining concerns as written below. Thus, I would keep my initial rating.

1. Table 1 results:

• The performance improvement of Hue-3 over CEConv-3 is marginal.
• The fact that CEConv-4 is worse is not meaningless, but it cannot be an evidence that the proposed method is better than the baseline because of the marginal gap between Hue-3 over CEConv-3.

2. Table 2 results:

• The performance improvements of Hue-3 and Hue-4 over CEConv-3 are marginal.

3. Table 4 results:

• It is not intuitively understandable how ResNet-gray and ResNet-aug could be worse than ResNet for color-shift generalization performance. Specific descriptions such as experiment settings and justifications are needed.
• If we compare Hue-4 with ResNet or ResNet-aug, the proposed method show worse performance in 3 columns (Cifar-10, Cifar-100, and STL-10) out of 6 columns. To me, the results from Table 4 are not good enough to support the argument that the proposed methods are more robust to color-shift.
• Additional benefits of the proposed method over ResNet-aug and ResNet-gray (more interpretable, having higher sample efficiency, naturally allowing for downstream tasks) are not persuasive enough yet. — Why is it important in practice achieving the additional benefits? For example, as for the “model sample efficiency”, the proposed methods show better performance when the # of training samples are less than 5% of the whole dataset. This setting sounds somehow unrealistic to me.

4. Question: what is Model sample efficiency of the baseline (CEConv)? — I am wondering if Fig. 4 is the only benefit of the proposed methods or the benefit that CEConv also has.

The error of CEConv-3 is already very low on 3D shapes.

The out-of-distribution error of CEConv-3 is small on the 3D shapes dataset, but the out-of-distribution error of CEConv-4 is significantly worse. This is because hue rotations of the RGB cube are only valid when the cardinality of the discretized hue group is one or three. In Figure 3, we show that the cardinality $N$ of the group (three for CEConv-3 and four for CEConv-4) impacts the invertibility of hue rotations. The invertibility error is negligible when N=1 and N=3, but greater than 7% otherwise. This is important since the equivariance error is determined by the invertibility error (it is only low when N=1 and N=3). It poses a problem in practice since complementary colors (i.e. colors that are 180 degrees apart) are arguably more prevalent in man made spaces, which suggests that even valued $N$ are more useful than odd valued $N$. Additionally, there are settings (i.e., fine-grained classification) where it may be preferable to select a large value for $N$ to better approximate the continuous hue group. In effect, the method proposed in [16] requires practitioners to make a choice between practicality and invertibility error, while ours does not.

Why not use their experimental protocol detailed in [16]?

The experimental protocol described in [16] does not provide sufficient details for reproduction. The batch sizes, learning rate scheduler parameters, and weight decay are not provided for any experiments. Additionally, train/test splits are not provided for the Caltech-101 dataset. We attempted to reproduce their results but ultimately decided to use the same hyperparameters for all networks (we provide all hyperparameters necessary for reproduction in Appendix D), and use a train-test split of 67%-33% for Caltech-101.

Only results with CE-conv3 and CE-conv4 appear in the current paper, while [16] often shows better results with CE-conv2.

Thank you for pointing out the notational conflict between our paper and [16]. In our paper, the $N$ in CEConv-N refers to the choice of discretization for the hue group so that a higher value of $N$ gives a better approximation to the continuous hue group. In [16], it refers to the number of layers in the network that are constrained to be color equivariant. This difference is significant and has several implications. First, their model is only equivariant when all ResNet stages are constrained to be equivariant (i.e., N=3 for CIFAR, N=4 otherwise), whereas all the networks we introduce are fully equivariant. Moreover, all of their experiments are performed with the same discretization of the hue group, i.e., cardinality three. Our experiments and analyses show that the performance of [16] deteriorates when $N$ is not equal to three, whereas our performance remains stable. We provide a comparison below to summarize the notational conflict and will use a different notational convention in the camera ready version.

• Ours: CEConv-N, N denotes the cardinality of the hue transformation group
• [16]: CEConv-N, N denotes the number of ResNet stages that are constrained to be color equivariant.

The authors do not not address luminance variations.

To address luminance variation, we developed a luminance equivariant GCNN and compared its performance to a conventional CNN architecture on the small NORB dataset [1]. The small NORB dataset consists of 48.6k 96x96 grayscale images captured under 6 different lighting conditions, 9 elevations, and 18 azimuths. We partitioned the dataset into a training set and two test sets; in each dataset two of the six lighting conditions were represented. When we trained and tested on the same luminance conditions, the conventional CNN and our new luminance equivariant GCNN performed comparably, around 92% accuracy. However, when we trained and tested on different luminance conditions, the performance of the conventional CNN dropped to 64% while our luminance equivariant GCNN maintained an accuracy of 78%. Thank you for pointing out this opportunity to extend the notion of color equivariance to luminance, we will definitely include these results in our paper!

[1] LeCun, Yann, Fu Jie Huang, and Leon Bottou. "Learning methods for generic object recognition with invariance to pose and lighting." Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004.. Vol. 2. IEEE, 2004.

Adding comparisons to common approaches for handling color variations would strengthen the paper.

We ran additional experiments to compare the performance of our color equivariant approach to baselines that use data augmentation or grayscale input images and found that our color equivariant approach performed on-par or better than the baseline color invariant approaches. We have included results from these experiments in Table 4. We’d like to point out that even though all methods performed similarly on classification, our color equivariant architecture is more interpretable by construction, has higher sample efficiency (see Figure 4 in our paper, and [2,3,4,5] who highlighting the sample efficiency of equivariant representation learning both empirically and theoretically), and naturally allows for downstream tasks such as color based sorting (see Figure 7) and color alignment.

While we did not compare against robust contrastive learning approaches (e.g., [6]), we suspect they would show similar results on our benchmarks and, since they are color invariant by construction, they would have the limitations of color invariant models described above.

[2] Behboodi, Arash, Gabriele Cesa, and Taco S. Cohen. "A pac-bayesian generalization bound for equivariant networks." Advances in Neural Information Processing Systems 35 (2022): 5654-5668.
[3] Elesedy, Bryn. "Group symmetry in pac learning." ICLR 2022 workshop on geometrical and topological representation learning. 2022.
[4] Zhu, Sicheng, Bang An, and Furong Huang. "Understanding the generalization benefit of model invariance from a data perspective." Advances in Neural Information Processing Systems 34 (2021): 4328-4341.
[5] Brehmer, Johann, et al. "Does equivariance matter at scale?." arXiv preprint arXiv:2410.23179 (2024).
[6] Lo, Yi-Chen, et al. "Clcc: Contrastive learning for color constancy." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

Would this approach also be applicable to other chromacity-luminance color spaces, such as CIELAB or YUV?

We think it would be possible to approximate the structure of each of the three channels of the YUV color space using the group $(\mathbb{Z}, +)$. This is the same approach we used for the saturation (Section 4: Saturation group and group action) and luminance (Appendix E.1: Luminance group and group action) space in our proposed architecture. How well this would work, however, depends on how approximation errors accumulate. To model HSL, we only have to leverage the approximation twice. To model YUV, we would need to leverage the approximation three times.

It’s not immediately clear to us if there are groups that capture the structure of the CIELAB color space but we will continue to think about possibilities.

Could other chromacity shifts in different color spaces be designed to further demonstrate the robustness of the model?

Yes! As long as there is a mapping between the HSL and the other color space, our network should maintain robustness to transformations of the input.

I wonder if this design could be beneficial for semantic segmentation.

That is an exciting direction we had not thought of! Semantic segmentation might benefit from the principled way our network can organize and discard color information. Our network might also be useful for generating superpixels that can be used in downstream processes. Since our design can be used with any CNN backbone, it would be interesting to see if incorporating it into an existing pipeline yields any benefit.

How would the method perform on large datasets like ImageNet?

To see how our approach compares to baseline models on a larger dataset, we trained our model (Hue-4), ResNet-18, and CEConv-4 on the Tiny-ImageNet dataset [1] and got a classification error (lower is better) of 52.95%, 53,62%, and 54.50% respectively. These results suggest our approach performs on-par with conventional architectures on larger datasets. The Tiny-ImageNet dataset is a downsized subset of ImageNet consisting of 100k 64x64 RGB images. Of the 1000 classes represented in ImageNet, 200 are represented in Tiny-ImageNet, each with 500 training examples, 50 validation examples, and 50 test examples. We chose to perform our experiments on Tiny-ImageNet due to time constraints. We will include results on the full ImageNet dataset in the camera ready version.

[1] Le, Ya, and Xuan Yang. "Tiny imagenet visual recognition challenge." CS 231N 7.7 (2015): 3.

Other common variations in real-world data, such as 3D geometric transformations or lighting changes are not addressed.

3D Geometric transformations such as 3D rotations are important real-world variations and are the focus of equivariant representation learning approaches such as [2,3,4,5]. In our paper, we exclusively consider color variation which has been shown to adversely affect predictive performance [6], and is significantly less explored (as far as we know, only [7] has explored equivariant representation learning in the context of color).

Luminance variation is also important in real-world applications. We developed a luminance equivariant GCNN and compared its performance to a conventional CNN architecture on the small NORB dataset [8]. The small NORB dataset consists of 48.6k 96x96 grayscale images captured under 6 different lighting conditions, 9 elevations, and 18 azimuths. We partitioned the dataset into a training set and two test sets; in each dataset two of the six lighting conditions were represented. When we trained and tested on the same luminance conditions, the conventional CNN and our new luminance equivariant GCNN performed comparably, around 92% accuracy. However, when we trained and tested on different luminance conditions, the performance of the conventional CNN dropped to 64% while our luminance equivariant GCNN maintained an accuracy of 78%. Thank you for pointing out this opportunity to extend the notion of color equivariance to luminance, we will definitely include these results in our paper!

[2] Esteves, Carlos, et al. "Learning so (3) equivariant representations with spherical cnns." Proceedings of the European Conference on Computer Vision (ECCV). 2018.
[3] Cohen, Taco S., et al. "Spherical CNNs." International Conference on Learning Representations. 2018.
[4] Thomas, Nathaniel, et al. "Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds." arXiv preprint arXiv:1802.08219 (2018).
[5] Fuchs, Fabian, et al. "Se (3)-transformers: 3d roto-translation equivariant attention networks." Advances in neural information processing systems 33 (2020): 1970-1981.
[6] De, Kanjar, and Marius Pedersen. "Impact of colour on robustness of deep neural networks." Proceedings of the IEEE/CVF international conference on computer vision. 2021.
[7] Lengyel, Attila, et al. "Color equivariant convolutional networks." Advances in Neural Information Processing Systems 36 (2024).
[8] LeCun, Yann, Fu Jie Huang, and Leon Bottou. "Learning methods for generic object recognition with invariance to pose and lighting." Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004.. Vol. 2. IEEE, 2004.

Reviewers jSPt suggested that color-equivariance might benefit semantic segmentation. To explore this, we compared the performance of a conventional CNN based foreground-background segmentation architecture with a hue-equivariant variant on the Caltech-101 dataset [1]. We found that our hue-equivariant architecture outperforms the conventional architecture with an accuracy of 90.33 (0.23) compared to 88.75 (0.12), and produces foreground-background segmentation masks that are more compact, have fewer artifacts, and better capture fine details (see Figure 20). The architecture details, and quantitative and qualitative results are provided in Appendix E.3. Thank you for suggesting this application of our method, we will definitely include these results in our paper!

[1] Fei-Fei, Li, Rob Fergus, and Pietro Perona. "Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories." 2004 conference on computer vision and pattern recognition workshop. IEEE, 2004.