Coloring Deep CNN Layers with Activation Hue Loss

We thank the reviewer for raising important points, requesting clarification of the method, and the importance of baseline performance. The primary contention of the review is that because maximum baseline performance is not reported, our results may be flawed and should be rejected. We understand baseline performance is important, however the goal of this paper present the idea of activation hue, linking the hue angle in human color perception to activation hue in deep network machine prediction. Unfortunately due to our limited computational budget, achieving maximum baseline performance (e.g. using extensive data augmentation, hyperparameter optimization) was simply not the best use of resources for this work. We thus opted for a convincing scenario with diverse datasets and tasks, but experiments in the context of limited data augmentation.

We thus hope the reviewer finds our paper acceptable based on the idea of activation hue and limited but convincing experiments. The reviewer may be unfamiliar with color hue, color hue is profoundly connected to human perception and cognition. To our knowledge, this is the first work explore hue angle in multi-channel activation maps, which has possible broad application as other reviewers mention.

We address all specific comments and questions below.

Weakness 2, Question 1: Regarding the method reviewer states “The idea of regularizing activation maps based on nearest neighbor information is interesting”, however then mentions “I have problems understanding the method”, “the proposed loss is not explained clearly”, “Please explain better how the proposed loss is implemented ”. It appears the reviewer misunderstood the hue loss in sections 4) Method and 5) Experiments, which involves estimating angular labels, and does not involve nearest neighbors as in section 3) Observations. We feel that adding the following text will clarify the implementation:

“We propose a lightweight hue loss term to regularize standard one-hot loss, requiring only two additional network outputs to predict hue-like angular $\theta$ labels parameterized as $(x,y)$ coordinates.”

Weakness 1, Question 2: The reviewer’s primary concern is that better baseline classification results are available in the literature, and thus any improvement is shadowed by the probability that the overall technique used in training CNN is flawed and the proposed idea recover only some of the flaws.

We agree with the reviewer that baseline accuracy is important, we commit to adding the following information regarding baseline accuracy specifically for Imagenet: Our baseline Resnet18 accuracy 0.635%, 9% lower than reported Resnet18 72.1%[He2016] but is higher than Alexnet 62.5%[Krizhevsky2012]. Our baseline Efficientnet accuracy of 71.4% is closer the reported Efficientnet[Tan2019] 76.3% which uses more elaborate augmentation including random crop

The reviewer is correct, for all datasets used, the highest baseline accuracies are achievable using different models (notably transformers), and higher baseline accuracy for the same models is achievable using extensive data augmentation, e.g. as in the original papers. However, our experiments were constrained by our computational budget for training CNN models, particularly ImageNet training which required on the order of weeks. our focus was not to achieve the highest possible baseline for specific networks and tasks, which requires extensive augmentation and computationally expensive training

Our goal was not to achieve highest baseline accuracy, but to demonstrate activation hue in most diverse range of contexts. We thus experimented with familiar CNNs and training from scratch, in the context of minimal data augmentation (horizontal flip, no random crop) where convergence may be achieved within 100 epochs. Results are lower that what is achievable in published baselines achieved with more extensive augmentation on individual datasets. For example, On Imagenet, our baseline Resnet18 accuracy 0.635%, 9% lower than reported Resnet18 72.1%[He2016] but is higher than Alexnet 62.5%[Krysevsky2012]. Our baseline Efficientnet accuracy of 71.4% is closer the reported baseline 76.3%[Tan2019] which uses more elaborate augmentation including random crop

Regarding larger differences in Table 1, e.g. “Resnet18 - Cars”, Resnet18 had simply converged to suboptimal solutions due to limited augmentation, also Efficientnet several cases of fine-grained classification. This highlights the possibility of activation hue regularization where limited augmentation is possible.

Regarding results for Table 1, not including VGG, Inception, Densenet. While various networks were important to motivate observations hue in Section 3), they were all outperformed by Efficientnet, particularly on smaller datasets. We thus presented the most improved (Resnet18) and best result (EfficientNetB0), to avoid redundancy.

We thank the review for insightful comments and suggestions. We hope that our clarifications below are sufficient for the reviewer to deem our work acceptable. We elaborate on these in order of importance.

Point 4: Regarding the reviewer’s evaluation that our method is “limited to few-shot learning scenarios”, we believe this is related to confusion in Section 3) Observations, where we motivate activation hue using nearest neighbor lookup in a manner similar to few-shot scenarios. The hue loss method we propose in Section 4) Method and Section 5) Experiments involves full training from scratch using hue labels, not few shot learning. We will add the following sentence where appropriate clarify our activation hue loss method:

“We propose a lightweight hue loss term to regularize standard one-hot loss, requiring only two additional network outputs to predict hue-like angular $\theta$ labels parameterized as $(x,y)$ coordinates.”

Points 1, 2: Regarding object centeredness: this is an interesting point which we have considered, with a surprising conclusion. First, the assumption of central objects is typical for image classification tasks and datasets, e.g. ImageNet, and it is true that classification degrades for images with small, off-center objects.

Our method is motivated by observations of class-related angular bias in activation layers, however surprisingly, this does not appear related to object position in the image. It is true objects tend to occupy the image center, however any systematic variations object position would be small relative to the cumulative receptive field size of filters in in deep layers, of translation invariant CNNs. Furthermore, our model improves such as textures (DTD), which does not contain centered patterns at all.

• We will elaborate on this in briefly in Discussion paragraph 2 if appropriate.

Point 3: We thank the reviewer for acknowledging our results supporting our hypothesis, we agree further in-depth analysis is necessary for general adoption in broader contexts. We propose an additional discussion sentence in response to similar comments other reviewers, mentioning future evaluation Vision Transformer, detection and segmentation tasks, larger scale networks.

Point 5: We agree an extensive comparative analysis with existing methods is important, we hope that clarification of our method and experiments, i.e. standard CNN training from scratch, is sufficient to allow the reviewer to better appreciate work and the potential usefulness of the activation hue.

We thank the reviewer. There are two specific clarifications of misunderstanding that may change the reviewers mind.

1. Regarding Few-shot experiments: We sincerely thank the reviewer for pointing out this specific error in the text of the Experiments, the current text reads: "Training and classification were evaluated in diverse few-shot learning tasks" where our experiments in this section are standard classification, and not few-shot learning.

The correct text should and will be: "Training from scratch classification was evaluated in diverse learning tasks" This is our error, it was originally refering to Section 3) Observations which involved a few-shot like scenario.

2. Regarding object centeredness: We agree classification should operate without assuming centeredness. Please note that our method evaluated in Section 5) Experiments does not require this assumption, and that experiments show improvement for the DTD Textures dataset where image structure of interest is specifically not centered.

We appreciate the positive comments, and that the reviewer has understood the motivation behind the activation hue. We commit to providing additional explanation the minimal complexity of our model, in addition to publishing the code implementation and humbly request the reviewer raise their assessment to accept.

We address specific weaknesses and questions mentioned below.

Weaknesses 1,3: The comment regarding scaling to larger networks is well taken, we propose to mentioning this in the discussion of future work applying activation hue to larger networks, tasks with spatial extend including detection, segmentation, and alternative architectures such as Vision Transformer mentioned by another reviewer. Activation hue may in principle be

Weaknesses 2, 4, Question 1: Regarding implementing activation hue, complexity, we add emphasize the lightweight nature of the implementation throughout the paper (abstract, introduction, Section 4) Method, Experiments. We will also provide a github code link in footnotes, which we neglected in the original submission. The following sentence is added:

“We propose a lightweight hue loss term to regularize standard one-hot loss, requiring only two additional network outputs to predict hue-like angular $\theta$ labels from $(x,y)$ coordinates.”

Weaknesses 2: regarding properties, convergence, functioning of the method: activation hue loss is implemented by a pair of output neurons which constrain network weights to be predictive of angular (x,y) class labels. We hypothesize this encourages and captures an activation hue intrinsic to each class, which leads to improved convergence.

Question 2: Fine grained results: it is true improvements are more marked for certain fine-grained tasks, Birds for Efficientnet, and Cars for Resnet18. It appears that Resnet may have difficulty converging given minimal augmentation in our experimental protocol (horizontal flips), may be related to the distinctiveness of the activation hue for these classes. We would emphasize this in a Discussion sentence.

Fixed possible citep

We appreciate the positive comments, and that the reviewer has read and grasped potential significance of activation hue. We address the weaknesses points raised as follows:

1. We agree the activation hue principle could be broadly applied to other architectures, tasks. We propose add a Discussion sentence to mention potential application to the Vision Transformer, in additional to detection and segmentation tasks, scaling to larger models as mentioned by other reviewers.

2. We thank the reviewer for mentioning label smoothing, we add this to our related work as follows, it is similar to the work of Rodriguez 2018 ‘beyond one-hot encoding’ in seeking to reformat the shape of training signals.

“Our final results training from scratch using one-hot $+$ hue loss are similar in spirit to work seeking to regularize the label and/or the activation space, including using real-valued rather than strictly one-hot training labels \cite{rodriguez2018beyond}, label smoothing which adds a constant to one-hot labels\cite{muller2019does},”

3. Similar comments from other reviewers allow us to recognize we did not sufficiently describe our implementation. The computational cost of implementing activation hue is minimal, equivalent to two extra output neurons for angular labels in x, y coordinate format.

"We propose a lightweight hue loss term to regularize standard one-hot loss, requiring only two additional network outputs to predict hue-like angular $\theta$ labels parameterized as $(x,y)$ coordinates."