Leveraging Graph Neural Networks to Boost Fine-Grained Image Classification

• The proposed learning scheme seems not to be devised for the task of FGVC.

From the reviewer's view, adding a graph neural network to refine the image representation and acquire a stronger contextual representation is versatile to many high-level tasks, from image recognition on such as ImageNet, detection, segmentation, and etc. Thus, it would be no surprise that such represention learning scheme, as a by-product, can improve the FGVC performance on top of some backbone networks.

• The proposed learning scheme does not have much insight for discerning the fine-grained patterns, which is critical for FGVC. Overall, in the FGVC community, one of the most important things for FGVC is to differentiate the fine-grained patterns from the entire image. Indeed the graph representation can improve the contextual information, or the relation between feature embeddings. However, the designed framework does not delve into depth to mine the relation between fine-grained patterns. Instead, it is still implemented through an image-level for contextual enhancement. It is of a style that the reviewer does not appreciate for FGVC and its representing.

• The technical novelties are somewhat incremental and marginal. Directly adding a graph neural network after a backbone is very ordinary and lacks insight.

• The compared methods and related work discussison are both very insufficient. In this work, the comparison and references are mainly from typical visual backbones. In both introduction and related work, more recent FGVC works from the past five years, especially from top-tier conferences such as CVPR and top-tier journals such as PAMI and TIP should be included and discuss. Besides, more comparisons should be made.

• The visualization is not good. The activation map is common for generic image recognition such as on ImageNet. Please follow the visualization in FGVC tasks, to activate the fine-grained patterns by such as GradCAM.

[a] Destruction and construction learning for fine-grained image recognition. CVPR 2019.

[b] Counterfactual attention learning for fine-grained visual categorization and re-identification. ICCV 2021.

[c] Fine-grained object classification via self-supervised pose alignment. CVPR 2022.

[d] Transfg: A transformer architecture for fine-grained recognition. AAAI 2022.

1. The main weakness of the paper is that some of the results do not seem to back up the proposed method. First of all, the main idea of the paper is that by processing the feature vectors of all the images in a batch as a nodes in a graph, GPH can exploit the relationships between the images in the graph to improve the feature embeddings and make them more suitable for fine-grained recognition.

   If that is the case, the proposed method should not be better with lower batch sizes. However, from Table 5 and Figure 3 it looks like models augmented with GPH perform very similarly with batch size of 1. How can a GPH augmented model perform similarly when the batch size is 1? Shouldn’t it perform the same as the baseline, as it is not using any extra information? Additionally, if the best model is ConvNext + GPH, it would be better if the results in Tables 4 and 5 included ConvNext-GPH.

   Another example, the results of DenseNet201-Attention seem really bad with batch size of 1. However, with batch size of 1, the result should be similar to the just DenseNet201, instead of much worse. Why is that the case?

2. Another aspect that is not clear is the choice of using GNNs to aggregate information from different images in the graph. The method proposed in the paper uses a fully connected graph, so there isn’t really a structure that the GNN can exploit. All the GNN is doing is aggregating features from a set of images without any relevant structural information, so why not use something like DeepSets [1] or attention?

3. Related to the previous point, the authors have used attention instead of GPH as a baseline, but it seems to obtain much worse performance. Why is that the case? From Table 2 the GPH-GCN version seems better than Attention, but I’m not sure that is correct. Considering that the graph is fully connected, GCN will amount to averaging the feature embeddings of all nodes at each layer, therefore it is not clear why that works better than attention. GCN should be equivalent to an attention that assigns $1/degree_i$ to the attention value for each node $i$.

4. The authors also comment on the batch design, i.e how to select which images go in the batch, since the output for one image depends on the other images in the batch. First of all, by using the "sequential" option (most images likely coming from the same class), the model is effectively looking at many images from the same class to make a prediction. However, in a real use case scenario, one does not know the class of the images to process, so therefore all validation scores should be reported using the “shuffled” option. It is not clear if the reported validation scores use the “sequential” or “shuffled” option. Validation scores using the “sequential” option should be invalid, since the method can exploit a pattern in the order of the data.

   Secondly, I would expect the predictions made with “sequential” to be much better than the shuffled option, since they are almost two different tasks. The former is easier, as it amounts to classifying the class shared by a group of images, while in the second one there is class variability, so it is harder. However, from Table 4 it looks like both are very similar. How were the values in that table obtained? Changing the method during training and evaluation? Or only during evaluation?

5. Figure 2 lacks details. It is not clear which samples from which dataset have been used, or which base model and GPH version are used to generate the image embeddings, as well as the projection method.

Finally, the authors should cite [2], which is a related paper in which a GNN is used to predict a label for one image considering other images in the batch.

[1] Zaheer, M., Kottur, S., Ravanbakhsh, S., Poczos, B., Salakhutdinov, R. R., & Smola, A. J. (2017). Deep sets. Advances in neural information processing systems, 30.

[2] Garcia, V., & Bruna, J. (2018, January). Few-shot learning with graph neural networks. In 6th International Conference on Learning Representations, ICLR 2018.

• My main concern is the lack of motivation for using a graph-based approach. The manuscript in its current version lacks any clear intuition for why a graph neural network on top of would improve performance in such a consistent way for specifically fine-grained image classification.
• Second, the contribution this paper makes is fairly limited. The paper proposes a very practical addition to current DNN-based image classification methods, but essentially does not go beyond showing that with additional compute, fine-grained image classification improves. Especially given the intimate connection between message passing on graphs and "conventional" convolutions (convolutions are really a special case of message passing), I would very much appreciate a more thorough analysis as to why a graph-based approach is beneficial, i.e. what specific benefits this method has in this setting. Also a more thorough analysis on the types of features that are being learned would be useful; is it really the GPH that leads to a more correct clustering in fig 2? Would simply adding more layers of DNN not yield the same results? These questions have not been adequately addressed in my opinion.

However, there are still some concerns to be addressed:

• The combination of GNN + DNN is quite straightforward and simple, thus, the novelty of this paper may be marginal.

• The authors claim that the proposed method is able to learn contextual information and relationships that are essential for fine-grained categorization. However, looking through the manuscript, it seems that the discussion and evidence are missing.

• The proposed method builds a fully connected graph, which will increase the complexity of the whole model as the training batch size increases. While the authors provide an ablation study of the test batch size, it will be more interesting to provide a detailed analysis of the training batch size, including the number of parameters, wall-clock training time, and performance.

• In Table 2, the optimal model accuracy varies across datasets with different GNN encoders. Please clarify how to choose the encoder if using the proposed method.

• It is challenging to deploy and evaluate the trained model because its test performance is tied to both the test batch size and the data sampling method. Please clarify.