Hierarchical Classification by Training to Diffuse on the Manifold

• The proposed hierarchical cross-modal contrastive fine-tuning loss is a straightforward extension based on CLIP, and there is a lack of explanation and theoretical justification for why this model is effective for hierarchical classification from a manifold learning perspective.

• The description of the proposed differentiable diffusion strategy in Lines 221-228 is unclear. For example, how can we train a linear mapping to replace the one optimized by Eq. (6)?

• In Lines 269-270, the authors claim that '... we strictly adhered to the code provided by Valmadre [1]. Our results for iNat21 are consistent with those presented by Valmadre [1].' However, this paper uses a more powerful backbone, ResNet-50, in contrast to the ResNet-18 adopted in [1]. The puzzling aspect is that this paper reports much lower performance on iNat21 compared to Valmadre [1]. What is the reason for such a performance degradation?

• The main contribution of this paper is a diffusion-based inference. In my view, it's a general method, so it can also be used in ImageNet-based fine-tuning methods. Why not use it directly in ImageNet-based methods, similar to Valmadre [1]?

• Eq. (4) introduces a crucial hyper-parameter as it plays an important role in injecting prior information from the connection matrix into the final prediction. However, there is no discussion about it in the experimental section or from a theoretical perspective.

• There are several typos, such as 'the neural network Remarkably' in Line 185, and Equation (6) in Line 218 and Eq. (6) in Line 221.

• HCCF loss trained with only L7 in Table 4 surpasses the existing loss in Table 1 largely. These results show the significant gain of HCCF is obtained mainly by contrastive training rather than considering the hierarchy. Contrastive finetuning of CLIP seems to be inspired by (Goyal et al., 2023).

• Some details are unclear. See the Questions section below.

Minor problems.

• Line 132. The two terms $log()$ are the same. This seems to be a typo.
• In Sec.3.2 $f$ is image feature embedding. In Sec.3.2 $f$, is a prediction output (softmax?). Using the same notation would be confusing.
• In Table 5, Advanced-top-down performs the best in Majority F1; however, the number is not bold.

I will use [1] to denote Valmadre (2022).

1. What is "hierarchical training" with CE loss in the ablative study? (L294; also "CE loss + Lxxxx" distinct from "Contrastive Loss + Lxxxx" in Table 3) This needs to be explicitly defined.
2. The proposed level-wise loss does not seem to produce consistent probabilities as defined in [1]; i.e. where the probability of an internal node is greater than or equal to the sum of its children.
3. The level-wise loss might not apply as well in hierarchies that contain leaf nodes of different depths (I believe the iNaturalist taxonomy has a uniform depth of 7).
4. The results in Table 4 lack sufficient significance to conclude that "for metrics like AP, AC, and Leaf F1, comprehensive training across all levels (denoted as L123467) outperforms other configurations" and consequently that "these findings diverge from the prevailing belief that top-1 accuracy benchmarks align with hierarchical metric rankings". Most of these numbers seem about the same, with the possible exception of Leaf F1. (However, I agree that L67 and L7 are significantly better for R@xxC and Leaf Top1.)
5. While the ablative study shows some important results, I would have liked to see further investigation into the importance of the image encoder. In particular, while the authors used CLIP pretraining for the baselines, the numbers are quite close to those of [1], which used standard ImageNet pretraining. I think it would be useful to highlight this in the paper (it shows that the method better leverages the pretrained model). Furthermore, since the label embeddings seem crucially important, it would be interesting to investigate whether the CLIP image encoder is even necessary by using ImageNet pretraining with the same label embeddings.
6. It would be useful to generate the operating curves as in [1], to highlight whether the method achieves a better trade-off or simply dominates the other methods.
7. No confidence intervals (error-bars).
8. The diffusion procedure was not evaluated on models that were trained using the "parameter sharing softmax" and "soft-max-margin" losses in Table 6.
9. It's inaccurate to characterize the inference procedure of [1] as "bottom-up" (e.g. Table 5): it returns the most-informative ($\approx$ deepest) label that exceeds a confidence threshold. In fact, if the threshold is > 0.5 and the probabilities satisfy the tree constraint, then it is equivalent to greedy top-down (since there can only be one path with confidence > 0.5). A better example of bottom-up inference would be Davis et al. "Hierarchical Semantic Labeling with Adaptive Confidence", where the inference algorithm starts at the argmax leaf node and proceeds upwards.
10. Several details are unclear, see questions.

Minor:

1. It seems as though the "Softmargin" and "Descendant softmax" rows might be interchanged between Table 1 and Table 2? The results of [1] showed the "softmargin" to perform consistently better than "descendant softmax", as in Table 2 but not in Table 1.
2. In (1), it seems that one of the summations should be over all leaf nodes, rather than $L(y)$.
3. The "parameter-sharing softmax" of Salakhutdinov et al. (2011), also considered in [1], is effectively a hybrid method: logits are obtained in a top-down fashion, normalization is performed at the leaf nodes, and then probabilities are obtained bottom-up.

Suggestions:

1. I would use terminology like "graph diffusion" at the start of the paper to make it clear that you're not considering generative diffusion (DDPMs, stable diffusion, etc.).
2. "down-top" should be "bottom-up", unless you mean something specific by this?