A Label is a Label is a Label: Relation Augmentation for Scene Graph Generation

Significance

The motivation focuses on long-tail distribution of relations, but does not fully justify and compare with a large body of work that outperforms the given method on all the metrics. (some papers discussed below). Also, the paper just uses a single dataset to compare long-tail while there have been other SGG datasets (GQA-LT, VG8k) that have a longer tail and the impact of augmentation could be useful there too. Providing more empirical analysis for the benefits of relation augmentation could better highlight its significance.

Methodology

1. The description of the visual augmentation method is unclear in places - for example, what is the value of alpha and how it is chosen.

2. For the semantic augmentation, additional analysis and justification for the plausibility criteria (similarity, co-occurrence etc) would be helpful. See the questions section for details.

Evaluation

1. More qualitative analysis of generated examples from the augmentation techniques would give intuition about their effects. Do they produce plausible, diverse relations?

2. Comparisons to a wider range of prior art in unbiased SGG is missing would better position the advances. Some recent work that uses augmentations and semantic upsampling are missing such as:

a) RelTransformer (https://arxiv.org/pdf/2104.11934.pdf) b) CaCao visually prompted model for SGG (https://arxiv.org/pdf/2303.13233.pdf) c) NARE (https://arxiv.org/pdf/2111.13517.pdf)

1. The presentation of the paper and the writing need to be improved. There are some mistakes in typos and style.
2. The main contribution of this paper is just a data augmentation, which has been used in other visual tasks. And when replacing the objects in the scene, how to ensure the relationship labels will not be changed.
3. The experiments and results are not convincing. Compared with some recent methods, the results are not SOTA. The authors should give more comprehensive comparisons and analyses with other SGG methods, such as VETO, NICE, and so on.

The main problem is: 1、 Even though the authors claim that they are the first to propose strategies to augment visual relations, several works have focused on relationship augmentation, such as:

Fast Contextual Scene Graph Generation with Unbiased Context Augmentation

Compositional Feature Augmentation for Unbiased Scene Graph Generation

Visual compositional learning for human-object interaction detection (HOI task, which is similar to SGG)

2、The equations and images are not well-defined. For example, in Equations (4) and (5), the < should be $\in$. What is x? What is b? It seems like the author uses the same symbol to reflect different meanings; for instance, the b may refer to a box region as well as a box size. The left subfigure of Figure 1 is enough.

3、The experiments are not sufficient. It would be better to give a recall, or there may exist the possibility to increase the mean recall by significantly sacrificing the accuracy of the recall. More state-of-the-art methods need to be compared, for example, unbiased methods in 2022 and 2023. More ablation experiments are required. How to choose $\alpha$ in the equation? How do you select top-k? How to choose the manner of perturbation methods? More analysis should be provided, for example, the exact relationships the proposed strategy improved and more explanation about why RelAug works.

This paper has the following weaknesses

1. Evaluation of Recall@K missing. The augmentation of a rare triplet may hurt the semantically similar common triplet. For example, if we augment the training with 'man-walking-on-snow', it may hurt the performance of 'man-on-snow'. Therefore, it is vital to observe how the recall@K is affected after the augmentation.

2. Evaluation of zsR@K missing. A great advantage of augmenting training samples is the improvement of the zero-shot capability of a model. For example, if we create a new training triplet 'boy-riding-snowboard' from the visual and/or semantic augmentation of the original triplet 'boy-on-snowboard, it may coincide with some zero-shot examples in the testing dataset. In general, it should improve the zsR@K metric and it would strengthen the claim of the paper. If the papers' augmentation does not improve the zsR@K, that also requires some investigation and should be reported/discussed.

3. Per-class bar plot of recall improvement missing: In Figure 1, authors demonstrated the per-class distribution of relationships before and after augmentation. It would be very interesting if we had a similar bar plot for recall-improvement of each class. Such bar plots are now present in almost every published SGG paper and they provide useful insight into the methods' robustness to different classes. I suspect the performance increase/ decrease of the classes would be proportional to its increase/decrease after the augmentation process. Such a bar plot would also provide the authors with insight on how to efficiently augment the training classes.

4. Missing ablation study of different semantic augmentation methods. In section 3.3, different perturbation method of semantic augmentation has been discussed. However, how each of these influences the performance is not demonstrated in the evaluation. Without such ablation studies, readers would find the detailed discussion without any ties to the results section.

5. Missing ablation study on grafting hyperparameter. Eqn. (6) describes the smoothed mixup-like augmentation of the visual domain. However, how sensitive the performance is with respect to the hyperparameter $\alpha$ is not studied. The performance should be equal to the baseline model with $\alpha=0$ and maximize to a certain value of $\alpha$. Both recall and mean recall can be observed with this sweep of $\alpha$.

I am quite concerned with the novelty. For semantic augmentation, it uses several strategies to perturb the relationships. However, I do not know why these strategies can lead to better performance. It seems it may lead to a mass of noisy labels. I wonder if I missed some points. For visual augmentation, it is similar to MixUp and CutMix except that it mixes the objects of the same category.

The authors merely compare with some methods that were published before 2021. However, more recent works that obtain better performance are intensively proposed. I think these works [1,2] should be included as the baseline to better demonstrate the effectiveness of the proposed algorithm. Besides, The authors merely compare with some baseline, and there exist methods [3] that also address the unbias SGG task. However, these works are also not included for comparisons.

Some minor issues with grammar errors and punctuation.

[1] Prototype-based Embedding Network for Scene Graph Generation, in CVPR, 2022 [2] RU-Net: Regularized Unrolling Network for Scene Graph Generation, in CVPR, 2022 [3] Stacked Hybrid-Attention and Group Collaborative Learning for Unbiased Scene Graph Generation, in CVPR, 2022