HyperDisGAN: A Controllable Variety Generative Model Via Hyperplane Distances for Downstream Classifications

Dear reviewer bUmq, Thank you very much for your helpful comments! We respond to each of your comments one-by-one in what follows.

[W1] The limitation of the binary classification.

Good question, thank you. We add this topic in Appendix I.1 (Page 18) in new version paper.

Binary classification is the foundation of classifications in Machine Learning. The current method can be extended to multi-class classification, but we would like to start with the basic binary classification problem which is the foundation and starting point of machine learning. This extension idea is straightforward. Unlike the cross-entropy, multi-hinge loss solves the multi-class classification by constructing the multiple hyperplanes, this will be specifically divided into two ways: (1) OneVsRest: It takes a multi-class classification and turns into multiple binary classification for each class. (2) OneVsOne: It takes a multi class classification and turns into multiple binary classification where each class competes against every other class. Obviously, for $n$ classes, OneVsRest constructs $n$ hyperplanes whereas OneVsOne constructs $\frac{n(n-1)}{2}$ hyperplanes.

Upgrading the HyperDisGAN for multi-class classification can be as follows: we firstly pretrain a multi-class classifier by the oneVsRest manner, and use $i^{th}$ hyperplane to classify between $i^{th}$ domain and not $i^{th}$ domain. Second, we can collect location information including the vertical distance between samples to hyperplane, and the horizontal distance between the intra-domain samples. Third, we can use only two generators (inter-domain and intra-domain) taking additional location information as input and one discriminator to control the generated samples' variation degrees by modifying the multi-domain generation method (such as the starGAN [a] and the starGAN V2 [b]) conditioning on classes. The controllable generated samples will be used to adjust the decision boundaries for this multi-class classifier.

Self-supervised learning benefit from the traditional data augmentation. For example, the augmentation in simCLR [c] directly flip/rotate the image and the class of the image is unknown. The proposed method cannot be used for self-supervised learning. Becasue training the HyperDisGAN needs to know the class label of each image in training dataset.

[a] Choi, Y. et al. StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation. CVPR 2018.

[b] Choi, Y. et al. StarGAN v2: Diverse Image Synthesis for Multiple Domains. CVPR 2020.

[c] Ting Chen. et al. A Simple Framework for Contrastive Learning of Visual Representations. ICML 2020.

[W2] Comparisons between the traditional and proposed method.

Thanks for your comments. Traditional augmentation (TA) sometimes cannot outperform the standard networks. That is the exactly the weak points we would like to raise and solve, i.e., relying on prior knowledge and specific scenes. For fairness of training on small-scale datasets, all experiments have been conducted on averaged over three runs. Furthermore, we test eleven classification models loading the ImageNet based pretraining-weights ("Training settings" in Page 6 in original version paper). Under this setting, making further breakthrough in classification performance is difficult even if the traditional augmentation is used. Therefore, we conduct the experiments combining the traditional augmentation and the proposed HyperDisGAN (TA+HyerDisGAN) to make breakthrough in classification performance, rather than aiming to replace the traditional augmentation methods.

[W3] Many hyper-parameters need to be tuned.

Thanks for your question. Actually, we don't need to tune all four hyper-parameters simultaneously. The four hyper-parameters are individual two paired parameters. The first pair $\lambda_{\rm verDIS}$ and $\lambda_{\rm crossCYC}$ is used to balance the loss of HyperVerDisGAN, whereas the second pair $\lambda_{\rm horDIS}$ and $\lambda_{\rm intraCYC}$ is used to balance the loss of HyperHorDisGAN. Therefore, we only need to tune the paired hyper-parameters for the HyperVerDisGAN and HyperHorDisGAN, respectively. Tuning these two versions of HyperDisGAN do not affect each other. See Appendix B, Table 3 (Page 14) in the new version paper.

Coutinued.

[W4] Formulation clearness for proposed method setting.

Thank you for your suggestion. The $X$ and $Y$ in the Section 3.3 of original version paper only denotes two domains. For the convenience of reading, we have added a clear formulation at the beginning of Section 3 (Page 3) in the new version paper:

Formulation:

Our goal is to learn cross-domain mapping functions between $X$ and $Y$ and intra-domain mapping functions inside $X$ and $Y$, given training samples $\lbrace x\_i \rbrace_{i=1}^N$ where ${x_i}\in{X}$, and $\lbrace y_j \rbrace_{j=1}^M$ where ${y_j}\in{Y}$. We denote the data distribution $x \sim P_X(x)$ and $y \sim P_Y(y)$, hyperplane vertical distance $d_x^v\in V_X$ and $d_y^v\in V_Y$ and hyperplane horizontal distance $d_x^h\in H_X$ and $d_y^h\in H_Y$. As illustrated in Figure 2, our model includes four mappings $G_{X2Y} : \lbrace X, V_Y \rbrace \rightarrow Y$, $G_{Y2X} : \lbrace Y, V_X \rbrace \rightarrow X$, $G_{X2X} : \lbrace X, H_X \rbrace \rightarrow X$ and $G_{Y2Y} : \lbrace Y, H_Y \rbrace \rightarrow Y$, where the generators generates images conditioned on both source image and target images' hyperplane distances. In addition, we introduce four adversarial discriminators $D_{X2Y}$, $D_{Y2X}$, $D_{X2X}$ and $D_{Y2Y}$, where $D_{X2Y}$ aims to discriminate between real images $\lbrace y \rbrace$ and generated images $\lbrace G_{X2Y}(x, d_y^v) \rbrace$; in the same way, $D_{Y2X}$ aims to discriminate between $\lbrace x \rbrace$ and $\lbrace G_{Y2X}(y, d_x^v) \rbrace$. $D_{X2X}$ aims to discriminate between real images $\lbrace x \rbrace$ and $\lbrace G_{X2X}(x, d_x^h) \rbrace$; in the same way, $D_{Y2Y}$ aims to discriminate between $\lbrace y \rbrace$ and $\lbrace G_{Y2Y}(y, d_y^h) \rbrace$.

[W5] Typographical error.

Thank you, we have eliminate typographical errors in the new version paper.

[Q1] Pre-trained generators.

Good question, we have clarified this topic to Appendix I.2 (Page 18) in new version paper.

Restart training the generators to generate samples will costs resources and efforts. The proposed method is possible to be extended to leverage pre-trained generators, if the practitioners want to use their own dataset having same classes as our train dataset. The current input distance parameters representing the real target samples' location are not random enough. Now we can extend the generators to take any random distances as inputs. The range of the random distances can refer to the coordinate axis in Figure 6 (Page 9) of original version paper. This is practical, because we can still use the auxiliary classifier to reconstruct the random distances and impose the random distance loss. In the test phase, the practitioners can load the pretrained generators and take their own input images and any random distance to generate the large amount of samples.

Dear reviewer aNvk, We appreciate your efforts and detailed comments to improve the manuscript. Please find our responses below.

[W1] Interpolation based data augmentation.

Thanks for your comments. The motivation of our proposed method is not weak.

Mixup does not bring high quality of synthesis and lack of explainability regarding the contributions of regions for the classification labels. Unlike this interpolation based methods, we build bridges between the image generation with the downstream classification, by accurately reshaping decision boundary for the downstream classification. The innovations based on hyperplane theory has good explainability. Our motivation is illustrated in the Figure 1 (Page 2) in original version paper, and we also upgrade the Figure 1 (Page 2) in the new version for better reading.

Although mixup is a way of mixing classification, it brutally combines random pair of classes which ignore other possible generated point in the generation space unlike a GAN based method. Due to the limitation of the page and the limitation of the mix-up, we choose other generator-based data synthesis methods for comparisons.

[W2] Practicality of multi-class classification.

Thanks for your question. We add this topic in Appendix I.1 (Page 18) in new version paper.

The proposed method is still practical for multi-class classification. The number of generators will not increase in proportion to the number of classes. We can take n-classes classification as the n binary classifications, and construct n hyperplanes by oneVsRest manner. For each hyperplane, we can still measure the vertical distance between samples to hyperplane, and the horizontal distance between the intra-domain samples. Moreover, starGAN [a] and starGAN V2 [b] only use one generator to transform images over the multiple domain. Therefore, what we need to do next is just let the starGAN's generator take additional distance parameters.

[a] Choi, Y. et al. StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation. CVPR 2018.

[b] Choi, Y. et al. StarGAN v2: Diverse Image Synthesis for Multiple Domains. CVPR 2020.

[W3] Negligible gained performance.

All experiments are conducted on four datasets, including three small-scale datasets and one tiny-scale dataset. Small data is where we need data synthesis most. That's why we also test not only accuracy but also auc score averaged over three runs. Most auc scores increase by using our methods especially on the small-scale datasets. Last but most important, we test extensive classifiers including eleven different architectures. Therefore, the proposed method consistently improved the performance of classification.

[W4] Experimental baseline.

Thanks for your comments. We have selected cycleGAN which is published in ICCV conference, and ACGAN and VACGAN which are famous and highly relevant to auxiliary classifier. Of course in the scientific community, there are many other variants of GAN generating samples, but there are some page limit.

[W5] Undefined words and algorithm explanation.

We have FULLY explained these words in original version paper:

The word cross-domain and intra-domain can be seen in the explanation of Figure 1 (Page 2): samples crossing the decision boundary are cross-domain, otherwise the samples are intra-domain; And in Related Work of ``Two-domain Image Transformation”, the related concept of domain is common in this research field.

The word "location" can be firstly seen in line (from 9 to 11) of abstract: "The locations are respectively defined using the vertical distance ... and horizontal distance ..."; And in the line (from 6 to 8) of Page 2: "... the vertical distance ... and the horizontal distance ... are taken as the controllable location parameters".

The word ``hyperplane" is in the first sentence of Section 3.1 (Page 3) : "We pretrain a auxiliary classifier by Hinge Loss (Rosasco et al., 2004) to obtain an optimal hyperplane ...". "hyperplane" is a common word in classification field of machine learning, such as SVM algorithm.

We have explained the overall algorithm in both Figure 2(a) and Figure 2(b) in Page 4. Figure 2(a) denotes the training procedure of HyperDisGAN and Figure 2(b) illustrates joint training downstream classification models on the both original and HyperDisGAN's generated samples.

Even though, we revised the full paper again to improve the quality. In new version, the overall algorithm and training procedure are detailed in Figure 2 (Page 4) and Figure 3 (Page 5).

Please let us try to be respectful. My review is based on the weighting of both strengths and weaknesses of the paper. I acknowledge your contribution and also highlight the main weaknesses of the proposed method. I do not believe that the contributions and potential impact outweigh the weaknesses. It doesn't look, at least to me, as an emotional review. Moreover, I do not have any "prejudice with GANs", since it is actually my main area of expertise.