Structural Adversarial Objectives For Self-Supervised Representation Learning

Thank you for your feedback!

Q1: Motivation of proposed methods.

A1: Our proposed loss functions seek to synchronize the distributions of real and generated images, as in Equation 2. Additionally, we aim to automate learning of semantic representations. This is achieved through the structuring of non-collapsed representations by optimizing their covariance (Eqn.2) and by creating local clusters through feature grouping (Eqn.6). Figure 2 illustrates the effectiveness of our proposed objectives, showing that by optimizing the discriminator with these objectives, the learned embedding aligns semantically similar data and separates dissimilar data.

Q2: Comparison to related work, ContraD

A2: We appreciate the reference to the ContraD study and plan to include it in our related work section. ContraD enhances representation learning in GAN frameworks by incorporating view-consistency contrastive learning methods. Their goal differs remarkably from our approach, which demonstrates that adversarial training objectives can inherently encapsulate feature learning capabilities. Our method presents a concise learning paradigm that does not rely on any form of view-consistency objectives, whether explicit or implicit.

Q3: Generation quality and evaluating our method on StyleGAN2-ADA

A3: In our work, the primary focus is on representation learning rather than image generation. This is why we used a ResNet-18 as the discriminator in all experiments, ensuring consistent comparison with other representation learning frameworks. Although the choice of generator is secondary, it is crucial that it does not become a limiting factor in our pipeline. We opted for BigGAN's generator with increased feature channels for this reason. We did experiment with StyleGAN's generator but found it less effective for our purposes.

Q4: Why did the authors choose to implement JSD as the loss function? Could a distance metric like Wasserstein-2 distance, commonly used in FID, also based on the assumption of Gaussian distributions, can be used?

A4: The JSD was chosen for its properties of symmetry, bounded values and allowing for the simultaneous optimization of mean and variance. Although the Wasserstein-2 distance similarly optimizes both mean and variance, its unbounded nature can lead to greater instability compared to JSD, making JSD a more stable and suitable choice for our objectives.

Q5: Breakdown of the running time.

A5: We have benchmarked the running times of each section in the discriminator's (D's) round, with the times reported as average values over 100 runs. The measurements were taken using a batch size of 256 and a single A40 GPU. The time taken for each section is as follows:

• Forward pass of the discriminator: 0.02s

• Forward pass of the generator: 0.07s

• Computing the model's approximated Jacobian: 0.27s

• Computing the adversarial loss: 0.007s

Running time for a single forward-backward pass is:

• A Complete D's round with our objectives: 0.78s

• A Complete D's round with vanilla GAN objectives: 0.29s

Our proposed objectives require about 2.7x the time of the GAN baseline during training, a reasonable time budget given the significant improvement in feature learning ability. It should be noted that, during inference time, there is no additional running time overhead compared to the GAN baseline.

Q6: Hyperparameter sweep

A6: We run ablation experiments by sweeping through the choices of $\lambda_h,\lambda_c, \lambda_s$ in CIFAR-10 and show the results as follows:

Table C: Ablation experiments on hyper-parameters. Our method is insensitive to the hyperparameter change within a reasonable range.

Thank you for your feedback!

Q1: Large-scaled experiment on ImageNet-1K.

A1: Please see the large-scale experiments section in our general reply.

Q2: It is also not clear why the authors use SVM and K-M for evaluating the learned representations in Table 1 and do not include linear probing, which is commonly used in the representation literature.

A2: We utilized Linear-SVM for evaluation, applying the default hyperparameters provided by SKlearn. This approach, compared to linear probing that uses mini-batch gradient descent, minimizes randomness by running over the entire dataset in a single batch. As Linear-SVM is also a linear classifier, we anticipate both classifiers reaching similar end states, a claim supported by the almost identical results in Table 2 for both Linear-SVM and Linear probing. We employed K-Means clustering to evaluate the features under more challenging conditions than linear classifiers, offering a unique perspective on the quality of the learned features.

Q3: The reconstruction-based self-supervised methods (e.g., MAE), which have been shown to outperform contrastive learning methods on ImageNet-1k, also do not rely on hand-crafted data augmentations. Hence, to demonstrate the contribution of this work, it is necessary to show that the proposed method can provide performance gain over them on large-scale datasets.

A3: Masked Autoencoders (MAEs) implictly employ the view-consistency objectives through a process of implicit matching between corrupted and clean images. On the other hand, they are sensitive to hyperparameters like patch size and mask ratio. In linear probing evaluations, MAEs underperform compared to contrastive learning methods. This is evidenced in MAGE [1], where the MAE-VIT-B achieves a 68$\%$ probing accuracy, in contrast to the 76.7$\%$ achieved by the MoCo v3.

Q4: I think the authors missed a very relevant related work (not my paper) which should be discussed and compared with: Li et al. MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesis. CVPR 2023.

A4: We thank reviewer for the suggestion of MAGE and we plan to include it in our related work section. MAGE enhances representation learning through masked autoencoders by adding a contrastive learning method. This differs significantly from our approach, which aims to demonstrate a straightforward learning paradigm free from any form of view-consistency objectives, whether explicit or implicit.

[1] MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesis

Thank you for your feedback!

Q1: The cluster property is well-known in the discriminator. Since DCGAN already show it, so I think it is not new in this paper to present it.

A1: While the discriminator's feature learning capability in traditional GAN frameworks, like DCGAN, is acknowledged, our work focuses on enhancing this aspect significantly. As evidenced in Table 4, conventional GANs exhibit limited feature learning ability. Our innovations in training objectives and regularization techniques have remarkably improved feature learning performance, as shown by the increase in K-Means clustering scores from 29.69$\%$ to 80.11$\%$, and Linear classifier scores from 69.31$\%$ to 89.76$\%$.

Q2: The presented method is not two much interesting, even authors give a comprehensive analysis. The paper is not new, and has less contribution to this community.

A2: We respectfully disagree with the assessment of our paper's novelty and contribution. Our approach, diverging from view-consistency objectives, explores a more generalized yet challenging direction in feature learning. Our method demonstrates significant improvements over GAN baselines in feature learning, showcasing a substantial advancement in the field.

Q3: The used datasets are small.I would like to use big datasets to support the proposed method.

A3: Please see the large-scale experiments section in our overall reply.

Q4: Also the frameworks are out of fashion. I think the well-known architecture (e.g., stylegan) is more convincing.

A4: Our primary goal is to advance representation learning, which is why we used a ResNet-18 as the discriminator across all experiments for consistent comparison with other representation learning frameworks. Although the choice of generator is secondary, it is crucial that it does not become a limiting factor in our pipeline. We opted for BigGAN's generator with increased feature channels for this reason. We did experiment with StyleGAN's generator but found it less effective for our purposes.

Q5: There are not much visualization results .

A5: We have included various visualizations in our paper, such as the learned embedding (Fig 1(b)), the dynamic updates of the learned embedding in a synthetic scenario (Fig 2), and generated images in the Appendix.

Thank you for your feedback!

Q1: Our description on generation quality and comparison with ICGAN

A1: Our work primarily focuses on representation learning, not on enhancing generation quality. Improved image quality emerges as a secondary benefit from our proposed objectives. These goals are not necessarily in conflict, though their relative importance might vary depending on application. Unlike ICGAN, which uses an external feature learner to improve their generation quality, our framework is designed to learn representations from adversarial training. We refer to ICGAN in our paper to highlight the fundamental differences between their approach and ours.

Q2: Large-scaled experiment on ImageNet-1K.

A2: Please see the large-scale experiments section in our overall reply.

Q3: Implementation details of compared methods.

A3: We provide implementation details for the methods we compared, including MAE, in Appendix A.2. For MAE, we used the average of all patch tokens from the encoder's output as the representation for the linear classifier.

Q4: Clarification on notations $z$ and $z^g$

A4: We use $z$ to denote the encoder's final representation, and $v$ as the input for the generator. This is in contrast to the traditional GAN framework, which only has a single latent space as the generator's input. Our model includes an additional latent space, which is the encoder's output.

Q5: Explanation of the Memory Bank

A5: The memory bank holds the encoder's representations of real images. It is implemented as a first-in-first-out queue, updated continuously by replacing the oldest features with new ones from the current mini-batch.

Q6: Explanation of the proposed adversarial objectives

A6: Our objectives aim to align the distribution between real and generated images (Eqn.2 for image generation) and to facilitate automatic learning of semantic representation. The latter is achieved by imposing structure on the non-collapsed representation, by optimizing covariance (Eqn.4), and forming local clusters by grouping features (Eqn.6). The effectiveness of our proposed objective is illustrated in Figure 2, where we show that by optimizing the discriminator with these objectives, the learned embedding aligns semantically similar data and separates dissimilar data.