Hierarchical-Latent Generative Models are Robust View Generators for Contrastive Representation Learning

3. About visualization of the hierarchical structure of latent spaces. We added a section in the Supplementary material (appendix A), where a schematic structure of a generic HL generative model is given. We hope that this can help in better understanding the HL models architecture. We also point the Reviewer to Brock et al (2018) and Karras et al (2020), where complete pictures of the architectures used can be found.

4. About the adaptability of HL models on other modalities. We thank the reviewer for the precious suggestion, which we did not consider in our paper and poses an interesting basis for possible future research directions. In our work, we considered image data since HL generative models are well established in such domain. In fact, images present a global-to-local structure that well fits the HL framework. Interestingly, similar structures can be found also for audio data. For example, Hono et al. (2020) proposed a HL model for speech synthesis. On the other hand, we did not find similar architectures for text data, where generative models do not usually consider a hierarchical structure with multiple input latent vectors.

References

Brock et al. (2018): Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2018

Karras et al. (2020): Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Ana- lyzing and improving the image quality of stylegan. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8107–8116, 2020.

Hono et al. (2020): Hono, Y., Tsuboi, K., Sawada, K., Hashimoto, K., Oura, K., Nankaku, Y., Tokuda, K. (2020). Hierarchical multi-grained generative model for expressive speech synthesis. arXiv preprint arXiv:2009.08474.

We would like to point out that the main scope of the paper is to show the benefits that a smart use of Hierarchical Latents can provide in the context of view generation for representation learning. Under this perspective, the baselines to consider are methods that use a generative model for view generation. To the best of our knowledge, the only two methods that actually perform such task are Li et al. (2022b) and Jahanian et al. (2021), which are extensively compared in the experimental sections. It is also valuable to note that we already extended the benchmark proposed by these papers, which solely utilized SimCLR as the self-supervised framework. For this purpose, all the baselines were re-implemented and further evaluated using the SimSiam framework, as reported in the right part of Table 2.

As for the comparison with supervised methods using real data, we included new results as an additional row in Table 4 of the Appendix. In detail, results are computed for the same ResNet50 backbone, trained on Imagenet-1k in a supervised manner and evaluated on all the transfer classification datasets presented in the paper. As expected, supervised training performs better than self-supervised. This is true for both the source dataset (Imagenet-$1$K) and most of the transfer target datasets. We thank the Reviewer for the suggestion, which broadens the experimental evaluation.

References

Li et al. (2022 b): Yinqi Li, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Optimal positive generation via latent transformation for contrastive learning. Advances in Neural Information Processing Systems, 35:18327–18342, 2022b.

Jahanian et al. (2021): Ali Jahanian, Xavier Puig, Yonglong Tian, and Phillip Isola. Generative models as a data source for multiview representation learning. In International Conference on Learning Representations, 2021

As per the relation between latent perturbations and the generation of views for representation learning, the used procedure (Eq. 3 of our paper) points at maximizing the distance between the two latent vectors $d(\mathbf{z}, T_\mathbf{z}(\mathbf{z}))$, while maximizing the mutual information between the generated views $g(\mathbf{z}), g(T_\mathbf{z}(\mathbf{z}))$. As demonstrated by Oord et al. (2018), maximizing a lower bound on mutual information is equivalent to minimizing InfoNCE Loss, which is the term optimized in Equation 3. This procedure has been previously introduced and theoretically motivated in Li et al. (2022b).

In our work, we extended such findings and procedures to generative models presenting multiple layers of latent variables, which we name Hierarchical Latent. In detail, we show a relation between the latent variables' hierarchical level and the amount of perturbation that can be applied to them, while still maintaining semantic consistency.

In the updated version of the paper, we included some insights in the paragraph above Equation 3, to better explain why InfoNCE loss can be used to find optimal latent transformations in the context of representation learning.

References

Li et al. (2022 b): Yinqi Li, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Optimal positive generation via latent transformation for contrastive learning. Advances in Neural Information Processing Systems, 35:18327–18342, 2022b.

Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018

1. Notation in Proposition 2.2: The purpose of the $\rightarrow$ operator in the Proposition was to denote that the two generated images $g(\mathbf{z}_i, \pmb{\theta})$ and $g(\mathbf{z}', \pmb{\theta})$ are mapped into the same semantic label $\mathbf{y}$. We acknowledge that it is ambiguous, and thus we changed it in the newly uploaded version of the paper:

$

\delta_i = \max_{\mathbf{z'}} \text{dist} (\mathbf{z}_i, \mathbf{z'}) \quad \text{ such that } \quad F_{\mathcal{T}}(g(\mathbf{z}_i, \pmb{\theta})) = F_{\mathcal{T}}(g(\mathbf{z'}, \pmb{\theta})) = \mathbf{y}
$

where $\mathbf{z'} \in \mathbb{R}^n$ is a generic vector in the generator's latent space and $F_{\mathcal{T}}$ is an oracle classification function for task $\mathcal{T}$, which assigns the true semantic label $\mathbf{y}$ to any image.

2. Paragraph above Assumption 3.1: The reviewer is right. In fact, in the paragraph and in Definition 3.1, we are not assuming any particular internal structure for the generator network $g$, which could possibly be a simple MLP for simpler tasks. What we want to stress here is that latent variables $\{ \mathbf{z}^0, \mathbf{z}^1, \dots, \mathbf{z}^{n-1} \}$ operate at progressive layers as inputs of the generator. This makes their contributions different, since later layers model more fine-grained details of the data.

3. Complexity: our method does not introduce any additional complexity, which only depends on the choice of the generator architecture. In our work, we utilize two well-known Generative Adversarial Networks (BigBiGan and StyleGan2), which, as any GAN, present the advantage of a constant inference time. We take advantage of this fact by also proposing continuous sampling, as outlined in the last part of Section 4.1.

1. Table 1 shows that different chunks require a different number of training samples for the Monte Carlo simulations to achieve a similar InfoNCE loss. The intuition is that walkers $T_\mathbf{z}$ responsible for fine grained details (i.e. acting on high level chunks) must be trained a lot more to obtain a level of perturbation consistent with previous walkers (i.e. low level chunks). Note that each training starts with the same hyperparameters (optimizer, learning rate) and is initialized as the identity mapping function (at training step 1 we have $T_\mathbf{z}(\mathbf{z}) = \mathbf{z}$).

2. The term task-agnostic can indicate almost any self-supervised learning method, since in general the downstream task is always unknown (as also in Li et al. (2022b)). However, we show in the experimental results that our method is robust across a range of downstream tasks, thus the term "robust" is used.

References

Li et al. (2022 b): Yinqi Li, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Optimal positive generation via latent transformation for contrastive learning. Advances in Neural Information Processing Systems, 35:18327–18342, 2022b.

As expressed in Section 3.2, the results of Table 1 are obtained utilizing the procedure reported in Eq. 3, which uses InfoNCE as a mutual information estimator. This procedure has been introduced and used by Li et al. (2022b) (the same formula is presented in Equation 6 of their paper). In the same work, a theorem and formal proof are given, supporting the validity of the procedure.

In short, the equation (their Eq. 6) consists of a min-max game between the minimization of InfoNCE Loss and the maximization of $d(\mathbf{z}, T_\mathbf{z}(\mathbf{z}))$. In this context, the minimization of InfoNCE Loss corresponds to a maximization of the Mutual Information between views, making it a valid choice for our purposes. This property of InfoNCE loss is demonstrated in Oord et al. (2018)

We thank the Reviewer for raising this point. In the new version of the paper, we modified the paragraph above Equation 3, in order to stress the relation between mutual information and InfoNCE loss.

References

Li et al. (2022 b): Yinqi Li, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Optimal positive generation via latent transformation for contrastive learning. Advances in Neural Information Processing Systems, 35:18327–18342, 2022b.

Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018

we acknowledge that the meaning of the $\rightarrow$ symbol is ambiguous. It was meant to denote that the two generated images $g(\mathbf{z}_i, \pmb{\theta})$ and $g(\mathbf{z}', \pmb{\theta})$ are mapped to the same semantic label $\mathbf{y}$. For this reason, the symbol has been removed and replaced with a new formulation:

Proposition 2.2: (Semantic Consistency in the Latent Space - (reformulation))

Let $g(\mathbf{z}, \pmb{\theta}) = \mathbf{x}$ be a generative model with parameters $\pmb{\theta}$ that maps from latents $\mathbf{z} \in \mathbb{R}^n$ to images $\mathbf{x} \in \mathbb{R}^d$. Consider a downstream task $\mathcal{T}$ with label $\mathbf{y} \in \mathcal{Y}$, and some distance metric dist defined in the generator's latent space. Then, for any random latent vector $\mathbf{z}_i$, the term $\delta_i$ indicates the maximum distance in the latent space of $g$ to ensure semantic consistency for task $\mathcal{T}$:

\delta_i = \max_{\mathbf{z'}} \text{dist} (\mathbf{z}_i, \mathbf{z'}) \quad \text{such that} \quad F\_{\mathcal{T}}(g(\mathbf{z}_i, \pmb{\theta})) = F\_{\mathcal{T}}(g(\mathbf{z'}, \pmb{\theta})) = \mathbf{y} \

where $\mathbf{z'} \in \mathbb{R}^n$ is a generic vector in the generator's latent space and $F_{\mathcal{T}}$ is an oracle classification function for task $\mathcal{T}$, which assigns the true semantic label $\mathbf{y}$ to any image.
********

Regarding the connection with Proposition 3.1 of Li et al. (2022b), it states:

"Given a well-trained unconditional generative model $g$, $\mathbf{z}$ and $\mathbf{z}'$ are two latent vectors, if the distance $d(\mathbf{z}, \mathbf{z}') \le \delta$, then $g(\mathbf{z})$ and $g(\mathbf{z}')$ will have the similar semantic label $\mathbf{y}$.

In the form of mutual information, we have:

$

|I(g(\mathbf{z}); \mathbf{y}) - I(g(\mathbf{z}' ); \mathbf{y})| \le \epsilon

$

where $\epsilon$ stands for tolerable semantic difference, and $\delta$ is the maximum shifted distance to maintain semantic consistency."

In our proposed reformulation (Proposition 2.2), 1) we omitted the mutual information part since not in the scope of our work: we are mostly interested in defining what the term $\delta$ represents, that is, the semantic consistency in the latent space of a generator $g(\mathbf{z}, \pmb{\theta})$; 2) we formalize the concept of "having the similar semantic label $\mathbf{y}$" by means of the oracle classifier $F_{\mathcal{T}}$.

In conclusion, both propositions state that for each latent vector $\mathbf{z}_i$, it exists a $\delta_i$ ensuring that an image $g(\mathbf{z}', \pmb{\theta})$ shares the same semantic label as $g(\mathbf{z}_i, \pmb{\theta})$, as long as $d(\mathbf{z}_i, \mathbf{z}') \le \delta_i$.

References

Li et al. (2022 b): Yinqi Li, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Optimal positive generation via latent transformation for contrastive learning. Advances in Neural Information Processing Systems, 35:18327–18342, 2022b.