Doubly Hierarchical Geometric Representations for Strand-based Human Hairstyle Generation

Thank you for your constructive feedback and suggestions, and your appreciation of our innovations.

[W1] Technical contributions

Please refer to the global rebuttal for a detailed discussion on our contributions. We will specify our innovations, and especially which components in our method are new with regard to existing strand hair modelling methods in related work to better show our contributions.

For some of your specific concerns:

[W1.1] novelty of DCT for hair curves not adequately highlighted

Thanks for the suggestion and acknowledging that utilizing DCT here is "innovative". We will specify in the contribution and related work that we do frequency decomposition on strand curves and introduce DCT in hair modelling, both for the first time.

[W1.2] k-medoids and comparison with other clustering techniques

First, we clarify that we do not perform clustering, but instead we sample guide strands (see W2 the discussion on Wang et al. (2009) for more details). It just happens that the optimal solution of the guide strand sampling can be achieved from the k-medoid clustering as resulting final medoids set. To discover this previously unknown property of this less famous clustering method in our application with a proof (Theorem 1) is considered our theoretical contribution. We compare our solution with grid sampling, because the latter is widely applied in recent hair modeling methods, especially the direct competitor HAAR.

k-means itself does not suffice as a sampling method. Unlike k-medoids, the cluster centers of k-means is not necessarily from the original data (existing strands). It can cause problems such as the root of the strand is not aligned on the head scalp. To make the k-means centers valid hair strands, we modify the solution with an additional projection step to project each root of a sampled k-means center to the nearest point on the head scalp, so it matches a valid strand. We show the results in Table D in the rebuttal PDF as an extension of Table 1 in the manuscript. Since k-means and k-medoids are similar in methodology, the performance gap is not large, while the k-means sampling may lose some accuracy due to the projection step. However as we showed in theory, the k-medoids is ensured to be the optimal solution.

[W2] Comparison with existing works

Please refer to the global rebuttal for evaluation and comparison with the SOTA method HAAR.

Thank you for introducing the work of Wang et al. (2009) which is relevant. We will add discussion in the related work. However, Wang et al. (2009) is not a generative method and the task setup is different. So we are not able to directly compare with this method. Also, as an early method there is no publicly available code or synthesized results. Wang et al. (2009) is a classic method based on Kwatra et al. (2005) that synthesizes texture variations from a given image example. So Wang et al. (2009) requires an exampler strand hairstyle (or combination of two if they are compatible) as well for the global shape and synthesizes only some local details.

For the hair representation, Wang et al. (2009) use PCA to encode each strand as a classic method while recent neural representations are more advanced. The hierarchy of coarse-to-dense hair strands is modeled by clustering, which is suitable for the spiky hairstyle in their illustration. But in more general cases, the densification of hair strands should concern more than one sparse guide strand. We believe that the relationship between dense strands and sparse guide strands should be modeled by sampling and interpolation rather than clustering. That is, each dense strand is affected by not only a single guide strand as from its own cluster, but also a collection of guide strands in its neighborhood. Our method is conceptually different from (and most likely more advanced than) Wang et al. (2009). Nevertheless, how to adapt the representation from Wang et al. (2009) to modern neural networks is non-trivial and worth exploring.

For these reasons, we believe that HAAR is a more recent and directly related SOTA method for us to compare with. But we will discuss Wang et al. (2009) in the related work section.

[W3] Details of nearest neighbors upsampling for fair and transparent comparison

For both our learning-based upsampling and nearest neighbor upsampling, we first need to sample a collection of root points for the dense strands on the head scalp UV map. In the usual pipeline of VAE generation and reconstruction, this is achieved by sampling from the learned density map. Specifically in the experimental evaluation here, we use oracle roots from the ground truth to ensure fair and direct comparisons.

For nearest neighbor upsampling, for each sampled root of the dense strand, we identify one guide strand whose root is the nearest to the sampled root. Then we make the dense strand at this root is the same as the nearest guide strand, i.e., we copy this guide strand and translate it to the sampled root.

Thanks for your suggestion. We will add the clarification to the appendix.

• Wang et al. (2009) Example-Based Hair Geometry Synthesis.
• Kwatra et al. (2005) Texture optimization for example-based synthesis.

Thank you for your valuable feedback and appreciation of our method and innovations.

[W1] On the effectiveness of DCT frequency decomposition, and separation of low- and high-frequency with varying threshold

We appreciate the constructive suggestion.

The designing choice to learn first low- then high-frequency components adheres to the well known principle of spectral bias [28] for learning and generalization of neural models. This principle has also been adapted by some implicit neural representation models with frequency band controlling in the phase of training / optimization, e.g. BACON [A] and SAPE [B]. Another advantage specific to our per-strand representation is that, as we mentioned in Sec 3.1, we downsample the low-pass filtered curve to the resolution of two times the frequency threshold, which, from the Nyquist sampling theorem and empirically from the quantitative evaluation, are accurate enough to represent the low-pass filtered curve. So the low-pass filtering can help with data compression and the computation efficiency.

For reasons above we expect frequency decomposition helps the representation quality in the neural network model. We show the additional ablation experiments on varying frequency threshold in reconstructing straight and curly hair strands with both low- and high-frequency details in Table C, with hair strands reconstructed by aggregating results from both our low- and high-frequency models. We observe that the frequency threshold in the range from 8 to 12 is optimal, and empirically we use 8 which is more efficient. When the frequency threshold is too low, the low-filtered signal does not capture enough information of the principle growing direction. And when the frequency threshold is too high, high-frequency structure cannot be encoded more efficiently by DCT coefficients, and the increased computation cost hinders optimization. Our representation makes use of both spatial and spectral domain with correct setup of frequency threshold.

We also clarify that separation of low- and high-frequency components does not affect clustering (and in fact we do not perform clustering, but instead we sample guide strands. It just happens that we observe that the optimal solution of the guide strand sampling can be achieved from the k-medoid clustering as resulting final medoids set. ) Using low-frequency strands for sampling is just an implementation choice for the concern of efficiency, because the low-frequency can be resampled to lower resolution of controlling points according to the Nyquist sampling theorem, which is more efficient to process on. And we think the principle growing direction from the low-pass filtered strand is representative enough for sampling the guide strands.

[Q1] Depth-to-space upsampling instead of deconv

In 2D CNNs for image reconstruction and generation, deconvs have well-known drawbacks of checkerboard artifacts [D] which can be resolved by depth-to-space upsampling [E] without losing computation efficiency. So we use depth-to-space upsampling by default.

[Q2] Density map

At the end of the conv branch in the PVCNN decoder, we output a $\mathbb{R}^{W × H}$ density map, same to the spatial resolution of the conv feature map. The density map is trained to optimize towards the GT probability values of a root points falling in each of the $W × H$ grid, for each training example. In training, we use oracle root points from the encoder and optimize the density map and strands simultaneously. During inference, we can sample root points from the probability maps when the oracle root points from the encoder are unknown, before generating strand details. Each sampled root points is assigned with a random 2D UV coordinate within the small square grid it comes from.

In practice, we output two density maps from the conv branch decoder, one for guide and one for dense strands, as mentioned in the appendix (the loss function part). At the implementation level, the sampling process can easily be helped by the torch.multinomial() function.

Although the density map sampling does not guarantee the exact same original root positions, empirically with a reasonable resolution of density map, we find that the resulting root positions correctly resemble the distribution of root points, and the reconstruction of the whole hairstyle is still significant advantageous over grid-based baselines. Our evaluation are all based on set chamfer measurements, thus not requiring strand-to-strand correspondence for evaluation.

• [A] Lindell et al. (2022) Bacon: Band-limited coordinate networks for multiscale scene representation

• [B] Hertz et al. (2021) Sape: Spatially-adaptive progressive encoding for neural optimization

• [C] Shen et al. (2023) Ct2hair: High-fidelity 3d hair modeling using computed tomography

• [D] Odena et al. (2016) Deconvolution and checkerboard artifacts.

• [E] Wojna et al. (2017) The Devil is in the Decoder: Classification, Regression and GANs

Dear Reviewer vBP9,

Thank you very much for your positive feedback, for sharing your valuable insights with your expertise, and for recognizing our contributions of "a novel pipeline that hasn't been explored before".

Regarding the performance gap from user study vs. HAAR, it is hard to quantify the performance gap by averaging the subjective scores between 1-10 and conclude as “marginal” improvement, since many users often provide conservative mid-range scores when feeling uncertain. We have observed a clear advantage in the quality of our results over HAAR, as evidenced by our data and experience, which shows notably fewer instances of failure or unnatural examples. This improvement is attributed to our more sophisticated and flexible hair representation and learning model design. We have highlighted some of HAAR’s common issues in Figure B of our rebuttal PDF. Due to space constraints, we will include a more qualitative comparison in the revised manuscript.

We believe that our work will contribute values to the NeurIPS community. Our research addresses a generative learning problem and engages with several prominent ML topics, including geometric deep learning, learning on sets, implicit neural representations, and graph convolution. We hope that our approach may inspire further exploration in these sub-communities, e.g. regarding hierarchical abstraction methods for learning structured non-Euclidean data representation. Thus, we believe that NeurIPS is an appropriate venue to share our approach and exchange thoughts, which could potentially inspire other emerging applications.

Thank you once again for your dedicated review, which helped in guiding improvements to our work, especially with regard to the suggested ablation experiment.

Best regards,

Submission 11742 Authors

Thank you for your valuable and constructive feedback.

[S1 and W1] Contributions

Please refer to our global rebuttal.

Although the methods of DCT and k-medoids are well established, these methods are never used in any prior work for hair modelling, because how to employ DCT and k-medoids to hair modelling is non-trivial and not straightforward. Our novel hierarchical hair representation proposes strand-level frequency decomposition and hairstyle-level optimal representative subset guide sampling, which enable using these techniques. The choices of DCT and k-medoids are well motivated against more popular variants DFT and k-means. Especially the discovery of k-medoids results as the optimal way of sampling guide strands and prove it theoretically (Theorem 1) is considered our theoretic contribution.

We use VAE as the generative model for our hierarchical strand hairstyle generation, which was also used to learn strand codec [32]. We do not claim any contribution on novel generative models there.

For technical contributions, apart from the more foundational methodology and theory on abstraction of hair strand data hierarchy (Sec 2), we would like to highlight also our several novel innovations with advantages in the corresponding neural model design (Sec 3) associated to the hierarchical hair parameterization. Please refer to our global rebuttal.

[W2] Relationship to existing works

Please refer to our global rebuttal. We will specify our innovations, and especially which components in our method are new with regard to existing strand hair modelling methods in related work to better show our contributions.

[W3] Evaluation against HAAR

Please refer to our global rebuttal.

[W4] Absence of evaluation metrics for strand hair generation quality

Indeed, quantitative evaluation of hair generation is hard, because currently there is no such a measurement to evaluate the generation quality of strand hair. Even the existing SOTA work HAAR did not evaluate the generation quality.

Evaluation of image generation can take domain-specific PSNR, SSIM measures, as well as FID and LPIPS that require a pretrained semantic encoder (eg. VGGNet). Unfortunately, strand hair has neither these domain-specific measures nor a VGGNet-like semantic encoder for strands to use FID and LPIPS measurements.

We will add lack of evaluation metrics as a limitation of the whold field of generative hair modelling in the discussion. Possible future direction could be to train a semantic encoder for strand hairstyles with self-supervised learning to enable FID and LPIPS measures, while it requires a large amount of high-quality data. We expect the most promising direction to achieve this would be to apply CT2Hair [A] in large scales for human hair capture and reconstruction, which requires specific equipments.

Instead, in the VAE generative framework, we quantitatively evaluate the VAE reconstruction as a way to show the quality and advantage of our hierarchical hair representation. The generation quality is shown by qualitative examples and a well-established human assessment study against the SOTA method HAAR in the rebuttal.

[Q1] Theorem 1

Theorem 1 implies that, if you want to sample a number of $k$ guide strands from the original hair with dense strands, then theoretically the optimal way is to perform k-medoids clustering on the dense hair strands and take the resulting set of medoids as guide strands. And this resulting set of guide strands (called the representative guide curve set as in Definition 1) has the smallest possible chamfer distance from the original dense strand set, from any other way to sample the same number of $k$ strands. (Note that we do not perform clustering, but we just use the resulting medoids as sampled guide strands. It just happens that the optimal solution of the guide strand sampling can be achieved from the k-medoid clustering as resulting final medoids set. )

In this way the extracted representative guide curves ensure the best possible retention of hairstyle characteristics for the modelling of hierarchical hair representation as used for training the neural generative model.

• [A] Shen et al. (2023) High-Fidelity 3D Hair Modeling using Computed Tomography.

Thank you for your valuable feedback.

[W1] Presentation quality and clarification

We will improve the proof details for better readability. For Sec 3 and Fig 4, we reviewed them and think the information is technically precise and clear. However, we understand that, since the whole field of generative hair modelling is very new, it may require some effort and time for readers to understand our methodology. So we make the following clarifications.

[W1.1] Sec 3 distinction between implementation/method details

We clarify that Sec 3.1 is the details of the parameterization of hair data for the neural model to process on, based on methods in Sec 2. These parameterization setup details are crucial to understand the following neural model design.

All the information in Sec 3.2 is the necessary method details to understand our method and motivation of neural model design based on the proposed hierarchical hair representation, while all the implementation details of the model are delayed to appendix C.1.

[W1.2] Fig 4

Each of the Fig 4 (b) - (d) are corresponding to a paragraph in Sec 3.2, which can be recognized by the subfigure and paragraph titles. We will add clearer pointers to connect them. In more details:

• Fig 4(a) illustrates the setup of guide / non-guide strands.
• Fig 4(b) The guide strand model. The main architecture illustration follows the PVCNN convention with conv and pointnet branches, plus components of 1D strand encoder and decoder and the VAE reparameterization to adapt to the generative hair modelling task.
• Fig 4(c) The densification model, combining bilinearly interpolated features (above) and the graph features (below) for decoding the dense hair strands. The graph features aggregate information from neighboring guide strands to sampled query locations on the scalp, as inspired by implicit neural representations (see the text in L224-233 for detailed explanation), which allows modelling arbitrary number of dense strands and density, and end-to-end joint training together with guides (b).
• Fig 4(d) Adding high frequency, which is similar to the architecture in (b).

[W1.3] Proof of Theorem 1

We update the proof with more explanations for better readability.

Proof: Assume that from the $k$-medoids algorithm, we obtain the set of medoids $\mathcal{U} = \{ u_1, \dots, u_k \}$ with each $u_i$ is from the set of dense hair strands $\mathcal{H}$. Then, from Eq. (2), $\mathcal{U}$ achieves minimum sum of cluster element-to-medoid distance $\sum_{i=1}^k\sum_{l^L \in \mathcal{S}_i}d(l^L, u_i)$.

Next, from the algorithm implementation, each element $l^L$ in $\mathcal{H}$ is closest (or equally close) to the medoid of its own cluster than that of any other cluster, so $\mathcal{U}$ is the subset of $\mathcal{H}$ with cadinality $k$ that achieves minimum $\sum_{i=1}^k \sum_{l^L: \min_{u_i \in \mathcal{U}} d(l^L, u_i)} d(l^L, u_i) = \sum_{l^L \in \mathcal{H}}\min_{u_i \in \mathcal{U}} d(l^L, u_i)$ which is the minimum sum of each dense strand $l^L$ with its nearest medoid $u$. After taking the average (divided by a constant $|\mathcal{H}|$), $\frac{1}{\mathcal{H}}\sum_{l^L \in \mathcal{H}}\min_{u_i \in \mathcal{U}} d(l^L, u_i)$ is in the form of a unidirectional chamfer distance from $\mathcal{H}$ to $\mathcal{U}$. So $\mathcal{U}$ achieves the minimum unidirectional chamfer distance from $\mathcal{H}$, from all possible subset of $\mathcal{H}$ with cadinality $k$.

Then we show that in the reverse direction, the unidirectional chamfer distance from $\mathcal{U}$ to $\mathcal{H}$, $\frac{1}{|\mathcal{U}|}\sum_{u_i \in \mathcal{U}}\min_{l^L \in \mathcal{H}} d(l^L, u_i)$, is constantly 0. This is easy to infer because $\mathcal{U}$ is a subset of $\mathcal{H}$, and each $u_i$ can find the same element from $\mathcal{H}$ that is closest to itself with distance 0. Aggregating both directions, we conclude that $\mathcal{U}$, from all possible subset of $\mathcal{H}$ with cadinality $k$, achieves the minimum (bidrectional) chamfer distance between $\mathcal{U}$ and $\mathcal{H}$, i.e., $\mathcal{U}$ is the representative subset of $\mathcal{H}$ with cardinality $k$ according to Definition 1.

[W2] Contributions See our global rebuttal. Some notes specific to your comments:

• Instead of "utilizing the DCT", here we claim the strand-level frequency decomposition in our novel hierarchical hair representation, which is for the first time used in hair modelling. Introducing DCT to hair strand modelling, again for the first time, can be a (less significant) contribution but not our main focus. How to apply DCT to hair modelling is non-trivial, without our novel representation design with frequency decomposition.

• Adaptation of geometric deep learning models such as PVCNN is also non-trivial, because hair strands are not point cloud data for direct application. We would refer to S3 in Reviewer vBP9's comments suggesting that this adaptation and modification in our model design are "quite clever and makes sense".

• We clarify that we do not "cluster" strands but we "sample" guide strands. See our reply to Reviewer Sftt [W1.2]

• We mainly focus on the learnable hierarchical generative hair representation and its neural model design. For broader ML topics, our novel method and neural architecture design is highly related to the topics of geometric DL, set-based modelling, graph NNs, implicit neural representations, and hierarchical abstraction of non-Euclidean data. In this way, our work can potentially inspire these communities on method design and extending the application to other novel application. Thanks for the comments and we will add this to the discussion.

• Besides the hair representation (Sec 2), we would like to highlight our several novel innovations with advantages in the corresponding neural model design (Sec 3) associated to the hierarchical hair parameterization. Please refer to our global rebuttal.

Dear Reviewer mnz6,

Thank you very much for your response. Your comments have been valuable in helping to improve the evaluation and readability of our paper. However, we still stand by and wish to defend the contributions we have made.

• [Regarding classic methods (DCT and k-medoids) for extracting the hierarchical hair representation]: Our major contribution here lies at the high-level hierarchical representation design, without which the direct application of aforementioned methods is impossible. For these individual classic methods, besides being introduced to hair modelling for the first time, we contribute by presenting the insights and theoretical justifications on why we opt for these classic methods, that are less utilized by recent researchers, instead of more common alternatives such as DFT and grid-sampling/k-means:

  • We use the DCT instead of the more popular DFT, by showing the motivational insights that DFT demonstrates the oscillation issue for frequency decomposition of strands as open curves.

  • We use k-medoids to sample guide strands, and show it is mathematically the optimal way to sample guide strands, which is more advantageous than the more commonly used grid-sampling / k-means in theory. In the meantime, using k-medoids for sampling and the theoretical optimality was never discovered before.

• [For existing methods (PVCNN) that inspire our neural model architecture design]:

  • Geometric deep learning models are first introduced to strand hair modelling. Also, adapting (rather than directly using) existing geometric deep learning models (such as PVCNN) to hair is not straightforward, because they were originally designed for point cloud data which is highly different from hair strands. In fact, at the implementation level, we were unable to directly use almost any existing code from PVCNN due to the huge difference, but we built the architecture entirely from scratch with the PyTorch framework.

  • In addition to the components related to existing methods, we would also highlight our several innovations in the learning model to handle flexible non-Euclidean parameterization, resolution-free representation to generate any number of strands with any density, and the end-to-end training of the high-dimensional strand hair data, etc. We hope the significance of these innovations can be acknowledged once a reader thoroughly follows the full pipeline of our methodology.

Overall, we fully understand the concern that using some existing techniques as components in (part of) the methodology can be considered less novel contributions. But we would like to present a slightly diverse philosophy from a well-famed researcher with a good reputation in the field: "A new use of an old method can be novel if nobody ever thought to use it this way"; "the novelty arose from the fact that nobody had put these ideas together before". (Black, 2022).

We would be grateful if you could consider re-evaluating our contributions and innovations, taking into account the full pipeline of our methodology and learning model design, as well as the insights presented in our work. Regardless of your decision, we sincerely appreciate the time and effort you have dedicated to the review. Your comments are invaluable in helping us to improve our work.

Best regards,

Submission 11742 Authors

• [Black, 2022]: Michael J Black (2022). Novelty in Science: A guideline for reviewers.