Decodable and Sample Invariant Continuous Object Encoder

We are grateful for the careful attention and time you have dedicated to reviewing our manuscript. Your constructive feedback is invaluable, and we have made concerted efforts to address each of your concerns, incorporating additional experiments and clarifications into the revised manuscript.

Weakness 1: It is necessary to provide a clear definition of "implicit representation" and "explicit representation" in the manuscript.

Thank you for highlighting the need for clearer definitions of "implicit representation" and "explicit representation" in our manuscript. We acknowledge the potential confusion among readers regarding these terms. In response to your suggestion, we have revised the definition of explicit representation in the Introduction's second paragraph: "Explicit representation refers to frameworks that generate outputs with fixed dimensions, such as fixed-length vectors." To enhance clarity, we have also included a brief comparison between explicit and implicit representations, elucidating their distinct roles and applications in our research context. This modification should provide readers with a more comprehensive understanding and eliminate ambiguities related to these key terms.

Weakness 2: The proposed method (HDFE) relies on the assumption that the input data follows Lipschitz continuity. It would be beneficial for the manuscript to include an analysis of the types of input data that adhere to Lipschitz continuity.

Thank you for highlighting this aspect. In response, we have included Appendix D: Suitable Input Types for HDFE, detailing various input types that conform to Lipschitz continuity. This addition aims to clarify the characteristics of appropriate inputs, offering both theoretical understanding and practical guidance for researchers applying our method.

Weakness 3: As a module that doesn't require any training, it is important to provide detailed guidance on selecting hyperparameters.

We concur that providing guidance for hyperparameter selection is crucial for HDFE's usability. We contend, however, that hyperparameter tuning should not pose a significant barrier to users. HDFE's simplicity is characterized by minimal hyperparameter requirements. Specifically, for dimensionality, higher values invariably yield superior representations, obviating the need for intricate tuning. The primary hyperparameter, the receptive field, is governed by the $\alpha$ parameter in Equation 6. Typically, $\alpha$ ranges from 10 to 30 for normalized inputs between $(0,1)$, simplifying the selection process. Moreover, we plan to release HDFE's source code along with comprehensive documentation, including hyperparameter guidance. We also envision incorporating automated mechanisms for receptive field optimization based on user-specific datasets, further reducing the tuning burden.

Weakness 4: The selection of weights ($w_i$ in Equation 1), hyperparameters ($\alpha$ and $\beta$), and the mapping functions and are highly dependent on the dataset. This means that if the task or input data changes, all these variables need to be carefully decided and tested.

We appreciate the opportunity to clarify potential misunderstandings:

Task Change vs. Input Functions: If the task changes (e.g., from classification to regression) but the input functions remain constant, the hyperparameters need not be altered. Only the downstream neural network requires modification.

Weights $w_i$: These are not hyperparameters but are determined through iterative refinement during input function encoding, thereby eliminating the need for user tuning.

Hyperparameter $\beta$: This remains constant as long as the function's range is normalized between 0 and 1. For functions normalized within $(0,1)$, we set $\beta$ to 2.5, ensuring the gradient of $\langle E_Y(y_1), E_Y(y_2)\rangle$ stays positive, as detailed in the manuscript.

Changes in Input Functions: The reselection of the $\alpha$ value (in Equation 4) is only necessary if there's a significant change in the input functions (e.g., shifting from 1D to 2D functions) or in the Lipschitz constant.

Please note that the hyperparameters $\alpha$, $\beta$, and the mappings $E_X$, $E_Y$ are fixed across both training and testing phases to ensure comparability of function encodings.

Weakness 5: There is a small concern regarding the experimental results on the PCPNet dataset. It is only comparable to the current state-of-the-art (SOTA) method, outperforming it in four out of twelve metrics, albeit with a slight drop in average performance.

We acknowledge the reviewer's concern regarding the experimental results with the PCPNet dataset. To address this, we have conducted a thorough analysis of the HDFE's performance compared to both the baseline and current state-of-the-art methods. For a comprehensive understanding, we invite the reviewer to refer to our detailed analysis presented in the global response section.

Question 1: Why is the representation learned by PointNet (Qi et al., 2017a) not easily decodable?

Thank you for your insightful question. We claim that PointNet's representations are not easily decodable based on our experiments where PointNet, as a function encoder, struggled to minimize reconstruction loss and failed to yield reasonable reconstructions. Detailed findings are presented in Appendix B: PointNet as Function Encoder.

While previous works like [1] may encode implicit functions (e.g., point cloud inputs) effectively, our analysis suggests they lack the capability for decodable representation of explicit functions, as evidenced in Appendix B. Additionally, these methods may not produce encoding invariant to sample distribution changes. For instance, if trained on uniformly sampled point clouds and tested on varied sampling distributions, their performance could deteriorate, as discussed in Appendix I.4.

We are thankful for this question, which has enhanced the completeness of our manuscript, and we are open to any further inquiries regarding our experiments or arguments.

[1] Learning Representations and Generative Models for 3D Point Clouds. PMLR

Question 2: When experimenting with batches, should the model visit all data samples to decide hyperparameters?

The hyperparameters $\alpha$, $\beta$, and the mappings $E_X$, $E_Y$ are predetermined and remain constant during both training and testing. The selection process, particularly for $\alpha$, involves identifying hyperparameters that minimize the function reconstruction loss. Once selected, these hyperparameters are fixed, eliminating the need for adjustments across batches. It's important to note that for meaningful comparisons of function encodings, they must be generated using consistent hyperparameters.

Question 3: why a high-dimensional input does not affect the size of the fixed-length vector representation. Could the author provide further explanation, possibly an extension of the paragraph on 'Scale to high-dimensional input'?

Our conclusion regarding the negligible impact of high-dimensional input on the size of the fixed-length vector representation is based on empirical findings. Detailed in Appendix I.6, our experiments assess the influence of input dimension and function complexity on reconstruction error. These studies reveal that while function complexity significantly affects the reconstruction error, the input dimension appears to have a minimal impact. This leads us to conjecture that HDFE can effectively encode high-dimensional data without incurring information loss due to increased data dimension.

We appreciate the reviewer's interest in this topic, aligning with our discussion in the "Scale to high-dimensional input" section. Our intention is to highlight a promising research avenue and provide a foundational understanding of its viability.

Question 4: in Section 2.3, the manuscript only shows the picking of and when the function output y is scalar. Are there more cases that can be considered?

Thank you for prompting further consideration of HDFE's application scope. To address functions with vector outputs, HDFE can be easily adapted by reshaping the $\Psi$ matrix in Equation 6. For cases where function inputs or outputs cannot be directly vectorized using Equation 6, we propose pretraining the input/output mappings $E_X$, $E_Y$ to align with the properties outlined in Section 2.1. This approach would enable HDFE to handle non-vector functions by employing customized mappings, as the principles for designing these mappings are thoroughly discussed in Section 2.1.

Question 5: Although HDFE is a deterministic function, is there any empirical result available to estimate the information loss from the raw input to the fixed-length vector representation?

To address the query about HDFE's information loss, we've included Appendix I.5: Information Loss of HDFE in our manuscript. Our empirical analysis reveals a linear relationship between information loss (quantified by reconstruction error) and function complexity, where complexity is measured by the integral of the absolute gradient. Interestingly, we observe that the reconstruction $R^2$ value appears uncorrelated with complexity in cases of sufficiently high input dimension. We trust this analysis will be insightful for the reviewer. We are grateful for this question, which has contributed to the manuscript's comprehensiveness.

We are sincerely grateful for your positive feedback on our research topic, contributions, and the organization of the paper. Your insightful questions have not only deepened our understanding but also guided us towards future research directions. We are pleased to address your concerns as follows:

Weakness 1: In Table 1, some metrics did not show improvement when comparing to the prior work HSurf-Net.

We acknowledge the reviewer's observation regarding Table 1's metrics in comparison to HSurf-Net. We understand the importance of this concern and have provided a detailed analysis in the global response section. This analysis explores the potential reasons behind these findings and their broader implications.

Weakness 2: The encoding capacity of the proposed method might be limited.

We recognize the concern about HDFE's limited encoding capacity for extensive input spaces, a point we also note in our conclusions. To address this, we propose partitioning the input space into disjoint subspaces and concatenating their respective encodings. While our current focus has been on low-dimensional functions, where capacity is less of an issue, expanding HDFE’s capacity for broader applications presents a significant future research direction.

Question 1: Can the proposed method be applied for any-resolution image encoder for complex natural images, e.g. ImageNet? What would the main challanges be for applying the method to the image domain?

The application of HDFE to complex natural image tasks, such as those involving ImageNet, introduces specific challenges, particularly its lack of translation invariance. This means the HDFE encoding for an image and its shifted version would differ significantly, making it less suitable for tasks requiring translation invariance like classification and detection. Nonetheless, HDFE may be well-suited for image-related tasks that do not necessitate translation invariance, such as image-to-image translation or image super-resolution. This area presents exciting avenues for future exploration, and we are grateful for your question, which has stimulated further consideration of HDFE's potential in image processing.

We are grateful for the time and attention you have devoted to carefully reading our manuscript. We are eager to address your constructive concerns and have incorporated additional experiments and clarifications in the revised manuscript. We intent to make our replies elegant and concise for easy understanding, and we encourage Reviewer oyrE to refer to the corresponding sections in the updated manuscript for detailed experimental insights.

Weakness 1: One concern is the computational cost of HDFE induced by the iterative refinement process.

We acknowledge that the need for iterative refinement, which entails additional computational effort, presents a significant challenge for large-scale tasks. However, in fact, the refinement process can be efficiently executed in a single iteration, yielding a highly accurate estimation of the sample invariant function encoding.

Details: The motivation of the iterative refinement is to balance the weights between dense and sparse samples. Such motivation can also be achieved by another cheaper one-shot refinement. After obtaining the initial function encoding by averaging the sample encoding, we compute the similarity between this initial encoding and all the sample encodings. The similarity can serve as a rough estimation of the sample density at a particular point. Therefore, if we were to balance the weights between dense and sparse samples, we can simply recompute the weights by the inverse estimated density. It turns out that this one-shot refinement can achieve similarity of > 0.95 when sample distribution varies. The computational cost is only 8.0 ms on an NVIDIA Titan-X GPU.

We kindly refer Reviewer oyrE to Appendix I.3: Practical Consideration of Iterative Refinement in the updated manuscript for the detailed procedure of one-shot refinement and experiments demonstrating its effectiveness. We highly appreciate your very constructive question that induces us to complete the manuscript.

---

Weakness 2: The authors claim that sample invariance is a crucial property for the machine learning tasks throughout the paper, but it lacks the supporting experiment revealing HDFE’s efficacy in those scenarios.

We heartly agree that experiments supporting HDFE's efficacy on sample invariance are important. We will show HDFE is significantly more robust to distribution variation than PointNet through two experiments: 1. point cloud normal estimation; 2. a synthetic function parameters regression problem.

1. Point Cloud Normal Estimation

   In our updated experiments, HDFE significantly outperforms the PointNet counterpart when sample variance exists. Specifically, HDFE achieves 4.79/7.37/0.52/0.46 lower errors than its PointNet counterpart in the setting of Density-Gradient (i.e. the density of the input point cloud varies) in the two baseline comparisons and two benchmark datasets. Through the comparison with the baseline, it is convincing that the significant improvement is attribute to HDFE's sample invariance property and the augmentation of the HDFE encoding.

   We highly appreciate your very constructive feedback and we have added a paragraph "HDFE promotes stronger robustness to point density variance." in the Section 3.2: Unoriented Surface Normal Estimation to make the analysis and discussion explicit to readers.

2. Synthetic Function Parameters Regression.

   We generate random functions parametrized by four coefficients and the task is to predict the four coefficients using the function samples. We artificially control the sample distribution to be varied or not varied across the training and testing phase. It turns out that PointNet's prediction $R^2$ decreases from 0.978 to 0.513 when switching from no variation to variation, while HDFE maintains its $R^2$ at > 0.9975 in both settings. We kindly refer the Reviewer oyrE to Appendix I.4: Effectiveness of Sample Invariance in the updated manuscript for the experiment details and analysis.

These experiments demonstrate HDFE's superior performance in scenarios with sample variance, thereby validating our claims about its efficacy in machine learning tasks. We highly appreciate your very constructive question that induces us to complete the manuscript.

---

Weakness 3: In the experiment section, it lacks the analysis why HDFE is more beneficial than the counterparts (e.g., PointNet) in terms of the performance. It would improve the understanding of HDFE if analysis on which component leads to the performance gap even when the noise is absent is provided.

We agree on your concern and we have investigated into the cause. We kindly refer the reviewer to the posted global response, where we explain our analysis in details.

Question 1: How long does the HDFE take compared to the baselines (eg, pointNet in Experiment 3.2)? Is the iterative refinement process applicable to a large number of samples? How long does it take for convergence in the process?

While the details are in the response to Weakness 1, we would like to mention that HDFE is much faster than PointNet. For an input of shape $5000\times 3$, assuming the complexity of multiplying an $m\times n$ matrix with an $n\times p$ matrix is $mnp$, then HDFE only has has complexity $5000\times3\times1024\approx1.5e7$. The computational cost of the one-shot refinement is negligible here. But PointNet has complexity $5000\times3\times64+5000\times64\times128+5000\times128\times1024\approx 70e7$.

Question 2: In the formulation on decoding, (i.e., equation between eq. (2) and eq.(3)), can you please clarify on why orthogonality property ensures that will produce a vector orthogonal to when the distance between two samples is large? Also what does the noise mean? Does it mean that it’s near zero so that it is a negligible component?

We recognize that the noise argument may not be straight-forward from the orthogonal property. For better rigor, clarification and readability, we replace the orthogonal property as similarity-preserving property, without affecting the correctness of HDFE:

​ similarity preserving: ⟨x ⊗ y, x ⊗ z⟩ = ⟨y, z⟩.

When $d_X(x_0, x_i)$ is large, $\langle E_X(x_0)\otimes E_Y(f(x_0)), E_X(x_i)\otimes E_Y(f(x_i)) \rangle\approx 0$. Since the unbinding operation is similarity preserving, we have $\langle E_Y(f(x_0)), E_X(x_i)\otimes E_Y(f(x_i)) \oslash E_X(x_0)\rangle\approx 0$. Therefore, $\mathbf{F}\oslash E_X(x_0)$ can be decomposed into $E_Y(f(x_0))$ and the sum of vectors that are orthogonal to it. Consequently, when we search for the $y$ to maximize the $\langle \mathbf{F}\oslash E_X(x_0), E_Y(y) \rangle$, those orthogonal vectors will not bias the optimization.

Note that the Fractional Power Encoding does satisfy the similarity preserving property, so the entire HDFE formulation is still correct. We have rewritten the decoding section based on the new definition and put the argument in Appendix E: Derivation of Decoding. We highly appreciate your constructive question for helping us improve the manuscript.

Question 3: In the formulation on decoding, it seems it misses $w_i$.

Thanks for your catch! We have corrected the typo.

Question 4: For an unbinding operation, element-wise division of complex vectors is used. But I don't think this operation is commutative, which violates the assumption. Can you please clarify on this?

This is a very sharp catch! We admit it is an overlook by us when we draft the manuscript. You are correct that the element-wise division of complex vectors is not commutative. Actually the commutative property is not required when deducing the decoding formulation, so the HDFE formulation is still correct. We have revised the definition of our binding and unbinding operations to address this (and Question 2) and ensure the logical consistency of our methodology.

Question 5: In experiment 3.1, how does the function prediction error is measured?

The function prediction error is computed by the mean absolute error between the predicted function and the ground-truth function, as mentioned in the Section 3.1 -> Dataset.

Question 5: And the paper states that “when decoding is not required, our approach achieves lower error than FNO”, but how can we compare to FNO, which directly predicts the solution?

We first encode the ground-truth solution into a vector and then decode the vector to reconstruct the solution. From this, we obtain a reconstruction loss $error_1$, which is visualized by the shallow red bars (Reconstruction Error) in Figure 3. Then we decode the predicted function vector to reconstruct the solution. From this, we obtain the error between the predicted solution and the ground-truth solution, denoted by $error_2$, which is visualized by the stack of Reconstruction Error and Function Prediction Error.

Our argument is, $error_2$ consists of two components. The first component arises when reconstructing the function from the encoding, which is measured by $error_1$. The second component arises when predicting the function encoding, which cannot be measured directly. But since $error_2$ only consists of two components, we can estimate the error arised from the second component by $error_2-error_1$, which is visualized as the Function Prediction Error in Figure 3.

Question 6: While the authors claim that HDFE is robust to point perturbation, the experiments on [PCPNet - PointNet + HDFE] in Table 1 shows that the performance boost becomes much less as the noise level increases. Can you please elaborate on this?

We believe it is caused by the trade-off between accuracy and robustness to noises. In Table 1, we present the result obtained by $\alpha=20$, which minimizes the average error. However, this parameter, compared with other selections of $\alpha$, does not obtain lowest error at all noise levels. Kindly refer to Appendix K, where we find a larger receptive field makes HDFE more robust to noise, while a smaller receptive field makes HDFE more accurate in clean inputs. We can increase HDFE's robustness to noise by choosing a large receptive field, but we will lose some accuracy when the input is clean.

Question 7: [Possible Typo]

Thanks for your catch! We have corrected the typo.