Inverse M-Kernels for Linear Universal Approximators of Non-Negative Functions

We would like to thank the reviewer for the highly positive comments, by which we are strongly encouraged! We feel sorry for the grammatical flaws in this paper. Despite this, we appreciate your thorough understanding of our work's content and value. We promise to improve the grammatical quality of this paper using a professional English proofreading service. Below we provide a detailed response to each of the comments.

There isn't a lot of kernels that are known to be M-kernels (it looks like we only know for 1-dimensional kernels).

As the reviewer pointed out, this paper succeeded in only showing two examples of strict inverse M-kernels, which could be a limitation of this paper. However, we are optimistic about the future. This paper successfully, for the first time, clarified a novel condition that allows a linear model to achieve both non-negativity and good representation power, that is, the condition of strict inverse M-kernels. By presenting the condition to a high-level NeurIPS audience, we believe that the open question of how to construct strict inverse M-kernels in a more general manner would be addressed eventually.

There are many small grammar/writing errors

We apologize for the inconvenience and appreciate the kind suggestions about writing, which we will reflect in our paper. We will ensure that the writing quality of the paper is improved through professional proofreading.

What is $\gamma$ on L209?

$\gamma$ on L209 is the hyperparameter of intersection kernel $k_{\text{int}}(x,x')$, which should be determined based on data. We will ensure that it is clearly stated in the paper. If $\gamma$ on L339 confused the reviewer, we apologize for it: $\gamma$ on L339 is the typo of $s(N+1)$.

On L213, is this tridiagonal formula well-known?

We derived the tridiagonal formula by using some known properties of symmetric Toeplitz matrices (e.g., see Trench 2001) as a reference. It is clear that the derived tridiagonal formula is a straightforward generalization of the inverse of Toeplitz matrices, but we cannot find any references that explicitly mention the tridiagonal formulae of one-dimensional exponential and intersection kernels. We will add the above discussion to the paper.

Trench, W. F. (2001). Properties of some generalizations of Kac-Murdock-Szego matrices. Contemporary Mathematics, 281, 233-246.

We would like to thank the reviewer for the highly positive and constructive comments. It is incredibly encouraging to know that a reviewer with practical experience in the estimation of non-negative functions finds value in our work! Below, we provide a detailed response to each of the comments.

The role of the function needs to be discussed in more depth...

Thank the reviewer for the pertinent comments. We agree with the reviewer about the role of $s(N)$ being ambiguous in the paper. According to the reviewer's suggestion, we will introduce $s$ as a non-negative factor at the definition of inverse M-kernels, and discuss a possible scaling of $s$ with data size $N$: The ideal case is $s = 0$ (strict inverse M-kernels), where inverse M-kernel model $f_{\text{IMK}}$ has greatest representation power, but we can only find such kernels on one-dimensional input scenario; By exploiting the strict path product condition (for details, see Future Work and Limitations), we can find inverse M-kernels with $s \sim \mathcal{O}(N)$ regardless of the input dimensionality, where $f_{\text{IMK}}$ has limited representation power for a large $N$; An essential next step is to investigate a more scalable factor (e.g., $s \sim \mathcal{O}(N^{1/2})$ ) according to observed data, which could relax the limited representation power of $f_{\text{IMK}}$ in multi-dimensional input settings.

please take the time to run this paper through a spell-checker!

Sorry for the inconvenience. We promise to improve the grammar quality of the paper by using English proofreading services. We appreciate the reviewer's efforts to read our paper despite the poor English.

We would like to thank the reviewer for carefully reading our paper and giving valuable comments. While the reviewer has positive evaluations of the core part of this paper (Soundness: good, Presentation: good, Contribution: good), we understand that the critical concern of the reviewer lies in the ambiguity of the paper's purpose and significance. We can clearly state the purpose and significance of our work, and therefore, we believe that we can fully dispel the concern raised by the reviewer. Below, we provide a detailed response to each of the comments.

The method is limited to fitting univariate non-negative functions, which is disappointing. Scalar functions can already be handled with existing techniques. This limits the papers significance to low in its current form... I'm afraid the paper has not been able to find a purpose for their good idea, or show what is the open problem that only it can solve.

This paper tackles the open problem of whether a linear model can simultaneously satisfy non-negativity and universal approximation. Through the novel concept of inverse M-kernels, we have, for the first time, demonstrated that the answer to the problem is 'Yes' at least in the case of one-dimensional input scenario. While non-linear models that satisfy both non-negativity and universal approximation have been proposed so far, no such linear model has been developed in existing techniques. The linearity of a model offers significant computational advantages in both learning and prediction of latent function, and as Reviewer wAiw kindly commented, its practical value is significant—even in the case of one-dimensional inputs. A comparison of CPU times between QNM (non-linear model) and our linear model in Tables 1 and 2 clearly demonstrates that our model achieved computational speeds tens to hundreds of times faster than QNM.

Based on the reviewer's comments, we found that our explanation of the paper's purpose and significance was inadequate. We will add the above discussion in the introduction.

In Fig1 the QNM is slightly better, in Fig2 the QNM is arguably much better.

This is a misunderstanding by the reviewer, likely due to his/her overlooking that the evaluation metric ($d_{\text{KL}}$) is the Kullback-Leibler distance (the lower, the better): In Figure 2, QNM is slightly inferior to our model regarding $d_{\text{KL}}$. We apologize for the unclear explanation. We will add the statement "the lower, the better" to the legend and the main text.

In FigB2 the new method is much better than baseline, but the fit is still quite poor.

Actually, the estimation result achieved by our model in Figure B2 is by no means poor. Event sequence data generated from a point process is inherently noisy, and it is challenging to estimate the intensity function from such data accurately. For example, the center of Figure 2 in (John & Hensman, 2018) shows the estimation result using a variational Bayesian method for the same underlying intensity function, where the result is similar to that of our model in Figure B2. To clarify it, we implemented and evaluated one of the SOTA methods, the structured variational approach with sigmoidal Gaussian Cox processes (Aglietti et al., 2019). Please see the following response about intensity estimation for more details.

Furthermore, all experiments are very simple, and baselines quite simple as well. For instance, there are no GP-based or Cox-process based competing methods.

Regression and Density Estimation:

In this paper, we focus on point estimation rather than interval estimation, where GP-based models usually reduce to kernel methods that include our model. An exception is the approach by (Pensoneault et al., 2020), which imposes the non-negativity constraints on finite virtual points by utilizing the confidence intervals of posterior distributions. It is a very different approach from the kernel method that cannot access posterior distributions. However, as mentioned in Section 2.2, the approach is out of the scope of our paper because it does not guarantee non-negativity at locations other than the points. We believe that NCM, QNM, and SNF are appropriate baselines in the literature.

Intensity Estimation:

As the reviewer pointed out, Gaussian Cox processes (GCPs) are the gold standard for intensity estimation. Actually, the intensity estimator defined in Section 3.2 is the MAP solution of the permanental process, a variant of GCP assuming that the square root of the intensity function is generated from a Gaussian process. Therefore, the reviewer's claim that there are no Cox process-based methods in the experiments is inaccurate. However, from the reviewer's comments, we found that other GCP-based methods should be included in the baselines to clarify the contribution of our paper within the context of intensity estimation: In the literature on GCPs, the advantage of permanental processes over other GCPs has been considered as the efficient estimation algorithm, and our work improves the predictive performance of the fast-to-compute permanental processes. We added the estimation result of the structured variational Bayesian approach with sigmoidal Gaussian Cox processes (Aglietti et al., 2019), denoted by STVB. We implemented STVB with the TensorFlow code provided by (Aglietti et al., 2019), where the number of inducing points were set as regularly aligned 100 points within the observation domain. Table C1 and Figure C1 (please see the pdf file in global response) show that our model achieves comparable predictive accuracy while being hundreds of times faster than STVB, which highlights our work's contribution.

John and Hensman (2018). Large-scale Cox process inference using variational Fourier features. International Conference on Machine Learning.

Aglietti et al. (2019). Structured variational inference in continuous cox process models. Advances in Neural Information Processing Systems 32.