Nonstationary Sparse Spectral Permanental Process

Q: My only concern is the lack of substantial novel contribution...... I feel that it is an elegant although not-too-difficult extension of GSSPP and I do not feel the contribution is significant enough for publication at NeurIPS. I enjoyed the paper and I feel the contribution is certainly very respectable and well designed and demonstrates the efficacy of the method.

A: The reviewers' main concern is that the method proposed in this paper seems "not-too-difficult" and therefore not very "novel". However, we politely disagree that a sufficiently complex method is the only significant contribution for publication at NeurIPS.

Our method may appear straightforward because we have presented it in a concise manner. From a high-level perspective, our main contributions are: (1) We used a nonstationary sparse spectral kernel to replace the stationary sparse spectral kernel, thereby constructing NSSPP, (2) we then derived the corresponding Laplace approximation inference algorithm, and (3) subsequently, we built a corresponding deep variant (DNSSPP) by stacking spectral feature mappings to further enhance expressiveness.

To the best of our knowledge, there has been no prior work applying (deep) nonstationary kernels in Gaussian Cox processes. This gap is what this work seeks to address. Each of these three steps above involves greater challenges when extending from the stationary case to the nonstationary case, both in theoretical derivation and algorithm implementation. Especially the third step, the deep variant (DNSSPP), has never appeared in previous Gaussian Cox processes work. This idea cleverly utilizes the relationship between spectral feature mapping and deep models, resulting in its design. This contribution has also been praised by the other two reviewers. For example, Reviewer WJmT stated: "The introduction of a deep kernel variant, DNSSPP, which stacks multiple spectral feature mappings to enhance the expressiveness of the model significantly, is another novel idea." Reviewer QimU commented: "In terms of originality and significance, this paper fills a clear gap in the existing literature on the topic, namely probing the usefulness of deep models in the permanental process setting. Whilst conceptually not a particularly complicated extension, there are obviously various nuances to moving to a deep kernel setting which are outlined clearly by the authors throughout the work."

We understand that different people may have different definitions of "novelty", which may be difficult to reconcile. We greatly respect your review and hope that the above response can lead you to reconsider the cleverly designed ideas in our work. We sincerely hope you can increase the rating.

Q: How is the marginal likelihood for DNSSPP computed? (The paper just says "numerically" and I couldn't see details in the appendix).

A: For DNSSPP, the nested structure of trigonometric functions leads to the intensity integral ($\mathbf{M}$ and $\mathbf{m}$) lacking an analytical solution, and we need to resort to numerical integration. Numerical integration has many choices, such as Monte Carlo integration or quadrature methods. In this work, we used Gaussian quadrature.

Q: Do the error bars represent a one/two standard errors or standard deviation over test set likelihoods?

A: It is one standard deviation.

Thank you for the response.

It seems all reviewers agree on technical aspects and only I differ by my subjective judgement on the impact for which it seems I may have been too harsh!

I am have to raised my score.

Q: As mentioned earlier, I think the rationale for using of a very small number of inducing points etc. for some of the baselines needs to be clarified in the text; is it the case that if you increase this number to 50+ that the baselines begin to outperform the models proposed by the authors?

A: Thanks for your suggestion. For the synthetic data, we used the same number of ranks (50 inducing points, eigenvalues, frequencies, etc.) for all baseline methods, except for DNSSPP because it has many layers and different layers can have different layer widths (frequencies). For the real data, we chose different numbers of ranks for the 1-dimensional dataset (10 for Coal) and the 2-dimensional datasets (50 for Redwoods and Taxi) for all baseline methods, except for DNSSPP. The reason is that the Coal dataset is very small (only 191 events) and has a simple data pattern. A higher number of ranks would not further improve performance but would significantly increase the algorithm's running time. Therefore, we set the number of ranks to 10 for Coal, but 50 for Redwoods and Taxi because the latter two datasets are larger and more complex, requiring a higher number of ranks to improve performance.

To demonstrate this, we further increased the number of ranks for all baseline models to 50 on the Coal dataset and compared the result to rank=10. The result is shown in Table 2 of the rebuttal PDF. As can be seen from the results, when we increased the number of ranks from 10 to 50, the performance of all baseline models did not improve significantly, but the algorithm’s running time increased substantially.

In summary, for the same dataset, the number of ranks for all baseline methods is kept consistent, except for DNSSPP because it is not a single-layer architecture. The choice of the number of ranks for the baseline methods was made considering the trade-off between performance and running time. Further increasing the number of ranks would not significantly improve model performance but would result in a substantial increase in computation time.

Q: You mention other variational methods are feasible, was there a certain specific reason that you decided upon a Laplace approximation-based approach?

A: Variational inference methods, to the best of our knowledge, require certain standard types of kernels, such as the squared exponential kernel, to ensure that the intensity integral in the likelihood has an analytical solution [18]. The advantage of the Laplace approximation is that it can analytically compute the intensity integral without restricting the types of kernels. Therefore, for NSSPP, we derived a fully analytical Laplace approximation inference algorithm.

When we further extended NSSPP to the deep variant DNSSPP, the introduction of the deep architecture meant that neither variational methods nor the Laplace approximation had analytical solutions. For convenience, we continued to use the Laplace approximation method employed in NSSPP.

Q: I'm aware that there has been some work by on understanding and exploring overfitting in the context of deep kernel learning (Promises and Pitfalls of DKL, Ober et al, 2021), it would be useful to touch on some work from this area during your discussion on overfitting just to clarify to the readers that this phenomenon is not arising solely due to the size of the datasets you are modelling.

A: Thanks for your suggestion. We agree with your advice. In fact, in the ablation study, we also observed some overfitting phenomena in deep kernel learning. We will provide more discussion on this issue, touching on related works from this area, in the camera ready.

Q: This is just a reoccurring minor typo but I'm fairly certain it's the Porto taxi dataset (as in the city of Porto, Portugal), not Proto, so just edit all mentions of this.

A: Thanks. We will fix all these typos in the camera ready.

Q: Performance comparison with a baseline that employs stacked mappings of stationary kernels.

A: Thanks for the suggestion. A baseline that employs stacked mappings of stationary kernels is an important baseline. To show the source of the performance improvement, we have re-compared with a deep stacked stationary kernel model.

The results are shown in Table 1 of the rebuttal PDF. The corresponding deep stacked stationary kernel model is referred to as DSSPP, which has the same deep architecture as DNSSPP. Therefore, DSSPP also has three implementations: DSSPP-[100,50], DSSPP-[50,30], and DSSPP-[30,50,30].

As can be seen from the results, the performance of the stationary kernel model significantly improves after introducing the deep architecture. On the stationary synthetic dataset, DSSPP performs comparably to DNSSPP. On the nonstationary synthetic dataset, DSSPP does not outperform DNSSPP due to the severe nonstationarity in the data. However, on any dataset, DSSPP outperforms other shallow stationary kernel models due to the introduction of the deep architecture. Therefore, this demonstrates that the source of performance improvement comes from the stronger expressiveness of both the deep architecture and the nonstationary kernel.