Constraining Gaussian Processes Regression with Quasi-Likelihood Constraint Relaxation

This paper is very far from the standard I would expect for an ICLR paper, and is much closer to a workshop paper. Having worked on GPs and published at machine learning conferences for many years, I have rarely seen a submission which is so far from ready to be submitted to a top conference. As a result, my review will be brief and summarise key flaws:

• The language used is very informal and not particularly appropriate for a paper. I would recommend making sure you write more formally and to the point. For example, the introduction reads more like an informal conversation over coffee than an introduction trying to convince the reader of the importance of the problem.

• There is no comparison to existing methods. The authors did review existing work, but then fail to compare against it. How do the advantages/disadvantages of this work show in practice? Do we actually get better approximations than if using competitors? When is it better and when is it worse? How does the computational cost relate?

• There is no discussion of nu. For example, in the example of "fixed values", why would we ever take nu greater than zero? I.e. why not do exact function evaluations, which is straightforward to do with GPs? Is it because this is only feasible for \Phi being linear? This needs to be discussed much more at length. Also, how should we choose \nu for applications? Can this be estimated? What is the impact of estimating?

• There is no discussion of limitations of this method. For example, no mention of the additional cost of using "virtual points", no discussion of the computational cost of approximating the posterior, no discussion of the exact type of algorithm used to approximate the posterior (line 382 mentions MCMC, but which MCMC algorithm? How good was this? How slow was it?).

• A much more minor concern is that I am not sure this paper is very relevant to the ICLR community. Given the AISTATS deadline is so close to that of ICLR, I think submitting the paper to AISTATS would have made much more sense as there would be a much bigger audience that would be interested in this line of work.

Overall, I think having to review this paper has been a waste of valuable reviewing time, and I encourage the authors to consider this when next submitting to a conference.

• The novelty compared to Hansen et al. and Riihimäki and Vehtari seems negligible to me. The authors do not argument improvements of their formulation of constraints compared to existing formulations.
• The experiments only show qualitative results, there is not even one quantitative comparison in this paper.

• The methods section lacks an in-depth analysis of the proposed method. For example, there is no discussion about the resulting posterior; the authors merely state that "the posterior can be approximated using variational methods or Markov Chain Monte Carlo," but this should be explored in more detail. They could, for instance, present the Evidence Lower Bound (ELBO) for the variational inference approach. Additionally, the authors should discuss the method's limitations and examine the implications of different choices for $\Phi$.

• The weakest part of the paper is the experiments section. The authors do not compare their method with other relevant approaches; the only comparison is against a heteroscedastic Gaussian Process. This seems unusual since this model does not handle the constraints addressed in the experiment. The authors should consider comparing their method with approaches such as Riihimäki and Vehtari (2010).

• Overall, I find it challenging to identify the main contribution of this work. Although the authors claim it is a framework for handling general constraints, they do not explore specific approaches to address the potential intractability of the posterior. The method essentially provides an alternative perspective on constraints without offering clear benefits. In contrast, previous approaches, tailored to specific constraints, have more effective ways of managing posterior intractability.

While the topic is interesting and timely, the current version of this manuscript has major weaknesses related to (1) limited description, missing details; (2) limited experimental validation; (3) applicability; (4) theoretical grounding and guarantees.

1. Limited descriptions, missing details: while the manuscript is well-written, it leaves many details of the proposed method to be filled in by the reader. This makes it difficult to judge its applicability in complex scenarios. For instance, it does not explicit state any posteriors except for (3)-(4), which is a particular case of the proposed method already put forward in [Hansen et al., 2024]. It is therefore hard to judge the novelty of the proposed method. Additionally, without explicit forms for the posteriors, it is hard to judge the computational complexity of the proposed approach, especially since no details are provided on the applications beyond the fact that they were sampled using MCMC. It is therefore unclear which MCMC technique is used (particularly since it must sample in function spaces), how the presence of constraints affects the convergence of these methods, and what effect does the specific MCMC method used has on how well the constraints are enforced. This particularly important since the advantage of GPs are their closed-form posterior that can be leveraged to reduce the computational complexity of inference. If MCMC methods are deployed, it is unclear why the retriction to GPs is important. The manuscript should include a more detailed discussion of all these aspects before it is possible to judge its contribution.

2. Limited experimental validation: The issue in (1) is exacerbated by the limited experimental validation. The applications considered are limited to low-dimensional settings with loose, linear constraints. This does not properly showcase the performance or limitations of the approach. In the first application, the "constraint component" ($\sigma^2 = 1$) in the posterior is substantially more lax than the "data component" ($\sigma^2 = 0.01$). Hence, while it modifies the solution (Fig. 2), it does not enforce the "divergence-free" constraint in (16). Doing so would require using a very small $\nu$, which would in turn affect the numerical stability of the MCMC method (see (4) below). Including additional experiments or theoretical results on the effects of $\nu$ is necessary to support the use of this method to enforce constraints. Even if this is not the goal of the paper, i.e., it seeks only to induce certain behaviors without guaranteeing them, it is still unclear to what extent $\nu$ affect how much that behavior is induced or how it interacts with other terms in the posterior. I cannot judge the second application as it lacks details on the choices of parameters altogether, but the effect of adding the constraint does not appear to be significant (Figure 3). Once again, this could be due to the data "taking over" the posterior, but it is impossible to tell. Finally, since the constraints in both applications are linear, it would appear that other baselines could also be considered. For instance, using virtual points to fix values and techniques from Section 2.2.2 or 2.2.3 (i.e., [Hansen et al., 2024] or (3)-(4)). Leaving out these comparisons also makes it difficult to judge the novelty of the method compared to the previous literature (see (1) above).

3. Limited applicability: The last point in (2) is particularly critical as the proposed method does not enforce constraints, but rather penalizes its violations. As such, the resulting inference is not representative of the constrained Bayesian problem. Substantial theoretical and experimental validation is required to showcase to what extent constraints are actually enforced (depending on $\nu$). Additionally, the curse of dimensionality issue related to virtual measurements (raised in line 129) is not tackled by this method (that itself relies on virtual points as per line 112). This limitation, however, is only discussed in the context of related work.

4. Theoretical grounding and guarantees: Since the manuscript relies primarily on MCMC methods for inference, it should more carefully examine that literature, particularly since penalties in MCMC are not commonly used in a myriad of context (including inducing constraints), see, e.g., [Robert and Casella, "Monte Carlo statistical methods"]; [Karagulyan et al., "Penalized Langevin dynamics with vanishing penalty for smooth and log-concave targets," NeurIPS'20]; [Gurbuzbalaban et al., "Penalized Overdamped and Underdamped Langevin Monte Carlo Algorithms for Constrained Sampling," JMLR'24]. While these methods do not target function spaces, they can still be used to sample paths from GPs. In the particular case of linear constraints, they can even be used to sample constrained paths (see [Pfortner et al., "Physics-Informed Gaussian Process Regression Generalizes Linear PDE Solvers," arxiv'22]).
   This last point raises another critical issue with the proposed method. Note that (3)-(4) are only valid for bounded linear operators under particular assumptions (see Thm. 1 and Cor. 2 in [Pfortner et al., 2022]). These are quite mild, but should still be mentioned in the paper. What is more, it is not clear to what extent this generalizes to arbitrary operators as in (5). In general, conditioning a GP on (5) is ill-posed and may lead to singular paths among other measurability issues. It is therefore unclear whether (7) makes sense without strong assumption on $\Phi$ (at the very least to guarantee the existence of the integral $p(\Phi \mid X, y)$).

In view of these issues, I cannot recommend the publication of the current version of this manuscript.