Probabilistic Feature Smoothed Gaussian Process For Imbalanced Regression

The novelty and motivation of the paper are unclear. First, regarding the novelty. The fundamental idea of this paper is similar to Sliced Inverse Regression (SIR), where the data is divided based on the space of the response variable, and the statistics of the explanatory variables in each division are utilized. Therefore, SIR should be cited as a key related work: Li, Ker-Chau. 1991. “Sliced Inverse Regression for Dimension Reduction.” Journal of the American Statistical Association 86 (June): 316–27.

Specifically, there are similar method, which applies kernel weights to each data point based on the value of the response variable to perform smoothing: Zhu, Y., & Fang, K. T. (1996). Asymptotics for kernel estimate of sliced inverse regression. Annals of Statistics, 24(3), 1053-1068.

Local SIR, which emphasizes data points with similar response values to estimate the dimension reduction directions: Chen, C. H., & Li, K. C. (1998). Can SIR be as popular as multiple linear regression? Statistica Sinica, 8(2), 289-316.

and Weighted SIR, which incorporates information from adjacent slices by weighting the mean vector of each slice.: Xia, Y., & Härdle, W. (2006). Semi-parametric estimation of partially linear single-index models. Journal of Multivariate Analysis, 97(5), 1162-1184.

I understand the above mentioned papers are for supervised dimension reduction and the submitted paper is for Gaussian process fitting, but the used technique is quite similar. The relationship between these methods and the positioning of the proposed method should be clearly explained.

I will comment on the motivation in "Question" later.

The presentation of the paper should be considerably improved. There are so many presentation issues that they cannot all be pointed out. I'll point out few of them:

• In regression problems, it is not as common to refer to the response variable y as a "label" compared to classification problems. In classification tasks, the term "label" is frequently used to denote discrete classes (categories). However, in regression tasks, where the response variable y takes continuous values, terms like "target" or "response variable" are more commonly used. That being said, using the term "label" in regression problems is not entirely incorrect. However, for greater precision, using terms like "target" or "response" is generally more appropriate in regression tasks.
• Please carefully read the example (template) of the ICLR paper. "The table number and title always appear before the table."
• On page 1, abbreviations such as SoR, DTC, FITC, and PITC are used without explanation, despite their first appearance. Similarly, on page 2, abbreviations like DIR, LDS, and FDS are used right from the start without being spelled out. While these are spelled out later, they should be spelled out upon their first appearance. Relatedly, this paper contains too many undefined notations and terms, or notations and terms that are used before being defined. Notations such as $K_{N,M}, K_M, \bar{f}, K_N, Q_N, f_{B_s}$ in Eqs. (1), (2), and (3) should not be used without definition. Given that this is a Gaussian process paper, one could generously assume that $k$ represents the kernel function, but it is still inappropriate from a writing perspective to use symbols like $k$ and $Q$ without defining them.
• While I understand that $\mathcal{Y}_n$ represents the partitioning of the response variable into intervals, what does $|\mathcal{Y}_b|$ in the lower part of p.3 represent?
• 'blockdiag' is not defined.
• The summation in Eq. (8) is used inaccurately.
• Why is the dimension of $\mathbf{f}$ in Eq. (11) $B$?
• Right below Eq. (11), it says 'log marginal likelihood (NLL)', but why is the abbreviation for log marginal likelihood 'NLL'?
• In Algorithm 1, $\mathbb{B}$ is referred to as the Bin index, but considering the definition of $\mathbb{B}$, calling it an 'index' seems inappropriate.

• Contribution: The paper’s contribution is limited, as it builds on the PITC framework, which is not novel. Simply adding a calibration layer to PITC may lack impact, particularly as this layer could potentially be integrated with more current methods.

• Feature Distribution Smoothing (FDS): This layer is insufficiently explained in the paper. Further discussion is needed to clarify how FDS substantially improves the results and what mechanisms make it effective.

• Computational Cost: Inducing point-based methods remain computationally expensive, a limitation that the paper does not adequately address. Including a discussion on computational complexity would strengthen the paper’s rigor.

• Quality of Estimation: The accuracy of estimation with inducing points is highly sensitive to the quantity and selection process of these points. However, the paper does not sufficiently consider these limitations.

• Hyperparameter Tuning for Bins: The paper would benefit from a discussion on selecting the number of bins and initializing this hyperparameter. It remains unclear whether increasing the number of bins would enhance result quality.

It is not clear (and made clear) what (implicit) assumptions are made for the smoothing to work. Also, I did not understand how Section 4.3 relates to the paper; these are worst-case bounds for an expected loss function approach on p(y|x) yet using Gaussian processes means that we already make a distributional assumption on p(y|x). It would have been better to study the relationship between p(x) and p(y|x) that has to be fulfilled for the approach to work well.

1. First, there seems to be a need for a clear discussion on the necessity of modeling imbalanced regression data, as addressed in this paper, using a Gaussian Process model. If I encountered such a data analysis problem, I would likely choose another regression model. For instance, regression models based on tree structures, such as random forest or boosting, are known to behave relatively robustly even when the response variable takes extreme values.

2. I am skeptical about whether it is appropriate to use only the values of the response variable for data partitioning in PITC. With this type of partitioning, regions with similar response variable values but significantly different input values would be represented by a single Gaussian Process model, which seems inappropriate.

3. In the numerical experiments, the performance of the proposed method is overwhelmingly better than that of other baseline methods; however, the experimental setup is overly complex and specific, making it difficult to properly interpret the results. I have the following concerns regarding the experimental settings among others:

   a) Why were the combined cycle power plant and concrete compressive strength datasets chosen? I am curious whether it was confirmed that these datasets represent imbalanced regression problems.

   b) How were the hyperparameters selected? Setting the GP hyperparameters the same for both the single GP model and PITC is clearly inappropriate—if optimal hyperparameters were specifically selected for the proposed method, that would be unfair.

   c) Why were the response variables divided into 100 intervals? Was it reasonable to make that many partitions? Modeling with 100 local GPs seems likely to result in highly overfitted results.

4. The proposed method appears to be merely a simple combination of existing methods, PITC and PFS, lacking sufficient originality to warrant acceptance at a top-tier conference like ICLR.