Thank you for the reviews and valuable suggestions. Here are our responses to the concerns raised:

Answer 1: Clarification on Figure 6

The scatter plot in the background represents the training instances $(x_i, y_i)$, with $p(y|x)$ defined in the southeast corner of page 7. Consider $y_r|x$ as the r-th quantile of $y$ given $x$; thus, $y_r$ is a function of $x$, and $(x, y_r)$ can form a curve $\\Gamma_r$ on the x-y plane. For $r_2>r_1$, the monotonicity of quantiles ensures that the curve $\\Gamma_{r_2}$ lies above the curve $\\Gamma_{r_1}$. In Figure 6, five estimated quantile curves (red curves) are plotted for $r$ in $(0.1, 0.3, 0.5, 0.7, 0.9)$ using different monotonic modeling methods at different training stages. Each column signifies a distinct modeling approach, and each row denotes a training stage. An accurate quantile value approximation should result in each estimated quantile curve being close to its actual quantile curve. Therefore, the estimated curves should:

• Not have excessively narrow gaps between them.
• Have $\\Gamma_{0.1}$ and $\\Gamma_{0.9}$ adjacent to the edges of the background scatter plot, with limited outliers.

Observing Figure 6 reveals that:

• The red curves for PosNN, MM, SMM, and SMNN (mostly monotonic by construction methods) are clustered in certain areas.
• CMNN, Hint, and PWL (mostly monotonic by regularization methods) exhibit too many outliers.
• GCM displays distinct gaps between the red curves and the fewest outliers.

Answer 2: More Intuition for the Experiments

Comparison between GCM and IGCM: Our analysis reveals that within public datasets, the IGCM outperforms the GCM. This improved performance is attributed to the fact that real-world scenarios do not always adhere to strict monotonic relationships; for instance, while BMI is correlated with diabetes, it is not necessarily causative. Typically, the selection of monotonic features is informed by experience or statistical analysis. Under these circumstances, implicit monotonic assumptions tend to be more applicable. Our experimental data bolsters this assertion, as IGCM demonstrates superior performance to GCM in five of the six datasets.

Comparison between strict monotonic methods: The strict monotonic methods evaluated include PosNN, MM, SMM, CMNN, SMNN, and the proposed GCM. The basic PosNN shows poor performance across most datasets, indicating that merely establishing a positively weighted neural network is insufficient for monotonic problems. Interestingly, the traditional MM method performs commendably, and its successor, SMM, exhibits a correlation, indicating that the core min-max architecture proves effective in optimizing the monotonic network. Recent developments like CMNN and SMNN introduce innovative monotonic network architectures, outperforming MM in several experiments. Our proposed GCM method consistently exhibits strong performance across all datasets, suggesting that reframing the monotonic problem as a generative one yields enhanced results compared to simply altering the network structure.

Comparison between nonstrict monotonic methods: Hint, PWL, and IGCM are non-strict monotonic methods, yet only IGCM addresses the non-strict issue through implicit monotonic modeling. This approach consists of two strict monotonic challenges: $p(r|k)$ and $p(y|k)$, establishing a nonstrict monotonic link between $y$ and $r$, which proves beneficial when compared to traditional non-strict monotonic methods.

Answer 3: Other Potential Improvements

Rapid Decision Making: As illustrated in Appendix D, GCM and IGCM provide the quickest inference for multiple revenue variables, providing a clear advantage in decision-making models. For instance, consider a robot tasked with determining a sequence of actions for a particular mission, where each action is directly proportional to total energy usage. If there is a model capable of forecasting energy consumption to help the robot minimize energy loss, this model must rapidly predict multiple action options. Here, the GCM method proves beneficial for swift decision making.

Scalability: GCM is inherently scalable. Unlike previous monotonic models that struggle to scale due to restrictions on non-monotonic normalization techniques (like LN, BN) and limitations on activation functions or linear projections, GCM can incorporate any structure to model $p(c|x,z)$ and $q(z|x)$. Thus, with more extensive datasets, GCM's scalability advantages may become more evident.

Monotone Multitask Modeling: For a set of tasks $y_1,\\cdots,y_n$ that are monotonic such that $(y_i=1)\\subset(y_{i+1}=1)$ and each $y_i$ is monotonic with respect to $r$, we can apply the monotonic cost variable sequence $c_1\\succ \\cdots \\succ c_k$ and let $y_i=\\mathbb I(r \succ c_i)$ to address it.

We hope these clarifications address your concerns and demonstrate the validity and applicability of our proposed method. Thank you again for your valuable feedback.

Thank you for the reviews and valuable suggestions. Here are our answers to the concerns raised:

Answer 1: Bounded Revenue Variable

This issue can arise in practice, for instance, when our model is trained with $r \\in (-10, 10)$, but is used for inference with values $r \\gg 10$. Such a discrepancy can lead to an out-of-distribution (OOD) problem, resulting in $\\Pr(y=1|x,r)=\\Pr(r>c|x)\\to1$. We address this by hypothesizing a non-zero probability $\\Pr(c=+\\infty|x)=p_0>0$, which yields a bounded estimate:

$$\\Pr(y=1|x,r)=\\Pr(r>c|x)<1-p_0.$$

This upper bound is independent of $r$. Alternatively, we can replace the revenue variable $r$ with a bounded monotonic function $h(r)$. Consequently, the probability $\\Pr(y=1|x,h(r))$ will be monotonic with respect to $h(r)$, effectively transforming the original unbounded problem into a bounded one.

Answer 2: Conditional Independence

The conditional independence $c\\perp r\\mid x$ is crucial for solving the strict monotonic issue of $p(y|x,r)$ by achieving $\\Pr(c\\prec r|x,r)=\\Pr(c\\prec r|x)$, with $r$ representing the revenue variable and $y$ as its monotonic outcome. As demonstrated in Lemma-2, for any strictly monotonic probability $p(y|x,r)$, such a latent variable $c$ is guaranteed to exist, allowing us to construct the $c$ model without worrying about its practical interpretation.

Conversely, in non-strict monotonic problems, finding such a $c$ is impossible. For example, a consumer might prefer buying a product at a lower price, but if the price is excessively low, it might seem dubious, resulting in the consumer not buying it, illustrating that $c$ (consumer cost) is affected by $r$ (inverse of the product price). In this case, an implicit monotonic model, detailed in section-4.4, can be utilized. Here, the concealed kernel variable $k$ denotes the product's authenticity, and then $r$ shows monotonicity with respect to $k$, enabling us to identify a cost variable $c$ so that $c\\perp k\\mid x$, instead of $c\\perp r\\mid x$. Here, $c$ and $k$ are both latent variables that the modeling for them will not obey the practical interpretation of $r$ and $y$.

Answer 3: Ablation on Hyperparameters in Variational Inference

The ablation of $D$, the latent variable dimension, and $N$, the sampling number, is detailed in Table-5 and Table-6 within Appendix C. Testing results for GCM using the Adult dataset are as follows:

The findings indicate that $D$, the latent dimension, impacts the test AUC, whereas the sampling number $N$ plays a lesser role. Another critical hyperparameter is $\\beta$, utilized in the ELBO as per the $\\beta$VAE approach, where the modified ELBO is expressed as follows,

\\text{ELBO}_\\beta=\\mathbb E_q \\log {\\text{Pr}( c\\curlyvee_y r| z, r)p({x},{r}| z)}-\\beta \\log \\mathbb E_q \\frac{q( z| x)}{p( z)}.

The test outcomes for varying values of $\\beta$ are:

The results reveal that $\\beta$ values within $\\{0,\\cdots,5\\}$ yield comparable results, with $\\beta=0$ achieving the best performance. This is attributed to the test AUC metric being primarily focused on the accuracy of $y$, rather than the whole generative model $p(x,r,y,z,c)$.

We hope these clarifications address your concerns and demonstrate the validity and applicability of our proposed method. Thank you again for your valuable feedback.

Thank you for your detailed feedback and suggestions. We appreciate the opportunity to address the concerns raised in the review.

Answer 1: Discriminative vs. Generative Models

We respectfully disagree with the assertion that generative models are inherently more challenging than discriminative models. The modeling difficulty depends on the task and the structure used. For example, naive Bayes, a generative model, is widely used for classification tasks and is comparable to discriminative methods such as logistic regression. In our paper, we have shown that a discriminative model $\\Pr(y=1|x,r)=G(x,r)$ can be transformed into a generative model $p(x,c)=p(c|x)p(x)$ using the relationship established in Lemma-2:

$$\\Pr( c \\prec r| x) =\\Pr( c \\prec r| x,r) = G(x, r),$$

and consequently

$$p( c | x) =\\partial G(x, c) / \\partial c.$$

This demonstrates that in monotonic modeling tasks, these approaches can be interconverted, so it is unfair to say that generative methods in monotonic modeling are more difficult arbitrarily.

Moreover, traditional monotonic networks face challenges such as requiring all weight matrices to be positive [1] and all activations to be monotonic and not fully convex or concave [2]. Common normalization techniques like layer-norm and batch-norm are also not allowed since they are not monotonic operators. As a result, it is hard to train a deep monotonic network. However, with the cost generative model, we can easily build a deep network for generating the cost variable $c$ and obtain the target variable $y$ via $\\{y=1\\}=\\{c\prec r\\}$. This allows us to use unconstrained weight matrices, non-monotonic activations such as silu and gelu, fully convex activations such as relu and softplus, and normalization techniques widely used in deep networks such as LN and BN. Training this generative network is also straightforward following the loss function demonstrated in (7).

[1] Monotonic Networks. Link

[2] Constrained Monotonic Neural Networks. Link

Answer 2: Challenge of Learning the Joint Probability Distribution

Previous studies show the outstanding performance of learning the joint probability in machine learning tasks, such as learning the joint distribution of an image via generative methods. For simple tasks such as the MNIST dataset, a simple four-layer VAE [3] is sufficient, while more complex tasks like CIFAR10 might require deeper models such as DDPM [4]. This paper has shown that the cost variable $c$ is essential for all monotonic problems $\\Pr(y=1|x, r)$, where $r$ is the monotonic revenue variable, $x$ is the non-monotonic variable, and $y$ is the response variable determined by $c\\prec r$. Since $x,r,c$ are all real vectors, we can adopt model structures similar to image generative models.

To design the generative model for $x,r,c$, we have chosen a basic structure similar to the VAE, as the tasks in our paper are not overly complex. Although $c$ is not directly observable, we can evaluate the model for $c$ using the observable variable $y=\\mathbb I(r\\succ c)$, and the testing AUC/ACC of $y$ indicates the performance of our generative model of $c$. Additionally, the ELBO $\\mathbb E_{z\\sim q}\\log p(x,r,y|z)-D_{KL}(q(z|x)\\|p(z))\\leq \\log p(x,r,y)$ serves as a lower bound for the evidence $\\log p(x,r,y)$, providing another evaluation metric.

[3] Auto-Encoding Variational Bayes. Link

[4] Denoising Diffusion Probabilistic Models. Link

Answer 3: Choice of Functions in (4) and (8)

We employ the standard normal distribution for the priors of $z_1$ and $z_2$. Each conditional likelihood $p(a|b)$ is expressed as:

$$\\mu_a, \\log \\sigma_a = \\text{MLP}(b), \\ \\ a \\sim \\mathcal N (\\mu_a, \\sigma_a^2).$$

In the GCM model, we use the reparameterization trick to derive the variable $a$, or we apply the CDF function of the normal distribution to compute the probability $\\Pr(a<a_0)$. The MLP here consists of one or two layers. We will provide further details in the appendix of future revisions.

Answer 4: Sufficiency of a Three-Layer Network

The number of layers is not a constraint for generative models. As mentioned in Answer 2, the VAE applied to the MNIST dataset uses only two layers with tanh activation functions and two affine layers without activations, yet it successfully generates high-quality handwritten numerals. In our experiments, the dimension of the revenue variable (see Table 3 on page 8) is less than 10, much smaller than the pixel count of an MNIST image. Therefore, we chose not to use an excessively deep generative network for our study.

We hope these clarifications address your concerns and demonstrate the validity and applicability of our proposed method. Thank you again for your valuable feedback.