Hard-Constrained Neural Networks with Universal Approximation Theorem

We thank the reviewer for their detailed analysis and constructive feedback. We have revised the manuscript based on your feedback, with key changes highlighted in blue. Below, we address the raised concerns point by point:

1. Novelty of the Paper

The reviewer correctly points out that the concept of using a differentiable projection layer or a differentiable parameterization of the feasible set to enforce constraints is present in prior works. However, existing methods are often restrictive with one or more of the following limitations:

• Iterative Refinement: They rely on iterative methods to adjust outputs toward the feasible set, which do not guarantee constraint satisfaction within a fixed number of iterations.
• Input-Dependent Constraints: They struggle to handle input-dependent constraints, as they require unique parameterizations for each input, such as determining a new interior point for every feasible set.
• Constraint Types: They support limited types of constraints such as linear equality constraints.

Our approach, especially HardNet-Aff, overcomes these challenges by introducing a closed-form projection that ensures the satisfaction of input-dependent affine constraints by construction. Furthermore, we rigorously demonstrate that the proposed architectures preserve the expressive power of neural networks by providing universal approximation guarantees.

We have refined the related work section to emphasize these points and included the references suggested by the reviewer.

2. Gradient Properties

We thank the reviewer for highlighting the importance of discussing the gradient behavior introduced by the projection layer.

It is true that the Jacobian of the projection layer is orthogonal to any violated constraint vector $a(x)$ (in terms of vector-matrix multiplication). However, this does not imply that $f_\theta(x)$, when initialized outside the feasible region, cannot reach values inside the feasible region through gradient descent (GD) optimization.

For example, in the experiment added in Appendix A.5, we train a HardNet-Aff model on a single datapoint, as shown in Fig. 4 (top right). Despite starting outside the feasible region, the model successfully reaches the target value within the feasible region through GD steps.

This is because the orthogonality between the Jacobian and the violated constraint vector does not restrict the direction in which $f_\theta(x)$ changes. Although infinitesimal changes in $f_\theta(x)$ result in $\mathcal{P}(f_\theta)(x)$ confined on the boundary of the violated constraint due to the orthogonality, larger update can shift $\mathcal{P}(f_\theta)(x)$ beyond the boundary.

Additionally, in Appendix A.5, we discuss cases where the gradient becomes zero due to the projection layer. This occurs when the gradient of the loss with respect to the projected output is spanned by the violated constraint vectors, even if the model output deviates from the target value. Notably, this condition includes the 1D output case with a single inequality constraint mentioned by the reviewer.

In practice, such cases are infrequent, especially when training on batched data. Moreover, zero gradients for certain datapoints are often offset by nonzero gradients from other datapoints, allowing the model to update and reduce the overall loss. This effect is demonstrated in the experiment on two datapoints in Fig. 4 (Appendix A.5).

Additionally, we can promote the model $f_\theta(x)$ to be initialized within the feasible region using the warm-start scheme outlined in Appendix A.8, which involves training the model without the projection layer for a few initial epochs while regularizing constraint violations.

To further address this issue, we recommend the warm-start scheme described in Appendix A.8, which involves training the model without the projection layer for a few initial epochs while regularizing constraint violations. This scheme can promote the model $f_\theta(x)$ to be initialized within the feasible region.

A summary of these points has been added as Remark 4.4, with detailed discussions in Appendix A.5.

I appreciate the authors' efforts in improving the manuscript and addressing my comments.

Gradient Proprieties I appreciate the inclusion of additional discussion on this aspect. As I had briefly mentioned in the original comment, I understand that it is possible to mitigate the gradient issues caused by projection using SDG or large lr. However relying on such effects in order to have a successful optimization is not ideal. Similarly, relying on a warm start procedure kind of defeats the purpose of having a hard constraint satisfaction during training. Again I'd like to thank the authors for including the new discussion in the appendix, as I believe the analysis was well done and welcome addition.

Treatment of Equality Constraints Unless I misunderstood something, most of the complexity whole procedure is caused by the handling of the equality constraints. It seems to me that if there are no equality constraints the projection operator, $\bar{A}$ reduces to $A$. I am failing to see why the rank condition is needed in this case. Wouldn't it be much simpler to convert equalities to pairs of inequalities, even if the number of constraints increases?

Novelty and Related works I'd like to start with a quite minor concern I had omitted in my original review. Unless I am missing something the universal approximation proof is a quite trivial consequence of the projection operation. I believe that reporting "unknown universal approximation" for most competitors might be a bit misleading. It seems to me that for at least some of the methods it follows from the reparameterization/projection onto the feasible set. For this reason, while I appreciate the author effort in the formal proof, I believe the importance of this contribution is a bit overstated.

As stated in my original comment, from my understanding of the related literature, a lot of effort is made specifically to avoid projection onto the feasible set. Further more, concerns about the experimental section and concerns about missing literature have been raised by reviewers ZsCV and rKKV, and have not been addressed.

Overall, given the author's efforts in addressing my concerns and improving the overall quality of the work, I'm willing to increase my score accordingly. However, given the remaining concerns, I believe the final result still falls short of the acceptance threshold given the high profile of this venue.

6. Regarding Figure 2, it looks like all models perform reasonably good in the region which the training data lie in, and the difference occurs outside of data coverage. I am confused why "Soft" seems much worse than others. If I understood it correctly, "Soft" penalizes when the model output violates the constraints. Since all training points are feasible, I intuitively expect "Soft" to behave similarly to "NN", but this is not the case. Could you please explain why such difference? Also, how would "Soft + proj" look like?

In Fig. 2 (left), the models at the initial epoch violate the constraints on many training points. As a result, the regularization in “Soft” alters the training process compared to “NN”, even if both models share the same initialization. This regularization would compromise the model’s fitting performance.

Interestingly, “Soft” appears to perform worse in terms of constraint violation than “NN” in this experiment, which contradicts its intended purpose of penalizing violations. We hypothesize that this occurs because, in this particular example, the feasible regions alternate between the upper and lower half-spaces. Penalization in one constraint region can have an adversarial effect on another, leading to ineffective point-wise regularization. This highlights a limitation of "Soft" in handling certain configurations of constraint regions.

"Soft + Proj" would address this issue by projecting all violated outputs onto the boundary of the feasible regions, thereby ensuring zero violations. Additionally, constraint satisfaction would likely improve generalization, resulting in better RMSE performance compared to "Soft.”

7. For the "safe control policy" experiment in Section 5.3, what do you think is the biggest advantage of the proposed method compared with non-learning methods such as model predictive control?

The main advantage of the proposed method compared to non-learning approaches like model predictive control (MPC) lies in computational efficiency. While MPC requires solving an optimization problem online for each initial state, our approach leverages offline training to learn a control policy for a set of potential initial states while satisfying safety constraints by construction. This leads to significantly faster inference during deployment, making it suitable for real-time applications.

8. Line 500 mentions that the constraint in Eq.(12) can be conservative, leading to worse performance compared to "Soft" and "DC3". Is it possible to adjust the level of conservativeness by changing $\alpha$?

The reviewer is correct that the conservativeness of the constraint in Eq. (12) can be adjusted by changing $\alpha$. A larger $\alpha$ reduces the conservativeness by making the constraint in Eq. (12) less restrictive. However, if the constraint becomes too non-conservative, initial roll-out trajectories may get stuck near the boundaries of obstacles, which can hinder the optimization of the policy.

While optimizing the choice of $\alpha$ can mitigate this issue, the current example effectively demonstrates the utility of HardNet-Aff in enforcing safety conditions during control policy optimization. We believe this highlights the practicality of our approach in ensuring constraint satisfaction.

We hope these clarifications address the reviewer’s concerns. Thank you again for your constructive comments.

We thank the reviewer for their thoughtful feedback and valuable comments. We have revised the manuscript based on your feedback, with key changes highlighted in blue. Below, we address the specific questions and concerns raised.

1. To use the closed-form projection algorithm Eq.(7), we need $n_{ineq}+n_{eq}$ to be no greater than the output dimension. Is this restrictive in practice? In the included experiments, which one of them uses closed-form projection?

The reviewer raises a valid question regarding the practicality of the condition $n_{ineq}+n_{eq}\leq n_{out}$ for using the closed-form projection in HardNet-Aff. This condition could be restrictive in practice, for instance, to enforce the constraints $0\leq f(x)\leq1$. In such cases, we can still utilize our method by choosing a subset of constraints to guarantee satisfaction and imposing the others as soft constraints. We have added this remark after Assumption 4.1.

That said, the condition still aligns with many practical applications as shown in the experiments. The closed-from projection is used for the results indicated with “HardNet-Aff” in Section 5.1, 5.2, and 5.3.

2. For Eq.(11), should $u$ also be a function of $t$, i.e., $u(t)$?

The reviewer correctly notes that u in Eq. (11) should explicitly be written as u(t) for consistency. We have updated the manuscript accordingly to prevent confusion.

3. In Table 4, why are rows 2 and 4 marked as red even though they are feasible?

Thank you for pointing out the inconsistency in Table 4. We have updated the manuscript accordingly.

4. I am a little confused about part iii) of Proposition 4.2. Namely, the projection preserves the distance from the boundary of the feasible set when $\bar{f_\theta}(x)$ satisfies the constraint. Would you mind sharing a geometric intuition?

We can use the example described in Fig. 1, where $f(x_2)$ satisfies the two linear inequality constraints—say $a_1^\top f(x_2)\leq b_1$ for the constraint with the almost horizontal boundary and $a_2^\top f(x_2)\leq b_2$ for the other. In this example, $\bar{f_\theta}(x_2)=f_\theta(x_2)$ as there is no equality constraint.

First, we can geometrically observe that HardNet-Aff projects $f(x_2)$ in parallel to the second boundary, which means the distance of $f(x_2)$ from the boundary is the same as that of $\mathcal{P}(f_\theta)(x_2)$, i.e., the distance from the boundary is preserved.

With more algebraical details, $f_\theta(x_2)$ satisfies the second constraint ($a_2$). Then, $a_2^\top \mathcal{P}(f_\theta)(x_2) = a_2^\top f_\theta(x_2)$ by Proposition 4.2 (iii), which implies $a_2$ and $\mathcal{P}(f_\theta)(x_2) - f_\theta(x_2)$ is orthogonal. This implies the movement from $f_\theta(x_2)$ to $\mathcal{P}(f_\theta)(x_2)$ is in parallel to the second boundary (which is also orthogonal to $a_2$). On the other hand, $f_\theta(x_2)$ violates the first constraint ($a_1$), so $a_1^\top \mathcal{P}(f_\theta)(x_2) = b_1$. This implies the projected output lies on the first boundary. This explains the location of the projected output colored in green in Fig. 2.

5. I am also confused about the $C_{\leq}(f(x))$ notation in line 359. What does $C$ denote? This is different from the $C(x)$ in Eq.(4), right?

$C_{\leq}$ is a new notation for computing inequality constraints and different from the $C$ in Eq.(4), as the reviewer pointed out. In the affine constraint case in Eq.(4), $C_{\leq}(f(x)) = A(x)f(x)-b(x)$. In general convex constraint case, $C_{\leq}$ could be a nonlinear function. To avoid confusion and indicate its dependency on $x$, we have updated the notations $C_{\leq}\rightarrow g_x$ and $C_{=}\rightarrow h_x$.

Thank you again for your time reviewing our paper. As the discussion period is coming to an end, if our response has addressed your concerns, we would be grateful if you could re-evaluate our work. If you have any additional questions or comments, we would be happy to have further discussions.

I urge the authors to address the weaknesses and questions within my review for me to be able to confirm my current evaluation and score.

We thank the reviewer for their detailed feedback and constructive critique. Below, we address the raised concerns and questions in detail. We have revised the manuscript to incorporate these points, with key changes highlighted in blue.

1. HardNet-Cvx

Acknowledgment of Prior Work:

We appreciate the reviewer pointing out the connection between HardNet-Cvx and prior works, such as DC3 and the "PP" baseline in RAYEN. While we acknowledge that the use of differentiable optimization methods (especially, Agrawal et al. (2019)) for orthogonal projections is not novel (as we have explicitly done so in the revised manuscript, in a note after the definition of HardNet-Cvx), we believe that our contributions are still significant.

While HardNet-Cvx is presented as a general framework—to complement HardNet-Aff—which can be implemented using various methods, we provide universal approximation guarantees for the general orthogonal projection, which is a nontrivial property, as discussed below. Furthermore, we believe the survey contribution of our paper is significant, as, to the best of our knowledge, no existing papers provide a detailed comparison of the different methods as well as a summary as presented in Table 1.

Universal Approximation Guarantees:

While existing universal approximation theorems show that $f_\theta$ can approximate a target function $f$ arbitrarily closely, these guarantees do not extend to satisfying constraints. We can project the close approximation $f_\theta$ to the feasible set to ensure constraint satisfaction, but the projection $\mathcal{P}(f_\theta)$ could be located far from $f$.

For instance, consider the target function $f:\mathbb{R}\rightarrow\mathbb{R}$ s.t. $f(x)=1\forall x$ with the constraint $y\leq 1$. A simple projection $\mathcal{P}(f_\theta)(x)=\begin{cases}f_\theta(x)&\text{ if }f_\theta(x)\leq 1 \newline 0 & \text{o.w.}\end{cases}$ could still differ significantly from $f$ if $f_\theta(x)>1$ for many $x$ even though $f_\theta$ is arbitrarily close to $f$. In contrast, we show that HardNet-Cvx and HardNet-Aff ensure $\mathcal{P}(f_\theta)$ remains arbitrarily close to $f$ when $f_\theta$ is sufficiently close to $f$.

Use Case in the ACAS Experiment:

The ACAS experiment employing HardNet-Cvx involves affine constraints with a larger number than supported by HardNet-Aff, as the reviewer noted. While an example with non-affine constraints would be more compelling, this experiment still demonstrates the utility of enforcing input-dependent constraints in practical scenarios. We acknowledge that HardNet-Cvx incurs higher computational costs, limiting its use in time-sensitive applications. However, this example highlights its flexibility for cases where the closed-form projection of HardNet-Aff is unavailable.

2. HardNet-Aff Assumptions

The assumptions for HardNet-Aff, particularly $n_\text{ineq} +n_\text{eq} ≤ m$ (number of constraints no greater than the output dimension), may indeed be restrictive for certain applications, such as interval constraints per output coordinate. In such cases, we can still utilize our method by choosing a subset of constraints to guarantee satisfaction and imposing the others as soft constraints. We have added this remark after Assumption 4.1.

3. Experiments

Performance of DC3:

While DC3 has demonstrated strong empirical performance in its original paper, its effectiveness is highly sensitive to hyperparameters, including the regularization coefficient for the soft penalty and the number of iterations and step sizes for gradient-based corrections. This sensitivity has been noted in prior literature, such as in RAYEN, which highlights performance variations with different regularization coefficients.

In our experiments, we used the official DC3 implementation, applying 10 correction iterations across all settings. This already resulted in significantly longer training times compared to other methods, as detailed in the tables. In the learning optimization solvers experiment, we also reran the experiments with the same hyperparameters and neural network model provided in the official DC3 implementation. While DC3 nearly satisfied the constraints, it showed a larger optimality gap than HardNet-Aff, as reflected in the revised manuscript. In prior settings, we used the same model without batch normalization and dropout layers (still with the same hyperparameters). In that setting, DC3 exhibited more severe constraint violations, further demonstrating its sensitivity to hyperparameter tuning.

Missing DC3 + Proj Baseline:

We agree that "DC3 + Proj" is an interesting and relevant baseline. We have now included its results in Table 3 and 4 and Figure 3 to provide a more comprehensive comparison.

Absence of DC3 from Table 5:

This experiment was implemented independently and did not include the integration of the DC3 implementation in the codebase, as we initially believed sufficient comparisons were provided in the other experiments. However, we plan to incorporate DC3 and DC3+Proj into this experiment in the camera-ready version.

Out-of-Distribution (OOD) Samples in Toy Example:

The reviewer is correct that evaluation on [-2, 2] while training on [-1.2, 1.2] introduces OOD samples, potentially contributing to performance gaps. This setup was intended to show the benefit of imposing constraints on generalization for unseen data. We have added additional experiments where training is conducted over the full [−2,2] region to evaluate the performance without OOD data (Appendix A.4).

Comparison of HardNet-Aff and HardNet-Cvx:

HardNet-Aff employs a non-orthogonal projection, which in general modifies function outputs more than the orthogonal projection in HardNet-Cvx. However, depending on $f_\theta$ and the constraint geometry, these larger changes can result in projections closer to the target value. Thus, the non-orthogonal projection does not necessarily lose something compared to the orthogonal one. Comparing the two approaches through visualizations of the optimization landscape is an interesting avenue for future work.

Experiments on Larger Networks:

Because the projections are applied only to the outputs of $f_\theta$, larger networks can be employed without additional computational burden on the projection process. Exploring how larger models influence method performance would be an interesting direction for future experimentation.

We hope these clarifications, revisions, and additional experiments address the reviewer’s concerns and highlight the value of our contributions. Thank you again for your constructive review.

We sincerely thank the reviewer for their detailed and constructive feedback, as well as for providing helpful references and suggestions to strengthen our paper. We have revised the manuscript based on your feedback, with key changes highlighted in blue. Below, we address the main concerns, questions, and comments point by point.

1. Neuro-symbolic AI literature

We thank the reviewer for their observation that our paper could better situate itself within the Neuro-symbolic AI literature, and we greatly appreciate the extensive references provided. After reviewing these works, we agree that they are highly relevant and can enrich our discussion, particularly C-DGM [3], as highlighted by the reviewer, and we have added them to the paper.

C-DGM was initially proposed to enforce constraints in generative models for tabular data, but its methodology can also be applied to general input-independent affine inequality constraints $Ay\leq b$, as noted in Table 1. However, its applicability to input-dependent constraints is limited, as it cannot efficiently process batched data in such scenarios.

To elaborate, C-DGM operates by iteratively computing a reduced constraint set $\Pi_i$ for each output component $y_i$. It then reprocesses these components in reverse order to project each $y_i$ onto an interval $[lb_i, ub_i]$ derived from $\Pi_i$. In the case of input-dependent constraints, batched data processing requires recalculating the reduced constraint sets and intervals for each input, resulting in significant computational overhead.

In contrast, HardNet-Aff efficiently computes the closed-form projection for batched data in a single step, making it well-suited for input-dependent constraints. Additionally, HardNet-Aff provides universal approximation guarantees, a feature not offered by C-DGM. However, for input-independent constraints, C-DGM has fewer restrictions on $A$, as it does not require $A$ to have full row rank. This flexibility allows C-DGM to treat equality constraints as pairs of inequality constraints, whereas HardNet-Aff employs a separate process to handle equality constraints.

We have incorporated a discussion of C-DGM into the related work section to acknowledge its contributions and limitations. For other referenced works, we have added a literature review on Neuro-symbolic AI methods in Appendix A.9 due to page constraints. This review also includes additional relevant works beyond those suggested by the reviewer.

2. At page 4, shouldn't the sup-norm be defined as $||f||_\infty = \sup_{x \in \mathcal{X}} ||f(x)||$?

Here, $||f(x)||_\infty$ indicates the maximum norm for the vector $f(x)$ so that the sup-norm for the function $f$ is defined with that specific norm. $||f(x)||$ indicates a general norm to be defined.

3. At page 5, I think it would have been great to have a small example with just a neural network with two outputs $y_0$ and $y_1$ and the constraint $y_0\geq y_1$.

We agree that including a small example would enhance the clarity and accessibility of the paper. We have added this example with slight modification to consider the input-dependent constraint $y_0\geq x y_1$.

4. At page 5, among the assumptions there is written that the constraints need to be feasible. Just to improve the readability of the paper and also make sure everything is well defined, it would help to add the meaning of the word feasible.

The reviewer’s suggestion to explicitly define the term "feasible" is well-taken. We have changed the condition to “For all $x\in\mathcal{X}$, there exists at least one $y\in\mathbb{R}^{n_\text{out}}$ that satisfies all constraints in (4).”

5. At page 5 the authors give the assumptions for which the number of constraints needs to be lower or equal than $n_\text{out}$. I think it would be really helpful to add a simple example with a set of constraints that cannot be captured.

We agree that an example of a constraint set exceeding $n_{out}$ would help illustrate the limitations of HardNet-Aff. We have included an example and noted that our method could be still utilized to enforce a subset of the constraints while imposing the others as soft constraints.