Neural Rule Lists: Learning Discretizations, Rules, and Order in One Go

Dear Reviewer, thank you for your in-depth, constructive feedback.

We want to comment further on Neurules's scalability and relative performance compared to other methods. In our experiments, the Juvenile dataset with 286 features has the highest dimensionality. On it, Neurules is the best performing method, with an $F_1$ score of $0.89$.

Furthermore, we supplement results on two additional datasets from the UCI repository that have hundreds of features: “Drug-induced autoimmunity” (477 samples x195 features) and “TUNADROMD” (4465 samples x242 features), which deals with Android malware. We use the same hyperparameters from the experiments and set the rule list size to $10$. We provide the results for each in the two tables at the bottom, excluding CORELS and SBRL due to their inability to scale to higher dimensions.

On TUNADROMD, we achieve an $F_1$ of $0.97$ compared to a best of $0.98$ (Classy, RLNet). For drug-induced autoimmunity, Neurules is the best performing method with a substantial gap between it ($0.78$) and the next best (RLNet, $0.73$). Drug-induced autoimmunity consists mainly of continuous features, where Neurules threshold learning ability comes in handy, while TUNADROMD is only binary.

In general, we agree that scalability is an important aspect for the real-world deployment of any rule list model. The current results indicate that Neurules is on par for large-scale datasets of discrete features or even superior on those with mainly continuous features. Neurules runtimes scale favorably due to the gradient-based optimization. We will add these results to the updated manuscript and expand the discussion on scalability.

Drug-induced autoimmunity

Results on TUNADROMD

Dear Reviewer,

Thank you for your in-depth, constructive feedback.

Below we address your concerns:

1. Comparisons: We respectfully disagree with the suggestion that our setup for CORELS is not fair or misrepresents its capabilities. Due to the delicate balance between performance and runtime when choosing the parameters, we performed a series of experiments for CORELS where we systematically varied the number of cut points. In Appendix G, Figure 7, we report runtime and performance for CORELS with an increasing number of cutpoints. We observe that, contrary to intuition, an increase in cutpoints has a slight negative impact on $F_1$ score. On the other hand, increasing the number of cut points leads to an exponential growth in the number of frequent itemsets considered during CORELs’ preprocessing, resulting in an exponential runtime increase. In theory, increasing the number of cut points should expand the search space and allow CORELs to discover better classifiers. However, in our experiments, this increase in resolution often leads to overfitting and timeouts, not improved generalization.

   With respect to the FICO dataset and the comparison to He et al. (2023), we note that the reported performance of CORELs is achieved under a very different experimental setup, including an extensive and carefully crafted discretization scheme and what appears to be significantly relaxed runtime constraints. Under such idealized and hand-tuned conditions, CORELs can outperform other models. But this underscores - rather than undermines - our central contribution: our method learns the discretization, rule structure, and ordering jointly in a single, scalable pipeline, requiring no manual pre-processing, and delivers competitive (often superior) performance in a fraction of the runtime.

2. Hyperparameters: Our evaluation avoids fine-tuning hyperparameters per dataset to ensure fairness and realism across a broad range of methods. In particular, neural approaches like ours often involve many hyperparameters that, if tuned per dataset, could lead to overly optimistic results and poor generalization. Moreover, tuning regularization parameters per dataset for combinatorial methods such as CORELs is especially problematic due to their high computational cost. As noted in the paper, many of these methods already approach or exceed runtime limits, and performing nested cross-validation across all datasets would break our computational budget. Instead, we determine the best overall hyperparameter configuration on a few holdout datasets. With the best overall parameters, the subsequent evaluation is conducted across all datasets to better reflect each method's robustness and out-of-the-box performance.

3. Rule complexity: The sparsity of the discovered rules is interesting due to its correlation with the interpretability of a rule. We provide in Figure 4b) the distribution of rule lengths, i.e., the number of predicates per rule, over all datasets. Below, we provide a table with statistics for all methods, which we will add to the Appendix:
   |Method | median | mean | std | |:-------------------|---------:|-------:|------:| | Neurules | 4.00 | 7.85 | 13.45 | | RLNet | 6.00 | 9.65 | 15.16 | | DRNet | 21.00 | 53.96 | 97.02 | | CORELS | 2.00 | 1.83 | 0.86 | | CLASSY | 3.00 | 2.89 | 0.90 | | RIPPER | 2.00 | 1.94 | 1.37 | | SBRL | 2.00 | 1.91 | 0.71 | Neurules learns more succinct rules than other neural approaches (RLNet, DRNet), showing the efficacy of our gradient shaping in that regard. Compared to combinatorial methods that impose a limit on maximum cardinality, the rules we find are longer. We don't think that this is necessarily a weakness; there may exist datasets where rules with longer clauses make sense. In general, we observe a power law-like trend of rule lengths learned by Neurules, where the majority of rules have four or fewer clauses.

4. Cardinality: We think that there are many datasets for which rules with five or more conditions are appropriate. Increasing the maximum cardinality does lead to a combinatorial explosion of possible rules and hence termination issues for CORELS. However, as a configuration with cardinality five achieved the best accuracy, it was selected in the hyperparameter optimization. Thank you for pointing out the typo in the Appendix. We will update the text.

Regarding your questions:

1. Loss discontinuities: You're right in observing that, in principle, discontinuities could arise from rule reordering or predicate activation. The idea of Neurules is to overcome the obstacle of discontinuity in a rule list using smooth, differentiable approximations. For rule ordering, we apply a Gumbel-Softmax relaxation, which smooths the argmax operation and allows gradients to flow even when priority values are near a crossover point via the additional Gumbel term in Eq. (12). Similarly, predicate inclusion is handled through soft conjunctions with slack, so the model can adjust predicate weights without abrupt changes in the loss.

   That said, as the temperature approaches zero, the approximations can become steep. Particularly, if an input value lies at the transition, the soft predicate is effectively discontinuous, such as e.g Fig. 3a for $x=0.3$. In very rare cases, this can lead to very large gradients. In practice, this has not posed a significant problem for us because we control the temperature annealing schedule and avoid hard transitions. While we did not find it necessary in our experiments, given that we precisely know when this happens, one practical safeguard would be to monitor this operation and stop gradient flow in this situation.

2. Comparison to CORELS: Although CORELS offers formal optimality guarantees, these apply only to the regularized training objective and only within the fixed rule space defined by mined itemsets. Although the method and paper are impressively rigorous, the authors do not provide a tight generalization bound. One could derive such bounds using standard tools like Hoeffding’s inequality, since the hypothesis space is finite, but they degrade rapidly as the rule space grows. Yet this bound would only hold for the regularized objective and does not allow us to make any statements about the performance on the test set.

   One can, as you suggest, use Neurules to extract predicates for CORELS. To test this idea, we take the FICO dataset and extract all boolean predicates that Neurules learns, excluding those with $w_{ij}=0$ and those that are always 0 or 1, respectively. We use these predicates to binarize the train and test sets for each cross-validation split. On this data, we re-run CORELS. We obtain an $F_1$ score of $0.68+-0.01$ with an average runtime of 5 minutes per fold compared to Neurules $0.70$. We hypothesize that the difference stems from the maximum cardinality constraint placed by CORELS, whereas Neurules predicates were learned in the context of longer, unrestricted rules. So while it is possible to combine Neurules predicate learning with CORELS search method, there are no immediate benefits in terms of performance, but potentially in sparsifying the learned rule list.

3. Sparsity: The sparsity pressure in our method arises from the slack term $\eta_i = \frac{\varepsilon}{\sum_j w_{ij}}$ in Eq. (7). This slack appears in the denominator of the relaxed conjunction and controls how sharply low-activation predicates are penalized. When ${\hat{\pi}}{ij}$ is close to zero, a small $\eta_i$ causes the denominator $(\hat{\pi}{ij} + \eta_i)$ to be large, which results in strong negative gradients on $w_{ij}$, pushing that literal's weight toward zero. Conversely, if $\hat{\pi}{ij} \approx 1$, the term contributes positively to the gradient, reinforcing the literal. The parameter $\varepsilon$ thus controls the strength of this push-pull: smaller values lead to stronger sparsity pressure. As $\varepsilon \to 0$, this pressure increases, but if $\varepsilon$ reaches exactly zero, the gradients can vanish or tend to infinity when $\hat{\pi}{ij} = 0$. In practice, we keep $\varepsilon$ small but strictly positive to ensure effective pruning without numerical issues. This mechanism eliminates literals with minimal contribution while retaining informative ones, and $\varepsilon$ provides a smooth, tunable control over the regularization strength.

Thank you for your continued engagement.

1. Comparison to CORELS: I am not sure where I stand on this issue. Alternatively, you can imagine your method, when flexibly choosing cutpoints, to be choosing from a maximally-granular set of potential cutpoints, but I'm not concerned about your method overfitting the cutpoints. I think that the issue arises when there are too many long rules to choose from, some of which end up being spuriously predictive of the label in the training set. In this sense, a good rule selection mechanism (like fpgrowth, for example) may allow for good generalization, even with a very fine discretization. As a note -- I am used to thinking about rules as being one-sided inequalities rather than intervals. I agree that having intervals, with 2^n cutpoints, could result in a huge explosion of the number of rules to consider.

2. Hyperparameters: I commiserate with this difficulty, but I maintain that tuning at least the regularization on every dataset individually is strictly necessary for me to trust the comparison. I think that the setup you settled on is a reasonable compromise, except for this point, where the right regularization is dependent on specific properties of each dataset. Another point, which this conversation brings to mind, is that the sparsity comparison between different methods depends on the regularization parameters chosen for each method in hyperparameter optimization.

3. Rule Complexity: With identical performance, I think that shorter rules are clearly better than longer rules. It is very important to note that complexity does not always trade off with accuracy, especially on problems with noisy labels (which are very common in the sorts of domains where interpretability is valued) [1, 2, 3]. My understanding of that study is not that longer rules are preferable, but rather that people have a cognitive bias towards longer rules. It may be that this results in more trust of models with longer rules, but it remains open whether this would translate into better performance or decision making on a real world task.

For clarity: I am inclined to raise my score to a 4 on the grounds that I am confident that your method is an advancement on other neural methods for learning rule lists. I remain skeptical of the comparison to discrete solvers for rule lists, namely CORELs, due to the lack of regularization tuning per-dataset and the restrictive discretization procedure. However, I acknowledge the difficulty in designing a perfect experiment that fits in runtime constraints.

[1] Semenova et al., 2019: On the Existence of Simpler Machine Learning Models

[2] Semenova et al., 2023: A Path to Simpler Models Starts With Noise

[3] Boner et al., 2024: Using Noise to Infer Aspects of Simplicity Without Learning

Extra typo that I found: line 527, the values of the "bs" parameters are missing comma separators.

Thank you also for engaging in this productive discussion.

1. Comparison to CORELS: I agree with all points raised, and I think that there could be interesting future work in leveraging your method to find good rules, potentially along with an optimal solver to maximally exploit them. That is outside of the scope of this work though, and for now I am satisfied with this conclusion.

2. Hyperparameters: Thank you for running this comparison, this is a very useful point of reference. I presume that CORELS' lackluster performance on f1 score arises mostly from a discrepancy in the optimization metric from the evaluation metric, and secondarily from the issues of feature selection that we discussed in regards to the CORELS comparison? Perhaps CORELS is less sensitive to regularization than I thought -- I will do my own benchmarking to check this for myself outside of the review process.

3. Rule Complexity: This is an interesting point, that I had not completely considered. I generally assume that a very long rule will capture such a small portion of the training data that it is indistinguishable from a spurious rule using the training data. But doing a postprocessing check with held out data to check whether a long rule is spurious is an interesting idea, and reminiscent of the model editing framework for generalized additive models proposed in [1].

Another benchmark, which is outside the scope of this paper, but which you may find interesting, is the STreeD algorithm (proposed in [2]) for decision tree optimization. This could be trivially restricted to optimize rule lists instead (go down only one side of a tree), and has built in optimization techniques for f1 score and other metrics usually secondarily optimized.

In light of the additional experiments showing that hyperparameter tuning is likely less of an issue than I initially believed, I am inclined to raise my score to an accept. I encourage the authors to consider including results with tuned regularization, at least, because although it does not appear to make a significant impact on the results, I think it improves the trustworthiness of the paper for somebody (like me) who is skeptical of hyperparameter tuning on held out datasets.

[1] Wang et al., 2021: GAM Changer: Editing Generalized Additive Models with Interactive Visualization.

[2] van der Linden et al., 2024: Necessary and Sufficient Conditions for Optimal Decision Trees using Dynamic Programming

Dear Reviewer,

Thank you for engaging so much in this discussion; we really appreciate it.

1. Comparison to CORELS: That is an interesting observation. Indeed, our approach can be viewed as modeling from a maximally granular set of potential cutpoints.
   As for pattern mining-based approaches, such as those based on fpgrowth, these are indeed able to find general patterns, yet come with their own set of challenges. As they return every itemset that passes the minimum support threshold, the result set tends to be very large due to down-ward closure property, highly redundant because due to noise it includes many variants of the same ground-truth rules, as well as includes many "spurious" rules that consists of combinations of true rules or frequent but independent items. Determining which of these are the most informative is an active field of study, but as of yet, remains a challenging task. This means spurious rules can still emerge even with sophisticated mining approaches, while the likelihood may be reduced, the fundamental issue remains.

2. Hyperparameters: To address your concern about regularization tuning, we conducted additional experiments by re-running CORELS on 6 datasets with regularization parameter $c \in \{0, 0.001,0.005, 0.01, 0.02, 0.03, 0.04, 0.05, 0.1\}$. Due to time constraints in the discussion period, we reduced the number of cutpoints from 5 to 3, though this modification yields nearly identical results on this subset of datasets. The results are presented in the table below.

   Dataset	NeuRules	0	0.001	0.005	0.01	0.02	0.03	0.04	0.05	0.1
   Android Malware	0.93	0.33	0.33	0.33	0.33	0.33	0.33	0.33	0.33	0.33
   FICO	0.7	0.69	0.69	0.69	0.69	0.69	0.69	0.69	0.69	0.68
   Heart Disease	0.8	0.68	0.68	0.68	0.68	0.68	0.68	0.68	0.68	0.07
   Hepatitis	0.81	0.76	0.76	0.76	0.76	0.76	0.76	0.76	0.48	0.63
   Rin	0.92	0.63	0.63	0.63	0.63	0.63	0.63	0.63	0.63	0.54
   Titanic	0.75	0.68	0.66	0.66	0.66	0.66	0.62	0.54	0.54	0.54
   Avg. F1	0.82	0.63	0.62	0.62	0.62	0.62	0.62	0.61	0.56	0.47

   Our analysis reveals that CORELS performance remains stable across the range $c \in \{0,0.001,0.005, 0.01, 0.02, 0.03,0.04\}$, with F1 scores showing minimal variation. Performance deteriorates only at higher regularization values $c \in \{0.05, 0.1\}$. This consistency across datasets supports our experimental design choice of using a fixed regularization parameter rather than dataset-specific tuning.
   Notably, CORELS performs best with no regularization $c=0$ on these datasets, yet still shows a performance gap compared to NeuRules. This gap persists consistently across all regularization settings and aligns with the results reported in our main paper.
   We want to emphasize that our rule grammar employs interval constraints rather than one-sided inequalities, which may differ from your previous experience with CORELS. Regarding sparsity comparisons, we systematically optimized all available hyperparameters (including sparsity regularizers where applicable) to maximize performance. This approach ensures that our sparsity comparisons reflect each method's best achievable performance rather than arbitrary parameter choices.

3. Rule Complexity: We understand each other regarding rule lengths. We are not arguing in favour of rules that include spurious terms. If a short rule is as good as a longer one, we also prefer the short one. However, it is important to distinguish that long(er) rules may be specialized rather than spurious. Short rules can typically describe common patterns (the norm) very well, whereas longer rules may be necessary to capture important long-tail behavior.
   Our analysis of NeuRules shows that the median rule length is indeed smaller than the mean – i.e. most rules are short – while some very long rules serve specialized functions. Upon closer inspection we find that most rules capture substantial parts of the training resp. test data, but there are also a few rules that capture only small portions of the training set and fire very infrequently on test data. Whether these are generally spurious (bad) or rare (good) needs further investigation, and it will be interesting to develop more advanced regularization or post-hoc pruning techniques to prune the bad apples.
   Last, but not least, we do argue that having spurious rules in rule-based models differs fundamentally from spurious correlations in other machine learning approaches, such as linear regression. In rule lists, spurious rules typically fail to fire on new data and cause minimal direct harm beyond occupying a slot in the rule list, which, while suboptimal, is less problematic than spurious parameters in continuous models. While it is not immediately clear how to do so during training, it is straightforward to prune unused or rarely used rules from the list during post-processing using a hold-out set.

Dear Reviewer, thank you for your in-depth, constructive feedback.

Below we address your questions and concerns:

1. Predicate FOL: Thank you for your comment. Although our predicates are parameterized and grounded in feature space, they serve an analogous role as in FOL: they act as the basic Boolean conditions used to construct a conjunctive rule, similar to how predicates in FOL form the structure of atomic formulas.

   In our work, we restrict these atomic conditions to threshold-based conditions over a single feature (e.g., $a_{ij}<x_j<b_{ij}$). In principle, for every individual feature $x_j$ there exist infinitely many predicates by combinations of $(a_{ij},b_{ij})$. Thus, in subgroup discovery/rule mining, the first step is usually to generate an initial set of predicates using fixed discretization. Instead, we propose a method that explicitly optimizes those, instead of pre-defining them a priori. After training, these learned predicates become fixed, binary-valued conditions that can be interpreted as atomic logical components. Both concepts are very similar in spirit but differ in the formalities. We will clarify this connection in the text.

2. Soft predicate: In [1], soft predicates refer to latent logical abstractions that emerge through structured sparsity over clause weights. These predicates are not explicitly defined; rather, they represent shared parameter patterns across clauses, serving as a soft form of predicate invention. In contrast, our soft predicates are explicit, parameterized threshold functions over real-valued features. Each outputs a value in [0,1], approximating binary inclusion and enabling differentiable rule learning.

   Although our formulation resembles fuzzy predicates-in that both yield values in [0,1], the underlying motivation differs. In fuzzy logic, the fuzziness reflects graded truth, such as "x is partially tall." In our case, the soft predicate is a training-time relaxation, used to enable end-to-end optimization via gradient descent. The goal of this relaxation is to learn the threshold by which we determine “x is tall”, by means of minimizing the predictive error. The learned predicates are ultimately interpreted as binary threshold conditions, although the intermediate softness could have similar semantics, albeit having a different functional form as say in the weak conjunction of Łukasiewicz, its primary purpose is to facilitate learning. As temperature annealing progresses, these soft predicates converge to interpretable, discrete conditions similar to classical logical atoms. We will clarify these distinctions in the revised manuscript to avoid confusion and better position our work in relation to both prior neural-symbolic methods and fuzzy logic approaches.

3. Threshold parameters: The lower and upper bound parameters $a_{ij}$ and $b_{ij}$ are learned via gradient descent. The formulation in Eq. (4) allows us to take gradients with respect to the bounds. In particular, in Appendix 1 we show Eq (4) is equivalent to Eq (21) $\hat{\pi}(x_j;a_{ij},b_{ij},t_{\pi})=\frac{1}{e^{(x_j-a_{ij})/t_{\pi}}+1+e^{(b_{ij}-x_j)/t_{\pi}}}$ In the limit of $t_{\pi}\to 0$, the predicate verifies whether $x_j > a_{ij}$ and $x_j<b_{ij}$, i.e. it behaves like a strict thresholding function would. We will clarify the intuition behind this formulation in the updated text.

4. Temperature: In our evaluation, we use a starting temperature of $t_{\pi}=0.2$, which is gradually annealed to $0.05$ (Appendix E.2). Additionally, in the hyperparameter sensitivity section (Appendix E.1), we re-run our experiments with different $t_{\pi}$. The results displayed in the upper left corner of Figure 5 show that under reasonable values of $t_{\pi}$, Neurules performance is stable.

5. Connection to column generation: Both [2] and Neurules seek to find accurate and intrepretable sets of rules, but differ in the kind of rule ensemble that is learned and scope of application. The column generation method focuses on learning CNF (AND-of-ORs) and DNF (OR-of-ANDs) rule sets on binary data. On the other hand, Neurules learns ordered rule lists (IF THEN; ELSE IF) on continuous data.

   Column generation casts rule learning as a mixed-integer program with provable guarantees, but to that end requires pre-mined candidate rules. In contrast, our method learns thresholds, rules, and their ordering jointly in a fully differentiable framework, without pre-discretization or candidate enumeration. While our approach does not offer global optimality guarantees, our evaluation shows that Neurules outperforms combinatorial methods for rule list learning that first mine candidate rules (SBRL,CORELS), and in the case of CORELS come with optimality guarantees over that search space. We will include this distinction in the revised text.

Dear Reviewer,

Thank you for your in-depth, constructive feedback. Below we address your questions:

1. Categorical features: We use one-hot encoding for categorical features and treat them as continuous features. We will make this clear in the updated manuscript
2. Ablation rule ordering: In the ablation, we compare to NeuRules with a fixed, predefined rule order, where $p_i$ is not learned during training. This is similar to using a fixed order as RLNet, but using our predicate and rule learning models. We will clarify this.
3. Gradient shaping: yes, we disable the gradient shaping mechanism by setting $\epsilon = 0$.
4. Fig. 4 colors: The red color marks the area where the gradient shaping leads to a performance loss. The blue signals where an improvement is achieved. The color of the markers is scaled as per the performance of the ablated model.
5. Notation consequent: Each rule has a consequent $\hat{c} \in \mathbb{R}^m$ that is a learnable parameter and represents the logits, which are turned into class probabilities using the softmax function. We acknowledge that denoting the consequent of each rule i as $c_i$ and the individual probabilities as $c_l$ is confusing, and we will adapt the notation to avoid this. We denote as $\hat{c}$ the logit parameter vector, and denote as c is the probability distribution over the classes induced by the learned logits.
6. Predicate: We agree that Equation (21) makes the predicate easier to understand, so we will substitute it in the main text.
7. Typos: We will incorporate your feedback. Thank you for your thorough review.

Lastly, we would like to provide further context to the rationale behind our differentiable relaxations. We only use relaxations that converge to their strict counterparts. For rule lists, the methodology of learning rule ordering and predicates is the first of its kind. For the logical conjunction, we introduce a novel relaxation approach with the slack $\epsilon$ and derive its gradient shaping (GS) properties. We evaluate GS in an ablation, showing significant improvements in accuracy and rule sparsity. We empirically find that Neurules as a whole performs better compared to other neural approaches, which employ different relaxations for logical conjunctions (RLNet, DRNet, RRL).