Beyond the convexity assumption: Realistic tabular data generation under quantifier-free real linear constraints

Q2. The authors did not compare the proposed method versus direct rejection sampling. Rejection sampling also guarantees 100% satisfaction of all constraints, and according to the reported timing in Section 4.3 (Table 5), it is not necessarily worse. e.g. for URL dataset, non-WGAN models has a CVR of about 10%. This means ~0.15/0.9=0.17s < 0.27s is needed, so it's even faster. Moreover, rejection sampling preserves naturally generated samples from generators instead of editing them, and there may be a chance that its utility can be even better.

A2. Again, we would like to thank the reviewer for this comment, which allowed us to directly compare our method with rejection sampling and show that using DRL is much better not only in terms of sampling time but also in terms of machine learning efficacy. We have already shown in the Table above that rejection sampling actually always takes longer than applying DRL. Moreover, we tested whether applying rejection sampling improves or deteriorates the machine learning efficacy. We report the results on our classification datasets (hence F1-score, weighted F1-score and Area under the Curve) in the Tables below. Notice that we could not report the results for the House dataset as the process timed out after 24 hours for rejection sampling. In each table we have highlighted in bold the best result and in red the worst result.

Efficacy in terms of F1-score (F1)

Efficacy in terms of weighted F1-score (wF1)

Efficacy in terms of Area under the Curve (AUC)

As it can be seen from the tables, WGAN+RS never yields higher Efficacy than WGAN+DRL. Similarly, except for one case (for TVAE, on the LCLD dataset, using F1 metric), TVAE+RS yields a lower Efficacy than TVAE+DRL in all other cases. Further, out of the cases where rejection sampling managed to produce a sample set without timing out, 20 out of 24 such cases showed a decrease in efficacy compared to the baseline unconstrained model. We registered the largest decrease for CCS: w.r.t. WGAN, the decrease was equal to 25.2% (24.2% and 24.4%, respectively) using the F1 metric (wF1 and AUC, respectively), while w.r.t. TVAE the decrease was equal to 30.1% (29% and 33.6%, respectively), using the F1 metric (wF1 and AUC, respectively).

These results clearly indicate that adding rejection sampling can indeed shift the learnt distribution, hence confirming that rejection sampling is often not a viable option, and surely in these cases using DRL should be preferred.

We thank the reviewer for their appreciation of the hardness of the problem we are handling. In the new version of the PDF we have highlighted all the new text related to this review in blue.

Below we provide point-by-point answers to all the questions.

Mathematical Statements:

Q1. Line 127: If $w_k=0$ and $b <0$, which satisfies your constraint that $w_k, b \in \mathbb{R}$, would the claim that $\Omega(\Phi)$ is non-empty still be true? (Although similar case is addressed in Line 275-277, it does not validate Line 127.)

A1. The claim that $\Omega(\Phi)$ is not empty holds if we assume that at least a variable appears in $\Phi$ which is the assumption under which wrote those properties (as specified directly below at line 131 of the original paper).

Q2. Line 170-172: What if $\Psi$s in $\Pi$ are not simplified and somewhat standardized? 1) If for some $\Psi$, $l_i^{\Psi} \ge r_i^{\Psi}$ (violating the constraint "whenever ..." in Line 159, if it could be violated), would the definitions of $l_i^{\Pi}$ and $r_i^{\Pi}$ still be valid? 2) If for some $\Psi, \Psi’ \in \Pi$, $[l^{\Psi}_i,r^{\Psi'}_i] \cup [l^{\Psi'}_i,r^{\Psi'}_i] \neq \emptyset$, would the definitions of $l^\Pi_i$ and $r^\Pi_i$ still be valid?

A2. The definitions of $l_i^{\Pi}$ and $r_i^{\Pi}$ are absolutely valid even if there exist $l_i^{\Psi} \ge r_i^{\Psi}$. Lemma 3.1 shows that, for every constraint, $DRL(\tilde{x})$ satisfies $\Pi$ and is optimal w.r.t. $\tilde x$. Notice that these boundaries are of interest only when $\tilde x$ violates the constraints; if the original sample does not violate the constraints then $DRL(\tilde{x})$ leaves the sample unchanged. 2) $l_i^{\Pi}$ and $r_i^{\Pi}$ are absolutely valid even when $[l_i^{\Psi}, r_i^{\Psi}] \cap [l_i^{\Psi’}, r_i^{\Psi’}] \neq \emptyset$ for the same reasons as above.

Q3. Line 259-260: Ambiguous (although can be guessed from textual explanation)

A3. We thank the reviewer for pointing this out. We have added the parentheses to improve readability. As explained in the text, it is indeed $(A \setminus B) \cup C$.

Q4. Line 285-287: Similar problem as in Line 170-172

A4. Yes, for the same reasons as point 2.

Implementation details not mentioned:

Q1. How the DRL is applied in all the experimented models is not explicitly stated, including Appendix.

A1. We included a diagram for each model to exemplify how the layer should be added in the topology of each of the considered models in Appendix A.

Q2. How constraints are determined (or if manually determined, what are they) is not mentioned in the main content nor appendix

A2. Some of the constraints were already included in the original datasets. This is true for URL, Heloc and LCLD. Regarding CCS and House, we manually annotated the constraints using our background knowledge about the problem, then we checked whether the data were compliant with our constraints and finally we retained only those constraints that were satisfied by all the datapoints. Simple examples of these constraints (from CCS) state simple facts like "if the feature capturing the number of cigarettes packs per day is greater than 0 then the feature capturing whether the patient smokes or not should be equal to 1" or "the feature representing the age should always be higher than the age at the first intercourse".

Firstly, we would like to thank the reviewer for their appreciation of the novelty and relevance of our method. Further, we would like to thank them for their comments, which have given us interesting avenues to explore for future work.

Below we provide point-by-point answers to all the questions.

Q1. The method proposed in this paper for post-processing already generated data may require further theoretical support. It isn't easy to assert that finding the data point with the shortest Euclidean distance within the constraint-satisfying set is the most reasonable approach. For example, the Mahalanobis distance might be a more effective metric.

A1. The Euclidean distance is computationally efficient, it is intuitive to understand and showed to work well in ensuring the satisfaction of the constraints while improving the Efficacy of the synthetic data. Additionally, using the Euclidean distance allowed us to follow the definition of optimality already given in [1], where only linear constraints are considered. This said, we totally agree with the reviewer that it is not the unique metric that can be used and it would be an exciting research question to study how the DRL layer would change when taking into account other metrics!

Q2. Furthermore, the paper does not compare information leakage and reproduction accuracy, essential performance metrics for generated data. It would be beneficial to demonstrate the trade-off between these two critical metrics in generative models. For example, it is helpful to report the Membership Inference Accuracy and Attribute Disclosure Risk of the considered models.

A2. As our method adds a layer (DRL) on top of existing generative models, we did not expect our method to have a big impact on the information leakage and reproduction accuracy. This is also confirmed by the results reported in the Table below, where the Identifiability Score (as defined in [2]) is shown. A 0-identifiability would represent a perfectly non-identifiable dataset and 1-identifiability would represent a perfectly identifiable dataset (hence the lower the better). Best results are in bold.

As expected, DRL did not have a big impact on the identifiability score as WGAN+DRL got the same score as WGAN twice, performed better than WGAN twice and worse only once. We computed these scores for a single model in order to start the discussion as soon as possible. We will include them for all models upon publication if the paper gets accepted, and if possible, even during this discussion period.

We thank the reviewer their appreciation of the novelty, relevance and soundness of our method and experimental analysis.

Below we give a point-by-point answer to all the questions. In the new version of the PDF we have highlighted all the new text related to their review in light blue.

Q1. The main method section (Sec 3) is a little bit dense and hard to read. I think providing a pseudo code is helpful here.

A1. We thank the reviewer for the suggestion of adding the pseudo-code. We added it in Section 3 and we believe it provides a nice bird-eye view of the method.

Q2. What is the computational complexity of DRL?

A2. The DLR compilation step (which only happens once before training) creates a number of constraints which critically depends on the number of non-convex constraints in $\Pi$ or derived from $\Pi$ with CP-resolution. In the worst-case, happening when for each variable $x_i$ the size of $\Pi^\pm_i$ is proportional to the size of $\Pi$, DLR can be super-exponential in the size of $\Pi$. However, we never experienced an exponential blow-up and hence memory problems in our experiments. It is also worth pointing out that once the DRL is built, starting from a sample $\tilde x$, the computation of DRL($\tilde x$):

1. takes linear time in the number of levels of DRL, which is at most equal to the number of variables occurring in $\Pi$, and
2. at each level $i$, the operations performed to compute DRL($\tilde x_i$) are standard (e.g., max, min, sum etc,) each of which has been heavily optmised by pytorch to run in parallel on GPUs.

In practice, in our experiments, we never experienced a significant degradation of the performance produced by DRL, as shown in Table 5.

Q3. What are the constraints for different datasets? Are they manually chosen by authors or are included in the original datasets?

A3. The URL, LCLD and Heloc datasets were already annotated with such constraints. Regarding CCS and House, we manually annotated the constraints using our background knowledge about the problem, then we checked whether the data were compliant with our constraints and finally we retained only those constraints that were satisfied by all the datapoints. Simple examples of these constraints (from CCS) state simple facts like “if the feature capturing the number of cigarettes packs per day is greater than 0 then the feature capturing whether the patient smokes or not should be equal to 1” or “the feature representing the age should always be higher than the age at the first intercourse”.

Q4. Why does satisfying the constraints help improve machine learning efficiency? Is that because we also transform the test dataset so that they satisfy the constraints?

A4. To explain why we get better machine learning efficiency, consider the House dataset, for which the target feature is the house price and we visualise the violation areas in Figure 4 (page 7) and 5 (page 23). For this dataset, we have among others two constraints simply stating that if a house is located in zipcodes 98004 or 98005 then the house price should be above 400,000USD and if the house is located in zipcodes 98006, 98007 or 98008 then the house price should be above 200,000USD. As we can see from the Figures all the standard generative models (TVAE, WGAN, TableGAN and GOGGLE) often violate the constraint, thus creating datapoints where for example a house located at zipcode 980004 is worth only 100,000USD. Now suppose we train a classifier on the datapoints generated with TVAE. Given these synthetic datapoints, the classifier will now wrongly learn that a house in zipcode 98004 can be worth well below 400,000USD, hence leading to worse predictions and higher Mean Square Error. This example clearly shows why by adding DRL we get higher machine learning efficacy. Finally, note that the constraints express known relationships between the datapoints that hold for all of them. Hence, no datapoint in the test set has ever been modified.

We thank the reviewer for their appreciation of (i) the novelty of the problem we are trying to solve and its hardness, (ii) the generality of our solution and (iii) the soundness of our method. In the new version of the PDF we have highlighted all the new text related to their review in orange.

Below we give a point-by-point answer to all the questions.

Q1. The authors mainly focus on proving that their proposed Neuro-symbolic AI layer can respect the complex constraints expressed by QFLRA. What about the impact of this new layer on the learned distribution? Is it possible that such a new layer may harm the learned distribution for satisfying the constraints?

A1. We thank the reviewer for raising this point. Please note that our paper has two focuses: (i) providing guarantees that our layer always satisfies the constraints and (ii) showing that more realistic synthetic datapoints (i.e., compliant with the background knowledge) also have better machine learning efficacy. The fact that we consistently improve the machine learning efficacy of the synthetic datasets is already an indicator that the new layer does not harm the learnt distribution, and actually makes it even more similar to the real distribution. To further prove this point, we also perform for each continuous feature the Kolmogorov-Smirnov test between the real datapoints and the samples from (i) WGAN and (ii) WGAN+DRL. The test measures the probability that two sets of samples come from the same distribution, thus the closer the result is to 1 the better the synthetic datapoints capture the real distribution. Below we report the results:

We computed these results for a single model in order to start the discussion as soon as possible. We will include them for all models upon publication if the paper gets accepted, and if possible, even during this discussion period.

Q2. It seems to me that the constraints of QFLRA may be very complex and even deciding if a QFLRA is satisfiable is a difficult problem. What would happen to the proposed model if the input QFLRA is not satisfiable?

A2. We thank the reviewer for this point, which allowed us to make our paper clearer. As written in the paper, CP resolution is refutationally complete: this means that if the set of QFLRA constraints is unsatisfiable then it is possible to derive a disjunction of linear inequalities in which each inequality (3) has $w_i = 0$ for $i = 1, \ldots , D$ and $b < 0$. At the end of the layer construction, we are able to automatically detect $\Pi$ unsatisfiability, returning a corresponding UNSAT_FLAG value in such a case. We have added this last phrase in the paper to make it even clearer. (page 6)

Q3. In the experiments, the authors adopt some simple QFLRA constraints, where the feasible region has a couple of disconnected parts. For these relatively simple QFLRA, can we first identify each disconnected subregion, then use traditional models to learn the distribution over each subregion and exploit a mixture model to combine distributions over all subregions to have an overall generative model?

A3. The goal of our method is to provide a general solution that works for any set of constraints. Additionally, it is not clear how to identify the disconnected components and then train in an efficient way a different unconstrained model that is guaranteed to be compliant with the constraints. If we understand correctly what the reviewer has in mind, this would amount to apriori $(i)$ compute the DNF of the input set of constraints. $(ii)$ make the disjuncts pairwise inconsistent (e.g., going from $(A \vee B)$ to $(A \wedge \neg B) \vee B)$), and then $(iii)$ consider the subspace corresponding to each disjunct (conjunction of constraints), which can be exponentially many. Even supposing we can find the disconnected components and train a different traditional model for each one of them, it is still not clear how to ensure that the constraints are not violated by unconstrained models which violate constraints having convex and connected solution spaces (as demonstrated by the fact that the unconstrained models often violate even the constraints expressed as linear inequalities)