Mind the Gap: Removing the Discretization Gap in Differentiable Logic Gate Networks

We thank the reviewer for their time and effort in helping to improve the paper. We will address the weaknesses and questions raised in the review.

Clarifying Experiments and Evaluation Scope

We would like to clarify that our experiments already evaluate multiple model configurations, not just a single 12-layer, 256k-width network. Specifically, we sweep across four depths (6, 8, 10, 12) (Figure 3) and nine $\tau$-values (0.01–2.0) (Figure 5) at a fixed width of 256k. These sweeps form the basis for our analysis of depth scaling, $\tau$-driven smoothing, and their effect on the discretization gap. We will make this more explicit in the revised draft (Section “Gap Scaling with Depth”) to avoid any ambiguity.

Clarifying Novelty Relative to Y. Kim, "Deep Stochastic Logic Gate Networks"

We thank the reviewer for pointing out Y. Kim, "Deep Stochastic Logic Gate Networks," in IEEE Access, vol. 11, pp. 122488-122501, 2023 and will add it to the related works section for completeness. Our paper focuses on analyzing the discretization gap in Differentiable Logic Gate Networks (DLGNs) by framing the problem as Neural Architecture Search and studying how the gap scales, how $\tau$-driven smoothing impacts optimization, and how these dynamics affect pruning and final performance. These aspects are not addressed in Kim (2023), which primarily explores empirical Gumbel-Softmax modifications to improve performance without analyzing gap scaling or optimization dynamics. Additionally, Kim (2023) adopts a different neuron parametrization and design that make its regime fundamentally distinct from ours:

1. It omits the $\tau$ -parameter (central to Lemma 1 and Fig. 5 in our paper),
2. Importantly, normalizes logits. In Kim (2023) this increases entropy in a way that would actually exacerbate the discretization gap, and
3. Introduces a learnable Gumbel scaling parameter that alters the Hessian-trace term, complicating any direct comparison to our analysis.

Because of these differences in focus and model regime, our theoretical and empirical analysis addresses a distinct research question.

On Gumbel Noise and Regularization

Kim (2023) interprets Gumbel noise as preventing overfitting via Bishop’s result that training with noise approximates Tikhonov/L2-regularization. This derivation assumes zero-mean Gaussian noise, not asymmetric Gumbel noise. Moreover, our results (Fig. 3) show DLGNs continue improving with additional training as the discretization gap narrows, without the accuracy decline typical of overfitting. This indicates that overfitting prevention is not the primary reason our method works.

Unused Gates

Regarding “unused gates”: in both DLGNs and GLGNs, all gates remain in the final LGN architecture and process signals. Many of the neurons in DLGNs maintain high entropy close to initialization or change only slightly by convergence, meaning they are not meaningfully utilized. The final architecture keeps the same gate count, and we expect inference time between the discretized LGNs to match between DLGNs and GLGNs.

Additional Evaluations

We agree that broader evaluation is important, and we have now added results on CIFAR-100 and a wider network to support generalization and scalability claims.

On CIFAR‑100 (depths 6–12, width 256k), we observe the same trend as Figure 3: the discretization gap grows with network depth for DLGNs, while GLGNs remain stable. For the largest model, the DLGN has not converged after 1M iterations, reaching a test accuracy/gap of 6.11% / 3.36% points, whereas the GLGN converges in ~400k iterations with 0.23% / 0.0099% points. CIFAR‑100’s complexity suggests that larger networks are required, where we expect the discretization gap to become even more significant for DLGNs, making GLGNs critical for scaling.

For a very wide CIFAR‑10 network (depth 6, width 2048k), the DLGN gap remains high (drop of 2.5% points of test accuracy at convergence) and does not shrink significantly over training, whereas GLGNs continue reducing the gap (drop of 0.188% points of test at convergence), consistent with our main results.

We thank the reviewer for their response.

This paper focuses on an often overlooked but important direction, LGNs.

We would like to add that we identify and analyze a key limitation in this subarea, the discretization gap, which, to the best of our knowledge, hasn’t been meaningfully explored in the literature. We believe it is important that this limitation is studied and added to the literature, as it directly impacts the scalability and practical use of DLGNs.

However, the findings are not that groundbreaking, given that similar ideas have been published by Kim (2023).

We respectfully note that our work addresses a different problem than Kim (2023). We analyze the discretization gap, its scaling, and convergence. None of which are studied in Kim (2023), yet all of which are critical for practical use in this subarea.

To faithfully highlight the novelty, it would be better to not only mention it in the related works but also benchmark against it to show the difference in performance, if any.

Several important details remain unclear in Kim (2023), e.g. how the learnable parameter $\beta$ is initialized (deterministically or randomly). Kim (2023) reports to follow the same approach as Petersen et. al (2022), yet reports a different learning rate (0.1 versus 0.01). Additionally, the codebase for Kim (2023) is not publicly available, which makes it difficult to faithfully reproduce or benchmark.

We thank the reviewer for their time and effort in helping to improve the paper. We will address the weaknesses and questions raised in the review.

Use of Gumbel in NAS.

The reviewer is correct that these and other papers in the NAS literature have used Gumbel noise before. We will make this clearer in the related work where we already included [1,2,3]. We would also argue that:

1. To the best of our knowledge, framing DLGNs as NAS is itself a novel and new insight. We hope future LGN work will be inspired by this, and further utilize advancements from NAS literature.

2. Lemma 1 generalizes beyond DLGNs, and would also theoretically justify why [1,2,3] would have a smoother loss landscape and discretization gap, which are not addressed. This would position [1,2,3] against the broader NAS / Smoothing DARTS literature, e.g. Chen and Hsieh: Stabilizing Differentiable Architecture Search via Perturbation-based Regularization, 2021 and Zela et. al: Understanding and Robustifying Differentiable Architecture Search, 2020.

3. The scale of the search spaces are not comparable. Thus, our work demonstrates that the methods have potential to scale to much larger search spaces.

Quantification of the flatness of the minima

We thank the reviewer for this thoughtful suggestion. We agree that further formalizing how Gumbel noise affects the curvature of local minima is an interesting direction. However, we believe this is challenging theoretically without assuming strong properties (e.g., convexity, L-Lipschitz gradients, or toy quadratic settings) that would not hold in our setting.

Our current analysis (Lemma 1) already shows that Gumbel smoothing reduces the expected trace of the Hessian, and Figures 5–6 empirically reflect this effect. A deeper mathematical treatment of curvature under noise may indeed be more suitable for follow-up work in a broader setting.

On Logic Simplification and Gate Counting

We did not apply the post‑training logic simplification step described in Appendix A.3 of [4], as the synthesis tool used there is not publicly available. Our results report raw gate counts for GLGNs and DLGNs so that differences between the default and Gumbel‑based methods directly reflect neuron utilization without additional processing.

Additional Evaluations

To account for limited evaluation, we have now added results on CIFAR-100 and a wider network to support generalization and scalability claims.

On CIFAR‑100 (depths 6–12, width 256k), we observe the same trend as Figure 3: the discretization gap grows with network depth for DLGNs, while GLGNs remain stable. For the largest model, the DLGN has not converged after 1M iterations, reaching a test accuracy/gap of 6.11% / 3.36% points, whereas the GLGN converges in ~400k iterations with 0.23% / 0.0099% points. CIFAR‑100’s complexity suggests that larger networks are required, where we expect the discretization gap to become even more significant for DLGNs, making GLGNs critical for scaling.

For a very wide CIFAR‑10 network (depth 6, width 2048k), the DLGN gap remains high (drop of 2.5% points of test accuracy at convergence) and does not shrink significantly over training, whereas GLGNs continue reducing the gap (drop of 0.188% points of test at convergence), consistent with our main results.

Thank you for the thoughtful and constructive rebuttal. I appreciate the clarifications and additional evaluations. I would like to offer a further comment regarding the novelty claim.

(Regarding the novelty of framing DLGN as NAS)

To the best of our knowledge, framing DLGNs as NAS is itself a novel and new insight.

I respectfully disagree with this claim. The DLGN formulation [1,2] itself already builds on the principles of DNAS [3]. Moreover, the logic synthesis community has increasingly adopted NAS methodologies to tackle logic circuit optimization problems [4,5]. In particular, I recommend reviewing [5, Sections 2 and A], which survey machine learning in logic synthesis. Therefore, while the application of Gumbel noise in this context is useful and empirically effective, viewing DLGNs through the lens of NAS is not, in itself, a novel framing.

Besides, I am satisfied with the overall contribution of the paper, and I decided to keep my assessment after reading the author response and other reviews.

References

[1] Deep Differentiable Logic Gate Networks, NeurIPS 2022.

[2] Convolutional Differentiable Logic Gate Networks, NeurIPS 2024.

[3] DARTS: Differentiable Architecture Search, ICLR 2019.

[4] Alan Minko et al. IWLS 2023 Logic Synthesis Contest Submissions, 2023. https://github.com/alanminko/iwls2023-ls-contest

[5] Towards Next-Generation Logic Synthesis: A Scalable Neural Circuit Generation Framework, NeurIPS 2024.

We thank the reviewer for their time and effort in helping to improve the paper. We address the reviewer’s concerns below:

Clarifying Model Evaluations

We would like to clarify that our experiments already evaluate multiple configurations, not just a single model. Specifically, we sweep across four depths (6, 8, 10, 12) (Figure 3) and nine $\tau$-values (0.01–2.0) (Figure 5) at a fixed width of 256k. These experiments form the basis for our analysis of depth scaling and $\tau$’s role in loss smoothing and the discretization gap. We will make this more explicit in the revised draft (Section “Gap Scaling with Depth”) to avoid any ambiguity.

Clarifying Terminology (DLGN vs. LGN)

Since our paper centers on the discretization gap, the distinction between Differentiable Logic Gate Networks (Differentiable LGNs, DLGNs), used during training, and fully discretized Logic Gate Networks (LGNs), used during inference, is critical. Some comments (e.g., regarding Lemma 1) appear to reference LGNs when DLGNs were intended. We will make this distinction clearer in the paper and correct mistakes.

Clarifying Novelty and Relation to Deep Stochastic LGN: Y. Kim, "Deep Stochastic Logic Gate Networks"

We thank the reviewer for pointing out Y. Kim, "Deep Stochastic Logic Gate Networks," in IEEE Access, vol. 11, pp. 122488-122501, 2023. We will add it to the related papers for completeness and highlight the differences to our paper. While both papers use variations of Gumbel-Softmax, our contributions differ substantially. Our work focuses on analyzing the discretization gap and its scaling behavior, framing DLGNs as a Neural Architecture Search problem. In particular, Kim (2023) does not study the gap or its scaling, nor does it analyze convergence speed or neuron utilization, all central to our theoretical and empirical results. Beyond scope, Kim (2023) adopts a different Gumbel-Softmax formulation and neuron parameterization that make its results incompatible with our analysis:

1. it omits the $\tau$-parameter (central to Lemma 1 and Fig. 5),
2. normalizes logits (as shown in Kim (2023) this increases entropy over the induced distribution, which would worsen the discretization gap), and
3. introduces a learnable Gumbel scaling parameter that alters the Hessian-trace term. These choices place Kim (2023) in a different training regime, so our theoretical and empirical results do not apply to their setup.

On Gumbel Noise and Regularization

Kim (2023) suggests Gumbel noise prevents overfitting, drawing on Bishop’s result that training with noise approximates Tikhonov / L2-regularization. However, that derivation assumes zero-mean Gaussian noise, not asymmetric Gumbel noise. Moreover, our results (Fig. 3) show DLGNs continue improving with training as the discretization gap narrows, without the accuracy decline characteristic of overfitting. This suggests that overfitting prevention is not the reason our model works.

We hope this clarifies the distinctions between our work and Kim (2023). Below, we address the reviewer’s remaining questions.

1. We believe the reviewer is referring to Differentiable Logic Gate Networks (DLGNs), as clarified in our terminology section, since LGNs denote the discretized (post-training) networks. In Figure 6, we visualize the loss landscapes of DLGNs and GLGNs, trained on the setup from Figure 3 (left). We will update the caption to clarify that $\tau = 1$ was used for the DLGN. Even at this baseline $\tau$ value, GLGNs already exhibit a visibly smoother landscape, supporting our core claim about Gumbel-induced smoothing. Lemma 1 formalizes this effect, showing that the expected trace of the Hessian scales like $1/\tau^2$, meaning smaller $\tau$ implies stronger implicit smoothing. This is also reflected empirically in Figure 5. We agree that additional $\tau$-specific loss visualizations could serve as a useful sanity check. However, we believe that the combination of the $\tau$-sweep (Figure 5), the visual comparison (Figure 6), and our theoretical derivation (Lemma 1) already provides a comprehensive view of the relationship between $\tau$, smoothing, and optimization performance.

2. Figure 4 compares training modes in our STE ablation. As stated in lines 235–236, Gumbel Soft refers to a DLGN with Gumbel noise applied to logits or equivalently a GLGN trained without STE. The labels (soft) and (discrete) correspond to the softmaxed network and its fully discretized LGN version. We will clarify this in the figure caption. This directly supports the statements in lines 49–50 and aligns with Lemma. Smoothing from Gumbel noise improves convergence. Notably, adding STE slows convergence, confirming that it helps reduce the discretization gap but is not the main driver of faster training.

3. As shown in Figure 4, Gumbel noise alone significantly reduces the gap between soft and discrete performance, whereas adding STE to Gumbel noise effectively removes it entirely. We will revise the abstract accordingly to reflect this distinction more clearly.

4. From our experiments it does seem to be the case that the models get caught in a local optimum when tau is small. And while the Gumbel noise does significantly help with convergence, GLGNs still have some of the same issues with vanishing gradients as observed in the original works by Petersen et al.

Additional Evaluations

We agree that broader evaluation is important, and we have now added results on CIFAR-100 and a wider network to support generalization and scalability claims.

On CIFAR‑100 (depths 6–12, width 256k), we observe the same trend as Figure 3: the discretization gap grows with network depth for DLGNs, while GLGNs remain stable. For the largest model, the DLGN has not converged after 1M iterations, reaching a test accuracy/gap of 6.11% / 3.36% points, whereas the GLGN converges in ~400k iterations with 0.23% / 0.0099% points. CIFAR‑100’s complexity suggests that larger networks are required, where we expect the discretization gap to become even more significant for DLGNs, making GLGNs critical for scaling.

For a very wide CIFAR‑10 network (depth 6, width 2048k), the DLGN gap remains high (drop of 2.5% points of test accuracy at convergence) and does not shrink significantly over training, whereas GLGNs continue reducing the gap (drop of 0.188% points of test at convergence), consistent with our main results.

We thank the reviewer for their response. We would like to respectfully clarify key points to ensure an accurate evaluation.

Kim (2023) does not mention or analyze the discretization gap, its scaling properties, or convergence behavior. This is the point of our work.

In this paper, only a single model is tested (depth: 12, width: 256k)

This point was addressed in our rebuttal, but not clarified by the reviewer in the response. We would appreciate if the reviewer still believes this is a concern.

Under limitations, the reviewer notes

As noted by the authors, the models should be tested over more datasets.

We added results for an additional dataset (CIFAR-100). If this remains a concern, we would be grateful for further clarification.

while new concerns have emerged.

We are unsure which new concerns the reviewer is referring to here. We would also appreciate clarification here.

... DSLGNs employ a learnable temperature parameter $\tau$

We respectfully note that Kim (2023) does not use a learnable temperature parameter. The learnable parameter ($\beta$ in the notation of Kim (2023)) scales the Gumbel noise, and is not equivalent to our $\tau$-parameter (and the temperature in the canonical Gumbel-Softmax parametrization). We have elaborated further in our general comment.

The key distinction lies in whether $\tau$ is fixed during training or not.

We believe there may be a misunderstanding about the role of $\tau$ in our work. The reviewer suggests the key distinction is whether $\tau$ is fixed during training, our point is more fundamental: we treat $\tau$ as a conceptual parameter that governs smoothness and the discretization gap, precisely the phenomena that Kim (2023) does not study. Some confusion may stem from the fact that our $\tau$ parameter is called $\lambda$ in Kim (2023). We will make this clearer in the final version. In addition, we kindly remind the reviewer that we raised two additional meaningful architectural differences.

Fundamentally, both methods are modifications of the Gumbel-Softmax mechanism and share the same underlying principles.

As stated in the rebuttal, we agree on this point. However, one of the main modifications would increase the entropy of the logic gate distributions. We refer to Figure 2 in Kim (2023). High entropy implies a higher divergence between the network during training and inference. This would increase the discretization gap (our Figure 7).

We would gently reiterate the importance of consistent terminology. For example, the reviewer refers to our work as “GSLGN”, whereas we consistently use GLGN or Gumbel LGN.

I appreciate the authors' time in clarifying my comments. Unfortunately, not all questions have been resolved and answered. Among them, the key issues are:

• Most importantly, the authors do not respond to the critical issue of isolating $\tau$ between the loss function and Gumbel noise perturbation. The lack of isolation makes the authors' interpretation inaccurate and consequently their claims unfounded. Temperature significantly affects the loss function's shape through $\mathcal{L}(\text{softmax}(z/\tau))$, which substantially impacts logit uniformity (acting as a normalization term) and final loss values. Therefore, the discretization gap shown alongside the loss landscape is not solely determined by Gumbel noise. As previously mentioned, the authors must address the $\mathcal{L}(\text{softmax}(z/\tau))$ component. Convergence behavior is also affected by these factors.

• The distinction between temperature $\tau$ and $\beta = \frac{1}{\tau}$ appears negligible (e.g., 1/0.25=4).

• Even though the authors emphasize the difference in formulations, the reviewer is not questioning whether the formulations are actually distinct. The key point is that both methods apply Gumbel-noise-based softmax functions and aim to synchronize one-hot-based sampling between training and inference phases to reduce the discrepancy, termed "discretization gap" by the authors. Gumbel-softmax was not proposed by the authors and is widely used for this purpose. Since the authors have not modified Gumbel-softmax, comparing formulations is irrelevant.

We thank the reviewer for their continued engagement and for raising this point again. We would like to clarify the role of the $\tau$ parameter in Lemma~1.

As stated in the paper, the expected loss scales as $\frac{\pi^2}{12\tau^2} \mathrm{tr}\big(H_f(z/\tau)\big).$ So as the temperature $\tau$ increases, the coefficient $1/\tau^2$ decreases, reducing the degree of implicit smoothing. We illustrate this with two representative choices: Small $\tau$ (e.g., 0.1) $\implies 1/\tau^2$ is large $\implies$ large smoothing, flat minima. Large $\tau$ (e.g., 2.0) $\implies 1/\tau^2$ is small $\implies$ almost no smoothing.

As a result, adjusting the temperature $\tau$ offers a mechanism to control the strength of this curvature-aware regularization, the convergence of the model, and to reduce the discretization gap implicitly.

Table 1 complements this theoretical insight by showing the practical effect of different $\tau$ values on convergence speed and final accuracy.

We would be grateful if the reviewer could kindly specify any further aspects regarding the role of $\tau$ they would like us to clarify.

We thank the reviewer for their time and effort in providing valuable feedback to improve the paper.

Claim on the number of unused gates

We agree that the terminology “unused gates” can be easily misinterpreted, as the neurons/gates are still processing the input during training and post-discretization. We use the 2.5% as a heuristic based on a standard two-sided 95% confidence, i.e. neurons whose entropy remain high and statistically similar to initialization. We will clarify this in the revised draft. Perhaps a term like “dead” or “inactive” neuron would better convey this idea.

Extension of Lemma 1 to general $d \in \mathbb{N}$

Lemma 1 holds for arbitrary $d \in \mathbb{N}$. Indeed, it would also apply to the general Gumbel-Softmax setting beyond DLGNs. We agree that stating the lemma in its full generality makes more sense - we will change this in the draft.

Introduction to LGNs

We thank the reviewer for pointing this out. We agree that the current introduction could be made more accessible, especially for readers unfamiliar with LGNs. In response, we will revise the introductory paragraph to clarify the LGN setup and improve readability without increasing length. Additionally, we will explicitly reference [25] as a recommended resource for readers seeking a more in-depth overview of LGNs. We appreciate the suggestion, as it meaningfully improves the paper’s approachability.

Mathematical typesetting

We will carefully revise the mathematical typesetting, addressing the noted issues (colons, Lemma 1 phrasing, and Algorithm 1’s input descriptions). We will ensure all variables are properly typed and all mathematical expressions are clearly presented in the revised draft.

On References and Citations

We thank the reviewer for this suggestion. We agree that citing foundational works (e.g., Bengio et al. for the straight‑through estimator, Adam, and other standard methods) will make the paper more polished and complete. We will add these references in the revised draft.

Lemma 2 typo

We apologise for this mistake and will correct it.

Once again, we would like to thank the reviewer for a thorough and thoughtful review of our paper!

We thank the reviewer for their time and effort in helping to improve the paper. We will address the weaknesses and questions raised in the review.

Advantages of GLGNs over NNs.

In both DLGNs and GLGNs, all gates remain in the final LGN architecture and are active during inference. Thus, the benefits of logic-gate-based computation (e.g., energy efficiency) apply equally to LGNs induced by either method. We refer to estimates from Petersen et al. [2], where a CLGN deployed on a Xilinx XC7Z045 FPGA performs inference on CIFAR‑10 in 9 ns per image achieving 60.38% test accuracy. Assuming a peak power consumption of 15 W, this corresponds to approximately 135 nJ per image. Although the CLGN architecture differs from our models, it shares the same neuron parameterization as DLGNs, so we expect a Gumbel-parameterized CLGN to have comparable inference-time energy characteristics.

On Advancing the Practical Availability of LGNs

As the reviewer notes, faster DLGN training directly advances the practical availability of LGNs. Our experiments also show that the discretization gap scales with architecture size and the number of logic gates. For more complex tasks, where larger and more expressive networks are needed, Gumbel LGNs makes scaling LGNs more feasible with less prohibitive training costs.