Factor Graph Optimization of Error-Correcting Codes for Belief Propagation Decoding

We thank the reviewer for the feedback and insights.

Our work builds upon the original Judea Pearl’s BP algorithm, published in AAAI82 as a marginalization method for Bayesian networks. Submitted to ICLR under the “probabilistic methods” area, our manuscript introduces a novel method for learning the structure of Bayesian networks (graph connectivity), as outlined in lines 64-70. Structure learning is a well-established subdomain of probabilistic graphical models. The error-correcting codes (ECC) application is chosen because it is one of the most prevalent applications of BP.

We address below the remarks about ECC. However, the work is best judged as a machine learning work submitted to a machine learning conference, which includes novel technical contributions including our graph connectivity masking paradigm and a novel optimization framework.

Weaknesses

1. Limited contribution: paper is too narrow for ICLR…deserves publication…more appropriate for a coding conference such as ISIT or ITW.

We respectfully disagree. As noted in our introduction, this work is the first to propose a gradient-based method for learning Bayesian networks/codes, as other reviewers also recognized.

2. See question 2.

3. See Question 3.

Questions

1. Sections 3 and 4 represent half of the paper. Can the reviewer point to specific complex presentations?

2. Our work introduces a new learning paradigm aimed at improving the performance of a given initialization of the factor graph/code. As with any first-order method applied to non-convex objectives, the method is sensitive to initialization and generally converges to local minima. Thus, the primary focus of this work is to present an efficient and empirically effective method, rather than exhaustively testing all sparse code variants.

In response to the reviewer’s recommendation, we conducted experiments with state-of-the-art (SOTA) 5G NR LDPC codes. The BER and BLER vs. SNR results, presented in Appendix L of the revised manuscript, demonstrate that our method can enhance these SOTA codes, highlighting the potential of our data-driven approach in modern ECC settings.

A comparison with SCL is now also included in Appendix I of the revised manuscript. Our results show that even in the challenging short-length setting, where achieving sparsity is difficult, our method significantly improves over existing low-density codes and approaches the ML bound with very few iterations, even under suboptimal initialization. With good initialization (using a sparse LDPC code), our method achieves state-of-the-art performance.

3. The chosen metric and display method follow established norms in the field of ECC published in the AI community. This metric emphasizes the exponent, is easily inverted ($BER = e^{-BER_{-ln}}$), and enables compact presentation across multiple codes and scenarios (18 codes on different channels).

Based on the reviewer’s suggestions, we have added BER and BLER vs. SNR curves in Appendix J of the revised manuscript. Additionally, Appendix L provides similar curves for short state-of-the-art 5G NR LDPC codes, demonstrating that our method outperforms these codes as well.

4. Our optimization framework primarily optimizes BER rather than BLER using cross-entropy loss, but this does not seem to significantly impact performance due to BP’s global marginalization effects. The figures are part of our ablation study, illustrating various impacts and effects of and on our method.

• Figure 3 shows that initialization plays a critical role in convergence. Interestingly, as shown, increased initial sparsity does not necessarily improve performance (e.g., at $p = 0.1$).
• Figure 4: Although loopy belief propagation is widely known beyond coding theory, empirical validation is needed to show that stochastic gradient descent (SGD) enforces sparsity through its inductive bias [1]. Figure 6 further supports that sparsity regularization has minimal impact on convergence, suggesting that gradient descent optimization and BP have similar sparsity enforcement effects.

These points are clarified in the revised manuscript.

[1] Soudry et al., The implicit bias of gradient descent on separable data, JMLR18

Dear Reviewer 7LZc,

As we near the conclusion of this year's discussion period, we would like to extend our gratitude for your thoughtful feedback, which has significantly enhanced our manuscript. We believe we have addressed the majority of the concerns raised by you and the other reviewers. In particular:

• We added BER and BLER vs. SNR curves in Appendices J and L. These curves allow a better visualization of our method’s impact.
• We added a performance comparison with SCL in Appendix I. The comparison shows our method can provide state-of-the-art sparse codes even on extremely short block lengths.
• We added performance on short 5G NR LDPC codes in Appendix L. The results demonstrate that our method can also outperform state-of-the-art short sparse codes.
• We added the codes’ column weight distribution in Appendix K. Our method introduces a distribution shift that adapts to BP's inductive bias in the short block length regime.

We would greatly value any further feedback you may have.

We thank the reviewer for the response.

I have evaluated your work as a machine learning paper with an application to error-correcting coding.

While we appreciate the reviewer's perspective, we note that their comments focus solely on classical ECC visualization and performance metrics for specific codes, without addressing the method itself. Our method generates codes for which BP is significantly better than the input codes, which required decades of research, using just a few hours of optimization on relatively weak GPUs.

Standard metrics plotted directly against SNR are universally recognized

We adopted the widely used tabular notation following the conventions established in numerous works across both IT and ML communities, including most if not all neural decoder contributions we are aware of.

This notation provides a clearer and more compact presentation of performance across multiple codes. Presenting results for 14 codes on three different channels in a single manuscript using traditional plots would be infeasible. Moreover, the numerical approach enables precise reconstruction of plot values, while extracting numerical values from plots for future reference is not exact.

While this practice is commonplace in many of the published work, we have included the requested plots to address the reviewer’s concern.

your method does not achieve better performance than state-of-the-art (SOTA) decoders.

1. Our method focuses on optimizing codes for the BP decoder and not on developing or training decoders. This is the essence of the work – using a novel method for learning Bayesian networks such that the BP becomes more effective. Again, our method is not about training a decoder.
2. We find the reviewer’s claim that Appendix J figure does not demonstrate our method’s ability to improve 5G NR LDPC SOTA difficult to reconcile with the evidence presented. We improve across all codes, except for the 3rd one, for which we obtain identical results.
3. We now also provide here the performance of the method on the 5G NR LDPC codes on other channels. For the (64,32) code, our optimized code shows a clear superiority across all three channels types (including the one already in appendix J). For (128,64) the results are very similar for all three channels. However, see point 4 below.
4. As extensively mentioned in the paper, the method is focusing on short-length codes (Sec 2), an important and challenging topic in modern telecommunications, particularly for BP decoding. From a hundred code-length the method requires perturbed gradients in order to extract from local minima (good codes), e.g., see lines 430,903.

Comparing your method to BCH and Polar codes with BP decoding is not meaningful

1. These codes serve as a clear demonstration of the descent capabilities of our optimization framework. Inferring an effective direction and algorithm for stochastic, non-convex, and non-differentiable optimization problems is highly non-trivial. We show that we are able to improve those codes (for the BP decoder) even though they were engineered to have a specific structure with favorable properties.
2. Neural BP manuscripts that use BCH and Polar codes have garnered thousands of citations from both the IT and ML communities, underscoring the widespread acceptance of benchmarks similar to those used in our manuscript.
3. Of the 14 codes tested in our original manuscript, 10 are sparse. Why focus on the BCH and Polar codes, if you believe these are not meaningful? Following the reviews, there are also three additional SOTA sparse codes that were added to the revised manuscript (Appendix J).

It is unclear what list size you used for the successive cancellation list (SCL) decoding of polar codes. Based on the results, it appears that you may have used a small list size, which significantly weakens the performance of SCL and renders the comparison unfair.

We clearly wrote in line 976: “The first and the second row of the SCL algorithm denote performance with a list length of 1 and 32 respectively.”

While your work introduces some interesting ideas, it does not demonstrate broad impact or significant advancements relative to existing methods to merit acceptance at a top-tier conference like ICLR... better suited for a more specialized venue...

1. Assuming the ICLR community is familiar with graphical models, our work is the first method to provide a first-order optimization method of Bayesian networks.
2. We do demonstrate the large superiority of our method for the vast majority of the factor graph initializations.
3. The review and follow-up responses focus on highly technical aspects, and very specific results, judged from a very narrow context, and focusing on a small portion of the results.
4. We believe IT conferences are not necessarily more adapted at judging graphical models and modern optimization methods than ICLR or other ML venues.

We thank the reviewer for the positive feedback and insightful comments.

Computational Complexity

Regarding the gradient computation, we indeed require gradients of size $n \times (n - k)$ to perform the first-order optimization. Our line search reduces the infinite search space to $n \times (n - k)$. However, due to the local proximity of the optimum to the working point, we limited the search to 50 steps in our experiments (see line 322 and Appendix C).

This complexity is minimal and manageable compared to training modern neural networks with up to trillions of parameters (as suggested by recent work in the field of ECC). Furthermore, our method is the first gradient-based optimization approach for designing codes and factor graphs, contrasting with prior heuristic or search-based methods. It requires an average of one hour of training for the tested codes (line 323) and is up to 140 times faster than existing search-based methods, such as the genetic algorithm in [Elkelesh et al., 2019] (line 409). Notably, all computational cost is incurred only during training, with no added cost to BP’s inference algorithm.

Impact of Initialization

The impact of initialization in our method stems from using first-order oracles to optimize non-convex objectives (cf. lines 20, 297). This challenge is common in first-order methods, including neural network optimization. We tested multiple initializations (sparse, dense, random codes) and observed that sparse initializations yielded the best results, positioning the algorithm near better local minima. However, for larger codes, the method can struggle to improve due to this pre-existing local minimum, as it may require escaping a potentially suboptimal extremum (line 425). This point is now emphasized in the paper’s conclusions.

L1 Regularization

If we misinterpreted the reviewer’s remark, please clarify. Given that $H_{ij}=bin(\Omega)_{ij}\in \{0,1\}$, the effects of L1 and (squared) L2 norms are similar. However, applying different regularization factors to weights and columns could yield unique impacts.

We note here that in Figure 4, we provide empirical evidence that SGD enforces sparsity through its inductive bias [1]. Figure 6 further suggests that sparsity regularization has minimal impact on convergence, implying that gradient descent and BP exert similar effects in promoting sparsity. Additionally, any constraint can be incorporated into the optimization problem.

Weighting of Iterations

We appreciate the reviewer’s recommendation regarding iteration weighting. Weighting the objective as suggested by Nachmani et al., (2016) using a multi-loss objective across iterations may indeed improve performance. We now added this reference. We note here that due to computational constraints, we optimized our model with only five iterations (line 320).

[1] Soudry et al., The implicit bias of gradient descent on separable data, JMLR18

We thank the reviewer for the valuable feedback and constructive remarks.

Missing performance on 5G LDPC codes.

Our work introduces a new learning paradigm aimed at improving the performance of a given initialization of the factor graph/code. As with any first-order method applied to non-convex objectives, the method is sensitive to initialization and generally converges to local minima. Thus, the primary focus of this work is to present an efficient and empirically effective method, rather than exhaustively testing all sparse code variants.

In response to the reviewer’s recommendation, we conducted experiments with state-of-the-art (SOTA) 5G NR LDPC codes. The results, presented in Appendix L of the revised manuscript, demonstrate that our method can enhance these SOTA codes, highlighting the potential of our data-driven approach in modern ECC settings.

Comparison with Elkelesh et al.

We are unsure what is the concern.

We implemented the Genetic Algorithm (GA) method from [Elkelesh et al., 2019] (originally in MATLAB) and conducted a direct comparison under the same conditions as our method, training both methods on 15 iterations and testing on 5 and 15 iterations.

The $k$ iterations mentioned refer to the number of GA training iterations. As a genetic algorithm, this approach is slower and limited in its beam-search capacity.

We also note in line 900 that combining our method with GA could help avoid poor local minima (similar to the perturbed gradients in Appendix A), especially with good initialization.

The proposed method appears to be a local optimization method highly dependent on the initial $H$ matrix

The sensitivity to initialization stems from using first-order oracles for the optimization of non-convex objectives (see lines 20, 297, 422, 426, 803, and 901). This limitation is inherent to all first-order methods in non-convex optimization, with neural network optimization being the most ubiquitous illustration.

Experimental results show that local convergence can yield substantial improvements, even with SOTA sparse codes (Appendix L).

Questions

1.Figure 2 shows the performance improvement in dB across various sparse codes, providing a compact summary (14 codes tested) of the conventional BER vs. SNR plots. In response to the reviewer’s suggestion, we added BER and BLER vs. SNR plots for various codes and channel configurations in Appendices J and L of the revised manuscript.

2.The definition is provided in line 327.

4.This experiment is part of the ablation study (Section 6) and demonstrates the method’s ability to integrate constraints/regularizations and their potential beneficial impact. The optimization framework can accommodate any structural constraint or desired code property.

5.LDPC codes, known for asymptotic optimality and efficient decoding, are relevant in modern communication, where short codes are increasingly important ([Liva et al., 2016], [1], lines 82-85). Supporting efficient short LDPC codes would enable a single decoder (BP) to be used on a device instead of deploying multiple decoders, as required in modern standards like 5G (short Polar codes and longer LDPC codes [ESTI, 2021]).

A comparison with SCL is now included in Appendix I of the revised manuscript. Our results show that even in the challenging short-length setting, where achieving sparsity is difficult, our method significantly improves over existing low-density codes and approaches the ML bound with very few iterations, even under suboptimal initialization. With good initialization (using a sparse LDPC code), our method achieves state-of-the-art performance.

7.Sparsity statistics are provided in Figure 5. The optimization does not affect the code’s girth (line 458), and calculating the minimum distance remains infeasible for these codes.

Per other reviewers’ suggestions, we now include the column weight distribution in Appendix K. We welcome further feedback on important analyses to include.

[1] Coşkun, Mustafa Cemil, et al. "Efficient error-correcting codes in the short blocklength regime." Physical Communication, 34 (2019): 66-79.

Dear Reviewer j4us,

As we near the conclusion of this year's discussion period, we would like to extend our gratitude for your thoughtful feedback, which has significantly enhanced our manuscript. We believe we have addressed the majority of the concerns raised by you and the other reviewers. In particular:

• We added BER and BLER vs. SNR curves in Appendices J and L. These curves allow a better visualization of our method’s impact.
• We added a performance comparison with SCL in Appendix I. The comparison shows our method can provide state-of-the-art sparse codes even on extremely short block lengths.
• We added performance on short 5G NR LDPC codes in Appendix L. The results demonstrate that our method can also outperform state-of-the-art short sparse codes.
• We added the codes’ column weight distribution in Appendix K. Our method introduces a distribution shift that adapts to BP's inductive bias in the short block length regime.

We would greatly value any further feedback you may have.

We thank the reviewer for the answer.

Local Optimization Limitation and not finding globally optimized codes.

Non-convex (stochastic and non-differentiable) global optimization is an NP-complete problem. The reviewer’s requirement for (finite time) global optimization is reducible to solving the most important open problem in computer science.

The main body of the paper should emphasize LDPC codes, as the work primarily utilizes the BP decoder. Instead of focusing on polar/BCH codes with a BP decoder, it would suffice to compare polar codes using an SCL decoder, given the focus on short-length codes.

1. These codes serve as a clear demonstration of the descent capabilities of our optimization framework. Inferring an effective direction and algorithm from stochastic, non-convex, and non-differentiable optimization problems is highly non-trivial. We show that we are able to improve those codes (for the BP decoder) even though they were engineered to have a specific structure with favorable properties..

2. Neural BP manuscripts that use BCH and Polar codes have garnered thousands of citations from both the IT and ML communities, underscoring the widespread acceptance of benchmarks similar to those used in our manuscript.

3. Of the 14 codes tested in our original manuscript, 10 are sparse. Why focus on the BCH and Polar codes, if you believe these are not meaningful? Following the reviews, there are also three additional SOTA sparse codes that were added to the revised manuscript (Appendix J).

4. The reviewer may have overlooked the revision. We have now included a comparison with SCL in Appendix I of the revised manuscript. Our results show that even in the challenging short-length setting, where achieving sparsity is difficult, our method significantly improves over existing low-density codes and approaches the ML bound with very few iterations, even under suboptimal initialization. With good initialization (e.g., using a sparse LDPC code), our method achieves state-of-the-art performance.

Consideration of Constraints

As an algorithmic framework targeting the ML community under the graphical models paradigm, our work is not designed to address the sum of all of industrial constraints. Such a requirement deviates significantly from the established benchmarks and scope of neural coding/decoding research.

lack of readability

The claim about readability issues is new, and did not appear before (!). We invite the reviewer to specify which sections and what simultaneously addressed points they find unclear so that we may address them effectively. If this refers to the two issues below, please consider these fixed.

I misunderstood the use of the term iteration (line 862).. While this was my error, other readers might also misinterpret it.

This line refers to the genetic algorithm baseline and not our method. Genetic algorithms are iterative by definition, and at each iteration a generation is created. We will refine the terminology in the revised manuscript so that it is clear that an iteration of the BP method is not the same as an iteration of the genetic algorithm.

definition on line 327 remains unclear

It refers to the degrees' sparsity level, i.e., $pn$ with $p\in (0,1)$ referring to the percentage and $n$ the number of variables nodes. We will make this definition clearer.

We thank the reviewer for their valuable feedback and insights.

Motivations for Short Codes

The primary motivation for short codes is practical, as outlined in lines 82-85. Alongside the survey by [Liva et al., 2016], further motivations are detailed in [1]. We summarize here the main reasons:

1. Larger blocklength codes approaching the capacity can be only marginally improved.
2. Short and medium blocklength codes are increasingly important in emerging applications, including IoT, smart metering networks, remote command links, and messaging services.
3. From a systemic perspective, larger computational resources are required to explore the optimization of larger codes.

Further explanations on these motivations are provided in the revised manuscript.

[1] Coşkun, Mustafa Cemil, et al. "Efficient error-correcting codes in the short blocklength regime." Physical Communication 34 (2019): 66-79.

BER vs. SNR Curves

The chosen metric and display method follow established norms in the field of ECC published in the AI community. This metric emphasizes the exponent, is easily inverted ($BER = e^{-BER_{ln}}$), and enables compact presentation across multiple codes and scenarios (18 codes on different channels).

Based on the reviewer’s suggestions, we have added BER and BLER vs. SNR curves in Appendix J of the revised manuscript. Additionally, Appendix L provides similar curves for short state-of-the-art 5G NR LDPC codes, demonstrating that our method outperforms these codes as well.

SCL Decoding Performance

A comparison with SCL is now included in Appendix I of the revised manuscript. Our results show that even in the challenging short-length setting, where achieving sparsity is difficult, our method significantly improves over existing low-density codes and approaches the ML bound with very few iterations, even under suboptimal initialization.

With good initialization (using a sparse LDPC code), our method achieves state-of-the-art performance.

PCM Analysis

Sparsity statistics are provided in Figure 5. The optimization does not affect the code’s girth (line 458), and calculating the minimum distance remains infeasible for these codes.

Per the reviewer’s suggestion, we now include the column weight distribution in Appendix K. We welcome further feedback on important analyses to include.

[Minor] Generator Matrix

Our method preserves the symmetry property of BP, making it independent of the generator matrix. However, the initial formulation supports alternative optimization schemes that could break this symmetry and lead to non-invariance with respect to the generator matrix (e.g., by optimizing Trellis graph connectivity). This clarification is added to the revised manuscript.

Dear Reviewer GzCC,

As we near the conclusion of this year's discussion period, we would like to extend our gratitude for your thoughtful feedback, which has significantly enhanced our manuscript. We believe we have addressed the majority of the concerns raised by you and the other reviewers. In particular:

• We added BER and BLER vs. SNR curves in Appendices J and L. These curves allow a better visualization of our method’s impact.
• We added a performance comparison with SCL in Appendix I. The comparison shows our method can provide state-of-the-art sparse codes even on extremely short block lengths.
• We added performance on short 5G NR LDPC codes in Appendix L. The results demonstrate that our method can also outperform state-of-the-art short sparse codes.
• We added the codes’ column weight distribution in Appendix K. Our method introduces a distribution shift that adapts to BP's inductive bias in the short block length regime.

We would greatly value any further feedback you may have.

We thank the reviewer for the feedback and remarks.

1.i. Novelty wrt [Choukroun & Wolf, 2024a]

The objectives in both papers describe the same standard channel coding pipeline. While [Choukroun & Wolf, 2024a] describes the end-to-end optimization of differentiable neural decoders (i.e., code and decoder), our method optimizes the code for the classical Belief Propagation decoder. Besides the regularization term, it can be observed that the expectation, the objective, and the optimization variables are different.

Our setting is fundamentally different (lines 201-207) since unlike [Choukroun & Wolf, 2024a] our decoder cannot be differentiably optimized as it requires the structure learning of the underlying graph/algorithm.

1.ii. Novelty wrt neural BP

Neural/weighted BP should never be implemented in a tensor form since it would lose all of the efficiency advantages of message-passing algorithms (sparsity).

We adopt the tensor form in order to learn the connectivity itself upon a complete factor graph. We are not learning the weighting of the variable nodes’ messages obtained upon a given fixed graph connectivity as in neural BP (line 48).

This is a very fundamental difference with neural BP.

1. mathematical or analytical contributions

From the ML perspective, the method develops new tools of high interest, such as a new way to optimize graphs for BP (not just for codes, i.e., structure learning) and a novel line search procedure tailored for quantized optimization. The information theoretical aspects of codes are not the main emphasis of our work. As a gradient descent method, the proposed framework's performance is tied to the initial graph, and this is emphasized repeatedly in the paper (cf lines 20, 297,422,426,803,901). Further analysis of the optimization over the cross-entropy loss can be done via classical convergence analysis of quantized networks [2], but this is beyond the scope of our work.

2. “Comparison with line-search methods” … “LS methods are used for non-convex optimizations wo assumptions…

Existing methods:

Inexact line search methods can be used for non-convex optimizations to find local minima. The existing zero/first-order line-search methods operate on a predefined continuous range (generally small) on which the search is performed.

In our binary setting, a continuous range is impossible to set as it can grow arbitrarily large according to the $\Omega$ values. Moreover, a standard line search would find local minima of the line-search objective because of the suboptimal range and the discretization approximation.

Our method:

As a binarization-aware method, we provide a provably (line 304) optimal line-search method that requires $\mathcal{O}((n-k)\times n)$ parallelizable computations.

2. Effectiveness of the grid-search approach … depends on the initial matrix

The impact of initialization is related to the use of first-order oracles for the optimization of non-convex objectives (cf lines 20, 297,422,426,803,901).

This is, of course, not unique to our work: any first-order method on non-convex objectives would be impacted by initialization, learning neural networks being the most ubiquitous example.

3. How much is the method fast compared with what

The method is the first gradient-based optimization for the design of codes/factor graphs. All other existing methods are heuristics or search-based methods.

Our method requires a single hour of training on average on the tested codes (line 323) and is up to 140 times faster (and better) than the existing search-based method (a genetic algorithm) [Elkelesh et al., 2019] (line 409)).

Suggestions

The suggestions and typos will be added to the revision.

3.The expectation should be with respect to the discrete number of iterations of the BP algorithm in order to ensure optimal decoding under any computational constraints (line 199). This can be similar in a sense to the multi-loss of neural BP. A citation has been added.

4.No assumption on the generator matrix is given in the paper. As the method is built to maintain the symmetry property of BP, the optimization framework is not dependent on the generator.

7.We may be missing your point but this is the discretization of the expectation above (suggestion 3). We added a reference to [Nachmani et al. 2016] for reference to the multi-loss.

10.“Cost function for the optimal learning rate” indeed. For example, since the BER metric is not differentiable, one may search for optimal BER during the line search for optimal performance on the loss of interest (BER), instead of the Bayesian (uncertainty) cross-entropy metric.

11.A citation will be added as this terminology already exists, e.g., [1].

[1] Kurmukova and Deniz Gunduz. Friendly Attacks to Improve Channel Coding Reliability. AAAI24.

[2] Li et al. Training quantized nets: A deeper understanding. Neurips 2017.

Dear Reviewer 6Jjp,

As we near the conclusion of this year's discussion period, we would like to extend our gratitude for your thoughtful feedback, which has significantly enhanced our manuscript. We believe we have addressed the majority of the concerns raised by you and the other reviewers. In particular:

• We added BER and BLER vs. SNR curves in Appendices J and L. These curves allow a better visualization of our method’s impact.
• We added a performance comparison with SCL in Appendix I. The comparison shows our method can provide state-of-the-art sparse codes even on extremely short block lengths.
• We added performance on short 5G NR LDPC codes in Appendix L. The results demonstrate that our method can also outperform state-of-the-art short sparse codes.
• We added the codes’ column weight distribution in Appendix K. Our method introduces a distribution shift that adapts to BP's inductive bias in the short block length regime.

We would greatly value any further feedback you may have.

I appreciate the authors' response.

1.ii. Novelty wrt neural BP

Although the tensor form loses sparsity property at a glance, one can use sparse matrix operations. So, this is not a critical point. I agree that the current formulation is defined on a complete graph, but the reviewer considers that it is not so novel.

2.“Comparison with line-search methods” … “LS methods are used for non-convex optimizations wo assumptions…

Although the authors claim that the existing methods find local minima, this is also the case with the proposed method, as the authors repeatedly stated in the response.

The reviewer considers that some concerns that need to be resolved still exist in the paper. Thus, I keep my current score.