A Foundation Model for Error Correction Codes

We thank the reviewer for the very supportive review.

Code Length

As with most neural decoding works (cf. Related Works section), we focus on short-length codes for three main reasons.

(i) On very large codes, some decoders (e.g., BP, SC) have been proven to achieve performance close to ML decoding on their respective codes of interest (e.g., LDPC, Polar, respectively), rendering the potential advantage of neural decoding solutions nonexistent/irrelevant.

(ii) The training and/or deployment of the existing neural decoding techniques is not trivial/straightforward for very large codes (i.e., code lengths of thousands of bits).

(iii) The emergence of applications driven by the Internet of Things created the requirement for optimal decoders of short to moderate codes. For example, 5G Polar codes have code lengths of 32 to 1024 for uplink (UCI) or 128 to 512 for downlink [1]. These points are emphasized in the revision.

[1] ETSI 3GPP TS 38.212 “5G NR Multiplexing and channel coding”, v.16.5.0, 2021-03.

Interpretability

We provide in the revision’s Appendix G the depiction of typical learned self-attention matrices. As can be observed, initially the attention is mostly focused on the syndrome values in order to perform the decoding, while in the final layer, the focus is also transferred to the information bits.

Integration

While being out of the scope of the paper, the deployment of the FECCT is indeed a challenge, requiring the application of many deep-learning acceleration techniques such as quantization (binarization) and sparsification (mainly of the self-attention module) as well as its FPGA/ASIC integration.

We thank the reviewer for the valuable feedback. We believe several points have been misunderstood and we have modified the manuscript to make it clearer.

“The reason is that error correcting codes are designed and deployed once and their training is not a big deal even if someone takes deep neural network decoders as opposed to classical BP methods”

While we agree with the reviewer that training is not a major limitation compared to deployment, we respectfully disagree with the reviewer regarding the integration/deployment side. Current error-correction systems (e.g., base stations) can/need to decode codes of different lengths and/or different families. For example, LDPC codes of different lengths are decoded on HW by omitting some of the integrated processing engines of the BP implementation for shorter codewords.

Moreover, deploying one single neural decoder for several types of codes and/or lengths is fundamentally different from integrating several neural decoders, especially in terms of die space and storage/memory (one model’s weights vs many models’ weights).

We remind here that existing neural decoders are designed and trained on one single code to be integrated into dedicated HW.

This point is emphasized in the revision.

“However, for short codes, we really do not need foundational decoders as one can design and easily train specialized neural decoders for the short codes that will very likely beat the performance of generalist decoders for all lengths.”

Following our previous answer, the deployment of a specialized neural decoder has major integration drawbacks.

Most importantly, the main goal of our work was to show our method, while being code/length-invariant (referred to as “generalist decoder” by the reviewer), can match and even outperform the “specialized” state-of-the-art neural decoder.

In addition to the points above that address the direct utility, foundation models support downstream tasks. Specifically, the ability to move quickly between codes enables the sampling and evaluation of random codes. Moreover, since our method defines a universal differentiable neural decoder, it opens the door to the optimization and definition of new families of codes.

These points are emphasized in the revision.

“What is the performance in the tables?What about reported numbers as performance. What are they? I like to see how these reported numbers translate to the error rate performance”

As described in Section 5, third paragraph, the reported numbers are the “negative natural logarithm bit error rates (BER) for three different normalized SNR values (Eb/N0)”, meaning we provide $-\ln(BER)$ (such that higher is better) for three Eb/N0 values. This dense presentation for multiple codes of the BER on the SNR of most interest has been borrowed from many of the previous neural decoding works.

“please compare the proposed generalist decoder at short lengths (less than 200 bits) with that of specialized decoders at those lengths. Because as I pointed out this is where these foundational decoders would show value if any.”

While our work focuses on the development of neural decoders, the previous work cited by the reviewer (e.g., [1],[2]) already provided a comprehensive comparison with specialized decoders. In our work, we compare with it.

The revised text now points to these results.

“it would be helpful to compare your proposed work with that of Choukroun & Wolf (2022b) which extends and enhances ECCT via Denoising diffusion in error correction codes and have far superior performance than ECCT.”

We wish to remind the main difference between ECCT and DDECC, as described in Section 6 paragraph 2. While ECCT focuses on the design of a new neural decoding architecture, DDECC is a generic iterative decoding algorithm applied to a given backbone architecture. As described in [2] Section 4.5, the ECCT has been chosen since it is the state-of-the-art decoding architecture.

Our model focuses on the first family of work (i.e., decoding architecture) that can, same as ECCT, be coupled with DDECC and makes it code and length invariant too. While these results would be interesting to see, they require many more experiments and much heavier resources.

This point is emphasized in the revision.

“Can the authors elaborate on the novelty relative to ECCT.”

We believe the paper differs from ECCT as much as ECCT differs from the original Transformer architecture.

ECCT brings three modifications to the original Transformer architecture. (i) An initial bit-wise embedding (i.e., $n$ encoding vectors, with $n$ the code length). (ii) Thresholded binary masking. (iii) A Linear (FC) prediction layer.

FECCT redefined these three contributions in order to (1) make them code/length-invariant, and (2) improve over them (c.f. Table 3).

(i) The initial embedding is restricted to three embedding vectors, one vector for each bit to be modulated by the channel’s output magnitude and a one-hot encoding defined by the syndrome values.

(ii) The masking is totally different in its definition and implementation and is defined as $softmax(QK^{T})\odot \psi(\mathcal{G}(H))$ compared to $softmax(QK^{T}+g(H))$. Here $g(H)\in \\{-\infty,0\\}$ denotes ECCT’s binary thresholding, while FECCT provides a fine-grained masking based on the Tanner graph distances.

(iii) The prediction module is fundamentally different than a Linear layer and is based on the parity-check matrix in order to remain invariant while being code-aware.

“-please plot error rate plots rather than the reporting used in the paper which is not insightful.”

We now also provide BER curves for several codes in Appendix F.

References

[1] Error Correction Code Transformer, Y. Choukroun and L. Wolf, NeurIPS 2022

[2] Denoising Diffusion Error Correction Codes, Y. Choukroun and L. Wolf, ICLR 2023

Thanks authors for clarifying a few things. However, I beLieve you misunderstood my inputs. (1) I am questioning the value of this work in real world. Again, I do not debate its intellectual exercise, which is good in my mind. But I disagree with authors in their argument for its real world implications for the following reasons: for long codes, we already have very good capacity achieving codes (e.g., LDPC and polar) that are rate compatible so we don’t have to worry about their rate variations. The authors are likely aware that LDPS codes, for example, can be punctured randomly and create very good rate compatible codes with BP decoding near their ML performance. So we should all agree that no neural based codes are needed for such lengths. Te need is in short lengths from practical impact, that is where all the potential resides for neural solutions. I would be, however, impressed if the authors can find decoders that are good for very short lengths (specially for Reed Muller codes) as we do not have good decoders for such cases. But the issue is that the authors show in appendix F that Fine tuned ECCT (and even initial ECCT, for the most part) always perform better than their codes in short lengths. So how we can justify the need for generalist decoder from practical point of view?

Regarding hardware complexity of decoding various codes, if the environment is a private network, it is setup to operate on the rate variation of the same type of code so I am not sure we need generalist decoder for that. If it is wireless cellular network, each network provider has its own parameter settings and uses the same code, that are preset, so I do not see any complications arise for decoding. So from practical point of view, I would rather to see a work that contributes to take one rate adaptive code with very good decoder at various rates rather than a generalist decoder for various types of codes and lengths at these short lengths. Again, I acknowledge that the work is intellectually good but I am not convinced about the usefulness of this line of work.

Thank you for the prompt reply! We are happy to hear that we have been able to clarify some issues and are very encouraged by the reviewer’s acknowledgment of the value of the work as basic research with good intellectual value.

The need is in short lengths from practical impact, that is where all the potential resides for neural solutions.

We agree with the reviewer that short codes are (including our work) and should be the focus of neural decoders. In the revision, we have added a paragraph to the Related Work section discussing this point.

But the issue is that the authors show in appendix F that finetuned ECCT (and even initial ECCT, for the most part) always perform better than their codes in short lengths. So how we can justify the need for a generalist decoder from the practical point of view?

This is a misunderstanding that stems from an unfortunate typo we had in the figure captions (now fixed in red). We deeply apologize for this. There is no such thing as zero-shot or finetuned ECCT, since ECCT is trained for a single code. Appendix F, as well as the Tables in the main text, show the performance of the SOTA ECCT compared with the performance of the proposed Foundational ECCT (FECCT) on zero-shot (i.e., unseen codes) and finetuned codes (i.e., short training on a code of interest) tasks. The results, along with the ablation study, support the fact that our method, while being length/code/rate-invariant and having fewer parameters, is able to reach and even outperform the code-dedicated SOTA ECCT.

To be clear, for 100% of the cases, our finetuned version outperforms the ECCT. In a limited number of zero-shot runs, this is not the case. As described in Section 6, the discrepancy in performance between the different families of codes is due to the underrepresentation of some code families in the training set.

Each network provider has its own parameter settings and uses the same code, that are preset, so I do not see any complications arise for decoding.

We respectfully disagree. A network provider does not define its own parameters but must acquire/use telecommunication infrastructures and hardware that follow a given standard. For example, 5G supports both LDPC and Polar codes [1], meaning that modern systems (supporting 2G to 5G standards) must be able to decode (on device) different families of codes (e.g., BCH, LDPC, Polar). We can also find similar multiple decoders in Concatenated error correction code.

[1] ETSI 3GPP TS 38.212 “5G NR Multiplexing and channel coding”, v.16.5.0, 2021-03.

...”it is setup to operate on the rate variation of the same type of code”... “So from practical point of view, I would rather to see a work that contributes to take one rate adaptive code with very good decoder at various rates rather than a generalist decoder for various types of codes and lengths at these short lengths.”

Existing neural decoders are not rate-adaptive at all. For example, the state-of-the-art ECCT requires a new distinct model for each rate (or even length) of interest. Our neural decoder is absolutely rate-adaptive by construction. As far as we can ascertain, this is the first time a neural decoder is length/rate invariant, enabling the potential practical integration of one neural decoder for any rate/length.

If needed, further specialization of the foundation model can be obtained simply by training it on a single family of codes. Since this is against the basic scientific ambition to generalize, we did not attempt this, but it is a valid future direction.

To summarize: our method is both conceptually and architecturally novel. It has multiple advantages on short/moderate codes: (1) The method reaches and outperforms the SOTA neural decoder (even on unseen codes), (2) The method has a smaller capacity than the SOTA (i.e. fewer parameters), and (3) The method is totally length/rate/code-invariant.

We thank the reviewer for the ongoing discussion and for acknowledging, again, the intellectual value of our work. The remaining concern seems to focus not on the potential of the method but on its capability to provide universally superior decoding performance. Allow us to clarify our motivation with respect to your remarks.

and the clarification of your simulation error.

We wish to note we did not have any simulation error, just a typo in the caption of one of the rebuttal figures (“fine-tuned ECCT” instead of “fine-tuned FECCT”). As mentioned in the previous answer, there is no such thing as “fine-tuned ECCT”, since ECCT is trained from scratch on each code.

unless the authors show that their decoder performs better than any other decoder out there, which is not true! This seems to be the core of the reviewer’s concern and it is exactly what our foundation neural decoder line of work aims to achieve. The main dispute is whether a first work in this direction needs to be the ultimate solution.

We wish to discuss a few points.

1. Our method obtains state-of-the-art in neural decoding almost always for inference-time zero-shot (remarkably) on codes that are unseen during training, and across all experiments for (i) codes from the training set without further training and (ii) finetuned on unseen codes.
2. Previous neural decoders have been shown to outperform or almost match the performance of the best classical decoders (cf. [1,2] for LDPC or BCH codes (BP) and [3] for Polar codes (SCL)).
3. Our method, as a machine learning-based solution in general and a neural decoder in particular, may not be the “best possible universal decoder on every possible code” (no free lunch theorem). Still, our approach contributes significant new scientific advancements in pursuit of this objective, benefiting both the machine learning and error-control coding fields.
4. Our state-of-the-art method is the first neural decoder to remain code, rate, and length invariant, enabling the performance and deployment advantages mentioned in the manuscript and the previous answers.
5. As previously mentioned, if the universal decoding is not of practical interest, our method can be trained or fine-tuned to decode a few or several families of interest with high accuracy and potential efficient integration.
6. Finally and as mentioned by another reviewer, this is the first time a learned "decoding algorithm rather than a complicated mapping” is proposed, enabling both the analysis of the learned decoding and the potential differentiable design of new codes.

Can the authors provide the comparison state of art decoder for ReedMuller

We provided the performance of our model on six families of codes and many different rates. Reed-Muller codes (related to the Polar codes) are out-of-domain (OOD) for our model as they are not present in our training set, so it is an interesting experiment, especially since it has not been tested in previous neural decoding works. Unfortunately, due to time constraints, we will not be able to provide such results within the remaining hours of the discussion phase.

In the context of OOD experiments, please note the OOD zero-shot experiment in Appendix H of the revision, in which the code is 7x the length of the maximum code length in our training set.

[1] Deep Learning Methods for Improved Decoding of Linear Codes, Nachmani et al., IEEE Journal of Selected Topics in Signal Processing, 2018

[2] Error Correction Code Transformer, Y. Choukroun and L. Wolf, NeurIPS 2022

[3] Denoising Diffusion Error Correction Codes, Y. Choukroun and L. Wolf, ICLR 2023

We thank the reviewer for the supportive review and helpful recommendations.

Code Length and Type

As with most neural decoding works (cf. Related Works section), we focus on short/moderate-length codes for three main reasons.

(i) On very large codes, some decoders (e.g., BP, SC) have been proven to achieve performance close to ML decoding on their respective codes of interest (e.g., LDPC, Polar, respectively), rendering the potential advantage of neural decoding solutions nonexistent/irrelevant.

(ii) The training and/or deployment of the existing neural decoding techniques is not trivial/straightforward for very large codes (i.e., code lengths of thousands of bits).

(iii) The emergence of applications driven by the Internet of Things created the requirement for optimal decoders of short to moderate codes. For example, 5G Polar codes have code lengths of 32 to 1024 for uplink (UCI) or 128 to 512 for downlink [1].

While the training/testing codes have been randomly selected from the available codes database, as described in the Appendix, we do provide the performance of LDPC (CCSDS) https://rptu.de/channel-codes/channel-codes-database/more-ldpc-codes#c94700

Finally, as the reviewer noticed, our current computing resources (described in the paper) cannot allow the efficient training of codes with thousands of bits.

These points are emphasized in the revision.

Differentiating the Text from ECCT

As far as we can see, the only part similar to the ECCT is Section 3.1 (now revised to be just the start of Sec 3), which reviews the standard coding setting background. We made several changes to further differentiate this subsection.

Performance Compared to ECCT on Larger Codes

We do not observe worse performance on larger codes (e.g., BCH or CCSDS codes), except for the Polar codes only. We believe that this arises from two main reasons. (i) The dataset is highly unbalanced, inducing a low representation of the Polar codes (cf. Section 4.4 paragraph 6, Section 6 paragraph 3, and Appendix C). (ii) Following [2,3,4], we standardized the parity-check matrices of Polar codes (but not of other codes), which may have had an adverse effect on the learned inductive bias. That being said, after fine-tuning, ECCT’s performance is outperformed on Polar codes as well, as described in Figure 4.

These points are emphasized in the revision.

Table 3 Explanation

We can observe from Table 3 that even while using ECCT masking, invariant initial (II) embedding remains better than using a distinct embedding for each bit (i.e., regular ECCT), where the bit position is taken into account by the FC layer. However, we can observe that using invariant output (IO) coupled with II loses positional information, inducing a drop in accuracy.

A similar effect appears also for ECCT+DM+II which improves over ECCT’s original initial embedding. The conclusion is that ECCT is worse than FECCT in all of its components: initial encoding, masking, and final encoding.

We note here that, as described in the caption of Table 3, ECCT has a larger capacity which makes an exact comparison difficult.

These points are emphasized in the revision.

References

[1] ETSI 3GPP TS 38.212 “5G NR Multiplexing and channel coding”, v.16.5.0, 2021-03.

[2] Error Correction Code Transformer, Y. Choukroun and L. Wolf, NeurIPS 2022

[3] https://github.com/yoniLc/ECCT

[4] How to Mask in Error Correction Code Transformer: Systematic and Double Masking, S. Park et al., arXiv:2308.08128

We thank again the reviewer for the valuable feedback. We hope that we have comprehensively addressed all the raised questions and remarks.

Should you have any additional inquiries, please do not hesitate to inform us. If we have satisfactorily addressed your concerns, we would appreciate your consideration in revisiting the score.

We thank the reviewer for the very supportive review.

The conclusion is rewritten and includes the future scopes of research as follows.

“Next steps should focus on the deployment of the method on existing error correction embedded systems via the utilization of Transformer acceleration methods along with extensive tuning and training of the codes and channels of interest. Additionally, the definition of a universal differentiable neural decoder may open the door to the optimization of codes and even to the learning of new families of codes.”