Symbolic Autoencoding with Straight-Through Gradient Approximations

Despite its merits, this work has a number of weaknesses, the most important of which have to do with the experimental section. Let me start by highlighting these.

• Unconvincing results: even within the setup proposed by the authors, the results are unconvincing. Indeed, as can be seen in Figure 4, the supervised baseline performs on par with the proposed method in two out of four datasets, at already the second smallest supervision ratio. It performs on par with the proposed method in three of the datasets, at the third supervision ratio. Or am I missing something?

Given that no information is provided about the training times of the proposed method, specially as compared with those of the supervised baseline, which is trained on a significantly smaller dataset, it is not clear what the advantage is in leveraging the non-parallel data to begin with.

• Insufficient baselines: in the related work section the authors highlight the connection of the proposed method with back-translation, which is depicted as the simplest approach which can leverage parallel and non-parallel data, specially in the general text sequence transduction setting studied by the authors. However, the latter is not considered as a baseline. Therefore, the authors do not compare with any method that makes use of both parallel and non-parallel data. This lack of comparison makes difficult the assesment of the proposed methology.

Similarly, there is little to no detail regarding the ICL baseline, whose performance is not discussed in the paper. It is known, for example, that ICL strongly depends on both the number and quality of examples (see e.g. Agrawal et al. [2023], Vilar et al. [2023]). I also have the feeling that the NLP community has moved from ICL to Chain-of-thought prompting, which makes the latter a more reasonable baseline. Note that the latter has been used in semantic parsing tasks (Tai et al [2023]).

• Unconventional tasks: Natural language translation and semantic parsing in low-resource settings are two major instances of the problem that could in principle be tackled by the proposed methodology. Nevertheless, the authors do not consider any of these.

In sum, these weaknesses have as a consequence that the reader does not fully grasp the relevance and usefulness of the proposed methodology.

Other weaknesses:

• The mathematical notation is very confusing and could significantly be improved. Some examples follow:

1. unless I am misunderstanding sth, the notation for probability distribution and density is constantly being exchanged (see e.g. 117 vs 121);
2. The notation for probability of an event changes within the same equation (see eq.18) and doesn’t match that in the main text;
3. Similar to the point above, see the eq. in lines 360 and 361. Instead of $\mathbb{P}$ the authors could use sth like $p_\theta(z=<EOS>|x)$ and thus make clear how the soft-mask approximation works. I might however be misunderstanding what the authors meant and, if so, I apologize;
4. again, unless I am misunderstanding, the class probabilities of the categorical distribution are labelled with both $\mathbf{p}$ and $\mathbf{s}$ in the same equations (see e.g. Eqs 8, 9 and 10).

• The organisation of the paper could also be improved: indeed, the authors spend a lot of time discussing how to optimise models which sample discrete variables. This procedure, whose description takes about 3 pages, is, in my view, well known by the ML community by now. Therefore, its discussion could probably be moved into the Appendix. In contrast, the reader would benefit from having e.g. the description of the dataset back into the main text.

• Lack of references: The authors make emphasis on the novelty of their autoencoder framework for latent sequences of symbols, but do not make reference to several other such methods as e.g. those of Zhao et al. [2018], Kaiser et al. [2018] or Sanchez et al. [2023].

References:

• Agrawal et al. [2023]: In-context Examples Selection for Machine Translation
• Vilar et al. [2023]: Prompting PaLM for Translation: Assessing Strategies and Performance
• Tai et al. [2023]: Exploring Chain of Thought Style Prompting for Text-to-SQL
• Ko et al. [2021]: Adapting High-resource NMT Models to Translate Low-resource Related
• Zhao et al. [2018]: Unsupervised Discrete Sentence Representation Learning for Interpretable Neural Dialog Generation
• Kaiser et al. [2018]: Fast Decoding in Sequence Models Using Discrete Latent Variables
• Sanchez et al. [2023]: Hidden Schema Networks

• The paper is difficult to follow and its structure could be improved in different ways:

  • First the authors do not explain in a clear way the main idea of their paper in the introduction. The authors only write (wrt. the unsupervised part): “$\Sigma AE$ performs two autoencoding tasks through discrete bottlenecks: one model encodes and reconstructs data from $X$ to $Z$, while the other does so from $Z$ to $X$.” This should be interpreted that you do x->z (as latent variable)-> x (back). But is not clear at all it should be like this here. While there is a Figure 1 describing the method, it is too cryptic to follow at that stage of the paper. I understood it after studying Section 3.1 which is in the Experiments section (on Page 8) , not even in the methodological section.
  • Different parts seem to be redundant / repetitive. Section 2.1.1 on REINFORCE / Annealing methods and Reparameterization trick should take only one or two lines since they are well known methods from literature and at least REINFORCE and Annealing methods are not used in the paper. Maybe the Gumbel-Softmax is ok to put since is directly used. Another example, why re-define the mask in Eq. 13 if it is already defined in previous sub-section?
  • There are sometimes symbols that are used without being defined and imprecisions on a notational level. For example in Line 360, there is $\mathbb{P}(O_k = \text{< EOS >})$, but $O$ is defined in the appendix. Or on Line 238 $\mathbf{s}$ was never introduced. I list other situations like this in the questions part.

• Experimental section is lacking.

  • While SCAN, PCFG SET, CFQ and COGS are valid benchmarks on compositional generalization, I do not understand why there isn’t any benchmark on natural language translation) (and relative evaluation on standard BLEU score), which is the standard symbolic transduction task in NLP (especially since the authors talk about the Rosetta Stone analogy in Figure 1).
  • There isn’t any quantitative evaluation of hidden sequence collapse phenomenon.
  • The authors do not discuss which quantisation method does work better: for example VQ DB seems doing pretty bad in Table 1 especially on CFQ (0.00 sentence reconstruction) but is achieving the best results on the weakly supervised benchmark (Figure 2), especially on CFQ.

1. The reliance on synthetic datasets, which limits applicability to real-world language.
2. The randomness by the Gumbel-Softmax discretizer reduces sentence-level accuracy consistency, impacting the model’s reliability in token-precise tasks.

Based on my reading of the manuscript, the authors may have missed quite a few earlier studies that effectively focused on the same problem. For instance, Miao & Blunsom (2016; https://arxiv.org/abs/1609.07317) proposed a discrete sequence bottleneck model for sentence compression, where they used reinforcement learning and semi-supervised learning. Learning a dynamic-length latent sequence was studied earlier in many different settings, and i can think of e.g. Evtimova et al. (2017; https://arxiv.org/abs/1705.10369) and Graves (2016; https://arxiv.org/abs/1603.08983). Of course these earlier studies do not follow the exact procedures in this manuscript, but these differences seem relatively minor (e.g. reparametrized gradient estimator vs. log-derivative trick, etc.) The authors should look at these papers as well as any follow-up studies to these and reposition their work more carefully.

One of the main promises by the authors already in the abstract is that their approach results in better “interpretability and structure”. Unfortunately, I cannot see why this is the case and do not see any evidence behind this. The authors need to give more thoughts on how they define interpretability, why such interpretability is desirable, how their approach improves interpretability defined in this way and how they can show this is the case. The same applies to “structure”.