A Joint Spectro-Temporal Relational Thinking Based Acoustic Modeling Framework

We value the reviewer's insightful queries and the valuable feedback provided. Below are our responses addressing the questions and concerns raised by the reviewer.

1. Are the baselines trained with cross-entropy objective?

AR: No, the baselines used in our experiments were not trained with the cross-entropy objective. Instead, when we fine-tuned the baselines, we followed Baevski et al. (2020) and optimized the standard connectionist temporal classification (CTC) loss function (Graves et al., 2006). CTC is a commonly employed method for end-to-end sequence modeling tasks in the field of speech processing.

2. Would it make sense to Impose left to right constraint between the time steps for causality?

AR: We agree that it is essential to consider causality when modeling sequences. In our proposed relational thinking modeling, we have already taken causality into account. As stated at the beginning of Section 3.1, for each time step $t$, we constructed the feature map $\\mathbf{C}\_t$ by incorporating the acoustic feature vectors from the current time step and the previous $w - 1$ time steps as

$$\\mathbf{C}\_t = [\\mathbf{c}\_{t - w + 1}, \\ldots, \\mathbf{c}\_t]. \\quad \\quad \\quad \text{(e)}$$

This formulation (e) explicitly excluded acoustic feature vectors from future time steps. Essentially, this aligned with the concept of a left-to-right constraint by considering only past and current information and not incorporating information from future time steps.

3. PER Analysis at the speaker level may give further intuition on how the model performs as compared to human perceptions.

AR: We appreciate the reviewer's suggestion regarding PER analysis at the speaker level. It is indeed a valuable perspective for understanding how the model's performance compares to human perceptions. In this particular study, we adhered to the established protocol of reporting PER over the entire split to maintain consistency with previous research in phoneme recognition, ensuring a fair comparison. However, we acknowledge the significance of speaker-level analysis and plan to explore this approach in our future work.

4. It would be good to see Fig.5 repeated with t1f8 and t8f1 models.

AR: As indicated in Table 1, the two joint spectro-temporal models, w20-t4f2 and w20-t2f4, demonstrated superior performance compared to the purely temporal and spectral models, w20-t8f1 and w20-t1f8. This suggested that the spectro-temporal models had a better grasp of the inherent relational information involved in speech. Therefore, we believed it would be more beneficial to conduct further analyses on these joint spectro-temporal models rather than the purely temporal or spectral model to gain more accurate and valuable insights.

5. Does feature mapping (Eq. 1) involve mixing of features within frequency bins? Is $\\mathbf{\\Lambda}$ diagonal or not?

AR: As mentioned in Section 3.1, in our proposed framework, we utilized a temporal convolution to implement the filtering operator $\\Xi$ in (1). Therefore, it is possible that this operation may involve linear combinations of features within frequency bins. However, we can also choose to implement $\\Xi$ with various other methods (e.g., pooling), which may not mix features within frequency bins.

$\\mathbf{\\Lambda}\_{t, d^{(f)}, d^{(t)}} \\in \\mathbb{R}^{D\_s \\times \\check{w}\_s}, \forall d^{(f)}, d^{(t)}$ are sub feature maps of $\\check{\\mathbf{C}}\_t$, which consist of the spectro and temporal domain feature information, and they are not necessarily diagonal.

6. Have you considered other types of spectra-temporal features for comparison? One example could be from, https://engineering.jhu.edu/lcap/data/uploads/pdfs/interspeech2012\_carlin.pdf

AR: We thank the reviewer for directing us to the reference. Given that our paper primarily aimed to explore relational thinking modeling in recognition tasks, we anticipate that such modeling should be agnostic to acoustic feature types. Consequently, we utilized the state-of-the-art acoustic features extracted by wav2vec2. However, we acknowledge and appreciate the significance of considering various types of acoustic features, which can greatly influence system performance. To this end, we have performed additional experiments using MFCCs as an alternate acoustic feature type (as detailed in A5: Generalization to Other Acoustic Features in Section 5.1). As highlighted by the reviewer's reference, it is clear that their proposed spectro-temporal features exhibited similar performance to MFCCs in clean speech scenarios, which aligned with our study. These experiments with MFCCs yielded consistent results and further validated the generalizability of our proposed framework.

We are grateful for the reviewer’s considerate questions and valuable feedback. Please find our responses to the questions and concerns raised by the reviewer below.

1. The innovation is not clear. The paper claims their innovation of using relational thinking based modeling method on spectro-temporal domain for acoustic modeling. However, from the description in the paper, there is no distinction between relational thinking based modeling and self-attention based modeling. For example, with several self-attention modeling layers stacked, it's equivalent to the so called relational thinking based model that pair-wised relation will be learnt among the transformed forms of each node (each time step), rather than the single step embedding. Therefore, theoretically, there is no difference between self-attention and relational thinking based method.

AR: Relational thinking and self-attention are two distinct mechanisms designed to capture different types of information. To illustrate the distinction, let us consider a simplified 2-layer self-attention network w.l.o.g., where each layer $l$ comprises two nodes $\\mathbf{h}\_1^{(l)}$ and $\\mathbf{h}\_2^{(l)}$. According to (13), the calculation of nodes in the subsequent layer $l + 1$ is as follows:

\\left [ \\mathbf{h}\_1^{(l + 1)}, \\mathbf{h}\_2^{(l + 1)} \\right ] = \\mathbf{W}\_v^{(l)} \\left [ \\mathbf{h}\_1^{(l)}, \\mathbf{h}\_2^{(l)} \\right ] \\left [ \\begin{array}{cc} \\alpha\_{1, 1}^{(l)} & \\alpha\_{2, 1}^{(l)} \\\\ \\alpha\_{1, 2}^{(l)} & \\alpha\_{2, 2}^{(l)} \\end{array} \\right ]. \\quad \\quad \\quad \\text{(b)}

As a result, the state of a node $\\mathbf{h}\_2^{(3)}$ after undergoing two layers of self-attention calculations is

\ $ \\mathbf{h}\_2^{(3)} & = \\alpha\_{2, 1}^{(2)} \\mathbf{W}\_v^{(2)} \\left ( \\alpha\_{1, 1}^{(1)} \\mathbf{W}\_v^{(1)} \\mathbf{h}\_1^{(1)} + \\alpha\_{1, 2}^{(1)} \\mathbf{W}\_v^{(1)} \\mathbf{h}\_2^{(1)} \\right ) + \\alpha\_{2, 2}^{(2)} \\mathbf{W}\_v^{(2)} \\left ( \\alpha\_{2, 1}^{(1)} \\mathbf{W}\_v^{(1)} \\mathbf{h}\_1^{(1)} + \\alpha\_{2, 2}^{(1)} \\mathbf{W}\_v^{(1)} \\mathbf{h}\_2^{(1)} \right ) \\quad \\quad \\quad \\text{(c-1)} \\\\ & = \\left ( \\alpha\_{1, 1}^{(1)} \\alpha\_{2, 1}^{(2)} + \\alpha\_{2, 1}^{(1)} \\alpha\_{2, 2}^{(2)} \\right ) \\mathbf{W}\_v^{(2)} \\mathbf{W}\_v^{(1)} \\mathbf{h}\_1^{(1)} + \\left ( \\alpha\_{1, 2}^{(1)} \\alpha\_{2, 1}^{(2)} + \\alpha\_{2, 2}^{(1)} \\alpha\_{2, 2}^{(2)} \\right ) \\mathbf{W}\_v^{(2)} \\mathbf{W}\_v^{(1)} \\mathbf{h}\_2^{(1)}. \\quad \\quad \\quad \\text{(c-2)} $

Even though (c-1) may exhibit a similar form to (12) (i.e., the graph embedding obtained by relational thinking, also as shown below),

$$\\mathbf{r} = \\sum\_{(i, j) \\in \\{ (i, j) | i < j, (i, j) \\in \\bar{\\mathcal{E}} \\}} \\bar{\\alpha}\_{i, j} \\bar{f}\_{\\theta}(\\mathbf{h}\_{i}, \mathbf{h}\_{j}), \\quad \\quad \\quad \\text{(d)}$$

particularly when we view $\\mathbf{W}\_v^{(2)} \\left ( \\alpha\_{1, 1}^{(1)} \\mathbf{W}\_v^{(1)} \\mathbf{h}\_1^{(1)} + \\alpha\_{1, 2}^{(1)} \\mathbf{W}\_v^{(1)} \\mathbf{h}\_2^{(1)} \\right )$ and $\\mathbf{W}\_v^{(2)} \\left ( \\alpha\_{2, 1}^{(1)} \\mathbf{W}\_v^{(1)} \\mathbf{h}\_1^{(1)} + \\alpha\_{2, 2}^{(1)} \\mathbf{W}\_v^{(1)} \\mathbf{h}\_2^{(1)} \\right )$ as node pair embedding functions from a linear family $\\mathcal{F} \\left ( \\mathbf{h}\_1^{(1)}, \\mathbf{h}\_2^{(1)} \\right )$, it is crucial to note that $\\mathbf{h}\_2^{(3)}$ is fundamentally still a weighted sum of node embeddings (as revealed by (c-2)) rather than a weighted sum of node pair embeddings as obtained by relational thinking (d), where $\\bar{f}\_{\\theta}(\\mathbf{h}\_{i}, \\mathbf{h}\_{j})$ is an arbitrary node pair embedding function. Therefore, the stacked self-attention mechanism and relational thinking are not entirely equivalent, and models cannot solely rely on the self-attention mechanism to effectively assess the importance of a pair of nodes, as discussed in Appendix D.

Furthermore, it is also important to highlight that our proposed relational thinking modeling is a variational approach, whereas the self-attention mechanism is a deterministic approach. As elucidated in Nan et al. (2023), deterministic models can produce incorrect predictions when confronted with unseen data, particularly due to the discontinuous and sparse nature of the latent space. In contrast, variational models address this challenge by mapping the input data to collections of distributions in the latent space rather than singular points. By capturing uncertainty in the latent space, variational modeling enhances model generalization, rendering the model more adaptable to diverse and unseen data. Also, we have further clarified our novelty; please refer to our responses to Reviewer Q5NM for more details.

Incremental Technical Contribution: The technical developments in this work appear to be an incremental advancement from Huang et al. (2020). The main modification is the extension of the input from one dimension to two dimensions, followed by a direct application of the relational thinking network proposed by Huang et al.

AR: In addition to extending the input from one dimension to two dimensions, our work differs from that of Huang et al. (2020) in three perspectives:

a) Our focus differs from Huang et al. (2020) as we examined a distinct application, one that readily extends to scenarios where the input and output sequences are of different lengths. This is a common scenario in sequence modeling tasks like speech recognition. To accommodate these real-world applications, we developed a novel loss function that can handle such cases. This stands in contrast to the loss function proposed by Huang et al. (2020), which is applicable only to cases where input and output sequences have equal lengths. This paper addressed the challenges that arise when dealing with input and output sequences of different lengths, a common scenario in sequence modeling tasks like speech recognition. To accommodate real-world applications, we developed a novel loss function that can handle such cases. This stands in contrast to the loss function proposed by Huang et al. (2020), which is applicable only to cases where input and output sequences have equal lengths.

b) Our paper included a comprehensive series of analyses toward the recognition results and the learned relational information, providing valuable insights into human speech perception and comprehension. We uncovered the patterns involved in the captured relations (i.e., the edges $\bar{\alpha}^{(t)}_{i, j}$ of the task-specific graph), which exhibited more similarities for phoneme classes within the same sub-group, while showed significant variations between phoneme classes from different sub-groups. We also conveyed that relational thinking modeling primarily enhanced the system by improving its performance in recognizing vowels. These insights represent a novel and previously unexplored aspect of research.

c) We conducted a theoretical analysis that explained the distinctions between relational thinking and the self-attention mechanism (in Appendix~D). Our analysis underscored that relational thinking captured a new type of (co-occurrence or pair-wise) information that could offer additional advantages beyond what the attention mechanism has achieved.

Dataset Limitations: The experiments rely heavily on the TIMIT dataset, which is relatively small in size. To firmly establish the proposed method's efficacy and robustness, it is imperative to test it on larger, more diverse datasets and under complex conditions.

AR: As clarified in Section 4, we used the TIMIT dataset because this study primarily aimed to delve deeper into the concepts of relational thinking with speech understanding and recognition tasks at the fine-grained level, serving as an initial step toward a comprehensive understanding of how this can be applied to speech. Such an analysis is feasible exclusively with the TIMIT dataset, which stands out as one of the very few datasets that provide precise information about the start and end instants of each phoneme. This unique attribute allowed us to conduct more in-depth analyses of the recognition results. For instance, it allowed us to uncover the patterns involved in the captured relations (i.e., the edges $\bar{\alpha}^{(t)}_{i, j}$ of the task-specific graph). These patterns exhibited more similarities for phoneme classes within the same sub-group, while they showed significant variations between phoneme classes from different sub-groups (as detailed in Section 5.2 and Appendix G.2). It also facilitated our ability to convey that relational thinking modeling primarily enhanced the system by improving its performance in recognizing vowels (as detailed in Section 5.3 and Appendix G.1). Without the manually annotated labels in the TIMIT dataset, we would not have been able to attain these insights. With the refined knowledge and deep understanding, we would like to extend and corroborate our findings on larger datasets beyond phoneme recognition in the future.

We thank the reviewer for his insightful questions and feedback. Please see below for our responses to the reviewer's questions and concerns.

This paper was very difficult to read and understand. It uses many words with very suggestive connotations like "Thinking" in the title, "unconscious", "mental impressions", etc. without the necessary strong justification for invoking them in the setting of a machine learning paper. These words obscure what is actually going on in the approach and are not necessary.

AR: The term "relational thinking" in neuroscience is commonly used to describe the human ability to handle abstract mental representations of the relationships among objects, attributes, and events (Alexander, 2016). This encompasses concepts such as "unconscious" and "mental impressions". While this is an established area of study in neuroscience, it is a relatively novel field of inquiry within the machine learning community, particularly concerning speech understanding and recognition tasks (Huang et al., 2020). We recognize the importance of striving for clarity and precision in scientific writing as highlighted by the reviewer. However, it may not be possible to clearly convey the essence of this concept without relying on the terminologies borrowed from neuroscience.

The task of phoneme classification on TIMIT is very old and is a reasonable first step in demonstrating the promise of an approach, but is definitely not sufficient to show that a model is learning a reasonable representation. Additionally, while the proposed system's results are good on the task (9.2% PER on the test set from Table 2), the reported wav2vec 2.0 baseline numbers (9.98% PER on the test set) are not the numbers that are reported in that paper (8.3% PER on the test set). It is not clear where the 9.98% number comes from.

AR: In Baevski et al. (2020), the authors explicitly mentioned that two versions of the wav2vec2 models, wav2vec2 BASE and wav2vec2 LARGE, were implemented. The authors reported a PER of 8.3% using wav2vec2 LARGE (in Table 3), while we adopted wav2vec2 BASE in our experiments (as highlighted in Appendix F.1.1 of our paper). The rationale behind this choice is that the TIMIT dataset is not particularly large, and the wav2vec2 BASE, which has the same architecture but fewer parameters, is considered sufficient for the task at hand. While Baevski et al. (2020) did not provide the performance of wav2vec2 BASE, we fine-tuned it for our experiments, with the corresponding PER being 9.98%.

As clarified in Section 4, we used the TIMIT dataset because this study primarily aimed to delve deeper into the concepts of relational thinking with speech understanding and recognition tasks at the fine-grained level, serving as an initial step toward a comprehensive understanding of how this can be applied to speech. Such an analysis is feasible exclusively with the TIMIT dataset, which stands out as one of the very few datasets that provide precise information about the start and end instants of each phoneme. This unique attribute allowed us to conduct more in-depth analyses of the recognition results. For instance, it allowed us to uncover the patterns involved in the captured relations (i.e., the edges $\bar{\alpha}^{(t)}_{i, j}$ of the task-specific graph). These patterns exhibited more similarities for phoneme classes within the same sub-group, while they showed significant variations between phoneme classes from different sub-groups (as detailed in Section 5.2 and Appendix G.2). It also facilitated our ability to convey that relational thinking modeling primarily enhanced the system by improving its performance in recognizing vowels (as detailed in Section 5.3 and Appendix G.1). Without the manually annotated labels in the TIMIT dataset, we would not have been able to attain these insights. With the refined knowledge and deep understanding, we would like to extend and corroborate our findings on larger datasets beyond phoneme recognition in the future.