Listening to Formulas: Pioneering Models and Datasets for Converting Speech to LaTeX Equations

Thank you for the detailed review. Let us address weak points and answer your questions.

W1.

It is hard to evaluate how representative our data is. There are too many fields of mathematics and natural sciences and we do not pretend to cover them all. However, firstly, we were fostering to cover the main LaTeX mathematical symbols, commands and letters, such as \alpha … \omega, \frac{}{}, \sqrt{}, \left(, …, etc. Secondly, we prompted the GPT to provide examples and pronunciation on topics that are typical for the BSc in math or physics, manually removing too simple samples or the samples that contain mainly the \text{} instead of special characters. Thirdly, we used 3 different formula sources. Fourthly, we tried to voice-over the equations with several voices in two languages. Finally, part of the equation is pronounced in two ways: sounds-like and name-like, e.g. (“f equals m a” and “Newton’s second law”). From these points of view, we assume our dataset to be representative. Moreover, we demonstrate that the Speech2Latex quality does not degrade significantly if we train the model on synthetically generated audios. Consequently, we encourage anyone to generate examples that are relevant to his/her topic almost for free and tune the model. However, we should admit, that usually there is an ambiguity of pronunciations and we cannot cover them all in audio, leaving it to the language model to handle it (“f equals m a”, “f is equal m a”, “f equals m multiplied by a”, “Newton’s second law”, “Newton’s second law of the motion”, … -> $F=ma$ or $\vec{F}=m\vec{a}$).

MathBridge was presented and became open-sourced during an intermediate stage of our work. However, if one looks closely at it, one can notice that it contains too many poor equations, as discussed in our work. We were physically unable to proceed with many formulas and considered 15,000 random samples. We removed the primary useless (too short, too many punctuation signs, too similar ones, etc.) and decided to switch to better-quality data.

The most representative dataset is OleehyO/latex-formulas of OCR-formulas, as it contains many comprehensive examples from real papers. We better should have started with this dataset, but we did not realise to use it during the early stage of our work.

W2.

We will add this information to the manuscript, which will be presented in the updated version. So far, the information can be found in the supplementary materials and the code.

Q1.

We evaluate the results in the LaTeX form, indeed (see, for example, table 4, where “GT” represents the ground truth equations, and “LaTeX” represents the S2L model’s prediction). Table 1 illustrates how different ASR models transcribe one particular audio to support our claims of which models are more suitable for the S2L task.

Q2.

No, we did not fine-tune ASR models. In the first part of the experiments, we used pre-trained Whisper to transcribe audios and tuned LLMs in a post-correction setting (Whisper transcription to Latex). We fine-tuned them in the audio-to-latex setting for the second part of experiments with audio-LLMs (Qwen-Audio and SALMONN). The reason why Qwen-Audio performs worse than other methods is probably due to the nuances of re-implementations. Qwen-audio's official repository does not provide training or fine-tuning scripts, thus, we had to re-implement the fine-tuning of the model on our own, which could lead to a worse performance than claimed by the authors.

We have made our dataset publicly (and anonymously) available https://huggingface.co/datasets/marsianin500/Speech2Latex.

We hope you will find these explanations helpful. Please feel free to ask additional questions; we will be glad to answer them.

Thank you for the detailed review. Let us address weak points and answer your questions.

We have chosen not to include links to the data in the submission for anonymity. However, we have made it publicly available while maintaining anonymity thus far https://huggingface.co/datasets/marsianin500/Speech2Latex.

Our motivation was to verify whether training on artificially generated audio gains similar quality on genuine test speech as training on human-annotated audio. XTTSv2 was chosen for audio generation because this model is publically available with pre-trained weights, generates good-quality audio and allows for voice adjustment (voice conversion). Due to the fact that generating a large number of audio samples using this TTS model is time-consuming, we have chosen to select only a limited number of reference voices for our hypothesis testing purposes.

The references would be expanded in the updated version of the manuscript:

Popović, M. (2015, September). chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the tenth workshop on statistical machine translation (pp. 392-395).

Popović, M. (2017, September). chrF++: words helping character n-grams. In Proceedings of the second conference on machine translation (pp. 612-618).

We decided to explain CER as a primary metric. Please refer to Appendix A.2 for the details of other metrics. We used both chrF and chrF++ in our experiments.

The mentioned uncertainties and miss-printings are fixed:

• Line 242: reference to the “Table 6”
• Lines 422-423: “Additional metrics for lower-case performance can be found in Appendix in Table 11, and for case-sensitive in Tables 9 and 10, respectively.”
• Line 479: reference to the “Table 13”

The undefined metric at the end of an abstract is the character error rate (CER).

The topics regarding metrics definition and the dataset creation procedure will be revised for clarity and simplicity.

Please feel free to ask additional questions; we will be glad to answer them.

I admit that based on the naming, I didn't open the file abstract.pdf of the supplementary data, but in fact it contained the (missing) appendices. I recommend to include the appendices in the paper as they do add some clarity. The dataset is made publicly available as described. In order to verify improvements to the clarity of the presentation I recommend to upload a revised version of the submission.

As a consequence I raise my contribution score from 1 to 2 (unfortunately, raising the final score from 3 to 4 isn't possible). Proof of adding clarity to the presentation may help to improve on the presentation score which may raise the final score further to 5.

Thank you for the detailed review. Let us address weak points and answer your questions.

W1.

Our main contributions are the dataset, the baseline training strategies, and the benchmarking for further research.

Although different combinations of ASR and LMs were studied thoroughly, to our knowledge, there is a lack of work devoted to processing structured but ambiguous data as equation pronunciations. In contrast, many works are dedicated to the NLP methods for processing structured and ambiguous data, e.g., chemistry formulas processing or text2code/SQL.

W2.

As we indicate at the end of the conclusion, simultaneous processing of the equations and ordinary speech is more prominent in the application.

The main problem is the absence of a suitable dataset for further voice-over. Mathbridge should have solved this issue as it also contains a short left and right context of the equation. Unfortunately, as discussed in the text, the Mathbridge dataset contains too many poor samples, making it hard to apply and annotate directly. We can use equations from a good-quality OleehyO/latex-formulas dataset, cover the equation with random artificial/GPT in the left and right contexts, and then annotate the obtained texts with TTS models. It should provide satisfactory results (as demonstrated in the text, trained on the TTS samples model performs relatively well compared to the model trained on the human data). Human annotation is more complex, expensive, and time-consuming, and we will probably use it for future work. Another way is to parse the relatively long formulas with context from the Arxiv article latex codes. One of the new model's possible applications is paper writing assistance.

W3.

For example, issues with brackets. We will add an analysis of the failure cases to the revisited manuscript.

Q1.

No, Qwen-Audio does not support the Russian language.

Q2.

The mentioned uncertainties and miss-printings are fixed:

• Line 214->242: reference to the “Table 6”
• Lines 416-420 moved to the introduction.
• Lines 422-423: “Additional metrics for lower-case performance can be found in Appendix in Table 11, and for case-sensitive in Tables 9 and 10, respectively.”
• Line 479: reference to the “Table 13”

For now, we have made our dataset publicly available: https://huggingface.co/datasets/marsianin500/Speech2Latex.

Please feel free to ask additional questions; we will be glad to answer them.

Thank you for the detailed review. Let us address weak points and answer your questions.

The mentioned uncertainties and miss-printings are fixed:

• Line 214->242: reference to the “Table 6”
• Lines 416-420 moved to the introduction.
• Lines 422-423: “Additional metrics for lower-case performance can be found in Appendix in Table 11, and for case-sensitive in Tables 9 and 10, respectively.”
• Line 479: reference to the “Table 13”

These errors will be fixed. The clarity and presentation of the revisited text will also be improved.

We disagree that the main task of the work is to convert text to LaTeX; this is merely one approach discussed in the manuscript. In contrast, we explored a multi-modal approach using audio LLMs, where audio is translated directly into LaTeX without explicitly converting it to phonetic text.

For now, we have made our dataset publicly available: https://huggingface.co/datasets/marsianin500/Speech2Latex.

Please feel free to ask additional questions; we will be glad to answer them.