Objective Soups: Multilingual Multi-Task Acoustic Modeling for Automatic Speech Recognition

Table-2: S2TT average(Ave.) BLEU score comparison using the CoVoST 2 dataset, including PT+FT, Two-stage static, Joint PT+FT (W/O MOO), PEFT, VS-ASR, VC-ASR, and VM-ASR (with UAS and USA optimization sequences). Results are provided for two model sizes: 100M and 58M.

From Table-1 and 2, we can observe that PEFT's performance is worse than PT+FT and Joint PT+FT W/O MOO. The main reason is we kept the pre-trained backbone parameters frozen during the fine-tuning.

Q2. Could you describe the baselines (PT+FT, static weighting, and PT+FT w/o MOO) in more detail?

We appreciate your inquiry. Below, we provide a more detailed explanation of the baseline methods:

1. Pre-training + fine-tuning (PT+FT). This baseline comprises two stages:

Pre-training. The model undergoes pre-training on a self-supervised learning (SSL) objective, such as CPC or Wav2Vec2, to learn robust, general-purpose representations from unlabeled speech data. During this stage, the backbone parameters are updated using:

$$\theta^{k+1} = \theta^{k} - \alpha\nabla_{\theta}l_{\rm u}(\theta^k),$$

where $l_{\rm u}$ is the SSL loss, and $\alpha$ is the pre-training learning rate.

Fine-tuning. Following pre-training, the model is fine-tuned on supervised tasks (e.g., ASR or S2TT) to adapt the learned representations. Fine-tuning utilizes the CTC loss across multiple tasks and languages.

• Backbone parameter update:

$$\theta^{k+1} = \theta^k - \frac{\beta}{NT} \sum_{t=1}^{T} \sum_{n=1}^{N} \nabla_\theta l_{ctc}(\theta^k, \phi_{t, n}^k)$$

where $\beta$ is the learning rate, and $T$ and $N$ represent the number of tasks and languages, respectively.

• Classification head update:

$$\phi_{t, n}^{k+1} = \phi_{t, n}^k - \beta \nabla_\phi l_{\mathrm{ctc}}(\phi_{t, n}^k, \theta^k),$$

where $\phi_{t, n}$ denotes the classification head parameters for task $t$ and language $n$.

2. Static weighting. This method follows the same process as PT+FT but introduces static weighting during fine-tuning. Instead of equal weights for all supervised objectives, a grid search determines suitable weights for each objective. The backbone parameters are updated as:

$\theta^{k+1} = \theta^k - \beta \sum_{t=1}^{T} \sum_{n=1}^{N} \mu_{t, n} \nabla_\theta l_{ctc}(\theta^k, \phi_{t, n}^k),$

where $\mu_{t, n}$ is the static weight for task $t$ and language $n$.

We used the following weights for different languages:
$[\text{En, Fr, De, Es, Ca}] = [0.18, 0.19, 0.27, 0.16, 0.20].$

3. Joint PT+FT without MOO. This baseline is inspired by VC-ASR but excludes multi-objective optimization (MOO). All supervised objectives are optimized jointly without dynamic weighting or conflict-aware gradient alignment, resulting in a simpler optimization process.

4. Parameter-efficient fine-tuning (PEFT). In the PEFT method, the backbone is pre-trained as described in PT+FT. During fine-tuning:

1. A single set of classification heads is fine-tuned using:

   $$\phi_{t, n}^{k+1} = \phi_{t, n}^k - \beta \nabla_\phi l_{\mathrm{ctc}}(\phi_{t, n}^k, \theta^k).$$

2. The fine-tuned classification heads are frozen, and the process is repeated for the next set.

This sequential optimization process ensures parameter efficiency by freezing previous sets while fine-tuning subsequent ones. We have added all the detailed descriptions in the Appendix-D.

W2 & Q1. Each MOO objective (VS-ASR, VC-ASR, VM-ASR) seems to have a simpler non-MOO counterpart, which I refer to as the 'VS-ASR' baseline, 'VC-ASR' baseline and 'VM-ASR' baseline, like to see each of these baselines.

We appreciate your suggestion. The baselines for the VS-ASR and VC-ASR methods are already included in the paper as follows:

VS-ASR: In this algorithm, we jointly trained all the conflicting objectives in a single optimization level using MOO. The PT+FT method or the static weighting method can be regarded as the baseline without MOO for VS-ASR (Please read the description of the PT+FT method presented in Q2).

VC-ASR: In this algorithm, we used unsupervised objective as a constraint and used MOO to optimize all the conflicting objectives. Joint PT+FT without MOO is the baseline for the VC-ASR method where no MOO is used (Please read the description of the Joint PT+FT without MOO method presented in Q2).

VM-ASR: Following your instructions, we trained the baseline for this method, which we term "parameter-efficient fine-tuning (PEFT)." In this PEFT method, the backbone is pre-trained, followed by freezing the backbone. Then, we fine-tune one set of classification heads, freeze those, and fine-tune the second set of classification heads, and so on.

Table-1:This table compares WERs of various ASR approaches, including PT+FT, Two-stage static, Joint PT+FT (W/O MOO), PEFT, VS-ASR, VC-ASR, and VM-ASR (with UAS and USA optimization sequences). Results are provided for two model sizes: 100M and 58M.

We appreciate the reviewer’s recognition of our findings and analysis. Below, we address the concerns one by one.

W1. The paper emphasizes multitask and multilingual learning but only considers 2 tasks. It was unclear to me whether the findings of the paper would generalize to a true multitask setup. It does already consider several languages, which is great.

We appreciate the reviewer’s positive feedback on our work. Regarding the concern about generalization, we respectfully disagree. In the context of MOO, multitask learning can involve different tasks even if they come from different datasets [a]. In our experiment, we use five languages, each with two tasks (except English), which adds up to nine classification heads for nine distinct tasks. Therefore, we can confidently say that our methods handle a true multitask setup and can manage a substantial number of tasks.

Reference

[a] Chen et al., "Three-way trade-off in multi-objective learning: Optimization, generalization and conflict-avoidance," NeurIPS 2023.

---

W3. Since the proposed methods are more resource-heavy than baseline, it would be fairer to report a memory-/speed-matched comparison between baselines and the proposed methods.

That's an excellent idea! However, conducting a memory-/speed-matched comparison would require significant adjustments to both the baseline and our proposed MOO methods, which is beyond the current scope. We will consider this for future work to provide a more thorough evaluation.

W3 & Q3. Many experimental details are missing and should be added to an Appendix to encourage reproducibility.

Thank you for pointing this out. We have included the following experimental details in the revised version of the paper:

1. Learning Rates for PT and FT in Joint PT+FT Baseline:

• Pre-training Learning Rate: The unsupervised pre-training starts with a learning rate of $\alpha=5e-4$ for 100 epochs, which is then annealed by a factor of 0.1 every 20 epochs. The model is pre-trained for 200 epochs.
• Fine-tuning Learning Rate: The maximum learning rate for fine-tuning is $\beta=5e -5$. A learning rate scheduler is used to monitor the test loss every 10 epochs. If the loss does not decrease during these 10 epochs, the learning rate is reduced by a factor of 0.1. The model is fine-tuned for a total of 100 epochs.

2. Optimizer:

• AdamW for both pre-training and fine-tuning

3. Acoustic Features and Token Vocabulary Size:

• Pre-training: We used raw speech data during pre-training to allow the model to learn representations directly from the audio. We utilize a context length of 20 frames (200 ms) and predict the next 12 frames, employing 12 negative samples for contrastive loss.
• Fine-tuning: The raw audio files were converted into 80-dimensional log-mel features. These features are commonly used in speech recognition tasks as they capture both temporal and spectral information effectively.
• Token Vocabulary Size: The token vocabulary size was set to 1000 for all languages except Chinese, where it was set to 5000.

4. Data Augmentation (SpecAugment):

• Data Augmentation: We employed SpecAugment, which includes time and frequency masking, during fine-tuning. All this training information has been added to the main paper's Appendix.

---

Q4. The authors should point to Appendix D that provides more details about the Librispeech/AISHELL experiment.

Thank you for pointing this out. We have added the information in Appendix D for more details on the LibriSpeech/AISHELL experiment.

Referencing Appendix D for More Experimental Details:

• The LibriSpeech/Aishell training follows the same procedures and hyperparameters as the CoVoST dataset training. We will clarify this setup in Appendix D.

---

Exclusion of Test-Other Results in Table 5:

• The test-other results were excluded from Table 5 to allow for a more focused comparison with the AISHELL dataset. This exclusion helps us better understand the effect of multilevel optimization and MOO across different languages.

Below is the results on "test-other."

---

CER for AISHELL Dataset:

Thank you for your comment. Yes, we used CER to evaluate the AISHELL dataset and have already updated the caption of Table 5 to clarify this.

---

Joint PT+FT W/O MOO Baseline in Table 5:

Thank you for pointing that out. We have included the performance numbers for the Joint PT+FT baseline in the updated version of the paper. You can find the relevant results in the revised table.

Table-5: This table compares WERs for ASR tasks on the LibriSpeech and AISHELL datasets across various approaches, including Two-stage PT+FT, Joint PT+FT W/O MOO, VS-ASR, VC-ASR, and VM-ASR (using UEC and UCE sequences). Results are provided for two model sizes: 100M and 58M.

From the table, we can observe that Joint PT+FT W/O MOO performs better than PT+FT and VS-ASR methods.

We appreciate the reviewer recognizing the strengths of our paper, particularly the benefits of MOO for multilingual speech-to-text tasks. Below, we address concerns one by one regarding baseline scores, the intuition behind optimization levels, and the experimental details.

W1 & Q1. Why do the baseline PT+FT numbers in Tables 1 and 2 fall short compared to prior Conformer-based work, such as the CoVoST 2 repository?

We respectfully acknowledge this difference and attribute the lower performance metrics to the following reasons:

Model architecture and setup: Unlike the GitHub Conformer models referenced, which use separate models for each language and task, we employ a single unified model for all languages and tasks. This approach increases training complexity because the model must balance diverse linguistic and task-specific objectives, often resulting in gradient conflicts during training.

To illustrate, we compare the multi-objective training results with those of single-objective models, both of which have 58M parameters:

This comparison underscores the trade-offs between single-objective and multi-objective training setups, with the unified model facing challenges due to competing objectives but offering benefits in scalability and efficiency.

---

W2 & Q2. Nothing is mentioned about the curvature of the loss (i.e., its flatness) in Appendix C. Please elaborate.

Below, we clarify why the self-supervised loss $c(x)$ possesses the desirable properties, including flatness, and how it supports our motivation for VC-ASR.

Generalization and flatness:

• As shown by [a], self-supervised learning methods optimize in flatter loss regions, indicated by smaller Hessian eigenvalues and lower sharpness metrics. Flat minima improve generalization and ensure stable optimization.

• For VC-ASR, this flatness allows $c(x)$ to generalize features across tasks, reducing the search space for Pareto solutions and aligning with conflicting objectives.

• Self-supervised losses provide robust feature representations that transfer well to supervised tasks, creating a stable foundation for multi-objective optimization.

Empirical Evidence:

• To illustrate, we analyzed the sharpness of the loss landscape for supervised, PT+FT, and VM-ASR training. Curvature is measured using the largest eigenvalue of the Hessian $( |\lambda_{\text{max}}|)$ and sharpness-aware metrics like $(C_\epsilon, A)$-sharpness. As shown in the table below, the VM-ASR model consistently demonstrates lower curvature and better flatness compared to the supervised and PT+FT models. And PT+FT method demonstrates lower curvature and better flatness compared to the supervised model.

Revisions to Appendix C:

• We acknowledge the lack of explicit discussion on flatness in Appendix C. In the revision, we will incorporate insights from [a] on how self-supervised losses lead to flatter optimization regions and enhance gradient compatibility.

Reference:

[a] Fradkin et al., "Robustness to adversarial gradients: A glimpse into the loss landscape of contrastive pre-training," ICML 2022.

Q3. Why En, Fr, and De have higher WER despite having more data hours in CoVoST2?

We believe that the higher WER for En, Fr, and De, despite having more data hours in CoVoST2, can be explained by:

Quality of test dataset: Similar to our paper, the CoVoST 2 paper also reported high WER and low BLEU scores for En, Fr, and De. See [a]. These discrepancies may be attributed to the use of more challenging test datasets.

Reference

[a] Wang et al., "CoVoST 2 and Massively Multilingual Speech Translation," Interspeech 2021.

---

Q4. Could you clarify the statement "In such scenarios, the Pareto front tends to be large and spread out"? How do these scenarios arise, and what are the implications?

Thanks for the question. Indeed, this is only an intuition based on our empirical studies which we detail below.

Intuition: when objectives are highly conflicting, the optimal vector-valued objectives can be far apart in the objective space. As a result, the Pareto front could become wide and spread out.

We will clarify in the revised paper that this is our intuition and needs further verification and move this sentence to future work discussion.

---

Q1. How separating conflicting tasks into different optimization levels reduces gradient conflict and improves WER?

To mitigate gradient conflicts we separate ASR and S2TT (VM-ASR) tasks into distinct optimization levels, thereby reducing conflicts among their objectives. This hierarchical approach first optimizes objectives at the lower levels before addressing those at the upper levels, rather than simultaneously optimizing all objectives. By doing so, we minimize the likelihood and magnitude of conflicts arising from competing objective gradients.

Moreover, increasing the number of optimization levels for a fixed total number of objectives reduces the average number of objectives per level. With fewer objectives at each level, the probability of gradient conflicts decreases, and the magnitude of maximum gradient conflicts between any pair of objectives at that level becomes smaller.

To substantiate this claim, we provide quantification of gradient conflicts across different methods below.

Quantification of gradient conflicts:

• To quantify the conflict between tasks, we calculate the cosine similarity of gradients for different pairs of losses in VC-ASR and VM-ASR training:

  • $\cos \omega_{A,S} = \frac{\langle \nabla L_{\text{ASR,CTC}}, \nabla L_{\text{S2TT,CTC}} \rangle}{\|\nabla L_{\text{ASR,CTC}}\| \|\nabla L_{\text{S2TT,CTC}}\|}$
  • $\cos \omega_{A,C} = \frac{\langle \nabla L_{\text{ASR,CTC}}, \nabla L_{\text{CPC}} \rangle}{\|\nabla L_{\text{ASR,CTC}}\| \|\nabla L_{\text{CPC}}\|}$
  • $\cos \omega_{S,C} = \frac{\langle \nabla L_{\text{S2TT,CTC}}, \nabla L_{\text{CPC}} \rangle}{\|\nabla L_{\text{S2TT,CTC}}\| \|\nabla L_{\text{CPC}}\|}$

• A lower cosine similarity (negative) indicates greater conflict between gradients, while higher (positive) values suggest better alignment.

• By comparing these values in single-level (VS-ASR) and multilevel (VM-ASR) training, we observe that multilevel training consistently results in higher similarity values, demonstrating its ability to reduce conflicts effectively.

• We also calculate the Number of Conflicts in one epoch, which represents the number of batches where significant gradient conflicts occur ($\langle \nabla L_{\text{t1}}, \nabla L_{\text{t2}} \rangle < 0$). Out of the total 1498 batches, 316 and 128 batches exhibit conflicts in VS-ASR and VM-ASR, respectively. The reduced conflicts in VM-ASR highlight its effectiveness in aligning gradients across tasks, resulting in more stable and efficient training.

The cosine similarity of the gradients of the losses and the number of conflicts were calculated after 200 epochs on the CoVoST dataset, with consistent training hyperparameters across both setups.

Reference

[a] Sato et al., "A gradient method for multilevel optimization. Advances in Neural Information Processing Systems," NeurIPS 2021.

[b] Saif et al., "Joint Unsupervised and Supervised Training for Automatic Speech Recognition via Bilevel Optimization," ICASSP 2024.

Q2. How a single MOO model reduces resource demands during deployment, making it a more efficient solution overall?

Thank you for your insightful question. A single MOO model can reduce resource demands during deployment by:

1. Reduced storage requirements: The MOO model (654.4 MB) handles five languages, with an encoder size of approximately 100M and each classification head about 0.5M. In the MOO setup, backbone parameters are shared across all objectives, making it a memory-efficient method. In contrast, deploying five separate models for five different tasks would require $5\times$ more backbone parameters, assuming each single-objective model uses the same encoder size as the MOO model.
2. Efficient inference: Loading the MOO model takes 0.27 seconds, whereas loading five separate models takes 1.25 seconds $(5 \times 0.25s)$, significantly reducing latency and overhead.

We appreciate the reviewer acknowledging our methods for handling conflicting objectives. Below, we address your questions and concerns one by one.

W1. The provided code requires cleaning and refinement for better readability and maintainability.

We appreciate your feedback and are working on improving the code to enhance its readability and maintainability. The updated version will be uploaded to GitHub soon.

---

W2. a. Important conclusions and interesting findings are relegated to the appendix. b. Key sections (e.g., Section 5.2) should be integrated into the main text.

We acknowledge that some interesting findings are included in the appendix due to page limitations. Our focus has been on presenting the most critical insights in the main paper, but we recognize the value of these additional results. We will try to incorporate some of these findings into the main paper in the revision.

---

W4. Limited model evaluation; broader testing across architectures and sizes is needed for stronger validation.

We focus on the Conformer model as it is a widely used ASR baseline, ensuring relevance to current research. To provide a broader evaluation, we have included Wav2Vec2 results in Appendix F, where our methods also outperform the baselines.

---

W5. Further theoretical analysis is needed (Notes as Future Work in Conclusion) to comprehensively understand the capabilities and limitations of the proposed method.

We appreciate your suggestion. Currently, our focus is primarily on improving the performance of the ASR and S2TT tasks. In the future, we plan to conduct a detailed theoretical analysis, including providing guarantees on the convergence rate of the studied algorithms, to gain a deeper understanding of the capabilities and limitations of our proposed methods.

We are glad that the reviewer finds our work novel and excellent. Below, we provide an answer to your question about computational complexity.

W1. The experimental results provided a stronger base for the results, however, it could be helpful to have more theoretical reasoning for the same.

Great suggestion! It is probably feasible to develop theoretical guarantees on the convergence rate of the studied algorithms. However, since we focus on empirical findings about how different approaches to handling multiple objectives help improve the performance of Multilingual Multi-Task ASR, it is beyond the scope of this paper. We thus leave it for future work.

---

Q1. Can there be any further improvement done for improving training time or reducing GPU memory?

Yes. The main computation bottleneck comes from the MOO algorithm that requires computing multiple objective gradients to find the conflict-avoidant (CA) direction. This bottleneck can be addressed by using the recently proposed approach Famo [a, Section 3.2]. Specifically, it uses an approximate method to compute the dynamic weight to compute the CA direction, without computing multiple objective gradients at each iteration, thus largely reducing both the training time and GPU memory.

However, the improvement in computational complexity is beyond the scope of this paper, thus we leave it for future work.

Reference

[a] Liu et al., "Famo: Fast adaptive multitask optimization," NeurIPS, 2023.