PhonATe: Impact of Type-Written Phonological Features of African American Language on Generative Language Modeling Tasks

Thank you for your review.

Reason to Reject 1: Perhaps the authors should have explained better why these phonological features make their way to text, no where in the paper transcription is mentioned, so I assume (from the cited papers) most of the time people write these phonologically transformed text on social media.

The reviewer is correct that most of the time these type-written phonological features are written by individuals and primarily occur on social media (paragraph 3 of Introduction). Our work focuses on the use of these features in written posts and does not use speech transcription. To our knowledge, there is little sociolinguistics research into why the spoken phonological features may affect orthography (paragraph 1 of Limitations), and we believe studying these reasons is outside of the scope of this work. We can, however, make more explicit that this research gap exists and emphasize that these features are primarily written by users on social media.

Reason to Reject 2: I understand that the authors have resource constraints but the two main LLMs used in the experiments are quite old and even Mistral is considered a small model. One cannot generalize these results and assume larger language models would behave similarly. Apart from that, I really think this paper fits better in a conference like *CL as the linguistic content is quite rich but the LLM part is quite limited.

Thank you for your suggestion. We are happy to evaluate a larger LLM in our robustness experiments. Additionally, we note that the scope of COLM is not just restricted to LLMs but language modeling more broadly, and we believe that the ethics and linguistics focus of our work falls within the Societal implications and LMs for everyone topics listed in the Call for Papers.

Thank you for your review.

Reason to Reject 1: The writing is unclear in many places and should be revised before publication…

Thank you for pointing out places that could clarified. Each annotator is asked to rate 50 augmented and natural texts such that 40 of each overlap. For these 40, the ratings are averaged. Concerning data subsets, we sample 5,000 texts with non-toxic labels from the toxicity datasets as well as 5,000 texts with positive labels from the sentiment dataset. We will update the descriptions to clarify.

Reason to Reject 2: The sentiment detection eval seems very odd, given that there is no ground-truth for the sentiment…

We directly measure the percentage of cases where the sentiment prediction does change and thus, we measure bias via this process. Prior work on other biases in sentiment detection similarly use differences in sentiment predictions (e.g., Goldfarb-Tarrant et al., 2023). We will more explicitly state why we consider this behavior biased and would be open to suggestions on improving the experiment.

Reason to Reject 3: The paper never really motivates the use of synthetic AAE instead of finetuning on a larger corpus of real AAE

We note in the beginning of the Introduction that AAL has minimal representation in common datasets. While the TwitterAAE corpus is large, our aim was to better isolate the impacts of specific phonological features (e.g. the robustness experiments). We will more thoroughly motivate using synthetic data in our controlled experiments in the revision.

Question 1: What was the error rate in the phonological transformations?

With multiple ways an author may spell a term reflecting phonological features, there are not necessarily correct or incorrect augmentations in all cases. Instead, we evaluate how natural the augmentations appear to AAL-speakers (percentages shown in Table 4).

Question 2: Why do you think finetuning works for lowering perplexity with Flan-T5 and not T5?

As we cannot reliably mask only AAL features, model losses rise for some spans without features. For T5 in particular, a small set of samples have a large rise, but the additional pre-training of Flan-T5 may mitigate these effects. We will expand our Results and Limitations sections to discuss this behavior and the limitation of random masking.

Question 3: Figures 2/3 are hard to decipher

Thank you for bringing this to our attention. We will make the figures clearer in the revision.

Thank you for your review.

Reason to Reject 1: While the paper states that fine-tuning on data augmented with synthetic AAL features is not intended as an approach to mitigate biases, there may be ambiguity in how the fine-tuning process was conducted and its implications on model behavior. Clearer explanations or guidelines on the fine-tuning methodology and its impact on model performance are necessary for transparency and reproducibility.

All model checkpoints, prompts, generation hyperparameters, and fine-tuning hyperparameters are described in Appendices D-F (we direct readers to the Appendix in section 3.4). Thus, we include sufficient details for reproducing our experiments, and plan to release our code. If there are particular details the reviewer feels are missing, we will include them in the revision.

Reason to Reject 2: The paper emphasizes the importance of centering AAL speakers in decision-making processes . However, there is a lack of clarity on how the research engages with or involves the AAL community in the development and evaluation of the PhonATe approach. Involving relevant communities in the research process can provide valuable insights and ensure that the study is culturally sensitive and beneficial to those it aims to represent.

The development of PhonATe is drawn from sociolinguistic studies of AAL conducted with AAL-speakers. Some of the authors of this work are from the AAL community, including sociolinguistics researchers. Additionally, we note in the paper that the PhonATe evaluation is conducted by 2 AAL-speakers (“Augmentation Quality” on pg.4-5). We will further elaborate on how the AAL community was involved in this work in the Ethics statement.

Reason to Reject 3: ... Considering the sensitive nature of the tasks evaluated in the study (toxicity detection, sentiment analysis), there should be a thorough discussion on how potential biases in annotation or data collection were addressed to ensure the reliability and fairness of the results.

We do not collect any data and instead rely on existing fine-tuning corpora. To mitigate the impacts of annotators beliefs and avoid relying on potentially biased AAL annotations, we use the demographic-alignment classifier introduced by Blodgett et al (2016) to filter out all texts from the fine-tuning datasets predicted to reflect AAL given that they may have biased annotations (Paragraph 1 of Data section). We will elaborate further on this in the revision.