Lawma: The Power of Specialization for Legal Annotation

Overlooked Generalizability: [...] Without comparisons on datasets from different jurisdictions or areas of law, it’s challenging to assess the model’s broader applicability. [...] Can the authors provide insights or preliminary findings on how well the Lawma models might transfer to legal data from jurisdictions outside the U.S. or to other branches and tasks of law?

We study how much task-specific fine-tuning might generalize in Section 4.4 and in particular Figure 9. We observe that fine-tuning only on the Court of Appeals database results in a mean case accuracy of 51.6%, compared to 82.4% for Lawma 8B (Section 4.4). That is, not fine-tuning on Supreme Court leaves a lot of accuracy on the table, 30.9 accuracy points to be precise. It is plausible that considering different jurisdictions or other branches of the law might result in even more stark results.

Our recommendation is therefore to always specialize on the particular legal tasks of interest, as performance may otherwise be poor. Fortunately, hundreds of labelled examples are often sufficient to obtain large performance gains (Section 4.3). Thus, for large-scale data annotation the following strategy may be highly beneficial: collect a few hundred labeled examples using human annotators, fine-tune an open-weights model, and use the fine-tuned model to annotate the remaining cases at scale.

Thank you for your thoughtful review.

Limited Novelty in Contribution: Although the paper effectively demonstrates the value of task-specific fine-tuning, the concept itself is not novel. Prior work has already shown that fine-tuned, specialized models outperform general-purpose LLMs [1, 2, 3].

Frontier LLMs are state-of-the-art in many domains, such as mathematical reasoning (e.g., MATH), code (e.g., HumanEval), reasoning over text (e.g., DROP), and graduate-level question answering (e.g., GPQA). Prior to our work, it was not at all evident that specialized open-weights models could match the performance of frontier models on legal annotation, let alone substantially outperform them. It is precisely for this reason that recent works that use LLMs for legal annotation rely heavily on frontier models (e.g, [1, 2, 3, 4]).

Please note that the references provided focus on more “classical” NLP tasks such as named entity recognition, natural language inference, or sentiment analysis. We will make sure to cite and discuss the provided references in the related work.

[1] Saromir Savelka and Kevin D Ashley. The unreasonable effectiveness of large language models in zero-shot semantic annotation of legal texts. Frontiers in Artificial Intelligence, 6, 2023.

[2] Michael A Livermore, Felix Herron, and Daniel Rockmore. Language model interpretability and empirical legal studies. Virginia Public Law and Legal Theory Research Paper, (2023-69), 2023.

[3] Jens Frankenreiter and Eric L Talley. Sticky charters? the surprisingly tepid embrace of officer-protecting waivers in delaware. European Corporate Governance Institute-Law Working Paper, (762), 2024

[4] Morgan A Gray, Jaromir Savelka, Wesley M Oliver, and Kevin D Ashley. Empirical legal analysis simplified: reducing complexity through automatic identification and evaluation of legally relevant factors. Philosophical Transactions of the Royal Society A, 382(2270):20230155, 2024.

Exploration of Advanced Prompting Techniques

We have now evaluated the models using zero-shot chain of thought (CoT). We follow the standard methodology of eliciting CoT by appending to the prompt “Let’s think step by step.” Since CoT requires two orders of magnitude more compute for evaluation than the standard QA approach, we only evaluate Llama 3 8B Instruct and Llama 3 70B Instruct. This required over 500 H100 GPU hours. We observe that CoT leads to modest improvements of performance for both the 8B and 70B model, on average of 2 to 3 accuracy points, see Figure 17 in Appendix E.5. Nonetheless, Lawma 8B still strongly outperforms Llama 3 70B, by over 20 accuracy points.

Narrow Focus on Text Classification

We study precisely the narrow specialization of models for legal annotation. This is a feature of work. We show that specialized models strongly outperform generalist models such as GPT-4. Such narrow specialized models have rich scientific applications for empirical legal studies and the broader “law as data” research paradigm. Entire subfields of legal scholarship, political science, economics and sociology build on law as data. Other NLP tasks (i.e. text generation) are very much of secondary importance for this domain of research.

What are some examples of typical errors Lawma makes, especially on complex tasks?

The types of errors made by Lawma are highly task dependent, and thus it is difficult to draw broad conclusions. Let us give one illustrative example of a failure case. For the Supreme Court issue area classification, Lawma tends to miss classify habeas corpus cases as “Criminal Procedure” rather than “Civil rights” cases, since the language of habeas corpus cases tends to be more similar to that of criminal cases. More broadly, we find that Lawma tends to excel in tasks for which only surface correlations (e.g., the “language” of the case) suffice for accurate prediction, but struggles for tasks that require deeper understanding of the substantive aspects of the case. Lawma also tends to perform worse for tasks that have fewer training data and larger number of classes.

Thank you for your thoughtful and positive review.

How did you make the prompts for the questions? Who wrote the explanation for each of the legal variables provided by USCAD? How are the authors sure that these are accurate representations of the classification?

The prompts follow the MMLU multiple-choice question answering style that is popular for LLM evaluations. For the general description of each task, we take the codebook’s (either SCDB or USCAD) description of the corresponding variable in the database. We make only very minor modifications to fit an instruction style (e.g., “This field identifies the forum that heard this case immediately before the case came to the court of appeals.” -> “Your task is to identify the forum that heard this case immediately before the case came to the court of appeals.”). Since the descriptions are directly taken from the databases’ extensive codebooks, we are sure that they accurately reflect the underlying legal annotation tasks.

What license does this fall under and will it be publicly released?

The benchmark, fine-tuning dataset, model weights, and all code used to construct CaselawQA and fine-tune the Lawma models are publicly available under an MIT License; however, they cannot be directly linked here to preserve anonymity.

Thank you for your thoughtful review.

There is little exploration of the relative difficulty of the different types of tasks they introduce—it would be useful to know more about which tasks that GPT-4 outperformed on, and where fine-tuning had minimal vs. substantial gains.

Fortunately, the Court of Appeals database offers one measure of relative task difficulty for human coders: intercoder agreement. We discuss how intercoder agreement compares with Lawma accuracy in Appendix C. Our findings speak to the non-trivial nature of the task suite as a classification benchmark: Lawma is far from the intercoder agreement rate for most tasks, see the newly added Figure 10. The linear fit in Figure 10 has slope 1.08 and intercept -15, indicating that Lawma is on average 15 accuracy points below the intercoder agreement rate irrespective of intercoder agreement. But there is large variability across tasks. In fact, there are many tasks (e.g, “CIRCUIT”) with perfect intercoder agreement rate, no class imbalance, and yet for which Lawma is far from the agreement rate.

Regarding what tasks GPT-4 outperformed Lawma, it is the following 7/260 tasks: songer_const1, songer_summary, sc_threejudgefdc, songer_casetyp1_1-3-3, songer_st_v_st, sc_respondentstate, songer_casetyp1_2-3-2. These tasks do not share much in common, are neither particularly easy nor challenging, and there are many similar tasks to those 7 (e.g., songer_const2, sc_petitionerstate, songer_casetyp1_1-3-2) for which Lawma does outperform GPT-4.

We find that fine-tuning leads to large performance gains across all tasks. Lawma 8B improves upon the performance of Llama 3 8B Instruct on average by 37 accuracy points, with the 5th percentile of improvement being 10 accuracy points. The tasks with lowest improvements tend to be those associated with finding specific case issues (songer_casetyp*), or the nature of the appellant (songer_appel*) or respondent (songer_respond*). These tasks are particularly challenging because they are highly specific and not much training data is available.

[Are the proposed tasks and their difficulty] comprehensive wrt the legal activities of humans?

The proposed tasks are derived directly from the Supreme Court Database and the Songer Court of Appeals Database, the two most widely used resources in empirical legal research. Our proposed tasks are therefore highly representative of the types of annotation tasks that concern empirical legal scholarship.

“The costs and error of existing methods is the single most important bottleneck in the empirical legal studies pipeline.” (39-40) is vague and needs a citation.

Michael A. Livermore and Daniel N. Rockmore. Law as Data: Computation, Text, & the Future of Legal Analysis. Santa Fe Institute Press, 2019.

What does "mixed answer" mean in Appendix G?

It means that the Court considered the question but gave a "mixed" answer - for example, when the Court supported the respondent in part and supported the appellant in part, or if two issues treated separately by the court both fell within the area covered by one question and the court answered one question affirmatively and one negatively.

Is there any transferability across tasks—i.e., if you train a model on a subset of the task, how does it perform on the held-out tasks?

Yes, we observe transferability across tasks, see Section 4.4 and in particular Figure 9. Specifically, fine-tuning only on the Court of Appeals tasks improves mean accuracy on the Supreme Court tasks by up to 18.8% accuracy points.

Do you anticipate that in the future all subtasks of legal reasoning need to be spelled out, or is there a critical mass of legal subtasks that accrue towards "legal AGI"?

We can only confidently speak about the current state of affairs. We observe that fine-tuning only on the Court of Appeals database results in a mean case accuracy of 51.6%, compared to 82.4% for Lawma 8B (Section 4.4). That is, not fine-tuning on Supreme Court cases results in a 30.9 accuracy points drop in performance. Our results highlight the importance of fine-tuning precisely on the target tasks of interest.

Fortunately, hundreds of labelled examples are often sufficient to obtain substantial performance gains (Section 4.3). Therefore, our recommendation for legal scholars is at present time the following: obtain a few hundred labeled examples using human annotators, fine-tune an open-weights model, and use the fine-tuned model to annotate the remaining cases.

Thank you for the thoughtful, detailed, and positive review.

the central result that fine-tuning can outperform prompting has appeared in many places

We believe that our results are highly interesting and consequential not because fine-tuning leads to performance improvements, but rather because we show that specializing a “small” open-weights model substantially outperforms the much larger GPT-4 model, which is what legal scholars currently tend to rely on when considering computational annotation of Court opinions. This is our central result.

there are several pre-existing benchmark legal datasets to which this adds another one, and not with a new type of task (all of them have many text classification tasks)

CaselawQA focuses on the annotation of entire Court opinions, which is of critical interest for empirical legal scholars, since the costs and error of existing methods is the single most important bottleneck in the empirical legal studies pipeline. We believe that the scale and richness of CaselawQA, containing almost all variables of the Supreme Court Database and U.S. Court of Appeals Database, makes it a valuable addition to the current ecosystem of legal benchmarks.

Would not it actually be useful to show performance on LegalBench? It would be a useful test of transfer, beyond section xx, and give people a better indication of how useful Lawma would be in general for legal tasks with/without doing further fine-tuning?

We study how much task-specific fine-tuning might generalize in Section 4.4 and in particular Figure 9. For LegalBench, we would expect even more stark results, as most LegalBench tasks differ substantially from those considered in our work. Put shortly, we do not recommend using the Lawma models for tasks beyond those considered in our work. Our recommendation is to always specialize models for the specific legal tasks they are intended to perform. Not doing so needlessly leaves a lot of performance on the table. Fortunately, we show that hundreds of labelled examples are often sufficient to obtain large performance gains, at least in many annotation tasks (Section 4.3).

Our results suggest the viability of the following strategy for large-scale legal annotation: to obtain a few hundred labeled examples using human annotators, fine-tune an open-weights model, and use the fine-tuned model to annotate the remaining cases at scale.

[Regarding Figure 7] These results indicate that more pre-training data still really helps (well, at least very clearly on the Supreme Court tasks).

We agree that pre-training on more data might still help substantially, at least for the Supreme Court tasks, whereas simply scaling model size might not. We will make this point more clear.

Thanks for your follow up to my comments!