Data Annotation in the Era of LLMs - Thoughts and Good Practices

The rise of large language models (LLMs) presents both opportunities and challenges for data annotation in NLP. In this talk, I will explore the evolving role of LLMs as annotators, particularly in tasks involving subjectivity. I will present recent work that will also be presented in the conference on how to evaluate whether an LLM is a good annotator: "The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs"—highlighting methods introduced in our paper—and discuss how LLMs compare to human annotators in consistency and reliability. I will also introduce new, unpublished research on best practices for identifying when and how LLMs can serve as reliable annotators for subjective NLP tasks. The talk aims to provide both theoretical insights and practical guidance for researchers and practitioners rethinking annotation pipelines in the LLM era.

Bio

Dr. Dror is an Assistant Professor (Senior Lecturer) at the Department of Information Systems, University of Haifa. She completed her Postdoctoral Research at the Cognitive Computation Group at the Department of Computer and Information Science, University of Pennsylvania, Working with Prof. Dan Roth. she completed her Ph.D. in the Natural Language Processing Group, supervised by Prof. Roi Reichart, at the Faculty of Industrial Engineering and Management at the Technion - Israel Institute of Technology. Her research involves developing statistically sound methodologies for empirical investigation and evaluation for Data Science with a focus on Natural Language Processing applications.

Engaging experts and LLMs in corpora development

Large language models (LLMs) have become ever more capable, surpassing human performance on a number of tasks. Recent findings showed that LLMs can effectively replace traditional crowdsourcing to a large extent, and model training has increasingly been driven by synthetically generated data. These developments have triggered new questions about corpora development. This talk explores two of them: First, what type of human annotation can still be useful? I discuss our efforts engaging human expertise to effectively capture implicit reasoning in discourse and pragmatics, revealing weaknesses in existing models in those aspects. Second, how can we leverage LLMs to reveal task nuances that may be unknown before annotation? I present Explanation-Based Rescaling (EBR), a method that uses an LLM to rescale coarse-grained human ratings into consistent, fine-grained scores using natural language explanations from annotators, while discerning task subtleties embedded in these explanations.

Bio

Jessy Li is an Associate Professor in the Linguistics Department at the University of Texas at Austin. She received her Ph.D. (2017) from the Department of Computer and Information Science at the University of Pennsylvania. Her research interests are in computational linguistics and NLP, specifically discourse and document-level processing, natural language generation, and pragmatics. She is a recipient of an NSF CAREER Award, ACL and EMNLP Outstanding Paper Awards, an ACM SIGSOFT Distinguished Paper Award, among other honors. Jessy is the current Secretary of NAACL.