The 13th Linguistic Annotation Workshop Co-located with ACL 2019 -- August 2019 -- Florence, Italy

• Home
• Call for Papers
• Programme
• Invited talk
• Committees
• Workshop Dinner

Invited Talk

Rebecca J. Passonneau

NLP Annotation and Essay Rubrics: Connecting Language, Learning and Critical Thinking

This talk will first present problematic issues in linguistic annotation that are inherent in the subjective nature of communication, then draw comparisons and contrasts between annotation in NLP, and rubrics in educational research.

The first part of the talk will discuss the elusive nature of the kinds of meaning that NLP annotation attempts to document. In my view, annotation can be described as the enterprise of making aspects of a message explicit that are implicit, and therefore subject to interpretation. I view meaning as essentially an ongoing negotiation over a somewhat fluid social contract. I will present some examples of annotation phenomena I am familiar with that illustrate this fluidity, such as coreference, word sense, and discourse segmentation. This will show that many questions we try to address through annotation do not have an objective answer that can be arrived at operationally. Despite this uncertainty, we as a community seem to believe that we can continue to chip away at the many different kinds of hidden dimensions of meaning by annotating them, and in this way lead to deeper revelations about language.

For many years, the main focus of NLP annotation has been to create datasets for training and evaluating machine learning algorithms, or to serve as benchmark evaluation datasets. By producing an algorithm that can automatically replicate human annotation, we demonstrate it is possible to computationally replicate deeper aspects of language, such as resolving coreferential expressions. But do we learn as much as we could about what coreference is? In education there is an enterprise that resembles annotation in many ways, but where the goal is to understand and have an impact on human learning. To test whether a proposed educational intervention has an impact on students' learning, researchers in education develop rubrics to assess students' reading and writing skills. The rubrics are tested for reliability, and are applied to students' written or oral discourse to compare students who receive a novel educational intervention to a control group. One of my current projects investigates how NLP could help automate educational rubrics. This same project has led me to consider how educational applications of NLP might shed light on the question of how meaning is constructed through language use, for example by students and instructors.

Bio

Rebecca J. Passonneau is a Professor in the Department of Computer Science and Engineering at Penn State University, in the area of natural language processing. Her main focus is computational pragmatics, meaning the investigation of how the same combinations of words have different meanings in different contexts, in spoken or written language. She received her PhD in 1985 from the University of Chicago Department of Linguistics, and worked at many academic and industry research labs before joining Penn State in 2016. Her work is reported in over 120 publications in journals and refereed conference proceedings. She has been a Principal Investigator on 26 sponsored projects with funding from 14 sources, including government agencies, corporate sponsors, corporate gifts, and foundations.