Probing Large Language Models for Scalar Adjective Lexical Semantics and Scalar Diversity Pragmatics

People often indicate messages on social media rather than directly say them by using scalar adjectives which vary in strength. For instance, saying ‘the movie is fine’ indicates that it is not excellent. However, not all scalar adjectives are equally strong in triggering implicatures. My master’s thesis focuses on (i) investigating whether LLMs understand the relative strength of scalar adjectives, and (ii) assessing and improving LLMs’ abilities to make more human-like decisions in reasoning about scalar implicatures triggered by those adjectives.

In this project, we develop a more robust algorithm to probe LLMs about their understanding for the strength of scalar adjectives compared to previous sota methods. In addition, we also find two training objectives which do not require large-scale annotated data by humans in scalar implicatures, but improve models’ performance in our task.