6 Benchmark Tasks for Language Modelling

This chapter assesses the progress of Natural Language Processing as a scientific discipline. We do so by considering applications of NLP that test the generality of a system or language model. The field of NLP comprises both practical applications and basic science.

Basic science refers to investigations where the primary goal is to predict phenomena or to understand nature. Basic science outcomes help us to develop technologies that alter events or outcomes. Early qualitative science NLP involved researchers looking at small examples of language phenomena and using them to better understand how human language processing works. This research has looked at what people do well and also when they make mistakes and seeing under what conditions computation models exhibit similar behavior. Inspiration for computational approaches comes from research in other disciplines, primarily psycholinguistics. Early work involved asking people to make judgements about whether they think a sentence is grammatical. More modern approaches use biosensors to track movements of the eyes or event-related potentials of the brain, as they process ambiguous sentences or resolve long distance dependencies, such as referring expressions^[1].

Basic science for NLP also includes developing models of human language. Language modeling involves learning parameters for complex mathematical expressions that measure or predict the likelihood of occurrences of a word or sentence for a given context. These models are used to implement methods for solving low-level NLP tasks that form the basis of more complex software applications. Four important examples of low-level NLP tasks include:

1. Labelling language units or pairs of units according to their grammaticality,
2. Labelling units with the type of sentiment or emotion they express
3. Labelling pairs of units as two whether they are semantically similar and
4. Labelling pairs of units as to their logical relationship.

They are sometimes called “benchmark” tasks for language processing because they are tasks that can be defined independent of a specific computational approach and thus offer a way to make them comparable.^[2] For example, having a method of correctly classifying sentences as ungrammatical might be useful for helping writers, teaching second language learners, or providing input to a classification-based parser. To date, there is no one modelling approach that can handle all these benchmark tasks well, which is a sign that the field of NLP has not yet solved the problem of creating general-purpose models of human language. In this chapter, we will consider each of the four  tasks.

6.1 Grammaticality Analysis

Grammaticality refers to whether an expression obeys the accepted syntactic conventions of a language, independent of whether those sentences make “sense”. A sentence can be both grammatical and meaningless; a classic example is “Colorless green ideas sleep furiously.”^{[3], [4]}. Grammaticality judgements are binary (yes or no). Since grammars are descriptions developed by observing the language of native speakers, decisions about what counts as grammatical must also come from native speakers of a language or expert observations. Grammaticality analysis can be used directly, such as for tools to support writing or to grade the writing of language learners, or it can be used to generate features for other tasks, such as assessing some cognitive disorders (e.g., the frequency of ungrammaticality has been associated with particular subtypes of autism spectrum disorder^[5]).

Approaches to grammaticality involve either comparing a given sentence against some model of perfect grammar or comparing a given sentence against some model of common errors, or a combination^[6]. Models of perfect grammar can consist of rules that enforce syntactic constraints at the sentence level. These constraints include that the subject and verb must agree (e.g., both be plural or singular); the pronoun must be correct form (case) for its role in the sentence (e.g., a subject “I” versus the object “me”); the noun must agree with its determiner (e.g., “This book” but not “These book”); the verb must in in the correct tense given its role (e.g. the present form of “be” expects the present participle, as in “am/is/are walking”); conjoined phrases should all have the same (parallel) structure; and words that take arguments, that is create requirements for the co-occurrence of other constructs, must occur with the correct pattern of arguments. There are also rules about the location of apostrophe when forming the possessive. Models of common errors can be rules that describe examples of violations of syntactic constraints and also rules for detecting problems that are primarily semantic, such as dangling modifiers, which are constructions where the entity being modified is either missing or ambiguous.

There are also specialized models of common errors, for different types of learners, such as children or people who are learning a second language. Children often have difficulty with homophones (such as “there”, “their” and “they’re”) or near-homophones (such as “then” and “than”) which sound similar, but are spelled differently. Second language learners have errors that occur when they mistakenly apply grammatical constraints from their first language that do not hold in the second language. For example, native speakers of Arabic sometimes omit the present form of “be” before an adjective or an indefinite article before a noun, because they are not used in these constructions in Arabic.

One of the first automated grammar checkers was EPISTLE^[7]. It takes what we would now consider to be a standard relaxation approach. The algorithm first attempts to parse a sentence while enforcing grammaticality constraints expressed as rules written by experts and then selectively relaxing some of the constraints, while noting the type and location of a violated condition. (The reason to look for known types of errors is to be able to decide whether a parser failed to derive a sentence because it lacks coverage or failed because the sentence being parsed has an error.) An alternative rule-based approach is based on extending a rule-based grammar to include explicit “malrules” that cover known types of errors and are marked with extra features that flag them as errors^{[8], [9]}. For example, a malrule might say that an erroneous noun phrase is one with a plural determiner and a singular head noun.

Both relaxation and malrules rely on search to find a rule that matches the suspect sentences. More recent approaches to grammaticality analysis use classification rather than search to determine the existence and type of the error.  One approach is to train models using a large corpus of grammatical text, then score unseen examples with a measure of their likelihood of being one of the training sentences and applying thresholds of similarity to determine if a writer’s usage is correct “enough” (that is close enough to a predicted example) given the context. More recent approaches use a corpus of erroneous examples to train models and then classify unseen sentences as one of the erroneous ones. Figure 6.1 shows some examples of errors found in writing assignments submitted by learners of English.

Resources for building models of correct grammar from data include syntactic tree banks (which are collections of sentences annotated with parse trees that have been vetted by experts)^[10]. Resources for building models of errors include sentences collected from published articles by trained scholars of linguistics that include examples of grammatical or ungrammatical sentences. One such collection is the Corpus of Linguistic Acceptability^[11]. Other collections of erroneous sentences include the NUS Corpus of Learner English (NUCLE)^[12], the Cambridge English Write & Improve (W&I) corpus, and the LOCNESS corpus, which are collections of essays written by second language learners that have been annotated. Another resource is ERRANT, a grammatical ERRor ANnotation Toolkit designed to automatically extract edits from parallel original and corrected sentences and classify them according to the type of error. Figure 6.1 shows 28 error types along with examples for each from the NUCLE corpus that have been annotated using ERRANT^[13].

6.2 Sentiment Analysis and Emotion Recognition

Sentiment Analysis attempts to capture the emotional aspects of language including opinions and evaluation. It originated from work that classified sentences as subjective or objective based on the particular words that they contain^{[14],[15],[16],[17]}. Particular words express the polarity of opinions expressed by a language unit, which might be positive (e.g., “liking”), negative (e.g., “not liking”), or neutral, which is often expressed using adjectives, such as “beautiful” vs “ugly”. Sentences are considered “objective” if they do not include any expressed opinions. Figure 6.2 shows examples of subjective and objective sentences. The sentence on the left is considered subjective because the word “fascinating” is considered positive (in contrast to saying “boring” or “trite”); the sentence on the right is objective because “increase” is considered factual (without implying the increase was too much or too little).

Sentiment often involves just labelling the general polarity, which might be positive, negative, or neutral. More fine-grained approaches classify particular levels of sentiment.  For example, the Stanford Sentiment Treebank uses continuous values ranging from 1 to 25, where 1 is the most negative and 25 is the most positive. Figure 3 below shows some examples (taken from movie reviews) and the scores that the sentences received using their algorithm.

Beyond sentiment, there are also approaches that try to capture various types of emotion, such as anger, excitement, fear, joy, sadness, etc. These tasks are not included in the benchmarks, but might be in the future. Training data would be more difficult to create as labelling examples with such categories requires specialized expertise. However, associating words with emotions has long been part of the qualitative analysis of human language performed by some psychologists^[18],[19] and incorporated into some tools, such as Linguistic Inquiry and Word Count (LIWC)^[20].

Freely available tools for sentiment analysis include VADER, TextBlob, and Sentistrength. Some of the most recent open source tools use Deep Learning (e.g., from the Open Data Hub). Resources for building sentiment analysis include movie and product reviews in the public domain, which typically include an explicit rating that can be mapped to a sentiment polarity value. Examples of public standardized corpora includes the Stanford Sentiment Treebank^[21] which was derived from a sentence-level corpus of movie reviews^[22]. The LIWC tools mentioned above are available for a licensing fee.

6.3 Semantic Similarity

Semantic similarity, also called semantic textual similarity, is the notion that two expressions mean approximately the same thing (e.g.,  they are paraphrases of each other). Similarity is thus a symmetric relationship – when comparing two units neither would be more general or specific than the other. Similarity can arise at the word level, through the synonyms, or at the sentence level, where one might reorder the parts of a conjunction or substitute an active construction for a passive one. Being able to detect when two expressions mean “nearly” the same thing is useful for assessing whether a student has answered a test question correctly, or when trying to determine the intent of a question or command without requiring the designer to list every possible expression explicitly. Systems that require one to say things in exactly one way make it difficult for users to learn or remember the required phrasing.

Systems that map expressions onto sentences in a formal logic to test subsumption (which they refer to as classification inference) are performing a type of semantic similarity analysis. A rule-based approach would create a deep-semantics  (such as an expression in first order logic or description logic) and then test whether a pair of concepts (A, B) are equivalent, that is, both A ⊆ B and B ⊆ A hold. Today, semantic similarity is a task that can be learned from data that includes pairs of expressions that have been previously deemed to be similar. Figure 6.4 includes some examples of similarity judgements given by staff at Microsoft.

Three resources for semantic similarity that have been proposed as a standard for evaluating work on similarity are: The Microsoft Research Paraphrase Corpus^[23], the Quora Question Pairs dataset^{[24], [25]}, and  the Semantic Similarity Benchmark corpus from SemEval 2017^[26]. The first two of these corpora include text collected from online news and questions posted to a community question answering website where equivalence judgements were provided by staff at the respective organizations (e.g., Microsoft and Quora). The SemEval data includes video and image captions created via crowdsourcing and judged by organizers and participants of the conference.  One newer dataset that may prove useful is the multi-domain question rewriting dataset, which was created from Stack Exchange question edit histories by researchers at Google, University of Chicago, and Toyota Technological Institute^[27].

6.4 Textual Entailment

Textual entailment,  also known as natural language inference,  is an approximation of real inference that has been formalized to allow for  using classification as a solution. There are three categories: a given text either entails another text, contradicts another text, or neither, which is classified as “neutral”. (There are examples in Figure 6.5.) By convention, the entailing text is known as “the text” or “the premise” and the other text is known as “the hypothesis”. A search-based system might create a semantics using first order logic and apply a theorem prover. A more efficient approach would use a graph-based representation, such as a subsumption hierarchy. Recent work by Young et al. (2014) explores combining a statistical measure (conditional probability or mutual information) and a graph structure, to form a hybrid structure that they call a Denotation Graph^[28].

Classification-based approaches for Textual Entailment have also been devised, using a training set where items have both generated and then labelled by hand for different entailment relations, typically by crowdsourcing, using the Amazon Mechanical Turk platform.  The conventionally used entailment categories for this approach are defined as follows:

Entailment: the hypothesis states something that is definitely correct about the situation or event in the premise.

Neutral: the hypothesis states something that might be correct about the situation or event in the premise.

Contradiction: the hypothesis states something that is definitely incorrect about the situation or event in the premise.

The collections of sentence pairs that have been created are designed to exemplify several known sources of entailment, from low-level word meanings and sentence structure to high-level reasoning and application of world knowledge. They capture four broad categories of phenomena: lexical semantics, predicate-argument structure, logic, and world and common sense knowledge, where each category may have several subtypes. We will now consider the general idea for each.

Lexical semantics or word meaning includes entailments based on concept abstraction or exclusivity, such as that a cat is a type of mammal and a cat is not a dog. It also includes morphological negation (e.g. agree vs. disagree, happy vs. unhappy, etc). It also includes entailments based on verbs, that in normal usage warrant certain inferences, such as saying “I stopped doing X” entails one was doing X previously and saying “I recognize that X” entails X. Also, saying “The cat is fat.” entails that the cat exists. (Linguists sometimes refer to these various types of expressions as presuppositions or factives.)

Predicate-argument structure includes a verb and its subject and object, which may tell you who did something and what was acted upon (their semantic role). The order of the roles depends on the syntax. For example both “The cat broke the vase.” and “The vase was broken by the cat.” entail that the vase broke.

Logic includes entailments that may arise because of connectives (conjunction, disjunction), negation, double negation, conditionals, and quantifiers. Logic also includes entailments based on specific types of entities, such as numbers and intervals of time, which have an associated magnitude and sequential order, and operators defined on them, such as less than or greater than, and before or after. Logic-based entailments (mostly) follow the semantics of mathematical logic. For example, “the cat sat on the mat and slept” entails “the cat slept” and “the cat slept” entails “the cat slept or the cat ate”, but “the cat slept or ate” is neutral about “the cat slept”. Conditionals include sentences like “if it is raining, the grass will be wet”, which would not entail that “the grass is wet” or “it is raining” or even “it is raining and the grass is wet”; however, a complex conditional such as “It is raining and if it is raining, then the grass will be wet” would entail that “the grass is wet.” Quantifiers of logic include the universal (all) and the existential (some) which create entailments based on their semantics, e.g., “All cats have fur.” entails “my cat has fur” and “my cat likes fish” entails that “some cats like fish”. Natural language includes additional quantifiers such as “most” or “most X in the Y”. Determining the entailments of these quantifiers requires judgements that combine an understanding of their general meaning and world or common sense knowledge.

World and common sense knowledge includes entailment relations based on knowledge of history, geography, law, politics, science, technology, culture, other aspects of human experience. An example might be that “Milwaukee has some beautiful parks.” entails that “Wisconsin has some beautiful parks.” Common sense includes entailment relations that are not exactly factual, but do not depend on either cultural or educational background. For example, “A girl was making snow angels” entails “a girl is playing in the snow” and “the grass is wet” entails “it might have rained”. Some examples from the Stanford Natural Language Inference dataset, showing the Text-Hypothesis pairs and the crowdsourced inference type, are shown in Figure 6.5

Textual entailment recognition is a classification task that can support several different software applications including text summarization, question answering, information retrieval, and machine translation. For example, summaries can be shortened by removing any sentences that are already entailed by other sentences in the summary. In question answering, any acceptable candidate answer found in a source document must entail the ideal answer, which we are presumed to already know. In information retrieval, search criteria might be given in terms of sentences that the desired documents must entail. Textual entailment being independent of any application has also been used as a benchmark task for natural language, one that could be used to evaluate and compare the effectiveness of natural language models across a variety of applications. Currently there are a number of software tools and datasets available for creating and evaluating systems for textual entailments. Functions for performing textual entailment are included in (or available for) AllenNLP and spaCy.

Some resources that have been proposed as a standard for evaluation include: the Multi-Genre Natural Language Inference Corpus, which includes examples drawn from transcribed speech, popular fiction, and government reports which were labelled via crowdsourcing^[29]; the Stanford Natural Language Inference corpus^[30]; the Stanford Question Answering Dataset^[31]; the Recognizing Textual Entailment datasets, which come from a series of annual challenges, first held in 2004^[32]; and the data from the Winograd Schema Challenge^[33].

6.5 Summary