Annotation Captcha

My project is a captcha that creates training data for natural language processing.

The project will contribute to semantic parsing research by creating annotated training data for classifying the entailment relation of a substitution edit between two texts. There are no existing, public corpora for this task, as far as I know. For instance, if the word ‘buddy’ is substituted for the word ‘friend’ in a text, this edit’s entailment relation can be classified as equivalency. That is, ‘friend’ and ‘buddy’ are synonyms, and are therefore equivalent. If an edit consists of substituting the word ‘hamburger’ for the word ‘food’, this edit can be classified as forward entailment, as ‘hamburger’ is a hyponym of ‘food’. Entailment relations can be composed across a sequence of edits between two texts to determine if one text is logically entailed by the other. State-of-the-art research in composing entailment relations is described in this paper. An implementation of the described system of natural logic would be able to automatically determine that the following hypothesis is entailed by the the following premise:

premise: Every firm polled saw costs grow more than expected, even after adjusting for inflation.
hypothesis: Every big company in the poll reported cost increases.

Textual entailment could be applied to various natural language understanding tasks, including question answering.

I will use six entailment relations:

  • Equivalence: the phrases in the edit are synonyms or aliases. For example, ‘Mitt Romney’ and ‘Romney’, or ‘dull’ and ‘boring’.
  • Forward entailment: the first phrase is a hyponym of the second phrase. For example, ‘apple’ and ‘fruit’.
  • Reverse entailment: the first phrase is a hypernym of the second phrase. For example, ‘fruit’ and ‘apple’.
  • Negation: the pair are antonyms. For example, ‘big’ and ‘small’.
  • Alternation: the pair are coordinate terms, meronyms or holonyms. For example, ‘red’ and ‘green’.
  • None: none of the above relations exist between the pair. For example, ‘hats’ and ‘lively’.

Lexical resources like WordNet cannot be used to automatically classify entailment relations as they are too incomplete; WordNet does not cover most proper nouns, coordinate terms, and any relation between phrases that include intersective modifiers (eg., ‘mini-robot’, ‘black rook’). Furthermore, WordNet does not include hypernymy and hyponymy relations for all parts of speech. Instead, humans will annotate the entailment relation, and synonymy, hyponymy, and hypernymy relations extracted from WordNet will be used as features by the classifier. Therefore, if a pair of terms is listed in WordNet as synonyms, this evidence will contribute to the pair’s classification as equivalence, but other pairs that are omitted from WordNet could still be classified as equivalence using different features (for example, string similarity).

Annotating entailment relations can be challenging, so care will have to be taken in presenting the annotation task.

Two phrases will be presented to the user. For example:

Lunch, Meal

Six buttons will be presented to the user. For example:

Choose the sentence that best describes the relation between lunch and meal:

  • Lunch is a type of meal.
  • Meal is a type of lunch.
  • Meal and lunch are the same. All types of meals are lunches and all types of lunches are meals.
  • Meal is the opposite of lunch.
  • Meal and lunch are two types of the same thing, just like red and green are two types of colors.
  • There is no relation between lunch and meal.

It may still be challenging to get accurate annotations, particularly for the alternation relation. Each pair will need to be annotated several times, and will only be accepted if the agreement between the annotators exceeds some threshold. As this is a captcha, some validation will be required to block bots. Any captcha can be defeated by determined hackers; hopefully the obscurity of this captcha will provide some defense. I may require users to type one of the six options presented to submit as an answer as a basic defense against bots.


Leave a Reply

Your email address will not be published. Required fields are marked *


You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>