Introduction

• Contextualized language models have changed the NLP landscape
Language models that work at the sentence level
• word-level: one vector for a word
• contextualized: one vector for each word in a sentence
Can assign different vectors to the same word
He is running a company.

He is running a marathon.

BERT

• BERT: Transformer-based language model with 100M+ parameters
• Improved the state-of-the-art on many tasks (GLUE, SQuAD, SWAG)
• 6 pretrained checkpoints: 4 English, 2 multilingual (mBERT)
Many BERT-variants
• difference in the architecture, the training objective or the training data
Huggingface Transformers library
• PyTorch model classes
• over 1000 pretrained checkpoints
• dataset repository

Evaluating contextualized language models

• Blackbox models, difficult to evaluate
• Extrinsic evaluation: various benchmarks, mainly in English
• Intrinsic evaluation: training metrics
• Diagnostic classifiers or probing is simple and popular
• With or without finetuning?

Agenda

1. Overview of the methodology
2. Morphological evaluation against various baselines
3. Input perturbations, Shapley values
4. Choice of subword
5. Evaluation for Hungarian

Methodology

Probing

Also called diagnostic classifiers
Probing addresses the following questions:
• Does the model encode certain information?
• Can we recover this information?
Typical steps:
1. freeze the model (i.e. no finetuning)
2. extract the representation of a word/sentence
3. train a small classifier to predict something
4. if it performs well, the model 'contains' the information

Types of probing

type of probing data: word, sentence, sentence pair

probing location: token, token span (edge probing), sentence

probed information: morphology, POS, syntax, semantics

Probing architecture

1. Subword tokenization
2. Feed it to a language model
3. Pick a layer or sum them
4. Pick a subword
Feed its vector to a small MLP
• 1 hidden layer with 50 neurons
• ReLU
• ~40000 trainable parameters
5. Compute loss against a label

Morphological evaluation

Morphological probing tasks

• Defined as $\langle$ language, morph tag, part-of-speech $\rangle$
• What tense is cut in in this sentence?
  I cut my hair yesterday. Past
• Data generated from the Universal Dependencies Treebanks
• Train/dev/test: 2000/200/200

Baselines

1. fastText
2. WLSTM: character LSTM on the target word
3. SLSTM: character LSTM on the full sentence

Languages and tasks

• 319 tasks
• 39 languages
• 13 language families
• 4 POS
• 22 tag

Task statistics

Tasks by language

Average accuracy by POS

• Most tasks are 'easy' for both mBERT and the baselines
• Verbal morphology is easier than nominal
• Focus on SLSTM from here on

Verbal vs. nominal inflection

By language

• mBERT > SLSTM in more than half of the tasks in 28 out of 39 languages
• mBERT better at all tasks in Afrikaans, Dutch, Farsi, Hindi, Italian, Lithuanian, Serbian
• mBERT > SLSTM in less than half of the tasks in Albanian, Basque, Estonian, Finnish, Hebrew, Hungarian, Latin, Turkish, Urdu

Input perturbations

Shapley values

Perturbations

Perturbation effect

Remove a source of information from the sentence
Retrain the probe from scratch
Quantify the information loss

0 if the perturbation doesn't affect the probing accuracy
100 if it breaks it completely
negative if there is an improvement

Mean effects

Relation between perturbations

Observation 1:

Linear regression with $\text{RMSE}=5.43$

$ \text{BOW} \approx 1.22 \cdot B_2 + 1.42 $

Observation 2:

Left context is more important than right context

1. $ \text{Effect}(L_2) > \text{Effect}(R_2) $
2. $\rho(\text{PREV}, \text{TARG}) > \rho(\text{NEXT}, \text{TARG})$

Shapley value

• Shapley, 1951
• Solution conecpt in cooperative game theory
• A coaliation of players cooperates and obtains certain overall gain
• Some players contribute more, some may contribute negatively
• The Shapley value is the individual contribution of each player
• 100: contributes all, 0: contributes nothing, negative: even worse

• Tokens are players
• Relative positions to the target token
• 0: target, -1: previous token, 1: next token

Shapley values 1.

• Target is the most important (59), context 41
• Left context contributes more than right context (24.1 vs. 16.9)

Shapley values 2.

Shapley values 3. - Hindi and Urdu

Shapley values 4. - German

Typology

WALS

World Atlas of Languages

linguistic typology with 100+ fields for each language, many missing values

• we train logistic regression classifiers to predict a single WALS feature from perturbation effects alone
• 3-fold cross validation macro F-scores
• 15 most 'predictable' features

Paper and code

• About to submit a TACL paper
• Collaboration with Noah Smith, Roy Schwartz and András Kornai
• Dataset and data generator: github.com/juditacs/morphology-probes

Choice of subword

First vs. last

Morphology results

POS and NER

Paper and code

• Subword Pooling Makes a Difference, EACL2021
• Collaboration with Ákos Kádár and András Kornai
• https://arxiv.org/abs/2102.10864
• Code: https://github.com/juditacs/subword-choice

Evaluation for Hungarian

Models for Hungarian

Data

Segmentation

• All models use subword segmentation
• Fixed vocabulary
• Multilingual models share the vocabulary over all languages

Segmentation

Results - morphology

Results - POS

Results - POS

Results - NER

Paper and code

• Evaluating Contextualized Language Models for Hungarian, MSZNY21
• Collaboration with Dániel Lévai, Dávid Márk Nemeskey and András Kornai
• https://arxiv.org/abs/2102.10848
• Code: https://github.com/juditacs/hubert_eval

Thank you for your attention