Why doesn't BertForMaskedLM generate right masked tokens? - nlp

Question

I am testing this piece of code:
from transformers import BertTokenizer, BertModel, BertForMaskedLM
tokenizer = BertTokenizer.from_pretrained("hfl/chinese-roberta-wwm-ext")
model = BertForMaskedLM.from_pretrained("hfl/chinese-roberta-wwm-ext")
from transformers import pipeline
def check_model(model, tokenizer):
fill_mask = pipeline(
"fill-mask",
model=model,
tokenizer=tokenizer
)
print('Fill blank: ')
fill_mask("我喜欢 {nlp.tokenizer.mask_token}.")
print('Fill blank: ')
fill_mask("这个品牌的面膜 {nlp.tokenizer.mask_token}.")
print('Check model ...')
check_model(model, tokenizer)
But it prints out this error message:
raceback (most recent call last):
File "/Users/congminmin/nlp/embedding/transformer/bert_roberta_wwm_test.py", line 21, in <module>
check_model(model, tokenizer)
File "/Users/congminmin/nlp/embedding/transformer/bert_roberta_wwm_test.py", line 15, in check_model
fill_mask("我喜欢 {nlp.tokenizer.mask_token}.")
File "/Users/congminmin/.venv/wbkg/lib/python3.7/site-packages/transformers/pipelines/fill_mask.py", line 162, in __call__
self.ensure_exactly_one_mask_token(masked_index.numpy())
File "/Users/congminmin/.venv/wbkg/lib/python3.7/site-packages/transformers/pipelines/fill_mask.py", line 90, in ensure_exactly_one_mask_token
f"No mask_token ({self.tokenizer.mask_token}) found on the input",
transformers.pipelines.base.PipelineException: No mask_token ([MASK]) found on the input

Answer 1

That is a string formatting issue. Currently when you call:
"这个品牌的面膜 {nlp.tokenizer.mask_token}."
the string you create is:
'这个品牌的面膜 {nlp.tokenizer.mask_token}.'
What you actually want to do is (formated string literals):
f"我喜欢 {fill_mask.tokenizer.mask_token}."
Output:
'我喜欢 [MASK].'
Full example:
from transformers import BertTokenizer, BertModel, BertForMaskedLM
tokenizer = BertTokenizer.from_pretrained("hfl/chinese-roberta-wwm-ext")
model = BertForMaskedLM.from_pretrained("hfl/chinese-roberta-wwm-ext")
from transformers import pipeline
def check_model(model, tokenizer):
fill_mask = pipeline(
"fill-mask",
model=model,
tokenizer=tokenizer
)
print('Fill blank: ')
print(fill_mask(f"我喜欢 {fill_mask.tokenizer.mask_token}."))
print('Fill blank: ')
print(fill_mask(f"这个品牌的面膜 {fill_mask.tokenizer.mask_token}."))
print('Check model ...')
check_model(model, tokenizer)
Output:
Some weights of the model checkpoint at hfl/chinese-roberta-wwm-ext were not used when initializing BertForMaskedLM: ['cls.seq_relationship.bias', 'cls.seq_relationship.weight']
- This IS expected if you are initializing BertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Check model ...
Fill blank:
[{'sequence': '我 喜 欢 他.', 'score': 0.20969171822071075, 'token': 800, 'token_str': '他'}, {'sequence': '我 喜 欢 你.', 'score': 0.2071659415960312, 'token': 872, 'token_str': '你'}, {'sequence': '我 喜 欢 她.', 'score': 0.13876770436763763, 'token': 1961, 'token_str': '她'}, {'sequence': '我 喜 欢 的.', 'score': 0.07542475312948227, 'token': 4638, 'token_str': '的'}, {'sequence': '我 喜 欢 它.', 'score': 0.05587303638458252, 'token': 2124, 'token_str': '它'}]
Fill blank:
[{'sequence': '这 个 品 牌 的 面 膜 好.', 'score': 0.15848451852798462, 'token': 1962, 'token_str': '好'}, {'sequence': '这 个 品 牌 的 面 膜..', 'score': 0.12413082271814346, 'token': 119, 'token_str': '.'}, {'sequence': '这 个 品 牌 的 面 膜 呢.', 'score': 0.09926403313875198, 'token': 1450, 'token_str': '呢'}, {'sequence': '这 个 品 牌 的 面 膜 啊.', 'score': 0.06865812838077545, 'token': 1557, 'token_str': '啊'}, {'sequence': '这 个 品 牌 的 面 膜 1.', 'score': 0.061997584998607635, 'token': 122, 'token_str': '1'}]