Do I need to train a tokenizer when training SBERT with MLM?

Question

I have trained a SBERT model with MLM on my own corpus which is somewhat domain specific using these guides:

https://ireneli.eu/2021/03/28/deep-learning-19-training-mlm-on-any-pre-trained-bert-models/ https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_mlm.py

When I saved a tokenizer with

tokenizer.save_pretrained(output_dir)

it created a set of files. So I opened a vocab.txt and tried to search for some domain specific words but I could not find them.

1. Do I need to train a tokenizer as well on my corpus?
2. If so, then do I need to retrain the SBERT model again using MLM? (which would be really disappointing since I don't have a GPU and had to train in the cloud paying for GPU)
3. Is the model I have now useless if I use it with the original tokenizer that lacks domain specific words?

Answer 1

Normally, the vocab file should have tokens (pieces of words) instead of words: is it the case?

Remember that you can select how tokens are made: either full words, or smaller pieces of them.

Then, if your vocab is new, you need to train your model from scratch because the tokens are the ground base of semantic reference of the model.

Note: you could have also good results using a pre trained model, overall if your data set is small and not very varied.