LysandreJik released this
Aug 15, 2019
· 3 commits to master since this release
RoBERTa (from Facebook), a Robustly Optimized BERT Pretraining Approach by Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du et al.
Thanks to Myle Ott from Facebook for his help.
Tokenizers get two new methods:
These methods add the model-specific special tokens to sequences. The sentence pair creates a list of tokens with the cls and sep tokens according to the way the model was trained.
[CLS] SEQUENCE_0 [SEP] SEQUENCE_1 [SEP]
[CLS] SEQUENCE_0 [SEP] [SEP] SEQUENCE_1 [SEP]
The tokenizer encode function gets two new arguments:
tokenizer.encode(text, text_pair=None, add_special_tokens=False)
If the text_pair is specified, encode will return a tuple of encoded sequences. If the add_special_tokens is set to True, the sequences will be built with the models' respective special tokens using the previously described methods.
There are three new classes with this release that instantiate one of the base model classes of the library from a pre-trained model configuration: AutoConfig, AutoModel, and AutoTokenizer.
Those classes take as input a pre-trained model name or path and instantiate one of the corresponding classes. The input string indicates to the class which architecture should be instantiated. If the string contains "bert", AutoConfig instantiates a BertConfig, AutoModel instantiates a BertModel and AutoTokenizer instantiates a BertTokenizer.
The same can be done for all the library's base models. The Auto classes check for the associated strings: "openai-gpt", "gpt2", "transfo-xl", "xlnet", "xlm" and "roberta". The documentation associated with this change can be found here.
Some examples have been refactored to better reflect the current library. Those are: simple_lm_finetuning.py, finetune_on_pregenerated.py, as well as run_glue.py that has been adapted to the RoBERTa model. The examples run_squad and run_glue.py have better dataset processing with caching.
thomwolf released this
Jul 16, 2019
· 4 commits to master since this release
pytorch-pretrained-bert => pytorch-transformers
Install with pip install pytorch-transformers
pip install pytorch-transformers
We went from ten (in pytorch-pretrained-bert 0.6.2) to twenty-seven (in pytorch-transformers 1.0) pretrained model weights.
The newly added model weights are, in summary:
The documentation lists all the models with the shortcut names and we are currently adding full details of the associated pretraining/fine-tuning parameters.
New documentation is currently being created at https://huggingface.co/pytorch-transformers/ and should be finalized over the coming days.
See the readme for a quick tour of the API.
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased',
input_ids = torch.tensor([tokenizer.encode("Let's see all hidden-states and attentions on this text")])
all_hidden_states, all_attentions = model(input_ids)[-2:]
Using tokenizer.add_tokens() and tokenizer.add_special_tokens(), one can now easily add tokens to each model vocabulary. The model's input embeddings can be resized accordingly to add associated word embeddings (to be trained) using model.resize_token_embeddings(len(tokenizer))
The serialization methods have been standardized and you probably should switch to the new method save_pretrained(save_directory) if you were using any other serialization method before.
### Reload the model and the tokenizer
model = BertForSequenceClassification.from_pretrained('./my_saved_model_directory/')
tokenizer = BertTokenizer.from_pretrained('./my_saved_model_directory/')
All models are now compatible with Torchscript.
model = model_class.from_pretrained(pretrained_weights, torchscript=True)
traced_model = torch.jit.trace(model, (input_ids,))
The examples scripts have been refactored and gathered in three main examples (run_glue.py, run_squad.py and run_generation.py) which are common to several models and are designed to offer SOTA performances on the respective tasks while being clean starting point to design your own scripts.
Other examples scripts (like run_bertology.py) will be added in the coming weeks.
The migration section of the readme lists the breaking changes when switching from pytorch-pretrained-bert to pytorch-transformers.
The main breaking change is that all models now returns a tuple of results.