LysandreJik released this
Oct 11, 2019
· 2 commits to master since this release
Two new models have been added since release 2.0.
Several updates have been made to the distillation script, including the possibility to distill GPT-2 and to distill on the SQuAD task. By @VictorSanh.
The run_glue.py example script can now run on a Pytorch TPU.
Several example scripts have been improved and refactored to use the full potential of the new tokenizer functions:
Enhancements have been made on the tokenizers. Two new methods have been added: get_special_tokens_mask and truncate_sequences .
The former returns a mask indicating which tokens are special tokens in a token list, and which are tokens from the initial sequences. The latter truncate sequences according to a strategy.
Both of those methods are called by the encode_plus method, which itself is called by the encode method. The encode_plus now returns a larger dictionary which holds information about the special tokens, as well as the overflowing tokens.
Thanks to @julien-c, @thomwolf, and @LysandreJik for these additions.
The two methods add_special_tokens_single_sequence and add_special_tokens_sequence_pair have been removed. They have been replaced by the single method build_inputs_with_special_tokens which has a more comprehensible name and manages both sequence singletons and pairs.
The boolean parameter truncate_first_sequence has been removed in tokenizers' encode and encode_plus methods, being replaced by a strategy in the form of a string: 'longest_first', 'only_second', 'only_first' or 'do_not_truncate' are accepted strategies.
When the encode or encode_plus methods are called with a specified max_length, the sequences will now always be truncated or throw an error if overflowing.
New contributing guidelines have been added, alongside library development requirements by @rlouf, the newest member of the HuggingFace team.
thomwolf released this
Sep 26, 2019
· 26 commits to master since this release
Following the extension to TensorFlow 2.0, pytorch-transformers => transformers
Install with pip install transformers
pip install transformers
Also, note that PyTorch is no longer in the requirements so don't forget to install TensorFlow 2.0 and/or PyTorch to be able to use (and load) the models.
All the PyTorch nn.Module classes now have their counterpart in TensorFlow 2.0 as tf.keras.Model classes. TensorFlow 2.0 classes have the same name as their PyTorch counterparts prefixed with TF.
The interoperability between TensorFlow and PyTorch is actually a lot deeper than what is usually meant when talking about libraries with multiple backends:
Training on TPU using free TPUs provided in the TensorFlow Research Cloud (TFRC) program is possible but requires to implement a custom training loop (not possible with keras.fit at the moment). We will add an example of such a custom training loop soon.
Tokenizers have been improved to provide extended encoding methods encoding_plus and additional arguments to encoding. Please refer to the doc for detailed usage of the new options.
To be able to better use Torchscript both on CPU and GPUs (see #1010, #1204 and #1195) the specific order of some models keywords inputs (attention_mask, token_type_ids...) has been changed.
If you used to call the models with keyword names for keyword arguments, e.g. model(inputs_ids, attention_mask=attention_mask, token_type_ids=token_type_ids), this should not cause any breaking change.
model(inputs_ids, attention_mask=attention_mask, token_type_ids=token_type_ids)
If you used to call the models with positional inputs for keyword arguments, e.g. model(inputs_ids, attention_mask, token_type_ids), you should double-check the exact order of input arguments.
model(inputs_ids, attention_mask, token_type_ids)
PyTorch is no longer in the requirements so don't forget to install TensorFlow 2.0 and/or PyTorch to be able to use (and load) the models.
The method add_special_tokens_sentence_pair has been renamed to the more appropriate name add_special_tokens_sequence_pair. The same holds true for the method add_special_tokens_single_sentence which has been changed to add_special_tokens_single_sequence.