Transfer Learning with Word Embeddings

[Don Horrell]

Hi Paul Azunre.
There are several pre-trained word embeddings available, but they generally cover the most common words.
Can I do something similar to Transfer Learning - start with a pre-trained set of word embeddings, then add my own domain-specific words somehow?

Cheers
Don.

[Paul Azunre]

Don Horrell wrote:Hi Paul Azunre.
There are several pre-trained word embeddings available, but they generally cover the most common words.
Can I do something similar to Transfer Learning - start with a pre-trained set of word embeddings, then add my own domain-specific words somehow?

Cheers
Don.

Don, the short answer is definitely absolutely YES!

Indeed the new breed of methods - pretrained models like ELMo, BERT, etc., etc., are all character-level. This means that you are actually no longer limited to in-vocabulary words, as used to be the case with word2vec and similar methods. Indeed even FastText, which preceded them, works at the sub-word level, and did ok with this.

Moreover, each method can be "fine-tuned" in your particular domain so that the general knowledge they contain can be adapted to the slang, differing meaning and structure inherent in the natural language you care about, beyond merely any particular words!!

Hope this is helpful!

[Lucian Maly]

Hi @Paul Azunre,

In your book, are you planning to cover this evolution from in-vocabulary context independent word embeddings to ones that take into account word order in their training? Or compare how Word2vec, Glove, ELMo, BERT generate different vectors for the same sentence? I have seen an attempt of something similar on the widely circulated sentence: He went to the prison cell with his cell phone to extract blood cell samples from inmates but nothing that was easy to digest...

Thank you.

[Paul Azunre]

Lucian Maly wrote:Hi @Paul Azunre,

In your book, are you planning to cover this evolution from in-vocabulary context independent word embeddings to ones that take into account word order in their training? Or compare how Word2vec, Glove, ELMo, BERT generate different vectors for the same sentence? I have seen an attempt of something similar on the widely circulated sentence: He went to the prison cell with his cell phone to extract blood cell samples from inmates but nothing that was easy to digest...

Thank you.

Lucian,

Yes, we cover the evolution from context-independent embeddings to the contextual ones, compare and contrast them, and provide examples (including code) on how to use them. Thanks for sharing the sentence, that is a very cool example! We have some examples which I think serve an equivalent purpose in the book and hope to add even more before it is done,

- Paul

[Don Horrell]

Thanks for your reply, Paul. I'm a little confused though.
The pre-trained FastText word embeddings I have downloaded map words to vectors, so in my case (using TensorFlow to do some NLP classification), I can only train my classifier on the words in the embedding list.
That is the crux of my original question - how can I add domain-specific vocabulary to pre-trained word embeddings. Will your book cover this?


Thanks
Don.

[Paul Azunre]

Don Horrell wrote:Thanks for your reply, Paul. I'm a little confused though.
The pre-trained FastText word embeddings I have downloaded map words to vectors, so in my case (using TensorFlow to do some NLP classification), I can only train my classifier on the words in the embedding list.
That is the crux of my original question - how can I add domain-specific vocabulary to pre-trained word embeddings. Will your book cover this?


Thanks
Don.

Don,

These are your options:
1. If your fastText embedding is in the .vec format, it is not enough. You will also need the .bin format with subword information to handle out-of-vocabulary words wit FastText, see also - https://stackoverflow.com/questions/47118678/difference-between-fasttext-vec-and-bin-file
2. Use ELMo or BERT, they will handle out-of-vocabulary words out of the box.
3. To get even better performance, fine-tune ELMo and BERT on your  data so they can learn your words better.

Hope this clarifies things,

- Paul