• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Tim Cooke
  • paul wheaton
  • Paul Clapham
  • Ron McLeod
Sheriffs:
  • Jeanne Boyarsky
  • Liutauras Vilda
Saloon Keepers:
  • Tim Holloway
  • Carey Brown
  • Roland Mueller
  • Piet Souris
Bartenders:

Transfer Learning with Word Embeddings

 
Ranch Hand
Posts: 81
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi Paul Azunre.
There are several pre-trained word embeddings available, but they generally cover the most common words.
Can I do something similar to Transfer Learning - start with a pre-trained set of word embeddings, then add my own domain-specific words somehow?

Cheers
Don.
 
Author
Posts: 14
5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Don Horrell wrote:Hi Paul Azunre.
There are several pre-trained word embeddings available, but they generally cover the most common words.
Can I do something similar to Transfer Learning - start with a pre-trained set of word embeddings, then add my own domain-specific words somehow?

Cheers
Don.



Don, the short answer is definitely absolutely YES!

Indeed the new breed of methods - pretrained models like ELMo, BERT, etc., etc., are all character-level. This means that you are actually no longer limited to in-vocabulary words, as used to be the case with word2vec and similar methods. Indeed even FastText, which preceded them, works at the sub-word level, and did ok with this.

Moreover, each method can be "fine-tuned" in your particular domain so that the general knowledge they contain can be adapted to the slang, differing meaning and structure inherent in the natural language you care about, beyond merely any particular words!!

Hope this is helpful!
 
Author
Posts: 90
7
Redhat Notepad Fedora Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi @Paul Azunre,

In your book, are you planning to cover this evolution from in-vocabulary context independent word embeddings to ones that take into account word order in their training? Or compare how Word2vec, Glove, ELMo, BERT generate different vectors for the same sentence? I have seen an attempt of something similar on the widely circulated sentence: but nothing that was easy to digest...

Thank you.
 
Paul Azunre
Author
Posts: 14
5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Lucian Maly wrote:Hi @Paul Azunre,

In your book, are you planning to cover this evolution from in-vocabulary context independent word embeddings to ones that take into account word order in their training? Or compare how Word2vec, Glove, ELMo, BERT generate different vectors for the same sentence? I have seen an attempt of something similar on the widely circulated sentence: but nothing that was easy to digest...

Thank you.



Lucian,

Yes, we cover the evolution from context-independent embeddings to the contextual ones, compare and contrast them, and provide examples (including code) on how to use them. Thanks for sharing the sentence, that is a very cool example! We have some examples which I think serve an equivalent purpose in the book and hope to add even more before it is done,

- Paul
 
Don Horrell
Ranch Hand
Posts: 81
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks for your reply, Paul. I'm a little confused though.
The pre-trained FastText word embeddings I have downloaded map words to vectors, so in my case (using TensorFlow to do some NLP classification), I can only train my classifier on the words in the embedding list.
That is the crux of my original question - how can I add domain-specific vocabulary to pre-trained word embeddings. Will your book cover this?


Thanks
Don.
 
Paul Azunre
Author
Posts: 14
5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Don Horrell wrote:Thanks for your reply, Paul. I'm a little confused though.
The pre-trained FastText word embeddings I have downloaded map words to vectors, so in my case (using TensorFlow to do some NLP classification), I can only train my classifier on the words in the embedding list.
That is the crux of my original question - how can I add domain-specific vocabulary to pre-trained word embeddings. Will your book cover this?


Thanks
Don.



Don,

These are your options:
1. If your fastText embedding is in the .vec format, it is not enough. You will also need the .bin format with subword information to handle out-of-vocabulary words wit FastText, see also - https://stackoverflow.com/questions/47118678/difference-between-fasttext-vec-and-bin-file
2. Use ELMo or BERT, they will handle out-of-vocabulary words out of the box.
3. To get even better performance, fine-tune ELMo and BERT on your  data so they can learn your words better.

Hope this clarifies things,

- Paul
 
With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime.
reply
    Bookmark Topic Watch Topic
  • New Topic