How to use pre-trained FastText embeddings with existing Seq2Seq model? - nlp

I'm new in NLP and I am trying to understand how to use pre-trained word embeddings like fastText with the existing Seq2Seq model. The Seq2Seq model I'm working with is the following. The encoder is simple and the decoder is Pointer Generator Network with CRF on the top. Both of them use an embedding layer.
The question: If I have my own dataset & vocab, how do I use both my own vocab and the one from the fastText? Do I have to use fastText weights in both the encoder and decoder?

Related

How to convert the output of pretrained Huggingface transformer model from classification to regression for fine-tuning on my data?

I am using a transformer model that was extended from huggingface (DNABERT). This is a pretrained classification model whose output I would like to convert to regression, then fine-tune that model on my own data. I imagine this process would be roughly the same for any BERT-based huggingface classification model. How would I go about doing this?

How to post-train BERT model on custom dataset

I want to get the BERT word embeddings which will be used in another down-stream task later. I have a corpus for my custom dataset and want to further pre-train the pre-trained Huggingface BERT base model. I think this is called post-training. How can I do this using Huggingface transformers? Can I use transformers.BertForMaskedLM?

How to use BERT pre-trained model in Keras Embedding layer

How do I use a pre-trained BERT model like bert-base-uncased as weights in the Embedding layer in Keras?
Currently, I am generating word embddings using BERT model and it takes a lot of time. And I am assigning those weights like in the cide shown below
model.add(Embedding(307200, 1536, input_length=1536, weights=[embeddings]))
I searched on internet but the method is given in PyTorch. I need to do it in Keras. Please help.

Autoencoder with Transfer Learning?

Is there a way I can train an autoencoder model using a pre-trained model like ResNet?
I'm trying to train an autoencoder model with input as an image and output as a masked version of that image.
Is it possible to use weights from a pretrained model here?

Yes! you can definitely do transfer learning using a pre-trained network, i.e. ResNet50 as the encoder in an autoencoder. For reference, check out the following link. https://github.com/hsinyilin19/ResNetVAE

From what I know, there is no proven method to do this. I'd train the autoencoder from scratch.
In theory, if you find a pre-trained CNN which does not use max pooling, you can use those weights and architecture for the encoder stage in your autoencoder. You can also extract features from a pre-trained model and concatenate/merge them to your autoencoder. But the value add is not clear, and the architecture might become overly complex.

Use pretrained models to further train current corpus

Is it possible to leverage the pretrained model e.g. GLOVE and use it to further train a corpus.
Any example will be very helpful.

Resources