mBert embedding, transformers - conv-neural-network

Please, I have to use the generated embedding by mBERT and use another model as a classifier like logistic regression, CNN, Naive bayes...
I'm wondering if this task can be done or not.
Thank you!

Related

Unsupervised finetuning of BERT for embeddings only?

I would like to fine-tuning BERT for a specific domain on unlabeled data and get the output layer to check the similarity between them. How can I do it? Do I need to fine-tuning first a classifier task (or question answer, etc..) and get the embeddings? Or can I just use a pre-trained Bert model without task and fine-tuning with my own data?

There is no need to fine-tune for classification, especially if you do not have any supervised classification dataset.
You should continue training BERT the same unsupervised way it was originally trained, i.e., continue "pre-training" using the masked-language-model objective and next sentence prediction. Hugginface's implementation contains class BertForPretraining for this.

Customizg loss function in Word2vec

I have list of co-occurences and I want to train word2vec model with my own customized loss_function.
What is the best way to approach this?
Is it possible to set gensim Word2Vec model with my own function?
If not, is there an example to an implementation with keras?
If not, must I define everything totally from scratch?
Thanks!

automatic classification model selecion

i want to know is there any method by which the computer can decide which classification model to use ( Decision trees, logistic regression, KNN, etc. ) by just looking at the training data.
even just the math will be extremely helpful.
I am going to be writing this in python 3, so if there's any built method in scikit-learn or tensorflow for this purpose,it would be of great help.

This scikit learn tool kit solves it :
https://automl.github.io/auto-sklearn/stable/index.html

LSTM with CRF in Keras

I don't really understand how to combine sklearn_crfsuite and Keras.
I have to made a classic LSTM and insteed of the last Activation, I use sklearn_crfsuite?
Someone have an example?
Thx,

You might want to look into the keras-contrib package, which has an implementation of CRF as a Keras layer.

How to train LSTM/GRU using unsupervised learning method before train it with supervised learning method in Keras?

Correct me if i'm wrong, but i saw an article somewhere that says that training neural net using unsupervised method before using it for supervised classification will give a better result. I assume it is like a transfer learning kind of stuff. But i'm wondering if i wanted to train the LSTM without the labels, it cannot be entered to the fit function of keras. Any idea how to do it?

Resources