I went through their github files as well as the official site, I can't find the named entity tagging training corpus they used in splotlight.
How Can I found the dataset instead of a trained model?
see This link https://github.com/dbpedia-spotlight/dbpedia-spotlight/wiki/Web-service
In here, method for setting up dbpedia lookup offline is explained. Also they have given 4 tar files which are
redirects_en.nt
short_abstracts_en.nt
instance_types_en.nt
article_categories_en.nt
these are supposed to be training data for it.
Related
I am beginner to Machine Learning and NLP, I have to create a bot based on FAQ dataset, Each FAQ dataset excel file contains 2 columns "Questions" and its "Answers".
Eg. A record from an excel file (A question & it's answer).
Question - What is RASA-NLU?
Answer - Rasa NLU is trained to identify intent and entities. Better the training, better the identification...
We have 3K+ excel files which has around 10K to 20K such records each excel.
To implement the bot, I would have followed exactly this FAQ bot approach which uses RASA-NLU, but the RASA,Chatterbot also Microsoft's QnA maker are not allowed in my organization.
And Spacy does the NER extraction perfectly for me, so I am looking for a bot creation using Spacy. but I don't know how to proceed further after extracting the entities. (IMHO, I will have to predict the exact question from dataset (and its answer from knowlwdge base) from user query to the bot)
I don't know what NLP algorithm/ ML process to be used or is there any easiest way to create that FAQ bot using extracted NERs.
One way to achieve your FAQ bot is to transform the problem into a classification problem. You have questions and the answers can be the "labels". I suppose that you always have multiple training questions which map to the same answer. You can encode each answer in order to get smaller labels (for instance, you can map the text of the answer to an id).
Then, you can use your training data (the questions) and your labels (the encoded answers) and feed a classifier. After the training your classifier can predict the label of unseen questions.
Of course, this is a supervised approach, so you will need to extract features from your training sentences (the questions). In this case, you can use as a feature the bag-of-word representations and even include the named entities.
An example of how to do text classification in spacy is available here: https://www.dataquest.io/blog/tutorial-text-classification-in-python-using-spacy/
About 5 years ago, I re-trained Stanford NER and it works somewhat, but new products often get missed. At that time, I retrained the entire NER model. What I would really like to do is to fine tune the Stanford NER model. Can that be done now? Someone asked that before but the answer is not clear to me.
How to create incremental NER training model(Appending in existing model)?
Also related:
How to extract brand from product name
The most recent paper I can find is this: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.904.3818&rep=rep1&type=pdf
I tried the NER command on https://stanfordnlp.github.io/CoreNLP/caseless.html and it gave an error indicating "english-caseless-left3words-distsim.tagger" etc. could not be found. Is there a place for download trained models?
(OK. The models are included in stanford-english-corenlp-2018-02-27-models.jar)
In Gensim's doc2vec implementation, gensim.models.keyedvectors.Doc2VecKeyedVectors.most_similar returns the tags and cosine similarity of the documents most similar to the query document. What if I want the actual documents themselves and not the tags? Is there a way to do that directly without searching for the document associated with the tag returned by most_similar?
Also, is there documentation on this? I can't seem to find the documentation for half of Gensim's classes.
The Doc2Vec class doesn't serve as a full document database that stores the original documents in their original formats. That would require a lot of extra complexity and state.
Instead, you just present the docs, with their particular tags, in the tokenized format it needs for training, and the model only learns and retains their vector representations.
If you need to then look-up the original documents, you must maintain your own (tags -> documents) lookup – which many projects will already have as the original source of the docs.
The Doc2Vec class docs are at https://radimrehurek.com/gensim/models/doc2vec.html but it may also be helpful to look at the example Jupyter notebooks included in the gensim docs/notebooks directory but also viewable online at:
https://github.com/RaRe-Technologies/gensim/tree/develop/docs/notebooks
The three notebooks related to Doc2Vec have filenames beginning doc2vec-.
I'm using OpenNLP for data classification. I could not find TokenNameFinderModel for disease here. I know I can create my own model but I was wondering is there any large sample training data available for disease?
You can easily create your own training data-set using the modelbuilder addon and follow some rules as mentioned here to train create a good NER model.
you can find some help using modelbuilder addon here.
It is basically, you put all the information in a text file and the NER entities in another. The addon searches for a paticular entity and replace it with the required tag. Hence producing the tagged data. It must be pretty easy to use this tool!
Hope this helps!
Could I ask about Stanford NER?? Actually, I'm trying to train my own model, to use it later for learning. According to the documentation, I have to add my own features in SeqClassifierFlags and add code for each Feature in NERFeatureFactory.
My questions is that, I have my tokens with all features extracted and Last column represents the label. So, is there any way in Stanford NER to give it my Tab-Delimeted file which contains 30 columns (1 is word , 28 are featurs, and 1 is label) to train my own model without spending time for extracting features???
Of course, in Testing phase, I will give it a file like the the aforementioned file without label to predict the label.
Is this possible or Not??
Many thanks in Advance
As explained in the FAQ page, the only way to customize the NER model is by inserting the data and specifying the features that you want to extract.
But, wait ... you have the data, and you have managed to extract the features, so I think you don't need the NER model, you need a classifier. I know this answer is pretty pretty late, but maybe this classifier will be a good place to start.