Where can I get free energy fraud detection statistic algorithms, with historic data, locations, types of grids, output as success rates etc, fraud probabilities?
There are CNN and other methods, but free codes are rare.
Many papers on networks, statistic,I need implementations with training datasets
I've tried many papers, there are algorithms, but without datasets, github also
Vitaly Ford algorithms, CNN-gru
Sheraz Aslam paper, Nasir Ayub
Related
I have a question is there already any free dataset available to test doc2vec and if in case I wanted to create my own dataset what could be an appropriate way to do it.
Assuming you mean the 'Paragraph Vectors' algorithm, which is often called Doc2Vec, any textual dataset is a potential test/demo dataset.
The original papers by the creators of Doc2Vec showed results from applying it to:
movie reviews
search engine summary snippets
Wikipedia articles
scientific articles from Arxiv
People have also used it on…
titles of articles/books
abstracts of larger articles
full news articles or scientific papers
tweets
blogposts or social media posts
resumes
When learning, it's best to pick very simple, common datasets when you're 1st starting, and then larger datasets that you somewhat understand or are related to your areas of interest – if you don't already have a sufficient project-related dataset.
Note that the algorithm, like others in the [something]2vec family of algorithms, works best with lots of varied training data – many tens of thousands of unique words each with many contrasting usage examples, over many tens of thousands (or many more) of documents.
If you crank the vector_size way down, & the training-epochs way up, you can eke some hints of its real performance out of smaller datasets of a few hundred contrasting documents. For example, in the Python Gensim library's Doc2Vec intro-tutorial & test-cases, a tiny set of 300 news-summaries (from about 20 years ago called the 'Lee Corpus') are used, and each text is only a few hundreds words long.
But the vector_size is reduced to 50 – much smaller than the hundreds-of-dimensions typical with larger training data, and perhaps still too many dimensions for such a small amount of data. And, the training epochs is increased to 40, much larger than the default of 5 or typical Doc2Vec choices in published papers of 10-20 epochs. And even with those changes, with such little data & textual variety, the effect of moving similar documents to similar vector coordinates will be appear weaker to human review, & be less consistent between runs, than a better dataset will usually show (albeit using many more minutes/hours of training time).
I'm working in a decently-sized data set, and wish to identify what # topics make sense. I used both NMF and LDA (sklearn implementation), but the key question: what is a suitable measure for success. Visually I have in many topics only a few height-weight keywords (the other weights ~ 0), and a few topics with more bell-shaped distribution of the topics. What is the target: a topic with a few words, high weight, rest low (a spike) or a bell-shape distribution, gradual reduction of weights over a large # keywords
NMF
or the LDA method
that gives mostly a bell-shape (not curve, obviously)
I also use a weighted jaccard (set overlap of the keywords, weighted; there are no doubt better methods, but this is kind-of intuitive
Your thoughts on this?
best,
Andreas
code at https://scikit-learn.org/stable/auto_examples/applications/plot_topics_extraction_with_nmf_lda.html?highlight=document%20word%20matrix
There are a few commonly used evaluation metrics that can give a good intuition of the quality of your topic sets in general, as well as your choice of k (number of topics). A recent paper by Dieng et al. (Topic Modeling in Embedded Spaces) uses two of the best measures: coherence and diversity. In conjunction, coherence and diversity give an idea of how well-clustered topics are. Coherence measures the similarities of words in each topic using their co-occurrences in documents, and diversity measures the similarity between topics based on the overlap of topics. If you score low in diversity, that means that words are overlapping in topics, and you might want to increase k.
There's really no "best way to decide k," but these kind of measures can help you decide whether to increase or decrease the number.
I classify clients by many little xgboost models created from different parts of dataset.
Since it is hard to support many models manually, I decided to automate hyperparameters tuning via Hyperopt and features selection via Boruta.
Would you advise me please, what should go first: hyperparameters tuning or features selection? On the other hand, it does not matter.
After features selection, the number of features decreases from 2500 to 100 (actually, I have 50 true features and 5 categorical features turned to 2 400 via OneHotEncoding).
If some code is needed, please, let me know. Thank you very much.
Feature selection (FS) can be considered as a preprocessing activity, wherein, the aim is to identify features having low bias and low variance [1].
Meanwhile, the primary aim of hyperparameter optimization (HPO) is to automate hyper-parameter tuning process and make it possible for users to apply Machine Learning (ML) models to practical problems effectively [2]. Some important reasons for applying HPO techniques to ML models are as follows [3]:
It reduces the human effort required, since many ML developers spend considerable time tuning the hyper-parameters, especially for large datasets or complex ML algorithms with a large number of hyper-parameters.
It improves the performance of ML models. Many ML hyper-parameters have different optimums to achieve best performance in different datasets or problems.
It makes the models and research more reproducible. Only when the same level of hyper-parameter tuning process is implemented can different ML algorithms be compared fairly; hence, using a same HPO method on different ML algorithms also helps to determine the most suitable ML model for a specific problem.
Given the above difference between the two, I think FS should be first applied followed by HPO for a given algorithm.
References
[1] Tsai, C.F., Eberle, W. and Chu, C.Y., 2013. Genetic algorithms in feature and instance selection. Knowledge-Based Systems, 39, pp.240-247.
[2] M. Kuhn, K. Johnson Applied Predictive Modeling Springer (2013) ISBN: 9781461468493.
[3] F. Hutter, L. Kotthoff, J. Vanschoren (Eds.), Automatic Machine Learning: Methods, Systems, Challenges, 9783030053185, Springer (2019)
I have searched the internet and there is more or less the same sentiment analysis of a sentence i.e Positive, Negative or Neutral. I want to build a sentiment analyzer that look for the following sentiments/emotions for a sentence.
happy , sad , angry , disaapointed , surprised, proud, in love, scared
It would be nice for you to explore a bit further what you tried so far and more in details of what you want to do. So, I'm answering this based on the assumption that you want to work with an emotion-based Sentiment Analysis. Actually there is an area of research that focus on identifying emotion from text.
In many cases, the problem is still treated as a multiclass classification problem, but instead of predicting sentiment polarity (positive, negative or neutral), people try to find emotions. The existing emotions vary in different research and different annotated data, but in general it looks like the ones you mentioned.
Your best chance to understand this area further is to look for papers and existing datasets. I'll list a few here for you and the emotions they work with:
An Analysis of Annotated Corpora for Emotion Classification in Text. Literature review of methods and corpus for such analysis.
Emotion Detection and Analysis on Social Media. Happiness, Sadness, Fear, Anger, Surprise and Disgust
This dataset is a good source for training data. Sadness, Enthusiasm, Neutral, Worry, Love, Fun, Hate, Happiness,
Can anybody please explain multitask learning in simple and intuitive way? May be some real
world problem would be useful.Mostly, these days i am seeing many people are using it for natural language processing tasks.
Let's say you've built a sentiment classifier for a few different domains. Say, movies, music DVDs, and electronics. These are easy to build high quality classifiers for, because there is tons of training data that you've scraped from Amazon. Along with each classifier, you also build a similarity detector that will tell you for a given piece of text, how similar it was to the dataset each of the classifiers was trained on.
Now you want to find the sentiment of some text from an unknown domain or one in which there isn't such a great dataset to train on. Well, how about we take a similarity weighted combination of the classifications from the three high quality classifiers we already have. If we are trying to classify a dish washer review (there is no giant corpus of dish washer reviews, unfortunately), it's probably most similar to electronics, and so the electronics classifier will be given the most weight. On the other hand, if we are trying to classify a review of a TV show, probably the movies classifier will do the best job.