I'm dealing with german law documents and would like to generate parse trees for sentences. I could find and use Standford CoreNLP Parser. However, it does not recognize sentence limits as good as other tools (e.g. spaCy) when parsing the sentences of a document. For example, it would break sentences at every single '.'-character, incl. the dot at the end of abbreviations such as "incl.")
Since it is crucial to cover the whole sentence for creating syntax trees, this does not really work out for me.
I would appreciate any suggestions to tackle this problem, espacially pointers to other software that might be better suited for my problem. If I overlooked the possibility to tweak the Stanford parser, I would be very grateful for any hints on how to make it better detect sentence limits.
A quick glance into the docs did the trick: You can run your pipeline, which might include the sentence splitter, with the attribute
ssplit.isOneSentence = true to basically disable it. This means you can split the sentences beforehand, e.g. using spaCy, and then feed single sentences into the pipeline.
Related
Can we detect the sentence stress (the stress on some words or pauses between words in a sentence) using common NLP packages such as spaCy or NLTK?
How can we tell content words from structure words using spaCy or NLTK?
Since all NLP programs detect the dependencies, there should be a possibility to identify which words are stressed in natural speech.
I don't think that NLTK or spacy support this directly. You can find content words with either tool, sure, but that's only part of the picture. You want to look for software related to prosody or intonation, which you might find as a component of a text-to-speech system.
Here's a very recently published research paper with code that might be a good place to start: https://github.com/Helsinki-NLP/prosody/ . The annotated data and the references could be useful even if the code might not be exactly the kind of approach you're looking for.
I assume you do not have a special training data set with labeled data in what words to stress. So I guess the simplest way would be to assume, that stressed words are all of the same Part-of-speech. I guess nouns and verbs would be a good start, excluding modal verbs for example.
NLTK comes with PoS-Taggers.
But as natural speech depends lot on context, it's probaly difficult for humans as well to identify a single solution for what to stress in a sentence.
My text contains text="Ravi beated Ragu"
My Question will be "Who beated Ragu?"
The Answer Should come "Ravi" Using NLP
How to do this by natural language processing.
Kindly guide me to proceed with this by syntactic,semantic and progmatic analysis using python
I would suggest that you should read an introductory book on NLP to be familiar with the chain processes you are trying to achieve. You are trying to do question-answering , aren't you? If it is the case, you should read about question-answering systems. The above sentence has to be morphologically analyzed (so read about morphological analyzers), syntactically parsed (so read about syntactic parsing) and semantically understood (so read about anaphora resolution and , in linguistics, theta theory). Ravi is called agent and Ragu is called patient or experiencer. Only then, you can proceed to pursue your objectives.
I hope this helps you!
You need lots of data.
What you are looking for might be reading comprehension, thus I recommend you read the papers(for instance: Reading Comprehension on the SQuAD Dataset) by the competitors from SQuAD.
For simple tasks(implying limited generalization) and if you don't have such large size corpus, you can use Regex by manually writing some rules.
It may not be 100 % accurate but still is there any written and tested CFG.
is it available with nltk data?
See this. And search for the word "probability". There are options for printing the probability of parse trees and you can interpret them as confidence, etc.
Short answer: no. Long answer: noooooooooooooo
This is a huge issue. One CFG couldn't possibly come even close to the complexity that English presents. Even the POS taggers aren't terribly accurate.
The very best spelling and grammar checkers look for invariant laws of English and violations therein. Search for English grammar rules and think about how you might use software to detect them.
I have a quick question and could not find the answer anywhere on the internet:
Can NLTK be used to Analyze the sentiment a certain word has within a sentence?
Like: Sentiment for iPhone: "Even though it is terrible weather outside, my iPhone makes me feel good again." = Sentiment: positive
Have you thought of breaking down the text into clauses ("it is terrible weather outside", "my iphone makes me feel good again"), and evaluating them separately? You can use the NLTK's parsers for that. This will reduce the amount of text you have to analyze, though, so it might end up doing more harm than good.
This won't help you in cases like "Microsoft Surface is no iPad, it's terrible" (where your target is "iPad"), since the sentiment is negative but the iPad wins the comparison. So perhaps you'll also want to check the syntactic analysis, and only examine sentences where your target word is the subject or object. Whether these will give you better performance is anybody's guess, I think.
I do not have much experience with NLTK but I have done some concept level sentiment analysis using NLP libraries in Java. Here is how I did it. The same approach should work for you if you are able to identify dependencies in NLTK. This approach works fine for simple rules but may not work well for complicated sentences.
I'm looking for tools for generating random but realistic text. I've implemented a Markov Chain text generator myself and while the results were promising, my attempts at improving them haven't yielded any great successes.
I'd be happy with tools that consume a corpus or that operate based on a context-sensitive or context-free grammar. I'd like the tool to be suitable for inclusion into another project.
Most of my recent work has been in Java so a tool in that language is preferred, but I'd be OK with C#, C, C++, or even JavaScript.
This is similar to this question, but larger in scope.
Extending your own Markov chain generator is probably your best bet, if you want "random" text. Generating something that has context is an open research problem.
Try (if you haven't):
Tokenising punctuation separately, or include punctuation in your chain if you're not already. This includes paragraph marks.
If you're using a 2- or 3- history Markov chain, try resetting to using a 1-history one when you encounter full stops or newlines.
Alternatively, you could use WordNet in two passes with your corpus:
Analyse sentences to determine common sequences of word types, ie nouns, verbs, adjectives, and adverbs. WordNet includes these. Everything else (pronouns, conjunctions, whatever) is excluded, but you could essentially pass those straight through.
This would turn "The quick brown fox jumps over the lazy dog" into "The [adjective] [adjective] [noun] [verb(s)] over the [adjective] [noun]"
Reproduce sentences by randomly choosing a template sentence and replacing [adjective], [nouns] and [verbs] with actual adjectives nouns and verbs.
There are quite a few problems with this approach too: for example, you need context from the surrounding words to know which homonym to choose. Looking up "quick" in wordnet yields the stuff about being fast, but also the bit of your fingernail.
I know this doesn't solve your requirement for a library or a tool, but might give you some ideas.
I've used for this purpose many data sets, including wikinews articles.
I've extracted text from them using this tool:
http://alas.matf.bg.ac.rs/~mr04069/WikiExtractor.py