For example, "parrots do not swim." Here the main verb is "swim". How can we extract that by language processing? Are there any known algorithms for this purpose?
You can run a dependency parsing algorithm on the sentence and the find the dependent of the root relation. For example, running the sentence "Parrots do not swim" through the Stanford Parser online demo, I get the following dependencies:
nsubj(swim-4, Parrots-1)
aux(swim-4, do-2)
neg(swim-4, not-3)
root(ROOT-0, swim-4)
Each of these lines provides information about a different grammatical relation between two words in the sentence (see below). You need the last line, which says that swim is the root of the sentence, i.e. the main verb. So to extract the main verb, perform dependency parsing first and find the dependency that reads root(ROOT-0, X). X will be the main verb.
There are several readily available dependency parsers, such as the one available with Stanford CoreNLP or Malt parser. I prefer Stanford because it is comparable in accuracy, but has better documentation and supports multithreaded parsing (if you have lots of text). The Stanford parser outputs XML, so you will have to parse that to get the dependency information above.
For the sake of completeness, a brief explanation of the rest of the output. The first line says that parrots, the first word in the sentence, is the subject of swim, the 4th word. The second line says that do is an auxiliary verb related to swim, and the third says that not negates swim. For a more detailed explanation of the meaning of each dependency, see the Stanford typed dependency manual.
Edit:
Depending on how you define main verb, some sentences may have more than one main verb, e.g. I like cats and hate snakes. The dependency parse for this contain the dependencies:
root(ROOT-0, like-2)
conj(like-2, hate-5)
which together say that according to the parser the main verb is like, but hate is conjoined to it. For your purposes you might want to consider both like and hate to be main.
To get the verb (or any other Part-Of-Speech) there are many supervised and unsupervised algorithms available like Viterbi Algorithm, Hidden Markov Models, Brill Tagger, Constraint Grammer, etc. Even we have libraries like NLTK(Natural Language Tool Kit) for Python (and similar is also available for Java) which have these algorithm already encoded in them. Annotating POS in any document or sentence is a complex job (especially when you desire high accuracy ) and you need an in-depth knowledge in this field, begin with the very basics first and continuous effort might lead you to develop an algorithm which has higher efficiency than the prevailing one.
Related
I'm dealing with german law documents and would like to generate parse trees for sentences. I could find and use Standford CoreNLP Parser. However, it does not recognize sentence limits as good as other tools (e.g. spaCy) when parsing the sentences of a document. For example, it would break sentences at every single '.'-character, incl. the dot at the end of abbreviations such as "incl.")
Since it is crucial to cover the whole sentence for creating syntax trees, this does not really work out for me.
I would appreciate any suggestions to tackle this problem, espacially pointers to other software that might be better suited for my problem. If I overlooked the possibility to tweak the Stanford parser, I would be very grateful for any hints on how to make it better detect sentence limits.
A quick glance into the docs did the trick: You can run your pipeline, which might include the sentence splitter, with the attribute
ssplit.isOneSentence = true to basically disable it. This means you can split the sentences beforehand, e.g. using spaCy, and then feed single sentences into the pipeline.
Can we detect the sentence stress (the stress on some words or pauses between words in a sentence) using common NLP packages such as spaCy or NLTK?
How can we tell content words from structure words using spaCy or NLTK?
Since all NLP programs detect the dependencies, there should be a possibility to identify which words are stressed in natural speech.
I don't think that NLTK or spacy support this directly. You can find content words with either tool, sure, but that's only part of the picture. You want to look for software related to prosody or intonation, which you might find as a component of a text-to-speech system.
Here's a very recently published research paper with code that might be a good place to start: https://github.com/Helsinki-NLP/prosody/ . The annotated data and the references could be useful even if the code might not be exactly the kind of approach you're looking for.
I assume you do not have a special training data set with labeled data in what words to stress. So I guess the simplest way would be to assume, that stressed words are all of the same Part-of-speech. I guess nouns and verbs would be a good start, excluding modal verbs for example.
NLTK comes with PoS-Taggers.
But as natural speech depends lot on context, it's probaly difficult for humans as well to identify a single solution for what to stress in a sentence.
I'm currently using different tools from Stanford NLP Group and trying to understand the differences between them. It seems to me that somehow they intersect each other, since I can use same features in different tools (e.g. tokenize, and POS-Tag a sentence can be done by Stanford POS-Tagger, Parser and CoreNLP).
I'd like to know what's the actual difference between each tool and in which situations I should use each of them.
All Java classes from the same release are the same, and, yes, they overlap. On a code basis, the parser and tagger are basically subsets of what is available in CoreNLP, except that they do have a couple of little add-ons of their own, such as the GUI for the parser. In terms of provided models, the parser and tagger come with models for a range of languages, whereas CoreNLP ships only with English out of the box. However, you can then download language-particular jars for CoreNLP which provide all the models we have for different languages. Anything that is available in any of the releases is present in the CoreNLP github site: https://github.com/stanfordnlp/CoreNLP
I'm looking for tools for generating random but realistic text. I've implemented a Markov Chain text generator myself and while the results were promising, my attempts at improving them haven't yielded any great successes.
I'd be happy with tools that consume a corpus or that operate based on a context-sensitive or context-free grammar. I'd like the tool to be suitable for inclusion into another project.
Most of my recent work has been in Java so a tool in that language is preferred, but I'd be OK with C#, C, C++, or even JavaScript.
This is similar to this question, but larger in scope.
Extending your own Markov chain generator is probably your best bet, if you want "random" text. Generating something that has context is an open research problem.
Try (if you haven't):
Tokenising punctuation separately, or include punctuation in your chain if you're not already. This includes paragraph marks.
If you're using a 2- or 3- history Markov chain, try resetting to using a 1-history one when you encounter full stops or newlines.
Alternatively, you could use WordNet in two passes with your corpus:
Analyse sentences to determine common sequences of word types, ie nouns, verbs, adjectives, and adverbs. WordNet includes these. Everything else (pronouns, conjunctions, whatever) is excluded, but you could essentially pass those straight through.
This would turn "The quick brown fox jumps over the lazy dog" into "The [adjective] [adjective] [noun] [verb(s)] over the [adjective] [noun]"
Reproduce sentences by randomly choosing a template sentence and replacing [adjective], [nouns] and [verbs] with actual adjectives nouns and verbs.
There are quite a few problems with this approach too: for example, you need context from the surrounding words to know which homonym to choose. Looking up "quick" in wordnet yields the stuff about being fast, but also the bit of your fingernail.
I know this doesn't solve your requirement for a library or a tool, but might give you some ideas.
I've used for this purpose many data sets, including wikinews articles.
I've extracted text from them using this tool:
http://alas.matf.bg.ac.rs/~mr04069/WikiExtractor.py
I am trying to build an NLP system for an assignment, for which I am allowed to use external libraries.
I am using parse trees to break down sentences into their constituent parts down to nouns, verbs, etc.
I am looking for a library or software that would let me identify which lexical form a word is in, and possibly translate it to some other form for me.
Basically, I need something with functions like isPlural, singularize, getInfinitive, etc.
I have considered the Ruby Linguistics package and a simple Porter Stemmer (for infinitives) but neither is very good.
This does not seem like a very hard problem, just very tedious.
Does anyone know of a good package/library/software that could do things like that?
Typically, in order to build a parse tree of a sentence, one needs to first determine the part-of-speech and lemma information of the words in the sentence. So, you should have this information already.
But in any case, in order to map wordforms to their lemmas, and synthesize wordforms from lemmas, take a look at morpha and morphg, and also the Java version of (or front-end to) morphg contained in the SimpleNLG package. There are methods like getInfinitive, getPastParticiple, etc. See e.g. the API for the Verb class.