I'm looking for tools for generating random but realistic text. I've implemented a Markov Chain text generator myself and while the results were promising, my attempts at improving them haven't yielded any great successes.
I'd be happy with tools that consume a corpus or that operate based on a context-sensitive or context-free grammar. I'd like the tool to be suitable for inclusion into another project.
Most of my recent work has been in Java so a tool in that language is preferred, but I'd be OK with C#, C, C++, or even JavaScript.
This is similar to this question, but larger in scope.
Extending your own Markov chain generator is probably your best bet, if you want "random" text. Generating something that has context is an open research problem.
Try (if you haven't):
Tokenising punctuation separately, or include punctuation in your chain if you're not already. This includes paragraph marks.
If you're using a 2- or 3- history Markov chain, try resetting to using a 1-history one when you encounter full stops or newlines.
Alternatively, you could use WordNet in two passes with your corpus:
Analyse sentences to determine common sequences of word types, ie nouns, verbs, adjectives, and adverbs. WordNet includes these. Everything else (pronouns, conjunctions, whatever) is excluded, but you could essentially pass those straight through.
This would turn "The quick brown fox jumps over the lazy dog" into "The [adjective] [adjective] [noun] [verb(s)] over the [adjective] [noun]"
Reproduce sentences by randomly choosing a template sentence and replacing [adjective], [nouns] and [verbs] with actual adjectives nouns and verbs.
There are quite a few problems with this approach too: for example, you need context from the surrounding words to know which homonym to choose. Looking up "quick" in wordnet yields the stuff about being fast, but also the bit of your fingernail.
I know this doesn't solve your requirement for a library or a tool, but might give you some ideas.
I've used for this purpose many data sets, including wikinews articles.
I've extracted text from them using this tool:
http://alas.matf.bg.ac.rs/~mr04069/WikiExtractor.py
Related
I'm dealing with german law documents and would like to generate parse trees for sentences. I could find and use Standford CoreNLP Parser. However, it does not recognize sentence limits as good as other tools (e.g. spaCy) when parsing the sentences of a document. For example, it would break sentences at every single '.'-character, incl. the dot at the end of abbreviations such as "incl.")
Since it is crucial to cover the whole sentence for creating syntax trees, this does not really work out for me.
I would appreciate any suggestions to tackle this problem, espacially pointers to other software that might be better suited for my problem. If I overlooked the possibility to tweak the Stanford parser, I would be very grateful for any hints on how to make it better detect sentence limits.
A quick glance into the docs did the trick: You can run your pipeline, which might include the sentence splitter, with the attribute
ssplit.isOneSentence = true to basically disable it. This means you can split the sentences beforehand, e.g. using spaCy, and then feed single sentences into the pipeline.
Can we detect the sentence stress (the stress on some words or pauses between words in a sentence) using common NLP packages such as spaCy or NLTK?
How can we tell content words from structure words using spaCy or NLTK?
Since all NLP programs detect the dependencies, there should be a possibility to identify which words are stressed in natural speech.
I don't think that NLTK or spacy support this directly. You can find content words with either tool, sure, but that's only part of the picture. You want to look for software related to prosody or intonation, which you might find as a component of a text-to-speech system.
Here's a very recently published research paper with code that might be a good place to start: https://github.com/Helsinki-NLP/prosody/ . The annotated data and the references could be useful even if the code might not be exactly the kind of approach you're looking for.
I assume you do not have a special training data set with labeled data in what words to stress. So I guess the simplest way would be to assume, that stressed words are all of the same Part-of-speech. I guess nouns and verbs would be a good start, excluding modal verbs for example.
NLTK comes with PoS-Taggers.
But as natural speech depends lot on context, it's probaly difficult for humans as well to identify a single solution for what to stress in a sentence.
I am making a "Part of speech Tagger". I am handling the unknown word with the suffix.
But the main issue is that how would i decide the number of suffix... should it be pre-decided (like Weischedel approach) or I have to take the last few alphabets of the words(like Samuelsson approach).
Which approach would be better......
Quick googling suggests that the Weischedel approach is sufficient for English, which has only rudimentary morphological inflection. The Samuelsson approach seems to work better (which makes sense intuitively) when it comes to processing inflecting languages.
A Resource-light Approach to Morpho-syntactic Tagging - Google Books p 9 quote:
To handle unknown words Brants (2000) uses Samuelsson's (1993) suffix analysis, which seems to work best for inflected languages.
(This is not in a direct comparison to Weischedel's approach, though.)
I have a quick question and could not find the answer anywhere on the internet:
Can NLTK be used to Analyze the sentiment a certain word has within a sentence?
Like: Sentiment for iPhone: "Even though it is terrible weather outside, my iPhone makes me feel good again." = Sentiment: positive
Have you thought of breaking down the text into clauses ("it is terrible weather outside", "my iphone makes me feel good again"), and evaluating them separately? You can use the NLTK's parsers for that. This will reduce the amount of text you have to analyze, though, so it might end up doing more harm than good.
This won't help you in cases like "Microsoft Surface is no iPad, it's terrible" (where your target is "iPad"), since the sentiment is negative but the iPad wins the comparison. So perhaps you'll also want to check the syntactic analysis, and only examine sentences where your target word is the subject or object. Whether these will give you better performance is anybody's guess, I think.
I do not have much experience with NLTK but I have done some concept level sentiment analysis using NLP libraries in Java. Here is how I did it. The same approach should work for you if you are able to identify dependencies in NLTK. This approach works fine for simple rules but may not work well for complicated sentences.
I am trying to build an NLP system for an assignment, for which I am allowed to use external libraries.
I am using parse trees to break down sentences into their constituent parts down to nouns, verbs, etc.
I am looking for a library or software that would let me identify which lexical form a word is in, and possibly translate it to some other form for me.
Basically, I need something with functions like isPlural, singularize, getInfinitive, etc.
I have considered the Ruby Linguistics package and a simple Porter Stemmer (for infinitives) but neither is very good.
This does not seem like a very hard problem, just very tedious.
Does anyone know of a good package/library/software that could do things like that?
Typically, in order to build a parse tree of a sentence, one needs to first determine the part-of-speech and lemma information of the words in the sentence. So, you should have this information already.
But in any case, in order to map wordforms to their lemmas, and synthesize wordforms from lemmas, take a look at morpha and morphg, and also the Java version of (or front-end to) morphg contained in the SimpleNLG package. There are methods like getInfinitive, getPastParticiple, etc. See e.g. the API for the Verb class.