With the advancement in technology, industry has been moving towards automation and intelligence. In this regards artificial intelligence and machine learning has played a vital role. Natural language processing (NLP) is a field of computer science and linguistics which focuses on methods to process the natural languages. So, which one is more reliable and efficient in natural language processing, Finite state machine [FSM] or Push down Automata?
Even though there are many techniques to do NLP, the state of art way is to use deep learning. Many significant improvements are shown in NLP using Deep Learning Techniques. This has happened because of the enormous amount of processing power which is available at low cost. If you want to read cutting edge techniques used in NLP domain or any other research domain, Go to google scholar (https://scholar.google.com/).
It seems like the real question you want to be asking are: "What are some efficient techniques in natural language processing?" But I will address your question first.
First of all, neither FSA (Finite State Automata) not PDA (Push Down Automata) are sufficient techniques to model language. FSAs can handle regular languages. They cannot, however, even answer the question of whether a word is a palindrome. PDAs are a little more powerful, and can answer such questions. Turing machines give universal computation and are useful for writing programs of arbitrary complexity.
Now to start bridging this gap. Natural languages are not regular languages. They thus cannot be handled by FSAs. Some context-free grammars such as LR(k) grammars are handled by PDAs, however natural human language is not context-free. As an example. The following three statements. "Jill drove to the grocery store to meet her friend Sally before she picked up her kids. Sally bought three boxes of cereal. Then she drove to the school." While this is poor grammar, it is "natural" in that they are utterances that people make and they are generally parseable by other people. The antecedent to the pronoun "She" in the third sentence clearly refers to Jill as she is the one with children. However, it is ambiguous and we have to infer that association.
The amount of ambiguity in context in natural human language makes it impossible to parse deterministically. Instead, we turn towards the fields of statistics and decision theory to make our inferences about the maximally likely model for the communication.
The locality but non-determinism in speech and writing are one of the things that make the application of machine learning techniques such as the utilization of deep recurrent neural networks so immensely effective by comparison to their classical rule-based counterparts.
While the term Neural Network is a bit of a misnomer as ultimately the human brain is far, far more complex than these rudimentary models from a neurological perspective, the general learning through approximate inference is ostensibly close to reality. We might better call these methods "Differentiable Computing" but that is a digression for another time.
In summary. The answer to your question you actually asked would be PDAs are going to produce better models than FSAs but both are going to be absolutely worthless by comparison to even rudimentary statistical methods.
If you are curious about NLP, I would actually recommend a course in machine learning and a follow up in deep learning.
Andrew Ng has a good series of courses that are targeted toward beginners. After that, I would follow up with Sirajs course on deep learning in Tensorflow.
I am researching formal and informal search heuristics. One of the best books on the subject I've found is Judea Pearl's Heuristics. Embarrassingly, I find myself unable to find a good search strategy that returns more material in this vein.
Things I am looking for:
Summary papers about advances in search
Books/papers that cover some of the history of the development of methods
Some idea about who is currently producing research in this space and their specialization
Additional keywords, search methods, and items that should appear on this list to broaden the search
I'm looking for non-technical material. Most works have a bunch of specific implementation detail and small, short bits about where the research came from and what it lead to (which leads to me chasing citation trails). This is totally fine, but hoping to find works that include more of the non-technical info all in one place.
Some works I've canvassed so far:
Search and Optimization by Metaheuristics. Techniques and Algorithms Inspired by Nature
Metaheuristics: from design to implementation
Artificial Intelligence, Evolutionary Computing and Metaheuristics: In the Footsteps of Alan Turing
Essays and Surveys in Metaheuristics
Essentials of Metaheuristics
Handbook of approximation algorithms and metaheuristics
Heuristics, Metaheuristics and Approximate Methods in Planning and Scheduling
Recent Advances on Meta-Heuristics and Their Application to Real Scenarios
Advances in Knowledge Representation
Applications of Conceptual Spaces: The Case for Geometric Knowledge Representation
Concepts, Ontologies, and Knowledge Representation
Handbook of Knowledge Representation
I realize this is more an academically oriented question and am also open to suggestions of where else to post such a question.
I've tried looking around but I can't seem to find an answer on what math I need before jumping into NLP. I was hoping to get a solid foundation in math before jumping into NLP.
From what I've gathered it's mostly:
Probability,
some Statistic,
Discrete Math
Thank you for your time.
As in most fields, you'll find once you dive in that the title "NLP" covers a fairly broad range of sub-fields. The math requirements vary widely depending on what you're trying to accomplish. So a little more detail about your goals would help.
That said, I can address parsing and the related fields I have some experience in, and offer very general comments on a few others.
You'll find discrete math and automata theory useful in any computer science discipline, so you can't go wrong there.
Some NLP work is closer to linguistics or psychology than computer science. So some linguistic theory might be helpful if that's where your interests lie, and some background in statistical hypothesis testing (applied statistics of the sort you might find in a social science department, although the more rigorous the better).
For morphology, tagging, parsing, and related fields, some probability theory is helpful (as is experience thinking about dynamic programming, although that's not really math background). If you're doing anything involving machine learning (which is most of NLP), it helps to understand some linear algebra.
That said, if your goals are more applied, you can accomplish quite a lot by applying existing tools, without detailed knowledge of the underlying math (it doesn't require any linear algebra to train an SVM, if all you need is a classifier).
For comprehension, there is no mathematics to model language. The 'model' means that a function maps language expression to number or utility. The 'comprehension' comes from representation and composition, which have exceeded the nature science.
I am wondering if anyone has any insight into this. I am thinking of going to grad school to get some computer science related degree. I have always been intrigued by people who are working on problems using statistical packages or simulation to solve problems. What would I study to get a good breadth of knowledge of these things? Do they fall into machine learning?
Thanks
My girlfriend is getting a degree in mathematics with an emphasis in Statistics and Operations Research.
She does a lot of work with SAS and other statistical software to maximize certain functions and predict the likelihood of future events. It may be more mathematics then you like, but you might try looking for masters of CS programs with an emphasis in Operations Research or Statistics.
There's a wide range of possible opportunities here. Let me add the following choices:
Physics with a focus on complex networks. This has applications in biology, epidemiology, sociology, finance, and computer science.
A good machine learning program, with statistics, data mining, text analysis, and computational learning theory.
Industrial engineering/operations research, with simulation, reliability, and process control.
I'd be happy to talk further about this, please put questions in comments.
I would assume that your school would offer some actual Statistics courses, probably in the Math department, which you could take to learn all about this.
Study a lot of mathematics, especially probability and statistics. I have a graduate simulation course right now, and I wish I knew more probs/stats stuff.
In Biostatics (at the U of Minnesota), we did a lot of simulation, in areas like Bayesian statistics, genetics, and others. Any strongly analytical program is a good candidate for teaching the skills you want, including: econ, econometrics, agronomics, statistical genetics... etc., etc., :)
While you're waiting, pick up R, Matlab (Octave is the free implementation), or your Turing-Complete language of choice, dig into Wikipedia, and get to work :)
I'd like to second Gregg Lind's recommendation of thinking about statistics in the biological sciences. It's well-funded, there's a lot of interesting work going on (both theoretical and applied!), and you can sound really cool at parties because somehow, someway you can always make some sort of connection from your work back to curing cancer. :)
Seriously though, a lot of great statistical work was done in the early 20th century by people like Haldane, Fiscer and Wright. More recent interesting work has been done on analysis or large data sets, multiple hypothesis testing, and applied machine learning. It's super exciting. Come join us!
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 9 years ago.
Question
So I've recently came up with some new possible projects that would have to deal with deriving 'meaning' from text submitted and generated by users.
Natural language processing is the field that deals with these kinds of issues, and after some initial research I found the OpenNLP Hub and university collaborations like the attempto project. And stackoverflow has this.
If anyone could link me to some good resources, from reseach papers and introductionary texts to apis, I'd be happier than a 6 year-old kid opening his christmas presents!
Update
Through one of your recommendations I've found opencyc ('the world's largest and most complete general knowledge base and commonsense reasoning engine'). Even more amazing still, there's a project that is a distilled version of opencyc called UMBEL. It features semantic data in rdf/owl/skos n3 syntax.
I've also stumbled upon antlr, a parser generator for 'constructing recognizers, interpreters, compilers, and translators from grammatical descriptions'.
And there's a question on here by me, that lists tons of free and open data.
Thanks stackoverflow community!
Tough call, NLP is a much wider field than most people think it is. Basically, language can be split up into several categories, which will require you to learn totally different things.
Before I start, let me tell you that I doubt you'll have any notable success (as a professional, at least) without having a degree in some (closely related) field. There is a lot of theory involved, most of it is dry stuff and hard to learn. You'll need a lot of endurance and most of all: time.
If you're interested in the meaning of text, well, that's the Next Big Thing. Semantic search engines are predicted as initiating Web 3.0, but we're far from 'there' yet. Extracting logic from a text is dependant on several steps:
Tokenization, Chunking
Disambiguation on a lexical level (Time flies like an arrow, but fruit flies like a banana.)
Syntactic Parsing
Morphological analysis (tense, aspect, case, number, whatnot)
A small list, off the top of my head. There's more :-), and many more details to each point. For example, when I say "parsing", what is this? There are many different parsing algorithms, and there are just as many parsing formalisms. Among the most powerful are Tree-adjoining grammar and Head-driven phrase structure grammar. But both of them are hardly used in the field (for now). Usually, you'll be dealing with some half-baked generative approach, and will have to conduct morphological analysis yourself.
Going from there to semantics is a big step. A Syntax/Semantics interface is dependant both, on the syntactic and semantic framework employed, and there is no single working solution yet. On the semantic side, there's classic generative semantics, then there is Discourse Representation Theory, dynamic semantics, and many more. Even the logical formalism everything is based on is still not well-defined. Some say one should use first-order logic, but that hardly seems sufficient; then there is intensional logic, as used by Montague, but that seems overly complex, and computationally unfeasible. There also is dynamic logic (Groenendijk and Stokhof have pioneered this stuff. Great stuff!) and very recently, this summer actually, Jeroen Groenendijk presented a new formalism, Inquisitive Semantics, also very interesting.
If you want to get started on a very simple level, read Blackburn and Bos (2005), it's great stuff, and the de-facto introduction to Computational Semantics! I recently extended their system to cover the partition-theory of questions (question answering is a beast!), as proposed by Groenendijk and Stokhof (1982), but unfortunately, the theory has a complexity of O(n²) over the domain of individuals. While doing so, I found B&B's implementation to be a bit, erhm… hackish, at places. Still, it is going to really, really help you dive into computational semantics, and it is still a very impressive showcase of what can be done. Also, they deserve extra cool-points for implementing a grammar that is settled in Pulp Fiction (the movie).
And while I'm at it, pick up Prolog. A lot of research in computational semantics is based on Prolog. Learn Prolog Now! is a good intro. I can also recommend "The Art of Prolog" and Covington's "Prolog Programming in Depth" and "Natural Language Processing for Prolog Programmers", the former of which is available for free online.
Chomsky is totally the wrong source to look to for NLP (and he'd say as much himself, emphatically)--see: "Statistical Methods and Linguistics" by Abney.
Jurafsky and Martin, mentioned above, is a standard reference, but I myself prefer Manning and Schütze. If you're serious about NLP you'll probably want to read both. There are videos of one of Manning's courses available online.
If you get through Prolog until the DCG chapter in Learn Prolog Now! mentioned by Mr. Dimitrov above, you'll have a good beginning at getting some semantics into your system, since Prolog gives you a very simple way of maintaining a database of knowledge and belief, which can be updated through question-answering.
As regards the literature, I have one major recommendation for you: run out and buy Speech and Language Processing by Jurafsky & Martin. It is pretty much the book on NLP (the first chapter is available online); used in a frillion university courses but also very readable for the non-linguist and practically oriented, while at the same time going fairly deep into the linguistics problems. I really cannot recommend it enough. Chapters 17, 18 and 21 seem to be what you're looking for (14, 15 and 18 in the first edition); they show you simple lambda notation which translates pretty well to Prolog DCG's with features.
Oh, btw, on getting the masters in linguistics; if NL semantics is what you're into, I'd rather recommend taking all the AI-related courses you can find (although any courses on "plain" linguistic semantics, logic, logical semantics, DRT, LFG/HPSG/CCG, NL parsing, formal linguistic theory, etc. wouldn't hurt...)
Reading Chomsky's original literature is not really useful; as far as I know there are no current implementations that directly correspond to his theories, all the useful stuff of his is pretty much subsumed by other theories (and anyone who stays near linguists for any matter of time will absorb knowledge of Chomsky by osmosis).
I'd highly recommend playing around with the NLTK and reading the NLTK Book. The NLTK is very powerful and easy to get into.
You could try reading up a bit on phrase structured grammers, which is basically the mathematics behind much language processessing. It's actually not that heavy, being largely based on set and graph theory. I studied it many moons ago as part of a discrete math course, and I guess there are many good references available at this stage.
Edit:Not as much as I expected on google, although this one looks like a good learning source.
One of the early explorers into NLP is Noam Chomsky; he wrote small books on the subject in the 50s through the 70s. You may find that engaging reading.
Cycorp have a short description of how their Cyc knowledge base derives meaning from sentences.
By utilising a massive knowledge base of common facts, the system can determine the most logical parse of a sentence.
A simpler place to begin with the building blocks is the look at the documentation for a package that attempts to do it. I'd recommend the Python [Natural Language Toolkit (NLTK)1, particularly because of their well-written, free book, which is filled with examples. It won't get you all the way to what you want (which is an AI-hard problem), but it will give you a good footing. NLTK has parsers, chunkers, context-free grammars, and more.
This is really hard stuff. I'd start off by getting at least a Masters in Linguistics, and then work towards my PhD in computer science, concentrating on NLP.
The problem is that most of us don't have the understanding of what language is. And without that understanding, it's bloody tough to implement a solution.
Other comments give some readings, which are probably fine if you want to get started playing around with a small subset of the problem, but in order to come up with a really robust solution, then there are no shortcuts. You need the academic background in both disciplines.
A very enjoyable readable introduction is The Language Instinct by Steven Pinker. It goes into the Chomsky stuff and also tells interesting stories from the evolutionary biology angle. Might be worth starting with something like that before diving into Chomsky's papers and related work, if you're new to the subject.