I'm designing a new search based web application in lucene. I've to share and re-open the IndexSearcher across multiple Threads. I've read about the SearcherManager in the api document. I'm not able to understand the uses of it. How should I share the SearcherManager across MultipleThreads.
see here for an article with code samples from one of the Lucene commiters
Related
All the examples I find for JHipster is for applications that have a one to one relationship between entities and screens.
I would like to have more than one entity being controlled by one screen.
Is there any example of this?
Is there a best practice?
I don't mind hand coding, but if I start to hand code I want to follow some best practice.
I normally google for this, but all the examples are too simple: one-to-one examples.
If you're looking for a full jhipster example with multiple entities per screen, you probably won't find one. You got to search by underlying technologies: angular, jackson, spring mvc and you'll find richer examples.
Few tips:
angular-ui router multiple named views per state can help
consider aggregating REST resources on server to avoid multiple requests per screen
I am fairly new to Azure and mobile services, and all the examples and tutorials I can find for the table and API scripts are fairly simplistic.
If I have some processes that are fairly complex and rely on pulling information from many different tables and processing contingent on that data, should I be doing that somewhere other than the API scripts? I am new to node.js as well so maybe that's the problem but I was wondering if there is a more appropriate place for business logic, such as some bridge I need to add to my stack?
There are a lot of examples of how to use MSSql object which is used to query tables and Node in general available. A healthy search will reveal just about anything you need. Since you said you are new to Node.js consider using the .NET backend instead. It is based on Entity Framework and there are lots of Entity framework examples out there for you too. Finally, there are some really good examples of complex logic being used in the back ends in the sample code available: http://azure.microsoft.com/en-us/develop/mobile/ios-samples/ (pick your client OS) and here: http://azure.microsoft.com/blog/topics/mobile/ and here: http://blogs.msdn.com/b/azuremobile/
Let us know if you have specific questions!
I would like to build a search engine for my website so I can quickly find relevant content. I've done quite a few google searches, discovered ElasticSearch and Solr (which both sit on top of Lucene), and whoosh (python-based).
But are all of these search engines just building an "inverted-index" on top of the data? What are some other algorithmic approaches for getting higher quality searches?
I was intrigued by this blog post using collaborative filtering on top of Solr, which returns related search queries:
http://www.opensourceconnections.com/2013/08/25/semantic-search-with-solr-and-python-numpy/
Are there other common techniques that I should be aware of? Are there other libraries sitting on top of ElasticSearch/Solr that I could just plug into, and use "out-of-the-box"?
Any links or tips would be greatly appreciated!
You haven't mentioned what tech stack you are working on.
If you use Ruby on Rails, I would recommend Tire, which is a gem that gives a DSL wrapper over ElasticSearch. Essentially, it allows you to index your data in Elasticsearch.
For Rails, Sunspot is a very popular gem that people use to interface with Solr.
For .NET - SolrNET is a great Solr client.
Other part of your question (around implementing a good search engine) is too broad - I would recommend reading a good book such as Lucene in Action to get a feel of what Solr/Elasticsearch could do.
I do have a few notes that I wrote a while back, you can read about some of my experience in search here.
Edit:
Since you work on python, I would recommend Haystack, although it is specific to Django. It is very versatile for our needs. However, if you are not using django, I can think of solrpy as a Solr client. Haystack works with both Solr and Elasticsearch.
i suggest you to learn Solr API, cause it was developed since 4 5 years so you can find lots of plug-ins like related search API in Solr, But in elastic search it is very easy to configure however it is very young engine so needs to be developed more.
Pyes is a well-documented Python client for Elasticsearch.
Also, this Youtube video provides a good overview of using Elasticsearch with Python.
I suggest you to use Google Custom Search Engine.
Here have a look.
https://www.google.com/cse/all
We have developed several search engines both on Solr and Elastic. Solr used to be the best as it provided most of the tools needed to admin and debug your indexes. Right now Elastic offers the same features as Solr either natively or via plugins. Plus it is easier to configure in high performance/high availability scenarios (easy to shard or cluster).
Your technology stack is irrelevant. Both Solr and Elastic have clients nearly for every language, plus you can access both via plain HTTP:
That said, each search engine applies to a problem domain. Tunning Elastic or Solr to retrieve relevant results is a bit of an art with some trial and error.
You will have to define analyzers for each field you'll search on and according to your search patterns and the kind of results you will be expecting.
Eventually, to create search engines with a single input that search across disparate attributes of a document type, may need the use of DisMax queries where you can boost results depending on the matching of the search terms to specific document fields.
To summarize: go for Elastic, and get some plugins or frontends. Two suggestions:
Inquisitor: for testing your analyzers
Elastic Head: for administration purposes
We are working for a client to redesign an existing system which basicaly deals with a lot of files.
The files(more than 5 million) are currently stored on the servers filesystem.The client wants the new system to store the file in S3.
The files also have metadata associated(name,authors name,price ,description etc.).
The search functionality is also to be redesigned.The following are the basic requirements
Full text search should be available on file descriptions.
Filtering should be possible on other attributes of files.
Also , based on the file description, the system should also be able to give recommendation for similar files.
I do not have experience with creating such solution before,so asking for help and suggestion.
I was thinking on the lines of following solutions:
Store the file meta data in MongoDB ,and use the search functionality (http://www.mongodb.org/display/DOCS/Full+Text+Search+in+Mongo)
Use Amazon DynamoDB.It provides api to scan/query the dataset.
Use Lucene/Solr(I havent worked with these yet,I still need to look deeper)
There was this project that I found,that is very similar to what I require
http://www.thriftdb.com - On the home page it says its a datastore with search builtin.
Please let me know if this question should be a community wiki.
Thanks in advance.
You're in luck, announced today:
http://aws.amazon.com/about-aws/whats-new/2012/04/11/aws-announces-cloudsearch/
About searching files and filtering by attributes, the best would be Sphinx Search Engine which is used in filestube (google was using it also years ago).
I dont know if it will work on amazon servers.
Amazon has a custom AMI for Lucene/Solr and we have been happily using it in our projects. Lucene has a powerful indexing capability and executes at exceptional speeds. I would strongly recommend using Apache Lucene/Solr for all your search needs.
We have a web app that allows users to upload documents, create their own documents, and so on. Uploaded files are stored on Amazon S3, created information is stored in a MySQL database. What I'm looking for is some sort of search engine, where I feed it all of our text documents, each with a unique ID, and it builds an index or whatever. Later, I can give it search queries, and it will pull out the best matching documents (via their ID), along with snippets of matching text.
Basically we want to allow our users to search through their repository of uploaded stuffs, along with anything that other users have marked as public. The solution should run on a standard Linux server, and ideally it would be open source, but I'll also consider paid solutions if they aren't outrageously priced.
So far, I've found three potential candidates:
MySQL Full Text Search - some reports I've read are that it's very slow
Apache Lucene - unfortunately written in Java, but I'll use it if I have to. Supposedly fast
Sphinx - doesn't seem to be as popular, ideally whatever solution I find will have lots of community support.
Please let me know if there are any other good choices that I've overlooked, or if you have experience with any of the above.
Take a look at Solr. It's based on Lucene, so it's very fast, and it's really easy to use from any platform.
Sphinx may be worth your consideration, as it works well with several common RDMS (notably MySQL)
There is also Xapian which is fast and is quite customizable.
It has support for custom indexers allowing one to index data that is not stored in a database which might be useful for your documents stored on S3.
I imagine that Google will have a solution that meets your needs. Start here: Google Enterprise
There is a Ruby port of Lucene called "Ferret". In addition to the Ruby API, you can get at the underlying c implementation called "cFerret".
Lucene is very good. And although it was originally written in java there is a php implementation http://framework.zend.com/manual/en/zend.search.lucene.html