There is an app that has elastic search via Algolia (elastic search) and algolia features RESTFul API to do operations on the data (add / remove doc etc).
There is also core persistence via PouchDB=CouchDB constant replication.
I want to make sure everytime a new document appears in CouchDB - it gets "replicated"/added to the algolia search.
Provided Algolia has restful API - am I correct that it would be relatively easy to setup filtered replication (more like copying) over to Algolia?
If not - what are the challenges?
It's really simple to replicate a couchdb database, by watching to changes in the _changes endpoint in some long-running script. Then you call saveObjects with the Algolia API client.
see also what we did to replicate the npm couchdb database into Algolia here: https://github.com/algolia/npm-search
Related
I am using mongodb to store items for an auction site
I want to enable fuzzy searching.
Should I query for 1000 results with no parameters then use a js library like fuse.js
Or should I rely on mongodb $regex alone to do the query?
mongodb isn't a great choice for a problem like this. There are lots of great text search utilities available, the most prominent these days being elasticsearch. You'd continue to store your data in mongodb, but you'd keep an elasticsearch instance synced to the mongodb database and perform your searches against elasticsearch. Mongoosastic is a good way to write to both concurrently or Transporter can be used shift the synchronization away from your database persistence flow.
Mongoosastic example:
https://blog.cloudboost.io/sync-mongo-with-elastic-and-save-months-of-development-time-and-cost-d281e0ca8fe4
Some other ways to sync including Transporter: https://code.likeagirl.io/5-different-ways-to-synchronize-data-from-mongodb-to-elasticsearch-d8456b83d44f
I am working with NodeJS and Cloudant (alternatively the DashDB Warehouse if that works better). I wonder if it is possible to have a function in NodeJS that gets called each time a document has been added to the database? I have checked out indexed views but can't really understand how to do it. Does anyone have any good tips regarding this or what documentation to look at?
You can listen to DB _changes in Cloudant (https://console.bluemix.net/docs/services/Cloudant/api/database.html) in continous mode.
Every document change in the Cloudant DB (create,update,delete) will be notified through this channel.
There are different nodejs libraries you can use with this purpose. This is
one example: https://www.npmjs.com/package/cloudant-follow
I am learning PouchDB with CouchDB and trying to wrap my head around intercepting documents to the couchdb server and performing an action on it wether it be creating other documents, updating the user table, etc.
On the server the json document will be treated through a business layer before it is submitted to the couchdb server, preferably in .net.
Is this possible? If not, is there a way to do so?
Thanks!
On the server side, you can listen to the _changes feed from CouchDB (docs here) and react whenever a document is added, modified, or deleted. This could be useful for reporting/messaging/aggregation/etc.
Alternatively, if you want to do some schema validation on the documents before they are accepted, then you should look into adding a design doc with a validate_doc_update field (docs here).
I am fairly new to both mongodb and node.js but recently got everything to work well for me until I reached the point where I needed to add a full text search to my website. From my research I figured out that Elasticsearch would be a good fit, but I couldnt figure out exactly how to get it to work with node.js and mongodb. I am currently using Heroku and MongoLab to host my application. Here are my questions.
How do I host Elasticsearch?
How do I make all my mongo data available to elasticsearch do I use a river or do I manually inset and delete all data?
I found something this river but I am not quite sure how to make this happen automatically and where to host it.
How do I query Elasticsearch from node.js? Is there a package that allows for this?
Edit:
Question 2 is really what I am struggling with. I have also included question 1 and 3 to help people that are new to the topic and coming from google.
1) either on your own server/VM/whatever.. or with a hosted service such as https://searchbox.io/
2) you can create a script to index your existing data and then index new data once its created, or use a river to index your current database.
3) ElasticSearch is a simple HTTP API, you can make your own requests using the 'http' module or simplify it with something like https://github.com/mikeal/request
you can also use a 3rd party library like https://github.com/phillro/node-elasticsearch-client
Searchly.com (Aka SearchBox.io)introduced a new feature crawlers includes MongoDB crawler.
It fetches data from a given collection and syncs periodically to ElasticSearch. Check http://www.searchly.com/documentation/crawler-beta/
You can host on your own server or use aws elasticsearch service or use elastic cloud provided by elasticsearch.
Try any of the following three solutions:-
i) Try using mongoosastic. NPM package
ii) Use mongo-connector.
iii) python script for indexing data to elasticsearch
elasticsearch-js. The javascript client library
Using the CouchDB river, it is possible to index CouchDB databases.
Is it also possible to index a CouchDB view with Elastic Search?
Not yet. See https://github.com/elasticsearch/elasticsearch-river-couchdb/pull/2
BTW you can checkout my pull request, build it and start to query views...