I am about to add the search engine to my Node application. I can see there is a npm package for ElasticSearch.
But why not simply using ElasticSearch standalone instead of nesting it with the application code?
The use case for me is:
Data is in a MySQL database with a true entity-relationship-diagram.
I intend to use Docker containers to ship my application elements (which can be modified as for now).
The results' positioning is intended to be complex, and I thought of a nightly cronjob that would re-evaluate the documents/results, because they depend on moving data (user's reputation or popular sales, for instance).
Don't you think a standalone ElasticSearch instance / grid is more enviable? I imagine that it would be more scalable, more secure, and easier to deploy in a cloud, for instance...
The npm package you refer to is a Javascript client for Node and in-browser use, that can be used to perform searches, index (create/insert) documents on the Elasticsearch server. What you refer to as Elasticsearch standalone is the server and yes, that can and should be entirely separate from the client application code.
It is definitely possible to not use a client library and access the Elasticsearch REST API directly using whatever HTTP client code, but then it is too easy to reinvent the wheel, badly.
Related
We can accept 20 times slower to avoid using python or C++ in our project. Are there a native module that still works?
There is no need to invent wheel. mongodb package is the simplest one
But if You insist there are many ways:
Easy way: You can use Rest API of mongo and do requests to it using request package
Moderate way: Open mongodb-core package and copy out what You need most, make Your own mongodb class.
Moderate way #2: fork mongodb package, manipulate it and save with new repository name
Hard way: If You want go hardcore (: read mongodb protocol and operate with it using net package to open socket connection to mongodb server.
How about Crest? It's a node wrapper around the MongoDB server that provides a REST API. With it you could talk to MongoDB over REST instead of with a native client, similar to CouchDB.
There are some other utilities listed here. Maybe you're okay with using Python outside of your app but in front of MongoDB to provide the REST API? If so then maybe those are some alternatives if you don't like Crest. Haven't used it myself so I can't vouch for its quality, but it is listed on MongoDB's own list so hopefully it's decent.
I am new on developing in Node.js, I would like to know if there's a way to have ElasticSearch embedded in a Node.js application for testing purposes or if there's another way to test interaction with ElasticSearch without having an ElasticSearch running instance (mock ?)
There are no existing mock libraries for Elasticsearch and Node at the moment. If you know exactly what queries you are going to be performing, you could use a mocking library to mock the entire ES client and assert simple behaviors.
What I'd recommend, however, is actually running ES in a development environment for testing. I find that mocking external services is a pretty tricky thing to do in general, and your tests will likely be much more robust / trustworthy if running against an actual instance of the server.
I'm learning more about ArangoDB and it's Foxx framework. But it's not clear to me what I gain by using that framework over building my own stand alone nodejs app for API/access control, logic, etc.
What does Foxx offer that a regular nodejs app wouldn't?
Full disclosure: I'm an ArangoDB core maintainer and part of the Foxx team.
I would recommend taking a look at the webinar I gave last year for a detailed overview of the differences between Foxx and Node and the advantages of using Foxx when you are using ArangoDB. I'll try to give a quick summary here.
If you apply ideas like the Single Responsibility Principle to your architecture, your server-side code has two responsibilities:
Backend: persist and query data using the backend data storage (i.e. ArangoDB or other databases).
Frontend: transform the query results into a format acceptable for the client (e.g. HTML, JSON, XML, CSV, etc).
In most conventional applications, these two responsibilities are fulfilled by the same monolithic application code base running in the same process.
However the task of interacting with the data storage usually requires writing a lot of code that is specific to the database technology. You need to write queries (e.g. using SQL, AQL, ReQL or any other technology-specific language) or use database-specific drivers.
Additionally in many non-trivial applications you need to interact with things like stored procedures which are also part of the "backend code" but live in the database. So in addition to having the application server do two different tasks (storage and rendering), half the code for one of the tasks ends up living somewhere else, often using an entirely different language.
Foxx lets you solve this problem by allowing you to move the logic we identified as the "backend" of your server-side code into ArangoDB. Not only can you hide all the nitty gritty of query languages, edges and collections behind a more application-specific API, you also eliminate the network overhead often necessary to handle requests that would cause more than a single roundtrip to the database.
For trivial applications this may mean that you can eliminate the Node server completely and access your Foxx API directly from the client. For more complicated scenarios you may want to use Node to build external micro services your Foxx service can tap into (e.g. to interface with external non-HTTP APIs). Or you just put your conventional Node app in front of ArangoDB and use Foxx to create an HTTP API that better represents your application's problem domain than the database's raw HTTP API.
It's also worth keeping in mind that structurally Foxx services aren't entirely dissimilar from Node applications. You can use NPM dependencies and split your code up into modules and it can all live in version control and be deployed from zip bundles. If you're not convinced I'd suggest giving it a try by implementing a few of your most frequent queries as Foxx endpoints and then deciding whether you want to move more of your logic over or not.
I'm looking into using gcloud node api to access the datastore api but was curious if it supported query caching in a similar manner to ndb? If not, what's the best way to make sure repeated queries are cached?
As far as I know, gcloud-node isn't planning to be a full-on ORM (like ndb is for Python). Also, as Patrick Costello noted in the comments above, NDB doesn't cache query results, but individual entities instead.
I think if you want caching of query results (or individual entities), you'd have to manually cache these by running your own Memcache server (http://memcached.org/) and interacting with it using memcached (https://www.npmjs.com/package/memcached)
I ended up using NsqlCache-datastore which is integrated into gstore-node. Guide: https://medium.com/google-cloud/how-to-add-a-cache-layer-to-the-google-datastore-in-node-js-ffb402cd0e1c
Looks like I can use the memcache app engine service accessible through this node library:
https://github.com/GoogleCloudPlatform/appengine-nodejs
My app has Node JS. I'm trying to connect NodeJS and Neo4j together. Can some one tell me, how to connect both? My queries need to work with labels on Neo4j. Please let me know which module should I use in Node Js to achieve this?I have spent lot of time already with-out luck.
Last I checked there are at least 4 popular and actively developed node.js modules (ordered by number of stars):
https://github.com/thingdom/node-neo4j (npm install neo4j)
https://github.com/bretcope/neo4j-js (npm install neo4j-js)
https://github.com/philippkueng/node-neo4j (npm install node-neo4j)
https://github.com/brikteknologier/seraph (npm install seraph)
They all support the Cypher endpoint, which was a requirement for my inclusion. One key feature that stands out from the list is that philippkueng/node-neo4j is the only one that has transactional API support. Another is the ability to ask for labels of nodes, and that is supported only by seraph and philippkueng/node-neo4j. (usually you can avoid needing to ask for labels of a node if you make your Cypher query ask for labels explicitly, which avoids a request back and forth)
On the other hand, it's really not hard to just implement a few HTTP requests, directly accessing the Cypher or Transactional Cypher endpoints, massaging the results as you see fit for your application.
Another cool new development I've seen recently was https://github.com/brian-gates/cypher-stream, which emits a stream of results from Cypher, enabling streaming JSON parsing, which is another performance-oriented feature lacking from the four listed above.
Edit: 03/2016 There is a new official JS driver for use with the new bolt protocol (binary). For new development this should definitely be considered. Bolt is planned for release in Neo4j 3.0. https://github.com/neo4j/neo4j-javascript-driver
Check out the koa-neo4j framework, it uses the official neo4j-driver under the hood.
One can write native Cypher (as .cyp files) in it on top of the latest stable neo4j (3.0.3 at the time of this writing) which, among other things, allows querying labels.
https://github.com/assister-ai/koa-neo4j
https://github.com/assister-ai/koa-neo4j-starter-kit
In a Neo4j enabled application, conducting queries directly from client side might not be the best choice:
Database is exposed to the client, unless some explicit security mechanism is in place; one can see the innards of the database by View page source
There is no one server to rule them all, queries are strings, scattered around different clients (web, mobile, etc.)
Third-party developers might not be familiar with Cypher
koa-neo4j addresses all of the above issues:
Stands as a middle layer between clients and database
Gives structure to your server's logic in form of a file-based project; finally a home for Cypher! All of the clients can then talk to an instance of this server
Converts Cypher files to REST routes, a cross-platform web standard that developers are familiar with, it does so on top of the widely-adapted koa server, ripe for further customization
Disclosure I was the original author of koa-neo4j
neode - Neo4j OGM for Node JS. here