I'm looking at using Bookshelf.js as an ORM for an Express project, with Knex. My only question is whether it supports streaming--if we have a query which returns many results, I'd prefer to deal with a stream rather than saving the results in memory. I am not seeing this functionality in the docs, but perhaps there's a way to do it?
Currently Bookshelf doesn't have that functionality, but you can use Knex directly which does. Of course you lose the benefits of using an ORM, but you gain a bit more performance in return, which is probably more important if you're dealing with huge amounts of data.
You can read more about it in Knex's documentation.
Related
The MongoDB docs discuss modelling computed data saying:
The application can either recompute the value with every write that changes the computed value’s source data, or as part of a periodic job.
Updating the data periodically is straight forward but not real-time enough for our use-case, so I wanted to know the best practice for hooking into every DB write in order to recompute some derived data?
Our application is written in NodeJS, using the native MongoDB driver.
I understand Mongoose offers a 'save' hook which sounds ideal for this, however it falls down pretty quickly as it doesn't apply to .update, .updateMany etc. (and Mongoose.prototype.save is pretty terrifying imo).
Does MongoDB support change events at the DB level which would be suitable, or perhaps the native driver (or another client?) exposes hooks designed for re-computing data?
The answer I got from an IRC channel:
Sequelize is an ORM that includes some query builder stuff; Knex is just a query builder, not an ORM.
ORMs don't actually fit very well in many use cases, it's easy to run up against the limits of what they can express, and end up needing to break your way out of them.
But that doesn't really explain the pros and cons of each. I am looking for an explanation, and possibly a simple example (use case) highlighting those similarities / differences.
Why would one use one over the other?
Sequelize is full blown ORM forcing you to hide SQL behind object representation. Knex is plain query builder, which is way too low level tool for application development.
Better to use objection.js it combines good parts of ORMs without compromising power of writing any kind of SQL queries.
Here is good article about it from the author of objection.js https://www.jakso.me/blog/objection-to-orm-hatred
Disclaimer: I'm knex maintainer and been also involved in development of objection.js.
Think of it like this which is the better performance and which is easier to learn.
As low level Database driver
For postgresql you can use pg as a query builder
As intermediate level you can use knex
As high level you can use ORM like sequelize, bookshelf, objection which is based on knex
Now low level doesn’t mean a bad thing. It’s the best performance you can get but the down side is you need to learn queries of the database you are using
Now knex is the same as a query builder the same cost operation
Now the highest level have the highest cost
But it’s easy to learn but the down size if you learn sequelize and decided to use objection they are different so you will need to learn another ORM
My suggestion if you want the best performance for a scalable complex backend server you can use query builder or knex
If you want to feel like dealing with objects instances like mongoose you can use Sequelize.
The only difference is the cost operating and it’s not large.
But ORMs have more functionality.
Of course you can refer to this article to understand more
About ORM
https://blog.logrocket.com/why-you-should-avoid-orms-with-examples-in-node-js-e0baab73fa5/
The MongoDB sorting functions are pretty neato. Can you use them on objects and/or arrays that have nothing to do with the database itself?
var mongo = require('mongodb'),
Server = mongo.Server,
Db = mongo.Db,
sortingFun = mongo.internalSortFilterFunction(); // By the miracle of imagination, this is a made-up line.
There is, for example, this awesome little node project called sift: MongoDB inspired array filtering. But there are more similar tools, different opinions, and projects merging and disappearing.
Considering it's popularity, MongoDB is quite probably gonna hang around. For that reason, plus the added bonus of being exactly similar instead of pretty similar, I was wondering if a specific object/model/function within node-mongodb could be linked from the require('mongodb') specifically for using the sorting and filtering functions on custom objects/arrays.
The sorting is done in the mongo server, not the client. It's also not particularily fast -- big collections should be pre-sorted, but that's another issue.
The mongo server is afaik written in C++ and uses custom types, separate from the JS engine, called BSON.
So if there is no sort implementation on the client for javascript, which would be an absurd feature, you can't use server sort.
Edit: If you really really want to use the sort, performance be damned, you could insert js objects into the DB, effectively converting them to BSON in mongo collections. Then sort it and pull it from the DB. Indexes etc will need to be recreated for every call to that function. Mongodb also refuses to sort for big collections sans index (limit being somewhere around 1000 I believe)
PS. I haven't read the source. I can't imagine a JS realtime, indexless sort that matches the speed of MongoDB's sort esp. when distributed (sharded). But you can write node.js modules in C++, and if BSON is similar enough to V8 JS objects (wouldn't think so), you might be able to port it. I wouldn't go down that road because it's probably not going to be a big speed increase compared to reimplementing it in JS, a reimplementation which would be a lot easier to create and maintain.
I'm working on a "real-time" website using Nodejs. Currently, I'm using Redis because I need high performance for read-access. The write accesses are not really significant for my use case.
In addition, Redis does not have a query language for the search. So, I create my indexes manually and I use some unions/intersections/... to find some values.
I think that it will be easier to use MongoDB with a embedded finding system and a ORM-like (Mongoose for example). The problem is that I'm not sure that MongoDB is the best choice for my usecase.
What is your advices about the NoSQL DB that I need ? Redis ? CouchDB ? MongoDB ? Cassandra ? etc.
I repeat: I want to have a real good performance for the read accesses and for the searches (the write accesses are not significant), the simplest possible (orm-like ? finding system ? etc.)
Thanks.
I believe that redis would be the better solution for the following reasons.
You require fast read access and redis provides the fastest solution since the keys are in memory, if not most.
Although mongodb is easier to query in the general case, your problem domain is narrow and once you decide how you would like to query the data, you can put the correct data structures and indexes in place.
I would say that Redis is a good fit for your DB, and you should look at something like Solr or elasticsearch to provide your searching.
CouchDB will do better in write heavy environment. I don't use it though.
MongoDB will do better on read heavy environment.
For search and indexing:
MongoDB would require separate index for each of your search criteria for better performance (at least this is what I remember).
Proper index is important in MongoDB. And no joins!!
Here are some links you might go through:
http://www.mongodb.org/display/DOCS/Comparing+Mongo+DB+and+Couch+DB
http://www.snailinaturtleneck.com/blog/2009/06/29/couchdb-vs-mongodb-benchmark/
http://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-redis
Hope these will help you find the right db
Goodluck
I've been learning Node.js so I decided to make a simple ad network, but I can't seem to decide on a database to use. I've been messing around with Redis but I can't seem to find a way to query the database by specific criteria, instead I can only get the value of a key or a list or set inside a key.
Am I missing something, or should I be using a more robust database like MongoDB?
I would recommend to read this tutorial about Redis in order to understand its concepts and data types. I also had problems to understand why there is no querying support similar to other (no) SQL databases until I read few articles and try to test and compare Redis with other solutions. Maybe it isn't the right database for your use case, although it is very fast and supports advanced data structures, but lacks querying which is crucial for you. If you are looking for a database which allows you to query your data then you should try mongodb or maybe riak.
Redis is often referred to as a data
structure server since keys can
contain strings, hashes, lists, sets
and sorted sets.
If able(easy to implement) you should use these primitives(strings,hashes,lists,set and sorted sets). The main advantage of Redis is that is lightning fast, but that it is rather primitive key-value store(redis is a little bit more advanced). This also means that it can not be queried like for example SQL.
It would probably be easier to use a more advanced store, like for example Mongodb, which is a document-oriented database. The trade-off you make in this case is PERFORMANCE, but I believe you should only tackle that if that is becoming a problem, which it probably will not be because Mongodb is also pretty fast and has the advantage that it can be queried. I think it would be advisable to have proper indexes for your queries(read>write) to make it fast.
I think that the main answer comes from the data structure. Check this article about NoSQL Data Modelling, for me it was very helpful: NoSql Data Modelling.
A second good article ever about Data Modeling, and making a comparison between SQL and NoSQL is the following: The Relational model anti pattern.