I have a nodejs project that using couchbase as database.
Just wonder if I store the temporary data in
1.redis
or in
2.couchbase directly.
As I know there is socket delay for couchbase, I think store temporary data in redis while store the permanent data in couchbase is better.
Is there any person has the experience on this?
Your comment welcome
I'm a big Redis fan, but in this situation I would use Couchbase only.
Couchbase is rather efficient, and comparable to the performance of memcached when the working set of your data fits in memory. Most of the time, an extra caching layer on top of Couchbase is not useful.
That said, if you really need a caching layer, or simply some storage for temporary data, you can simply create a memcached bucket hosted in the Couchbase cluster. So you would have an "eventually persistent" bucket for your persistent data, and a memcached bucket for the temporary data.
The bucket types are described here:
http://docs.couchbase.com/couchbase-manual-2.5/cb-admin/#data-storage
In that context, adding Redis as a extra storage layer does not really make sense.
Couchbase has a managed cache built into it, even for Couchbase buckets. So it already has a caching layer and adding another one on top just sounds superfluous.
I am not sure what you mean by a socket delay in Couchbase. Can you perhaps explain more about that? That is not something I have ever seen before and sticks out as suspect to me. I would try and troubleshoot this and figure out what that is before looking to add redis to the mix and have yet another layer to manage and code against. Without know more about the socket delay, it is difficult to make more recommendations.
It's an old question, but I'll have my take at it as well, if nothing else then for the people coming across it via google, just as I did.
I agree with he accepted answer, in that CouchBase has the most recently used Documents in RAM. In that aspect, it does the same as Redis. The advantage of CouchBase is of course that the data can reliably spill over the RAM limit, and the server disk limit, automatically, by adding more nodes.
However, I have a project where I am considering using Redis along side CouchBase. It's basically thought as a caching server, but for the "calculated" items. Such as html-snippets or other things. CouchBase is a fantastic document store, but making lists and other structures, doesn't come that easy, especially not without a lot of views. So I'm thinking to use Redis as a temporary datastore for the ad-hoc data manipulation needed, and CouchBase as the main datastore.
Related
Trying to decide between DynamoDB and CouchDB for my website. It's a static site (built with a static site generator) and I'm planning on using a JavaScript module to build a comment system.
I'm toying with using PouchDB and CouchDB so that synchronizing is easy. I'm also considering DynamoDB.
I have a performance question. From these databases, do any of them push data out to edge locations so that latency is reduced? Or is my Database essentially sitting on one virtual server somewhere?
From what I know, neither of these solutions utilise edge locations ootb.
Since you're mentioning PouchDB, I assume you want to use a client-side database in your app?
If that's the case you should keep in mind that, in order to sync, a client-side DB needs to have access to your cloud db. So it's not really suitable for a comment system since all client could just drop comments of others, edit them, etc.
I am building a simple game in node.js using socket.io. My web experience with node.js has typically involved saving everything to a relational database and keeping nothing in memory. I set up a relational database for the state of a game. I am using sqlite3 for development and I might use something like PostgreSQL or MySQL for production.
My concern is that, every time an event is emitted from the socket the whole game-state is loaded into memory from the server. I feel that in practice this will be less efficient than just keeping all of the game-state data in memory. Events will probably be emitted every 5 seconds or so during a game. All of the game data is temporary, none of it will be needed after the game is over. A game-state consists of a set of about 120 groups of small strings and integers (about 10 per group but subject to change).
Is it good practice to keep this type of data in memory?
If not, should I stick with relational databases or switch to a third option like a file-based storage structure?
Should I not load the whole gamestate in for every event even though that will lead to a lot more read/writes (at least triple)?
I would not keep this data in the memory of your NodeJS application. Its best avoid storing state in your app server. If you really need faster read access than sql provides consider using a cache like Redis or Memcached as a layer between your app and db.
All that being said its best not to prematurely optimize you code. Most SQL engines have their own form of cacheing, and optimizing your sql queries is a better place to start if your experiencing performance issues. Postgresql Query Optimization
But don't worry about it until its an actual problem (because most likely it never will be).
Sounds like relational, SQL type database is a huge overhead for your specifics. Do you have idea how big your data is and how many users you'd like to handle? Then you could compare it with your's server ability. If result is negative (couldn't handle with mem) - then i'd go with some quick nosql, like mongo. For yours example its sounds like the best choice. It'll be faster to get data for single session, easier to dump, more elastic in structure.
Need to setup a server backend web-service and contemplating either some MongoDB solution or other NoSQL and cache concoction. I've read several articles indicating how Couchbase is so much faster than MongoDB which isn't a slouch itself. Here's for reference:
http://www.couchbase.com/press-releases/couchbase-dominates-cassandra-datastax-and-mongodb-newly-released-nosql-performance-benchmark
http://prnewswire.com/news-releases/mongodb-30-with-wired-tiger-new-benchmark-measures-performance-vs-couchbase-server-302-300053144.html
So my question how true is this? Has anyone else tested and can confirm such orders of magnitude performance difference?
If so, is there a way to improve MongoDB performance by integrating some cache for it? I think Couchbase is actually a 'cache' with CouchDB store added, how can MongoDB be used/integrated in some manner to provide similar performance?
Why not just use Couchbase if its better?
Well, I was concerned by reading many places about its "lack of documentation". Then I was alarmed by reading this:
"...Couchbase forum threads which are habitually abandoned by Couchbase reps when a developer points out a pretty huge flaw in their code, intentionally or unintentionally..."
http://scalabilitysolved.com/dont-use-couchbase-unless-you-really-really-want-to/
Just go to the bottom of that article linked above and read the entire comment at the bottom by Erutan. Basically if one goes to Couchbase website it does seem that the company is really pushing their "Enterprise" version mainly which is fine, but it is worry-some when people think that they might be purposefully not providing documentation and perhaps I misunderstood, but from what I gather from that Couchbase user's comments, some think that bugs might be left in the code "intentionally" to steer people to the enterprise version?
On the PLUS side, it does seem that all the code is Apache licensed so anyone is free to fix any bugs.
Anyway, for me, I was leaning towards MongoDB for various reasons, although performance was one of them, until happened on some couchbase benchmarks. Looking forward to some affirmations or challenges to these couchbase performance superiority claims and possible solutions to bolster MongoDB setup.
So is Couchbase way faster than any other non-memory proven/stable NoSql?
CouchBase is fast but not the fastest one. I tested it, and in my scenarios Tarantool was 20% faster in terms of requests per second. Both of them are at order of magnitude faster than MongoDB. Maybe you should consider using one of the in-memory with persistence databases instead of MongoDB as your primary data store. One database is more consistent than a database and a cache layer on top of it.
I recently wanted to process some large data in nodejs and then put them into mongodb, at first I tried to process data in memory (javascript arrays),but when I decide to use node cluster, I figured out that by using node cluster it is not good way to use cache object. however, it seems impractical solution to store data in arrays even for short time. I think it would be better to store it in some memory databases such as redis.
so my first question is whether I can use redis via node cluster or not? and whether redis is the best solution or not?
thanks in advance.
When using the cluster module, an external data store is by far the simplest way of sharing state. Think of each worker as an individual client accessing the data store. Redis happens to be perfectly fit for the job, but the same can be said for just about any database. For short term storage, redis and memcached are both popular solutions. For arrays, redis beats memcached because it has an array like type that you can push to and pop from.
An extremely popular library for Node.js connecting to redis is node-redis. I've personally used it and highly recommend the API.
The Setup:
Imagine a 'twitter like' service where a user submits a post, which is then read by many (hundreds, thousands, or more) users.
My question is regarding the best way to architect the cache & database to optimize for quick access & many reads, but still keep the historical data so that users may (if they want) see older posts. The assumption here is that 90% of users would only be interested in the new stuff, and that the old stuff will get accessed occasionally. The other assumption here is that we want to optimize for the 90%, and its ok if the older 10% take a little longer to retrieve.
With this in mind, my research seems to strongly point in the direction of using a cache for the 90%, and then to also store the posts in another longer-term persistent system. So my idea thus far is to use Redis for the cache. The advantages is that Redis is very fast, and also it has built in pub/sub which would be perfect for publishing posts to many people. And then I was considering using MongoDB as a more permanent data store to store the same posts which will be accessed as they expire off of Redis.
Questions:
1. Does this architecture hold water? Is there a better way to do this?
2. Regarding the mechanism for storing posts in both the Redis & MongoDB, I was thinking about having the app do 2 writes: 1st - write to Redis, it then is immediately available for the subscribers. 2nd - after successfully storing to Redis, write to MongoDB immediately. Is this the best way to do it? Should I instead have Redis push the expired posts to MongoDB itself? I thought about this, but I couldn't find much information on pushing to MongoDB from Redis directly.
It is actually sensible to associate Redis and MongoDB: they are good team players. You will find more information here:
MongoDB with redis
One critical point is the resiliency level you need. Both Redis and MongoDB can be configured to achieve an acceptable level of resiliency, and these considerations should be discussed at design time. Also, it may put constraint on the deployment options: if you want master/slave replication for both Redis and MongoDB you need at least 4 boxes (Redis and MongoDB should not be deployed on the same machine).
Now, it may be a bit simpler to keep Redis for queuing, pub/sub, etc ... and store the user data in MongoDB only. Rationale is you do not have to design similar data access paths (the difficult part of this job) for two stores featuring different paradigms. Also, MongoDB has built-in horizontal scalability (replica sets, auto-sharding, etc ...) while Redis has only do-it-yourself scalability.
Regarding the second question, writing to both stores would be the easiest way to do it. There is no built-in feature to replicate Redis activity to MongoDB. Designing a daemon listening to a Redis queue (where activity would be posted) and writing to MongoDB is not that hard though.