I have been reading through some instructions on using redis and memcache. Can someone input on which would be more suitable?
Our scenario, here is to cache results from a database query and store it for a period of time, if the same value is called it should be used from the cache if its within the time interval.
Can someone share an example of how this can be achieved . just a simple example would do it?
Both Redis and Memcache should work for your use case. But Redis has more features compared to Memcache. Redis can provide persistence of cached data, Memcahe doesn't have persistence. Redis can provide high availability as well as clustering. Memcache doesn't have clusering and HA (some libraries impose HA, clustering from client side, but that is less reliable). Redis provides more data structures and features. Overall, Redis is a better choice as it is kind of Memcache++++.
Related
I have an application having more than 2 TB of data to be stored in cache, the data will be accessed using NodeJS APIs. For a NodeJS app which would be a better choice, Hazelcast or Redis(or RedisLabs)? Considering following criteria?
NodeJS API Support, including connection pooling. Looks like HazelCast doesn't have NodeJS API
I understand that in benchmarking Hazelcast is faster due to multithreaded implementation, and its scalable as well. But can we effectively leverage these good features using NodeJS(need Set datastructure)? Lastly, we can have multiple shards in RedisLabs which will be like having multiple threads or processes working on their respective chunk of data, in that case I believe the Hazelcast's edge due to multi-threaded nature would be true for Redis but not for RedisLabs, Any comments in this?
Hazelcast Node.js client in fact does exist and currently provides following features
implementation of Open Client Binary Protocol, Redis uses text-based protocol
Map
Get
Put
Remove
Smart Client - clients connect to each cluster node.
Since each data partition uses the well known and consistent hashing algorithm, each client can send an operation to the relevant cluster node, which increases the overall throughput and efficiency. The client doesn't need to be restarted in case of adding or removing nodes from the cluster.
Distributed Object Listener
Lifecycle Service
In terms of comparing Hazelcast and Redis server-side features, you find comprehensive doc here.
Thank you
Well, I would suggest if you are using very complex data handling/processing you should go with HazelCast, and by nature nodejs is single threaded so if you are using it just to store key-value dont go with it.
There is official API you can use for (NodeJS + hazelcast) but with very limited functionality, uses only KeyVal
If you are just using cache as key-value store Redis is good, fast, FREE!, it can handle huge data as well with some extra setup take look at
http://redis.io/topics/partitioning
in terms of support pricing RedisLabs are less costly and if you use Redisson with Redis it can give you all data structure which Hazelcast use :)
BitSet, Set, Multimap, SortedSet, Map, List, Queue, BlockingQueue, Deque, BlockingDeque, Semaphore, Lock, AtomicLong, CountDownLatch, Publish / Subscribe, Bloom filter, Remote service, Spring cache, Executor service, Live Object service, Scheduler service
RedisLabs is having much more userbase + contributors, HazelCast is bit less if you comapre users, so if your data is 2TB if its just key-value .. Redis be best
I recently wanted to process some large data in nodejs and then put them into mongodb, at first I tried to process data in memory (javascript arrays),but when I decide to use node cluster, I figured out that by using node cluster it is not good way to use cache object. however, it seems impractical solution to store data in arrays even for short time. I think it would be better to store it in some memory databases such as redis.
so my first question is whether I can use redis via node cluster or not? and whether redis is the best solution or not?
thanks in advance.
When using the cluster module, an external data store is by far the simplest way of sharing state. Think of each worker as an individual client accessing the data store. Redis happens to be perfectly fit for the job, but the same can be said for just about any database. For short term storage, redis and memcached are both popular solutions. For arrays, redis beats memcached because it has an array like type that you can push to and pop from.
An extremely popular library for Node.js connecting to redis is node-redis. I've personally used it and highly recommend the API.
I am using MongoDB for persisting data, and Redis for session storage. I need to cache a complex aggregation query done on MongoDB, so I was wondering what your opinions are on Caching on NodeJS, specifically caching with Redis or CouchDB.
Which one is more performant? Correct me if I'm wrong, but is having a different database specifically for cache better in terms of scalability?
It depends what you want to do; but basically, Redis is much more performant than CouchDB, and it is much more suitable for a cache system. Redis is an in-memory database (with options to sync to disk), with features for expiring data after a certain time.
Redis is used often for caching, so it may be better in terms of scalability. If you already have Redis running now, then you will have no additional maintenance overhead compared to CouchDB.
The Setup:
Imagine a 'twitter like' service where a user submits a post, which is then read by many (hundreds, thousands, or more) users.
My question is regarding the best way to architect the cache & database to optimize for quick access & many reads, but still keep the historical data so that users may (if they want) see older posts. The assumption here is that 90% of users would only be interested in the new stuff, and that the old stuff will get accessed occasionally. The other assumption here is that we want to optimize for the 90%, and its ok if the older 10% take a little longer to retrieve.
With this in mind, my research seems to strongly point in the direction of using a cache for the 90%, and then to also store the posts in another longer-term persistent system. So my idea thus far is to use Redis for the cache. The advantages is that Redis is very fast, and also it has built in pub/sub which would be perfect for publishing posts to many people. And then I was considering using MongoDB as a more permanent data store to store the same posts which will be accessed as they expire off of Redis.
Questions:
1. Does this architecture hold water? Is there a better way to do this?
2. Regarding the mechanism for storing posts in both the Redis & MongoDB, I was thinking about having the app do 2 writes: 1st - write to Redis, it then is immediately available for the subscribers. 2nd - after successfully storing to Redis, write to MongoDB immediately. Is this the best way to do it? Should I instead have Redis push the expired posts to MongoDB itself? I thought about this, but I couldn't find much information on pushing to MongoDB from Redis directly.
It is actually sensible to associate Redis and MongoDB: they are good team players. You will find more information here:
MongoDB with redis
One critical point is the resiliency level you need. Both Redis and MongoDB can be configured to achieve an acceptable level of resiliency, and these considerations should be discussed at design time. Also, it may put constraint on the deployment options: if you want master/slave replication for both Redis and MongoDB you need at least 4 boxes (Redis and MongoDB should not be deployed on the same machine).
Now, it may be a bit simpler to keep Redis for queuing, pub/sub, etc ... and store the user data in MongoDB only. Rationale is you do not have to design similar data access paths (the difficult part of this job) for two stores featuring different paradigms. Also, MongoDB has built-in horizontal scalability (replica sets, auto-sharding, etc ...) while Redis has only do-it-yourself scalability.
Regarding the second question, writing to both stores would be the easiest way to do it. There is no built-in feature to replicate Redis activity to MongoDB. Designing a daemon listening to a Redis queue (where activity would be posted) and writing to MongoDB is not that hard though.
I've written an application with a CouchDB backend. I have invested a lot of time into CouchDB and so I'm reluctant to move everything over to a different NoSQL database (like Redis).
The problem is that I now need to implement a rate limiting (based on IP address) feature.
There are plenty of examples on how good Redis is for this kind of task, however because I don't want to drop CouchDB for other tasks this means I would essentially be running (and supporting) two databases (1 for most data, 1 for rate limiting) and so...
Is running CouchDB in tandem with Redis unheard of?
Is CouchDB itself suitable for handling rate limiting itself?
Is running CouchDB in tandem with Redis unheard of?
Redis is commonly used in complement with other storage solutions (MySQL, PostgreSQL, MongoDB, CouchDB, etc ...). Like many other NoSQL solutions, Redis is not adapted to all kind of workloads or situations. The authors of Redis are pragmatic and open people, and they routinely suggest to use other solutions rather than Redis, when they are more adapted to the situation.
Redis is therefore a good team player, and it is generally easy to integrate in an existing infrastructure.
Here is an example of usage of Redis with CouchDB.
Is CouchDB itself suitable for handling rate limiting itself?
CouchDB has a number of useful features to implement the rate limiting strategy described in Chris O'Hara's article. For instance, it supports bulk operations on several documents (with optional atomicity). A "bucket span" can be stored in a single document. In-place incrementation of counters can be covered by using update handlers.
IMO, the main missing feature would be automatic item expiration (which CouchDB does not provide AFAIK). So you would have to design a clever mechanism to get rid of obsolete data on top of CouchDB.
The main problem is CouchDB is not really designed for this kind of workload: it is a log structured document oriented database. Each time a counter has to be incremented, it would involve JSON unpacking/packing operations, some Javascript code to be executed, and writing a new revision of the whole document in append only files. You can find a good article describing how CouchDB stores its data here.
I suspect a rate limiting strategy implemented on top of CouchDB would not scale very well (too many I/Os, too much CPU consumption, inefficient network protocol). For instance, CouchDB is a RESTful server; I would not feel comfortable to initiate client HTTP operations (REST queries to CouchDB) to rate limit each incoming HTTP query of my system.
Redis is much more adapted to this kind of workload (fast, in-memory, no I/O, efficient client protocol, no JSON parsing/formatting, incrementations are native atomic operations, etc ...)
You can do rate limiting with Memcached - it has a nice counter increment command as you mention, plus obsolete data is automatically purged from the cache in due course, so it has all the benefits of Redis for this application without the annoying duplication of capability (and complexity) that running Redis on top of CouchDB would bring.
http://simonwillison.net/2009/jan/7/ratelimitcache/
You could add memcached to your own setup easily enough or you could investigate CouchBase whose current server product integrates a CouchDB derived database with Memcached compatibility baked in:
http://www.couchbase.com/memcached
Personally I dislike the way Couchbase forked from CouchDB, but for your application it might be a perfect fit.