I have a Queue which is backed-up by Redis, with multiple node connections to that Redis server, and I need to make sure that it won't get the same key twice, so it will never run the same task more than once.
i'm using node-redis for this task:
client.set("some_key", data);
client.get("some_key", function (err, data) {
//..
});
How can I make sure when getting that key that no other node process will be able to get it too? if I will set it as expired only after getting the value it won't be enough when 2 process will try to get the same value at the same time.
I don't think there's any other way than to wrap it in a MULTI
MULTI
GET some_key
DEL some_key
EXEC
So using node-redis, something like
client.multi()
.get("some_key", data).del("some_key").
.exec(function (err, replies) {});
You can ensure atomicity of operations in Redis with MULTI/EXEC blocks and/or Lua scripting. In your case, you can do the GET followed by a DEL immediately afterward using either of the above approaches to ensure a single read.
I'd use a list with pop operations instead of a key. In particular you add new items to, say, the right, (via rpush) and pop them off the left ( via LPOP).
If you are storing a bunch if data in the key currently, such as a hash, use a unique identifier as the key of the hash and add that ID to the list instead. That way you get the get-and-remove capability in a simple fashion without needing transactions and multiple commands with the ability to store job data as well.
When the job succeeds delete the data key, if it fails you can re-queue it.
Related
I want to store all objects of a class in redis cache and be able to retrive them, as I understand hashmaps are used for storing objects, but they are require a different key to be saved. So I can't save them all under key e.g. "items" and retrieve them by that key. Only way I can do it is something like this:
items.forEach(item => {
redis.hmset(`item${item.id}`, item);
}
But this feels wrong and I have to have a for loop again when I want to get this data. Is there a better solution?
Also there is a problem of associated objects, I can't find anywhere how they are stored and used in redis.
As I understand, you want to save different keys with same prefix
You can use mset to store them
For retrieving the data you use the mget
with your keys as params
In case you still want to use the hmset
Use pipline in the loop
So the call to redis will be only one with the sync action
whats up! I am using redis with express and nodejs. when looking how to insert or retrieve data from redis, I saw two ways, one like this:
req.session.surname = 'toto'
console.log(req.session.surname)
and the other way is looking like this:
client.set('surname', 'toto')
client.get('surname', (err, data) => {
console.log(data)
})
Is there a difference between these two methods ?
Thanks for any help. Cheers !
There is no major difference between these two methods. In the first one you could use any other session store like mongo-db if you need more reliability (since redis is memcached there is a possibility to lose the data as data will be stored in RAM only). Second one is just set and get the desired value to the key for general usage where there is no 100% reliability is needed. Also you will face issue when processing request concurrently as there is no concurrency control for mem-cached DB like redis.
If you need 100% reliability (if you don't want to lose data easily) you can go with mongo-db. In mongo-db data will be stored persistently also we can control concurrency as well.
I need to design a node.js app in this scenario:
User requests to /page with a cookie which has a unique token
Node.js creates a record with this token on redis(?) with a 5 mins TTL. tokens has n type. So redis(?) should store tokens with types.
If user comes back again before record expire, record's TTL reset to 5 mins again.
If user doesn't come back again and record expires, a function is triggered.
And at last, I also need the count of records belongs to a specific type (i.e. type 27).
What would be the best way to achieve this problem? Is redis right choice for it? How can I count tokens and trigger 5. step if I use redis?
Using https://github.com/Automattic/kue or something like that would be the nice choice?
Edit:
It seems possible duplicate with Event on key expire
. In general, I ask, is redis suitable for this problem? If you think it is, could you please give an example pattern I should follow. If not, what are any other solutions maybe even without redis or 3rd party tool.
Also as I said at the 5. item in the list above, redis seems not suitable at querying keys with a pattern as described here:https://redis.io/commands/keys
. Because there seems no way to put expiration on an individual member of a set (TTL for a set member) , I cannot put tokens in a type set or hash . Thus I need to query keys in order to get count of a group (type) and this seems not efficient as described at the link above.
Even if https://redis.io/topics/notifications solves the problem at the 4. item in the list above, I also ask for "Which pattern/algorithm should I go and how" . I don't want you to code for me, but just need a spark. A tutorial, guide, video etc.. would be great. Thanks!
You have many options in order to achieve what you want, but I'm going to provide one here, which is a very efficient one in terms of retrieving speed, but it requires some more space. It all depends on your use-case, obviously.
Say token is 12345, type is A
When adding a token (we'll do it with a transaction):
MULTI
SETEX tokens:12345 300 ""
SET type:12345 "A"
INCR types:A
EXEC
When the key expires (after 300 seconds, or whenever Redis sees it as expired) we get notified using Keyspace notifications (https://redis.io/topics/notifications) listening for the EXPIRED event:
PSUBSCRIBE __keyspace#0__:expired
When that subscription receives a message, in your code you'd need to:
MULTI
GET type:12345 # Returns A
DEL type:12345
DECR types:A
EXEC
In order to get the elements of a specific type:
GET types:A
Any NodeJS Redis client would work just fine for this.
I want to make clear this is only one of the multiple options you have; one of the Redis advantages is it's flexibility.
so I have two scripts : One that will pass data to redis and another one that will get the data from redis.
The script that will get the data from redis is written with node.js and the script that will send the data to redis is written in ruby.
I am looking for the rnode.js script to get the data from redis as soon as the ruby script sends it.
I was thinking about a continuous monitoring from node.js to check if there is an update into redis and as soon as there is an update , node.js grab the data. If you have a better way to do this , I'll consider it as well
I would like help clue about how should I write the node.js script to continuously monitor redis and grab any new data.
Thanks
One way of doing it is to use Redis' lists as queues.
Assuming that the Ruby script stores the new "data" under a key called data:1, have it also RPUSH the key's name to a list called new_data for example. This list is essentially a queue of all new data.
Now, have your 'rnode.js' script do a blocking left pop (BLPOP) on the new_data list. Whenever new data arrives, the script will unblock and you'll be able to process the news. Once finished, return to blocking pop.
I'm new in Node.js and Cloud Functions for Firebase, I'll try to be specific for my question.
I have a firebase-database with objects including a "score" field. I want the data to be retrieved based on that, and that can be done easily in client side.
The issue is that, if the database gets to grow big, I'm worried that either it will take too long to return and/or will consume a lot of resources. That's why I was thinking of a http service using Cloud Functions to store a cache with the top N objects that will be updating itself when the score of any objects change with a listener.
Then, client side just has to call something like https://myexampleprojectroute/givemethetoplevels to receive a Json with the top N levels.
Is it reasonable? If so, how can I approach that? Which structures do I need to use this cache, and how to return them in json format via http?
At the moment I'll keep doing it client side but I'd really like to have that both for performance and learning purpose.
Thanks in advance.
EDIT:
In the end I did not implement the optimization. The reason why is, first, that the firebase database does not contain a "child count" so I didn't find a way with my newbie javascript knowledge to implement that. Second, and most important, is that I'm pretty sure it won't scale up to millions, having at most 10K entries, and firebase has rules for sorted reading optimization. For more information please check out this link.
Also, I'll post a simple code snippet to retrieve data from your database via http request using cloud-functions in case someone is looking for it. Hope this helps!
// Simple Test function to retrieve a json object from the DB
// Warning: No security methods are being used such authentication, request methods, etc
exports.request_all_levels = functions.https.onRequest((req, res) => {
const ref = admin.database().ref('CustomLevels');
ref.once('value').then(function(snapshot) {
res.status(200).send(JSON.stringify(snapshot.val()));
});
});
You're duplicating data upon writes, to gain better read performance. That's a completely reasonable approach. In fact, it is so common in NoSQL databases to keep such derived data structures that it even has a name: denormalization.
A few things to keep in mind:
While Cloud Functions run in a more predictable environment than the average client, the resources are still limited. So reading a huge list of items to determine the latest 10 items, is still a suboptimal approach. For simple operations, you'll want to keep the derived data structure up to date for every write operation.
So if you have a "latest 10" and a new item comes in, you remove the oldest item and add the new one. With this approach you have at most 11 items to consider, compared to having your Cloud Function query the list of items for the latest 10 upon every write, which is a O(something-with-n) operation.
Same for an averaging operation: you'll find a moving average to be most performant, because it doesn't require any of the previous data.