I'm not quite sure if I'm overthinking this, or just approaching it the wrong way in node.
I want to have an object (in this specific case it will be a component tree) in memory that will be accessible to all sessions. I'd also like to have it update on some kind of "heartbeat" separate from user sessions.
To have a object that is available to all sessions you need some storage outside the node.js process as the comments already mentioned. The natural choice would be redis because it is fast and also "speaks" JavaScript. This is the best way to share the data between processes.
To update the local copy of the data in the storage in a "heartbeat" like interval you can use setInterval.
Related
Why should you use Redis to optimize your NodeJS application?
Why POST, PUT and DELETE methods should never be cached?
How is the caching process?
Why do we cache?
Things to install to use Redis on NodeJS?
What is an example of an app that uses the Redis implementation?
Is there any alternative or better than Redis?
Is it too hard to implement Redis in NodeJS?
What happens when we don’t use Redis?
Can we use Redis in any OS?
According to several sources that i have been searched.
By using Redis we can use cache database that gives clients faster data retrieval.
a. The POST method itself is semantically meant to post something to a resource. POST cannot be cached because if you do something once vs twice vs three times, then you are altering the server's resource each time. Each request matters and should be delivered to the server.
b. The PUT method itself is semantically meant to put or create a resource. It is an idempotent operation, but it won't be used for caching because a DELETE could have occurred in the meantime.
c. The DELETE method itself is semantically meant to delete a resource. It is an idempotent operation, but it won't be used for caching because a PUT could have occurred in the meantime.
We can simplify the method like this :
a. client request data X with ID "id1".
b. the system will check the data X in cache database on RAM.
c. if the data X available in cache database, clients will retrieve the data from cache database in RAM.
d. if data unavailable in cache database, the system will retrieve the data from API and then deliver it to clients also save it on cache database at the same time.
To shorten the data retrieval time.
Redis in npm.
Twitter, GitHub, Weibo, Pinterest, Snapchat.
Memcached, MongoDB, RabbitMQ, Hazelcast, and Cassandra are the most popular alternatives and competitors to Redis.
The redis community is quite large, you can see lots of tutorials and manuals. You should be fine
No cache , slower speed to query data and slowing performance
Redis works in most POSIX systems like Linux, *BSD, and OS X, without external dependencies, but there is no official support for Windows builds.
According to many sources,
It'll gives clients faster retrieval of similar/repeated data. Therefore it's called cached database.
Because, Commands (POST,PUT,DELETE) may include many variable, thus differ to each client. Also, not worth the cache. you might want to read more about CQRS.
One of the many methods, in oversimplified terms:
a. client request certain data A with request ID req-id-1.
b. cache will be stored in high speed memory (RAM).
c. if another client request data with ID req-id-1, instead of reading from slower drives, it'll deliver from cache in RAM.
d. if data A is updated, cache with req-id-1 will be deleted. and repeats to step a.
same as answer 1.
redis or ioredis in npm, and a redis process running.
Mostly app/site with a lot of GET request such as news portal. If it's well known, high probability it implements redis.
this is opinionated question. here's a list of redis-like DB,
as long as you read the manual/tutorial, it should be fine.
no cache, thus, saving ram but slower query.
works in most POSIX systems
I'd like to store some info. in a node.js array variable (to be a local cache) that my middleware would check before making a database query.
I know that I can do this w/redis and it's generally the preferred method b/c redis offers snapshots for persistence and is quite performant, but I can't imagine anything being more performant than a variable stored in-memory.
Every time someone brings up this topic, however, folks say "memory leaks" make this a bad idea. But why? Why is node.js bad at managing server-side vars?
Is there a preferred method (outside of an external k/v db store) of managing a server-side array/cache through node.js?
The problem with using a node variable as storage is that by using it you have made your application unable to scale. Consider a large application which serves thousands of requests per second, and cannot be run on a single machine. If you spin up a second node process, it has different values for your node storage variable.
Let's say a user making an API call to your application hits machine 1, and stores a session variable. They make a second API call and this time are routed by your load balancer to machine 2. Their session variable is not found and you throw an error.
If you are writing a small application and have no expectations of scaling up in the near term, by all means use a node variable - I've done this myself before for auth tokens on small websites. You can always switch to redis later if you need to. Of course, you need to be aware that if your node process restarts, the contents of your variable will be lost.
I have a forum which contains groups, new groups are created all the time by users, currently I'm using node-cache with ttl to cache groups and it's content (posts, likes and comments).
The server worked great at the begging but the performance decreased when more people start using the app, so I decided to use the node.js Cluster module as the next step to improve performance.
The node-cache will cause a consistency problem, the same group could be cached in two workers, so if one of them changed, the other will not know (unless you do).
The first solution that came to my mind is using redis to store the whole group and it's content with the help of redis datatypes (sets and hash objects), but I don't know how efficient this could be.
The other solution is using redis to map requests to the correct worker, in this case the cached data is distributed randomly in workers, so when a worker receives a request that related to some group, he checks the group owner(the worker that holds this group instance in-memory) in redis and ask him to get the wanted data using node-ipc and then return it to the user.
Is there any problem with the first solution?
The second solution does not provides a fairness (if all the popular groups landed in the same worker), is there a solution for this?
Any suggestions?
Thanks in advance
It's been a while that i am using node for my applications and i was wondering where a global or local variable is stored? (in RAM or CPU cache maybe. guessing RAM. right?) and is it a good idea to store some JSON's that are most of the times static as a global variable and access them right away.
would it be faster than Reading from in-memory database like Redis?
For example let's see i am talking about something like website categories list which is a JSON with some nodes in it.
Most of the times this JSON is constant and even if it gets changed i can refresh the variable with new value because one server app handles all requests.
And when node app starts i can have this initializer function that reads the JSON from in-disk database.
Currently i am using Redis for this situation and when app starts i'm reading this JSON from mySQL and keep it in redis for faster request handling.
But i'm wondering is it a good practice to keep JSON as a global variable and how would it be compared against having it in Redis performance wise?
P.S: I know redis has consistency and keeps value in disk too but i am reading them from mySQL because redis is a caching mechanism for a small part of schema and using initializer gives me a manual sync if needed.
Thanks
I would prefer Redis. Because even if you restart node application data will be there and putting global variables in memory has one disadvantage that at run time if you want them to be changed you are just left with choice of restarting whole application.
Plus while running application you should always query Redis to get data whenever you want.So in future if you want these values to be dynamic it will directly reflect by just changing it in Redis.
You can keep it anywhere you want. You can store them as files and require them while starting your app. I'd prefer this if they do not change.
If you update them, then you can use any database or caching mechanism and read them. It's up to you.
Yes, the variables are stored in memory. They won't persist if the app crashes. So a persistent storage is recommended.
I have a functionality where user post data containing few userid and some data related to those userid and I am saving it into postgresql database. I want to save this returned userid in some object.
I just want to check if userid is present in this object and then only call database. This check happen very frequently so I can not hit db every time just to check is there any data present for that userid.
Problem is, I have multiple nodejs instances running on different server so how can I have a common object.
I know I can use redis/riak for storing key-value on server, but don't want to increase complexity/learning just for a single case.(I have never used redis/riak before.)
Any suggestion ?
If your data is in different node.js processes on different servers, then the ONLY option is to use networking to communicate across servers with some common server to get the value. There are lots of different ways to do that.
Put the value in a database and always read the value from the common database
Designate one of your node.js instances as the master and have all the other node.js instances ask the value is on the master anytime they need it
Synchronize the value to each node.js process using networking so each node.js instance always has a current value in its own process
Use a shared file system (kind of like a poor man's database)
Since you already have a database, you probably want to just store it in the database you already have and query it from there rather than introduce another data store with redis just for this one use. If possible, you can have each process cache the value over some interval of time to improve performance for frequent requests.