I build an API, which will send data to another API when has been collect 10 hashes. The client sends 1 hash per hour.
For example:
The client POST hash to API
API need to store it somewhere until the hashes number becomes to 10
When the number of hashes becomes 10 need to send data to another API and start from 0 again
My question related to the 2nd point. I can store the hashes in the array, the problem is that the data will be lost when the server will be shut down suddenly.
This is the only data which I need to store in API, so I don't want to use DBS.
By the way, it's my first time of developing API, so will be glad to your help.
Thanks in advance.
Sorry but your only options of storing data are either memory or disk.
If you store data in variables, you're using memory. It is fast and instant but it's not durable as you already said.
If you store data in database, you're using disk storage. It is slower but it is durable.
If you need durability, then database is your only option. Or maybe if you don't want to store the data in your machine, you could use cloud database such as firebase database.
Maybe your problem will be solved with Redis.
I had one feature where I needed to use some user's pieces of information on the server side in runtime and it could not be persisted at the database.
So, I used this.
In simple words, the redis will save the information in your cache and you can retrieve when you need.
There's no disk use and are more stable than a hand made memory control.
I hope this helps you.
Related
I'm using node js along with mongo db. I need help for something related to caching. I have certain data like trending posts which will be updated for every 15 mins. But for a given instance of time period all the users making the api call for the trending posts data will get same response. Hence is there any way to save the cache data from db and update it periodically for 15 mins.
Note: I doubt if storing the data in json format in a file in server will be helpful? Is it prone to attacks or malicious usages? This data contains some confidential information too. So I guess that is not a good idea to store it in files. So is there anyother methods???
As mentioned by Kedar, this is where you would use some type of distributed cache like Redis.
There are NPM modules like mongodb-redis-cache
(https://www.npmjs.com/package/mongodb-redis-cache)
They syntax is quite simple, just append it onto the select. (it does require a redis server though, if you are not familiar with Redis or setting one up, I would look at a more managed solution then like on Digital ocean etc )
The idea here is that each call to Mongo will first check if there is a 'cached' key/value in redis, if not it runs the query, otherwise it will just pull from Redis.
I'm looking for a way to encrypt the entire DB and keep the ability to search for data although it's encrypted.
I have seen a lot of questions regarding encryption of at rest data in Mongo, but none of it got an answer that can help one complete a full flow for their application.
I hope to present here my findings and get feedback and more ideas (I still have some questions).
Encryption options:
1.mongoose-encryption.
Complete solution! Can encrypt all fo the db with minimal work for you!.
2. Procona mongodb - I didn't had a chance to test it, I've spent hours trying to install and get it to run, without luck (this is probably just me though..).
3. Create get and send methods to encrypt and decrypt your data in the Module level.
My requirements for at rest data encryption are:
Application layer does not need to be involved in the encryption- decryption process. Should be like we don't even have the data encrypted (for the most part).
We can perform search and lookups on encrypted data.
I don't know how to do that but hopefully search for partial words and phrases in encrypted text fields.
Of course that all data is encrypted expect for Object IDs.
My approach:
I want to try and use mongoose-encryption to use all the benefits of this amazing plugin.
I also want to add to the schema the Hash of the Real value in the encrypted field so I could preform find operations on encrypted field.
The problem:
I can't seem to find the correct mongoose Hook to temper with the non-encrypted data before mongoose-encryptions hides it. So I can't generate my Hash.
This doesn't work:
Users.pre('save', () => {
this.hashedName = hash(this.name)
console.log(":(")
});
Also as mentioned above, searching for partials and phrases in encrypted data.
With my approach we could find someone named "Danielle" but we can't search in Hash for users with a name that starts with "Dani".
Please give me your opinion as well for my approach. I know that this is a topic without easy to find solutions.
If you want to encrypt the data on disk, encrypt the entire disk and encrypt the swap. If someone gets a copy of the database (e.g. you forgot to put auth on the database and someone connects to the database and dumps the data) the plaintext is exposed.
If you want the database to store encrypted data only, use client side encryption. This requires key management on the client side but makes it so that someone dumping your database doesn't get the plaintext.
I'm writing an auth module that contains several functions so my server can authenticate with an oAuth2 system using client_credentials. In the module I want to cache / save the credentials since they don't expire for some time (I'll refresh as needed).
Whats the best way to store the credentials?
Should I just create a var at the top of my node module? Should I create a class and instantiate it (const auth = new MyClass()) where my makes subsequent API calls (with the Bearer token)?
Creating a var variable means to store data in RAM, which is not a comprehensive approach.
I suggest you look at Redis, which was developed mostly for your purposes.
Redis is used as a database and for cache, since it’s super fast because the data is stored “in-memory” contrary to other databases in which the data is usually stored “on-disk.”
Moreover, there some thoughts about the fact that Node.js is less efficient at storing data than Redis. This article will clear more about my point.
So, in general, I guess using Redis will bring you more advantages in storing credentials.
Why use a DB like Redis over writing your own server side key: value JSON - for example?
For objectivity, let's measure value by the following metrics:
Flexibility
Speed
Reliability
Thinking they’re pretty case by case as is. I could structure data, through objects, sets, etc to read/write/query with more flexibility. I’m trying to learn why the feature set of these services are rich enough to warrant plugging them in per operation type i.e. user sessions, shopping cart, recommendations, product catalog, user activity logs, etc. I know, at least, some leverage memory; so, the possibility of them being more performant is nice. But, I have no concrete data yet.
Why not build your own JSON server? For the same reason, you don't write your own operating system. It is always better to use the components which are time tested (even better if they are free).
However, there are other servers to store JSON docs in the market like CouchDB, MongoDB etc. Redis doesn't have native support for JSON, it will store JSON as serialized string.
If you need really fast read/write on complete JSON strings use Redis (fast memory based data structure cache), if you need to query parts of JSON use others JSON stores like Mongo DB etc.
You would need to be more specific about your use-case but even so they would probably close it like this one https://softwareengineering.stackexchange.com/questions/273127/best-strategy-for-storing-static-data-json-file-vs-db-mongo-vs-redis
I want to create a database which stores the user data for a temporary period of time (i.e) Until the user logout.
Once the user logged out, I want all the details they are provided to get deleted.
My question is, can we use mongoDB for temporary storage. Until the user logout!
Or is there any other solution for this?
My answer is yes, you can use mongoDB for temporary storage. You can even set a TTL for the document so it will expire after an amount of time.
Here is the link for the documentation: https://docs.mongodb.com/manual/tutorial/expire-data/
Another solution would be to store that kind of information in a database that persists only in memory like Redis. That way you could gain some performance even.
You can use any database to do that. Just delete the data when the user logs out.
Redis may be a good fit for that because it stores the data in RAM, see:
http://redis.io/
But really any database can do it. You will just have to remove the data on logout.
Since you tagged your question with "mean-stack" then I assume that you're using Express. There are some modules that can help you with what you're trying to do. See the Compatible Session Stores in the express-session documentation.
Some of the more relevant modules from that list:
connect-mongo (for Mongo, as you asked)
connect-mongodb-session (another one for Mongo)
connect-redis (for Redis, which is well suited for that use case)
connect-sqlite3 (for SQLite, an embedded RDBMS that you don't have to install)
session-file-store (for storing session data in files)
or you can use the default in-memory session storage for testing (not suitable for production)