I want to save the "state" of my application each time it is changed, and load it each time the application is booted up.
The "state" will be a simple object with a handful of variables in it, the idea is to JSON.stringify it to a file, and JSON.parse it when needed.
From what I understand, this cannot be done using Node's fs, since files on Heroku are not permanent.
I cannot use S3 either, because it's not free (free plan only lasts a year), and this is a hobby project of mine - I am not willing to pay for it.
Another recurring suggestion, is to use some sort of a database, but I think that is a waste of time, since I will only be dealing with one very small file.
Essentially, my question is, how can I achieve something that is closest to this?:
WRITE("filename.txt",JSON.stringify(x));
x=JSON.parse(READ("filename.txt"));
(P.S: I've read somewhere, can't seem to remember where, that Heroku gives free 100MB (Which would be way more than enough). What is that? Does it have anything to do with my code?)
I can think of a few ways to do this for free. They all pretty much boil down to "What free service allows me to read/write arbitrary file content, and access via an API?"…
Do you use or already pay for Dropbox (or something similar?). If so you could you the Dropbox API for Node.js to save/load your application state.
You could use the Github Gist API and just update the same Gist over and over.
Otherwise, you mentioned databases. Sure, a database would be overkill tech-wise, but given your constraints (and the fact that you can get a small db for free on Heroku), and how much overhead implementing one of the aforementioned APIs would be, it might be the best option.
Hope this helps.
Related
I want to build an API that the user can download files, but there is one problem, the files can be huge, something like more than 100GB sometimes. I'm thinking about making the API using nodejs, but I don't know if it gonna be a great idea to make the file download features using node, some users may spend more than a day to make a download, node is single thread and I'm afraid that can hold to much time and make the others request slower, or worse, block them.
I gonna use clouding computing to host this API, I gonna start to study serverless hosts to see if it worths it in my case. Do you have any idea what I should use to make the download feature? There is an open-source code to use as an example?
Very closely related: How to protect strings without SecureString?
Also closely related: When would I need a SecureString in .NET?
Extremely closely related (OP there is trying to achieve something very similar): C# & WPF - Using SecureString for a client-side HTTP API password
The .NET Framework has class called SecureString. However, even Microsoft no longer recommends its use for new development. According to the first linked Q&A, at least one reason for that is that the string will be in memory in plaintext anyway for at least some amount of time (even if it's a very short amount of time). At least one answer also extended the argument that, if they have access to the server's memory anyway, in practice security's probably shot anyway, so it won't help you. (The second linked Q&A implies that there was even discussion of dropping this from .NET Core entirely).
That being said, Microsoft's documentation on SecureString does not recommend a replacement, and the consensus on the linked Q&A seems to be that that kind of a measure wouldn't be all that useful anyway.
My application, which is an ASP.NET Core application, makes extensive use of API Calls to an external vendor using the HttpClient class. The generally-recommended best practice for HttpClient is to use a single instance rather than creating a new instance for each call.
However, our vendor requires that all API Calls include our API Key as a header with a specific name. I currently store the key securely, retrieve it in Startup.cs, and add it to our HttpClient instance's headers.
Unfortunately, this means that my API Key will be kept in plaintext in memory for the entire lifecycle of the application. I find this especially troubling for a web application on a server; even though the server is maintained by corporate IT, I've always been taught to treat even corporate networks as semi-hostile environments and not to rely purely on corporate firewalls for application security in such cases.
Does Microsoft have a recommended best practice for cases like this? Is this a potential exception to their recommendation against using SecureString? (Exactly how that would work is a separate question). Or is the answer on the other Q&A really correct in saying that I shouldn't be worried about plaintext strings living in memory like this?
Note: Depending on responses to this question, I may post a follow-up question about whether it's even possible to use something like SecureString as part of HttpClient headers. Or would I have to do something tricky like populate the header right before using it and then remove it from memory right afterwards? (That would create an absolute nightmare for concurrent calls though). If people think that I should do something like this, I would be glad to create a new question for that.
You are being WAY too paranoid.
Firstly, if a hacker gets root access to your web server, you have WAY bigger problems than your super-secret web app credentials being stolen. Way, way, way bigger problems. Once the hackers are on your side of the airtight hatchway, it is game over.
Secondly, once your infosec team detects the intrusion (if they don't, again, you've got WAY bigger problems) they're going to tell you and the first thing you're going to do is change every key and password you know of.
Thirdly, if a hacker does get root access to your webserver, their first thought isn't going to be "let's take a memory dump for later analysis". A dumpfile is rather large (will take time to transfer over the wire, and the network traffic might well be noticed) and (at least on Windows) hangs the process until it's complete (so you'd notice your web app was unresponsive) - both of which are likely to raise some red flags.
No, hackers are there to grab as much valuable information in the least amount of time, because they know their access could be discovered at any second. So they're going to go for the low-hanging fruit first - usernames and passwords. Then they'll move on to trying to find out what's connected to that server, and since your DB credentials are likely in a config file on that server, they will almost certainly switch their attentions to that far more interesting target.
So all things considered, your API key is pretty darn unlikely to be compromised - and even if it is, it won't be because of something you did or didn't do. There are far more productive ways of focusing your time than trying to secure something that already is (or should be) incredibly secure. And, at the end of the day, no matter how many layers of security you put in place... that API or SSL key is going to be raw, in memory, at some stage.
I've just started trying to use NodeJS and socket.io to create a simple multiplayer online game (similar idea to online chess). I apologise if the answer to my question is really obvious because I have tried googling around, but I think I am missing some key bit of understanding.
Basically, I need to store a few things on the server while the application is running. For example:
I need to store which socket connections are hosts, and which are players.
I need to store the current state of each game (e.g. in the case of chess, where the pieces are and whose turn it is)
It would also be nice to be able to store all the socket.io "rooms".
Feel free to answer the question at this point, information below is for extra reference.
There are a few things that I have tried or seen online:
When I google something with "persistence", I get results based on saving to a database or something, I don't think this is what I want.
I have tried just adding variables at the top of the NodeJS file, like I would with global variables in an ordinary JS file. This seems to work, but just feels wrong to me, if someone could explain how this works it would be great.
I have also seen things called session variables, I think this might be what I want.
I have seen applications that do this by just passing the information back and forward between to client and server, but I would prefer that the client couldn't just edit the information to "hack" to game.
Any help or explanation appreciated.
Nothing wrong with saving to a database. If your server crashes and restarts a few seconds later, you don't really want everyone's data to just be obliterated. I think you're thinking about it in the way that databases are always long-term and slow. But really, there are DB technologies great for this type of thing, and oft used with socket.io.
The one I'd probably opt for is Redis, which is super fast and stores data in-memory. This means that it's not constantly writing to disk, and it's a bit of a halfway house between having full persistent storage like with MySQL, and the slightly dodgy method of just keeping it in Node memory via variables.
When reddit created "Place", that massive multiplayer drawing with a tonne of concurrent users, they used Redis and Cassandra together. You can read a bit about it here.
I'm new in cooperative web developing and I've recently got a new colleague to work with in my node.js based web app. We work remotely and therefore we need some environment to manage our files.
We work in different areas, but from time to time, we make tiny changes in each others codes.
My need is some file manager (maybe online, maybe a node.js module) to work safely, prevent overwrites and keep track of changes.
If it's free, it's the best. If it has a small price, it's good and affordable, but if it comes with a heavier price tag, i might reconsider that option unfortunately.
(i also gladly accept any idea about cooperative web developing)
As #jfriend00 pointed out, GitHub is free and quite useful for collaboration and, as the name implies, it is based on git which is the defacto standard for source code management these days.
I'm building a website that will be an open-source, user-contributed content kind of thing, and I think if developers had access to nightly production SQL dumps, they'd be more likely to check out the code from github and play with it.
In line with that idea, I'm considering either:
Not collecting private user information at all, using open-id for accounts and making heavy use of memcache for things like session authentication.
Anonymizing sensitive data before publishing
Sometimes I get carried away with "wouldn't it be cool if...?" ideas, so I'm hoping for a sanity check here. Any obvious flaws in either approach? Is this a sane idea?
Speaking generally, I think you should do both. Any private data you collect is simply a liability for you, and not just because you intend to publish your databases. The less you can collect, the better.
By the same token, however, you probably realize that it is not just IDs and passwords which are sensitive. Remember the AOL search data leak? Or the Netflix database publication? Even without having IDs, people managed to figure out the real identities of some of the accounts, simply by piecing together trails of user behavior, and corresponding that with data from other places. Some people are embarrassed by their search histories and their movie rentals. Go figure.
Therefore, I think the general rule should be to collect as little as possible, and anonymize what is left. Even if you don't store the identity of the person corresponding to a certain account, you may want to scramble what the various logins did.
On the other hand, there some cases where you simply don't care about this kind of privacy. In Wikipedia, for example, pretty much everything you can do on the site is public anyway. At least, everything which gets recorded in the database. If the information is already available through the API, there is no point in hiding it in a database download.
In addition to collecting less data and anonymizing the data you do collect, you could add a bit/flag for the users to select whether their data is included or not. You could make it a CC license flag to give users the warm'n'fuzzies while filling your need.
Sounds like a pretty good idea. The one thing you have to be careful with though is security, since hackers will know the exact schema of your DB. Although this isn't impossible to deal with, just look at most open source projects. But you will need to put a little extra emphasis on security since say a potential SQL injection is now made much easier.
Another thing is to make sure doubly that the sensitive data is anonymized. Also, some people may (wrongly) try and claim their copyrights on user submitted content is being violated, so you may want to specify a CC license or something just to make everything extra clear and prevent future headaches (even if you're right anyway).