Creating a Node.js server, on Heroku.
I only need to store data for around 30min-1 hr at a time, then, I can release the data.
Heroku recommends not using SQLite because it is an on memory database, and will get reset every time the server goes to sleep.
Since I don't need the data for very long, is it okay if I go through with this?
If you're curious, the project is to track timestamps of summoner spells that occur in a game of League of Legends. Since League of Legends games only last around 30min-1hr, I don't need the hold data for very long.
SQLite is not an in-memory database. It stores the tables in a database file. In-memory tables are just an optional feature. The advantage of SQLite is that no setup or administration is needed, since it is embedded in your application. It has a small footprint, is very well tested and is actively developed. SQLite is the right choice.
Maybe the confusion comes from misunderstanding "embedded in your application". It is the database engine code that is embedded, not the database and its tables.
Related
I am building a Node.js application which uses a few global variables to track data such as online users and statuses, information about other servers, and ongoing events, but having this information be lost in the event of server restart/crash is not ideal.
As these things are frequently read & modified, I figure it would not be a good idea to put that extra strain on my existing MySQL database. I have looked into Redis but unfortunately my application is hosted on a Windows server so I would have to use an old unsupported version of it which isn't ideal.
I'm currently considering setting up a NoSQL database such as MongoDB, but I'm not sure if this is an efficient solution and if it would be too much on my relatively weak server to have an application and 2 different databases running.
What would be the best solution for persistent storage of data that needs to be frequently accessed and updated by an application?
Making my comments into an answer...
If it's a reasonable amount of data, you can just write JSON to a single data file. No database required. Just overwrite the file with a new block of JSON to save the new state. This is very fast, efficient and simple. I've used this before as a quick and easy way to regularly save snapshots of state that you want to be able to reload if your server restarts. Read the state into memory upon server start, then use it from memory, then regularly save a new snapshot to disk however often your application desires.
If some data changes a lot and some data doesn't change very much, you can break the data into multiple files so you're writing less data on the more frequent interval. Obviously, there is a threshold of amount of data or frequency of writes or complexity of data access where a database would be warranted, but you should at least consider the simpler option first and only add a new database when you think you really need it.
If you cluster your servers in the future, that would speak to a multi-user database (one with appropriate concurrency management features) to be your master keeper of state, but you're going to have other design issues to work through if you're trying to share multi-user state (like online status) across all clustered servers as you can no longer keep that in memory for any server unless all state changes are broadcast to all servers so they can update their in-memory copy of the data or unless you make users sticky to a particular server (which complicates load balancing in clustering). That does somewhat call for a redis-like central store that all clustered servers can access.
I am making a website with nodejs and mongodb which records the username of the currently online users. I wonder whether it would be better practice to store this in an array created during the website's runtime or should I store it in a database?
I agree with explorer. Generally, when an app is in production, you store information in some sort of database. This insures that your application uses the least possible RAM, assuming that you write decent code. Also, if your application crashes for some unforeseen reason, you can recover quickly and your data isn't lost.
i got a question about handling Big Data in offline application in Electron.
i made my application which handles a bunch of data (actually a 250MB Sqlite Database)
i'm handling the sqlite data with sql.js library, because the sqlite npm library doesent't pass the build phase (even with some tricks found online)
what i'd like to know is if this is the best solution to handle offline data, with update capability.
my app actually connects to my server, syncs with the online database, and allow the user to work offline.
the sync is made by downloading the whole DB every time, because if i perform a massive INSERT statement (even with TRANSACTION, pragma and everything else) the sql.js library hase some issues with memory, and crashes.
now they're asking me to add more data, so the DB size will grow.
is any other option that i may evaluate?!
i was evaluating everything, from localstorage to PouchDB, but i don't want to bring down everything without good reasons...
any help will be appreciated
I am building a simple game in node.js using socket.io. My web experience with node.js has typically involved saving everything to a relational database and keeping nothing in memory. I set up a relational database for the state of a game. I am using sqlite3 for development and I might use something like PostgreSQL or MySQL for production.
My concern is that, every time an event is emitted from the socket the whole game-state is loaded into memory from the server. I feel that in practice this will be less efficient than just keeping all of the game-state data in memory. Events will probably be emitted every 5 seconds or so during a game. All of the game data is temporary, none of it will be needed after the game is over. A game-state consists of a set of about 120 groups of small strings and integers (about 10 per group but subject to change).
Is it good practice to keep this type of data in memory?
If not, should I stick with relational databases or switch to a third option like a file-based storage structure?
Should I not load the whole gamestate in for every event even though that will lead to a lot more read/writes (at least triple)?
I would not keep this data in the memory of your NodeJS application. Its best avoid storing state in your app server. If you really need faster read access than sql provides consider using a cache like Redis or Memcached as a layer between your app and db.
All that being said its best not to prematurely optimize you code. Most SQL engines have their own form of cacheing, and optimizing your sql queries is a better place to start if your experiencing performance issues. Postgresql Query Optimization
But don't worry about it until its an actual problem (because most likely it never will be).
Sounds like relational, SQL type database is a huge overhead for your specifics. Do you have idea how big your data is and how many users you'd like to handle? Then you could compare it with your's server ability. If result is negative (couldn't handle with mem) - then i'd go with some quick nosql, like mongo. For yours example its sounds like the best choice. It'll be faster to get data for single session, easier to dump, more elastic in structure.
So here's my deal.
I'm using node on the express framework. The website i'm working on grabs scraped data and stores it for each user on the website. That data can then be displayed on the users page whenever they want to access it, so the data will be scraped, put in a database or storage, whatever i decide the best way to do it is, and then pulled back out for the user.
I'm trying to figure out what the best database setup would be. There will potentially be large amounts of data per user, especially over long periods of time. I've read some stuff about using redis to cache some data like the user login info and that basic stuff, and then using mongodb for the big data. But I don't know, i'm new to database stuff so I am open to some new teachings and some ideas from the masters.
What would you guys suggest I do? I want it to be fast and be able to handle multiple queries at the same time, but really, I have no idea what i'm talking about, so please help me.
What would you guys suggest I do?
This really depends on the nature of your data, how you model your domain and how you want to persist it. I would first try to figure out the basic model and based on that choose the most suitable database system. Don't jump at quick conclusions around caching with redis when you don't even know if you will need it in the first place.
Suggestion might also depend on how much time you want to spend with database layer of your application. Some database systems provide more functionality than others depending on their concepts. If you are a beginner choose a single mainstream solution that is well documented with established community like MongoDB or MySQL that will cover all your needs from the beginning so that you won't end up managing multitude of systems.