below is architecture of my applications.
sensor↔parser app↔database↔application1↔ethernet↔server
application2 and application3 are same level of application1.
database = sqlite3
problem is that too many transaction occured on database system.
parser app and applications are queries whole range of database for checking any differences every second.
so i would like to change architecture or database.
is there any database which has better performance than sqlite3?
or which part do i have to change?
I would switch out sqlite3 in favor of MySQL or PostgreSQL, those database systems are meant to handle multiple clients where as sqlite3 won't be able to do this because everything is stored in a single file. Each (write) access therefore has to block the entire database instead of only a single row of the table in question.
Related
Creating a Node.js server, on Heroku.
I only need to store data for around 30min-1 hr at a time, then, I can release the data.
Heroku recommends not using SQLite because it is an on memory database, and will get reset every time the server goes to sleep.
Since I don't need the data for very long, is it okay if I go through with this?
If you're curious, the project is to track timestamps of summoner spells that occur in a game of League of Legends. Since League of Legends games only last around 30min-1hr, I don't need the hold data for very long.
SQLite is not an in-memory database. It stores the tables in a database file. In-memory tables are just an optional feature. The advantage of SQLite is that no setup or administration is needed, since it is embedded in your application. It has a small footprint, is very well tested and is actively developed. SQLite is the right choice.
Maybe the confusion comes from misunderstanding "embedded in your application". It is the database engine code that is embedded, not the database and its tables.
Here is my situation. I have an extensive REST based API that connects to a MongoDB database using Mongoose. The API is written as a standard "MEAN" stack application.
Currently, when a developer queries the API they're always connecting to the live production database. What I want to do is have an exact duplicate database as a "staging" database, where new data will be added first, vetted over a period of time, and then move to the live database. Then I want developers to be able to query either one simply by modifying their query.
I started looking into this with the Mongoose documentation, and it appears as though the models are tied to the DB connection, and if I want to have multiple connections I also have to have multiple models, one for each connection. This would be a nightmare of WET code and not the path I want to take.
What I want to do is not touch any of my code at all and simply have a switch that changes to the proper database for a given query. So my question is, how can I achieve this? Is it possible? The documentation seems to imply it is not.
Rather than trying to maintain connections two environments in the same code base have you considered setting up stage version of your application? Which database it connects to could be set through an environment variable or some other configuration option.
The developers would still then only have to make a change to query one or the other and you could migrate data from the stage database to production/live database once you have finished your vetting process.
I want to use my sqlite3 database with multiple threads in parallel. I read that using connection pools makes the access threadsafe but I still get errors while inserting data.
(make-thread
#'(lambda()
(dotimes (i 100)
(with-database (db ("/path/to/db")
:database-type :sqlite3 :pool T)
(do-stuff-with db)))))
When using multiple threads in this fashion in this error
While accessing database #
with expression "INSERT INTO ...":
Error 5 / database is locked
Is it even possible to do a multi threaded insert with an sqlite3 database? If yes how?
SQLite does not support concurrency of multiple write transactions. From the SQlite site:
SQLite supports an unlimited number of simultaneous readers, but it will only allow one writer at any instant in time. For many situations, this is not a problem. Writer queue up. Each application does its database work quickly and moves on, and no lock lasts for more than a few dozen milliseconds. But there are some applications that require more concurrency, and those applications may need to seek a different solution.
Cl-sql has been written to give a "unified" interface for the typical client-server relational DBMS, like other "standardized" libraries (e.g. JDBC or ODBC), but SQLite is an "untypical" database management system: in practice it is a library that offers SQL as language to access a simple "database-in-a-file", and a few other functionalities of DBMSs. For instance, it has no real concurrency control (it uses the Operating Systems functions to lock the db file), so it cannot be considered a "real" DBMS, and cl-sql cannot offer nothing more than the functionalities of the underlying system.
So, if you need concurrent insertions into a database, you should use something else, for instance PostgreSQL.
I am using a C++ shell extension DLL which used to read, write data into the SQLite database tables. There another application ( exe) which used to access all the tables.
Sometimes, my dll displaying an exception "The database file is locked" when I try to Delete/Insert/Update to the SQLite Database tables. This is because the other application was accessing the tables at this time.
Is there any way to resolve this issue from my DLL? Can I use the solution as mentioned in the link : "http://stackoverflow.com/questions/6455290/implementing-sqlite3-busy-timeout-in-an-ios-app"
In the current code, I am using CppSQLite3.cpp method execQuery(const char* szSQL) to execute the SQL query.
Please advice.
First of all you should know that SQLite does a Database level locking. When you start a transaction and the other application tries to write something to the same database, then you get Database is locked and SQLite automatically tries executing that same query after sqlite3_busy_timeout interval.
So the trick is to make sure you keep your transactions are as short as possible i.e
1. Do a begin transaction
2. Update/Delete/Insert
3. Commit
and not have anything else between these 3 steps.
And also increase your sqlite3_busy_timeout interval to suite your application depending on how large your transactions are.
You can try WAL mode, where reading and writing to SQLite can be done at the same time. But it comes with its own set of disadvantages. You can refer SQLite documentation.
SQLite has some restrictions regarding multiple users and multiple transactions. That means you can't read/write on a resource from different transactions. The database will be locked when the database is being updated.
Here are some links that might help you
http://sqlite.org/c3ref/busy_timeout.html
http://www.sqlite.org/c3ref/busy_handler.html
Good Luck
I currently developed an app that connects to SQL Server 2005 database, so my DAL objects where generated using information from that DB.
It will also be possible to connect to an Oracle and MySQL db, all with the same table structures (aside from the normal differences in fields, such as varbinary(max) in SQL Server and BLOB in Oracle, and so on). For this purpose, I already defined multiple connection strings and multiple SubSonic providers for the different DB's the app will run on.
My question is, if I generated my objects using a SQL Server database, should the generated objects work transparently with the other DB's or do I need to generate a different DAL for each database engine I use? Should I be aware of any possible bugs I may encounter while performing these operations?
Thanks in advance for any advice on this issue.
I'm using SubSonic 2.2 by the way....
From what I've been able to test so far, I can't see an easy way to achieve what I'm trying to do.
The ideal situation for me would have been to generate SubSonic objects using SQL Server for example, and just be able to switch dynamically to MySQL by just creating at runtime the correct Provider for it along with its connection string. I got to a point where my app would correctly connect from SQL Server to a MySQL DB, but there's a point where the app fails since SubSonic internally generates queries of the form
SELECT * FROM dbo.MyTable
which MySQL doesn't support obviously. I also noticed queries that enclosed table names with brackets ([]), so it seems that there are a number of factors that would limit the use of one Provider along multiple DB engines.
I guess my only other option is to sort it out with multiple generated providers, although I must admit it does not make me comfortable knowing that I'll have N copies of basically the same classes along my project.
I would really love to hear from anyone else if they've had similar experiences. I'll be sure to post my results once I get everything sorted out and working for my project.
Has any of this changed in 3.0? This would definitely be a worthy reason for me to upgrade if life is any easier on this matter...