When using db.SaveAll(collection) and a Sqlite database it seems to trigger the below error if the collection is too large:
SqliteException: SQLite Error 1: 'too many SQL variables'.
The collections are not that big, maybe a couple of thousand entries at most. Is this a Sqlite limit? Can I configure this?
I have done workaround of inserting items one by one (works extremely slow), would just like to know for future reference if this is intended behaviour or I am doing something wrong.
Thanks.
Edit: seems the insert limit for sqlite is 500. I am guessing I have to batch this myself
The answer was to do inside a transaction. It changed it from 10mins to insert 10,000 records to 7 seconds.
using (var dbLite = _sqlLiteFactory.Open())
{
using (IDbTransaction trans = dbLite.OpenTransaction())
{
collection.Each(x => dbLite.Save(x));
trans.Commit();
}
}
Related
I receive 3 post calls from client, let say in a second, and with nodejs-mongodb immediately(without any pause, sleep, etc) I try to insert the data that is posted in database using updateOne. All data is new, so in every call, insert would happen.
Here is the code (js):
const myCollection = mydb.collection("mydata")
myCollection.updateOne({name:req.data.name},{$set:{name:req.data.name, data:req.data.data}}, {upsert:true}, function(err, result) {console.log("UPDATEONE err: "+err)})
When I call just 1 time this updateOne, it works; 2 times successively, it works. But if I call 2+ times in succession, only the first two ones correctly inserted into database, and the rest, no.
The error that I get after updateOne is, MongoWriteConcernError: No write concern mode named 'majority;' found in replica set configuration. However, I always get this error, also even when the insertion is done correctly. So I don't think this is related to my problem.
Probably you will suggest to me to use updateMany, bulkWrite, etc. and you will be right, but I want to know the reason why after 2+ the insertion is not done.
Have in mind .updateOne() returns a Promise so it should be handled properly in order to avoid concurrency issues. More info about it here.
The error MongoWriteConcernError might be related to the connection string you are using. Check if there is any &w=majority and remove it as recommended here.
So I have a NodeJS+KnexJS setup on a PostgreSQL DB, and am using the .whereRaw() method so I can use a CASE statement in my WHERE clause.
The query was tested in my CLI before migrating to code. Here is the code that is being used.
var qry = knex.select(....); // ignore the select, not important.
qry.with('daspecs', function(qy) {
qy.select('spec_id').from('drawings').where('uid', query.d);
}).whereRaw('CASE WHEN (select "spec_id" from "daspecs") IS NULL THEN true ELSE c.spec_id = (select "spec_id" from "daspecs") END');
The SQL that KnexJS is generating (output using qry.toString()) is correct, and I can even copy and paste this to my psql CLI and it returns the results I want (12 records), but for some wierd reason the KnexJS query seems to return a completely different set of results (1106 records).
Not sure where to go next, since KnexJS is giving me the right SQL, but seems like it's executing something else, and not sure how else to diagnose what it is actually doing (I've tried the knex.on('query'...) event).
Any alteration on the final SQL would result in an error (i've tested), so at the point of ruling out missing pieces.
Has anyone had any experience or issues with KnexJS saying one thing, but doing another, in particular, with the whereRaw command?
Same code works fine when letting couch auto generate UUID's. I am starting off with a new completely empty database yet I keep getting this
error: conflict
reason: Document update conflict
To reiterate I am posting new documents to an empty database so not sure how I can get update conflicts when nothing is being updated. Even stranger the conflicting documents still show up in the DB with only a single revision, but overall there are missing records.
I am trying to insert about 38,000 records with _bulk_docs in batches of 100. I am getting these records (100 at a time) from a RETS server, each record already has a unique ID that I want to use for the couchDB _id instead of their UUID's. I am using a promised based library to get the records and axios to insert them into couch. After getting the first batch of 100 I then run this code to add an _id to each of the 100 records before inserting
let batch = [];
batch = records.results.map((listing) => {
let temp = listing;
temp._id = listing.ListingKey;
return temp;
});
Then insert:
axios.post('http://127.0.0.1:5984/rets_store/_bulk_docs', { docs: batch })
This is all inside of a function that I call recursively.
I know this probably wont be enough to see the issue but thought Id start here. I know for sure it has something to do with my map() and adding the _id = ListingKey
Thanks!
We have a collection with millions of records in mongoDB. its taking lots of time and time out to count and create pagination with these records. whats the best way to do it using nodejs. I want to create a page where I see records with pagination, count, delete, search of records. Below is the code which doing query to Mongo with different conditions.
crowdResult.find({ "auditId":args.audit_id,"isDeleted":false})
.skip(args.skip)
.limit(args.limit)
.exec(function (err, data) {
if (err)
return callback(err,null);
console.log(data);
return callback(null,data);
})
If the goal is to get through a large dataset without timing out then I use the following approach to get pages one after another and process the paged resultset as soon as it becomes available:
https://gist.github.com/pulkitsinghal/2f3806670439fa137210fc26b134237f
Please focus on the following lines to get a quick idea of what the code is doing before diving deeper:
Let getPage() handle the work, you can set the pageSize and query to your liking:
https://gist.github.com/pulkitsinghal/2f3806670439fa137210fc26b134237f#file-sample-js-L68
Method signature:
https://gist.github.com/pulkitsinghal/2f3806670439fa137210fc26b134237f#file-sample-js-L29
Process pagedResults as soon as they become available:
https://gist.github.com/pulkitsinghal/2f3806670439fa137210fc26b134237f#file-sample-js-L49
Move on to the next page:
https://gist.github.com/pulkitsinghal/2f3806670439fa137210fc26b134237f#file-sample-js-L53
The code will stop when there is no more data left:
https://gist.github.com/pulkitsinghal/2f3806670439fa137210fc26b134237f#file-sample-js-L41
Or it will stop when working on the last page of data:
https://gist.github.com/pulkitsinghal/2f3806670439fa137210fc26b134237f#file-sample-js-L46
I hope this offers some inspiration, even if its not an exact solution for your needs.
When running the templates against a database with 1400+ tables I get the following error. The server shows hundreds of connections. Does anyone know if this is a problem with template generation in general, or with these templates specifically. Other, smaller DBs generate ok for me.
Running transformation: System.InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
at System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection)
at System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory)
at System.Data.SqlClient.SqlConnection.Open()
at Microsoft.VisualStudio.TextTemplating8D8967BD3E8719BDA6DD9945992440F1.GeneratedTextTransformation.GetCommand(String sql) in c:\POS\POS.Win\Templates\SQLServer.ttinclude:line 13
at Microsoft.VisualStudio.TextTemplating8D8967BD3E8719BDA6DD9945992440F1.GeneratedTextTransformation.LoadFKTables(String tableName) in c:\POS\POS.Win\Templates\SQLServer.ttinclude:line 179
at Microsoft.VisualStudio.TextTemplating8D8967BD3E8719BDA6DD9945992440F1.GeneratedTextTransformation.LoadTables() in c:\POS\POS.Win\Templates\SQLServer.ttinclude:line 131
at Microsoft.VisualStudio.TextTemplating8D8967BD3E8719BDA6DD9945992440F1.GeneratedTextTransformation.TransformText() in c:\POS\POS.Win\Templates\ActiveRecord.tt:line 21
I had this exact problem with a 250 table database. Poking around the SQLServer.ttinclude, I found this:
var cmd=GetCommand(sql);
cmd.Parameters.AddWithValue("#tableName",table);
var result=cmd.ExecuteScalar();
cmd.Dispose();
if(result!=null)
pk=result.ToString();
Changed it to this:
using (var cmd=GetCommand(sql))
{
cmd.Parameters.AddWithValue("#tableName",table);
int x = 0;
if (table == "tbl_Address")
x++;
var result=cmd.ExecuteScalar();
if(result!=null)
pk=result.ToString();
cmd.Connection.Close();
}
The connection was not getting closed and you run out of connections in the pool after 100. Any DB with more than 100 tables would hit this issue. It's pretty awesome that I could fix this without touching SubSonic.Core.
Now if I can just spend the time to figure out Git, I would post this fix back to the subsonic project. :-D
Well, I might suggest that running T4 on 1400 tables is probably not the best idea. Given that - you can spelunk the T4 code (in SQLServer.tt) to see how we load the tables (in LoadTables) and doctor the connections as required.
1400 classes generated - would be fun to guess the file size of the final output...
I had this same problem with a very large database (lots of tables, views, and SPs). I added the following to my web.config connection string for Sql Server 2005 - Connect Timeout=60;
This seemed to do the trick.
You can also dig throught the .tt and .ttinclude files and set the cmd.CommandTimeout too.
Also, here is the const to set for specific tables:
const string TABLE_SQL=#"SELECT *
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_TYPE='BASE TABLE' and table_name = 'myTableA' or table_name = 'myTableB'