I want to select data from one table and insert into another in batches, using different sql connections. The two tables are set up exactly the same. At the moment I have:
destination.withBatch(1000) { stmt ->
source.eachRow(selectQuery) {
String insertString = """
INSERT INTO dest_table
VALUES (
${it[0]},
${it[1]});
"""
try {
stmt.addBatch(insertString)
}
catch (Exception e) {
println insertString
}
}
}
Something seems to happen to the data types in this process, because it gets very unhappy inserting a string like 'a:string' because of the colon.
I could do '${it[0]}' to enforce it is treated as a String, but this will cause problems when I come to other data types.
Furthermore, my error handling is definitely not working correctly. I want it to print out the inserts that it was unable to execute, and then carry on gracefully.
Thanks
It's likely that groovy sql is creating a prepared statement from your sql string, and that anything with a colon in it is parsed as a parameter placeholder.
So, I'd suggest going with the flow, and binding your data values separately, instead of putting them inline in the sql statement. This will also probably improve performance, as the prepared statements can be cached by the database.
Related
Is there a way to "tag" or put names on SQL statements in jOOQ so when I look at the Performance Insights of AWS RDS, I can see something more meaningful than the first 500 chars of the statement?
For example, Performance Insights shows that this query is taking a toll in my DB:
select "my_schema"."custs"."id", "my_schema"."custs"."other_id", "my_schema"."custs"."cid_id", "my_schema"."custs"."valid_since", "my_schema"."custs"."valid_until", "my_schema"."custs"."address", "my_schema"."custs"."address_id_1", "my_schema"."pets"."id", "my_schema"."pets"."cst_id", "my_schema"."pets"."tag", "my_schema"."pets"."name", "my_schema"."pets"."description", "my_schema"."pets"."owner", "my_schema"."pets"."created_on", "my_schema"."pets"."created_by", "my_schema"."pets"."modified_on",
But as it comes chopped, it's not straight-forward to know which jOOQ code generated this.
I would prefer to see something like this:
Customer - Pet Lookup
or:
(Customer - Pet Lookup) select "my_schema"."custs"."id", "my_schema"."custs"."other_id", "my_schema"."custs"."cid_id", "my_schema"."custs"."valid_since", "my_schema"."custs"."valid_until", "my_schema"."custs"."address", "my_schema"."custs"."address_id_1", "my_schema"."pets"."id", "my_schema"."pets"."cst_id", "my_schema"."pets"."tag", "my_schema"."pets"."name", "my_schema"."pets"."description", "my_schema"."pets"."owner", "my_schema"."pets"."created_on", "my_schema"."pets"."created_by", "my_schema"."pets"."modified_on",
There are at least two out of the box approaches to what you want to achieve, both completely vendor agnostic:
1. Use "hints"
jOOQ supports Oracle style hints using the hint() method, at least for SELECT statements. Write something like:
ctx.select(T.A, T.B)
.hint("/* my tag */")
.from(T)
.where(...)
The limitation here is the location of the hint, which is going to be right after the SELECT keyword. Not sure if this will work for your RDBMS.
2. Use an ExecuteListener
You can supply your Configuration with an ExecuteListener, which patches your generated SQL strings with whatever you need to be added:
class MyListener extends DefaultExecuteListener {
// renderEnd() is called after the SQL string is generated, but
// before the prepared statement is created, let alone executed
#Override
public void renderEnd​(ExecuteContext ctx) {
if (mechanismToDetermineIfTaggingIsNeeded())
ctx.sql("/* My tag */ " + ctx.sql());
}
}
Using regular expressions, you can place that tag at any specific location within your SQL string.
So I have a NodeJS+KnexJS setup on a PostgreSQL DB, and am using the .whereRaw() method so I can use a CASE statement in my WHERE clause.
The query was tested in my CLI before migrating to code. Here is the code that is being used.
var qry = knex.select(....); // ignore the select, not important.
qry.with('daspecs', function(qy) {
qy.select('spec_id').from('drawings').where('uid', query.d);
}).whereRaw('CASE WHEN (select "spec_id" from "daspecs") IS NULL THEN true ELSE c.spec_id = (select "spec_id" from "daspecs") END');
The SQL that KnexJS is generating (output using qry.toString()) is correct, and I can even copy and paste this to my psql CLI and it returns the results I want (12 records), but for some wierd reason the KnexJS query seems to return a completely different set of results (1106 records).
Not sure where to go next, since KnexJS is giving me the right SQL, but seems like it's executing something else, and not sure how else to diagnose what it is actually doing (I've tried the knex.on('query'...) event).
Any alteration on the final SQL would result in an error (i've tested), so at the point of ruling out missing pieces.
Has anyone had any experience or issues with KnexJS saying one thing, but doing another, in particular, with the whereRaw command?
I find myself doing a lot of queries to fetch just the first couple of items of a big set, e.g. to show the three most recent news articles or blog posts on the homepage of a website.
As long as this query only involves predefined or custom Parts, I can do something like this:
public IEnumerable<ContentItem> GetTopArticles(int amount)
{
var cultureRecord = _cultureManager.GetCultureByName(_orchardServices.WorkContext.CurrentCulture);
var articles = _orchardServices.ContentManager.Query().ForType("Article")
.Where<LocalizationPartRecord>(lpr => lpr.CultureId == cultureRecord.Id)
.OrderBy<CommonPartRecord>(cpr => cpr.PublishedUtc)
.Slice(0, amount);
return articles;
}
I'm assuming this will more or less be the same as a SELECT TOP [amount] ... in SQL and will have good performance on a large number of records.
However, sometimes I use Migrations or Import to create Content Types from an external source and want to conditionally check a field from the generic Part. In this case I don't have a Part or PartRecord class that I can pass as a parameter to the ContentQuery methods and if I want to do a conditional check on any of the fields I currently do something like this:
public IEnumerable<ContentItem> GetTopArticles(int amount)
{
var articles = _orchardServices.ContentManager.Query().ForType("Article")
.OrderBy<CommonPartRecord>(cpr => cpr.PublishedUtc)
.List()
.Where(a => a.Content.Article.IsFeatured.Value == true)
.Take(amount);
return articles;
}
This is really wasteful and causes large overhead on big sets but I really, REALLY, do not want to delve into the database to figure out Orchard's inner workings and construct long and complex HQL queries every time I want to do something like this.
Is there any way to rewrite the second query with IContentQuery methods without incurring a large performance hit?
I'm working on something similar (being able to query model data with a dynamic name). Sadly, I haven't found anything that makes it easy.
The method I've found that works is to do plain SQL queries against the database. Check out this module for syntax on that if you do later find yourself willing to delve into the database.
I am new to Cassandra and trying to write a program in C# for insertion and deletion.
I want to know if there is a way i can use variables instead of values in the insert command ?
When i try the following:
string s1="A";
string s2="B";
session.Execute("insert into users (lastname, firstname) values (s1,s2)");
The error occurs: A first chance exception of type 'Cassandra.SyntaxError' occurred in Cassandra.dll
Assuming that you are using the DataStax CQL3 C# Driver, the best way to go about this is to use a prepared statement. Once you set that up, you bind your variables and Execute, like this:
string strCQL = "INSERT INTO users (lastname, firstname) VALUES (?,?)";
string s1 = "A";
string s2 = "B";
PreparedStatement preparedStatement = session.Prepare(strCQL);
BoundStatement boundStatement = preparedStatement.Bind(s1,s2);
session.Execute(boundStatement);
Please don't ever build a CQL statement with string.Format (or string concatenation) and execute it. Cassandra/CQL can also be subject to injection-based attacks, so you should always use a prepared statement and bind your variables to it. Also, if you have a statement that you are going to run multiple times (ex: within a loop), you can get better performance by preparing it prior to the loop, and binding/executing within.
You need String.format or better yet use prepared statements.
http://www.datastax.com/documentation/developer/csharp-driver/2.1/csharp-driver/reference/21features/namedParameters.html
Ok, so I'm working with an ObservableList, which is working fine, but now I need to use the observable list to insert rows into and update rows in an SQL database table. I've found little info on working between JavaFX and SQL databases ... all the examples of data tables have the data created in the java code. I had hope when I saw "update SQL database" in this post:
Update sql database from FoxPro data on Glassfish server
but it was not applicable to my situation.
So the question is, how do I start the code to read from the ObservableList so I can run my SQL Insert statement? If you could point me to an example code where an ObservableList is used and an SQL table is created/added to/updated I would greatly appreciate it.
Thanks!
UPDATE TO QUESTION:
I can't really post relevant code here because the relevant parts are what I don't have. However, I'm thinking what I need to do is something like this:
mylist.moveToFirst();
while (mylist.next()) {
make connection // think I got it
INSERT INTO mytable (name, address, phone) VALUES (observablename, observableaddress, observablephone // think I got this as well
Obviously I'm applying my knowledge of other areas to ObservableList, but I am doing it to demonstrate what I don't know how to do with my ObservableList (mylist).
Again, thanks for any help.
Tying up loose ends today, and this question has not really been answered. I reposted a newer question with more specifics once I learned more about the situation, and that question also went unanswered, but I did figure it out, and posted an answer here: Understanding my ObservableList.
However, to be neat and tidy, let me post here some code to help me remember, as well as help anyone else who looks at this question and says, "YES, BUT WHAT IS THE SOLUTION?!?!?"
Generically, it looks something like this:
I like to open my connection and prepare my statement(s) first.
Use the iterator to get the variables from the list
within the iterator, add the variables to the prepared statement and execute.
I read somewhere about batch execution of statements, but with as few updates as I'm doing with each list, that seemed too complicated, so I just do each update individually within the iterator.
Specifically, here is some code:
Connection con;
con = [your connection string]; // I actually keep my connection string in its own class
// and just call it (OpenDB.connect()). This way I can swap out the class OpenDB
// for whatever database I'm using (MySQL, MS Access, etc.) and I don't have to
// change a bunch of connection strings in other classes.
PreparedStatement pst;
String insertString = "INSERT INTO People (Name, Address, Phone) VALUES (?, ?, ?)";
pst = con.prepareStatement(insertString);
for(Person p : mylist) { // read as: for each Person [a data model defined in a class
// named Person] which in this set of statements we shall call 'p' within the list
// previously defined and named 'mylist' ... or "For each Person 'p' in 'mylist'"
String name = p.name.get(); // get the name which corresponds to the Person in this object of 'mylist'
String address = p.address.get(); // ditto, address
Integer phone = p.phone.get(); // ditto, phone. Did as integer here to show how to add to pst below
pst.setString(1, name); // replace question mark 1 with value of 'name'
pst.setString(2, address); // ditto, 2 and 'address'
pst.setInt(3, phone); // ditto, 3 and 'phone'
pst.executeUpdate();
And that's how I did it. Not sure if it's the 'proper' way to do it, but it works. Any input is welcomed, as I'm still learning.
In JavaFX you usually get to be the person to create the example :)
ObservableList supports listeners, these receive events which tell you what has been added or updated by default. There is a good example in the javadocs here.
To get update events you need to provide an 'extractor' to the method creating the list here. This should take an instance of the object in the list and provide an array of the properties you want to listen to.
Try this:
SQLEXEC(lnConn, "Update INVENTORY SET brand = ?_brand, model = ?_model, code =?_code, timestamp =?_datetime where recno=?_id ")