So I have a NodeJS+KnexJS setup on a PostgreSQL DB, and am using the .whereRaw() method so I can use a CASE statement in my WHERE clause.
The query was tested in my CLI before migrating to code. Here is the code that is being used.
var qry = knex.select(....); // ignore the select, not important.
qry.with('daspecs', function(qy) {
qy.select('spec_id').from('drawings').where('uid', query.d);
}).whereRaw('CASE WHEN (select "spec_id" from "daspecs") IS NULL THEN true ELSE c.spec_id = (select "spec_id" from "daspecs") END');
The SQL that KnexJS is generating (output using qry.toString()) is correct, and I can even copy and paste this to my psql CLI and it returns the results I want (12 records), but for some wierd reason the KnexJS query seems to return a completely different set of results (1106 records).
Not sure where to go next, since KnexJS is giving me the right SQL, but seems like it's executing something else, and not sure how else to diagnose what it is actually doing (I've tried the knex.on('query'...) event).
Any alteration on the final SQL would result in an error (i've tested), so at the point of ruling out missing pieces.
Has anyone had any experience or issues with KnexJS saying one thing, but doing another, in particular, with the whereRaw command?
Related
I use jooq to generate objects against a local database, but when running "for real" later in production the actual databases will have different names. To remedy this I use the <outputSchemaToDefault>true</outputSchemaToDefault> config option (maven).
At the same time, we have multiple databases (schemas), and are using a connection pool to the server like "jdbc:mysql://localhost:3306/" (without specifying a database here).
How do I tell jooq which database to use when running queries?
I have tried all config I can think of:
new Settings()
.withRenderSchema(true) // true/false seems to make no difference.
.withRenderCatalog(true) // true/false seems to make no difference.
.withRenderMapping(new RenderMapping()
.withDefaultSchema("my_database") // Seems to have no effect.
// The above 3 configs always give me an error saying "no database selected".
// Adding this gives me 'my_database.my_table' does not exist - while it actually does.
.withSchemata(new MappedSchema()
.withInputExpression(Pattern.compile(".*"))
.withOutput("my_database")
));
I have also tried using a database/schema name, as in not configuring outputSchemaToDefault. But then, adding the MappedSchema code above, but that gives me errors with "'my_databasemy_database.my_table' does not exist", which is correct. I have no clue why that code gives me the database/schema name twice?
Edit:
When jooq tells me that the db.table does not exist, if I put a break point in a good place and get the sql from jooq and run exactly that against my database it does work. But jooq fails to run it.
Also, I'm using version 3.15.3 of jooq.
I solved it. Instead of using .withInputExpression(Pattern.compile(".*")), it seems to work with .withInput("").
I'm still not sure why it works, or if this is the "correct" way of solving it. But at least it is a way forward.
No clue why using the pattern, I got the name twice though. But that one I'll leave alone.
I am connected to a database using JDBC and I am trying to run an update query. First I am typing the query, then I am executing it (in the same way I do the SELECT which works perfectly fine).
caseoutputUpdateQuery = "(UPDATE dbo.CASEOUTPUT_TEST SET NOTIFIED = 'YES') alias_output "
spark.read.jdbc(url=jdbcUrl, table=caseoutputUpdateQuery, properties=connectionProperties)
When I run this I have the following error:
A nested INSERT, UPDATE, DELETE, or MERGE statement must have an OUTPUT clause.
I tried to fix this in different ways but there is always another error. For example, I tried to rewrite the query in the following way:
caseoutputUpdateQuery = "(UPDATE dbo.CASEOUTPUT_TEST SET NOTIFIED = 'YES' OUTPUT DELETED.*, INSERTED.* FROM dbo.CASEOUTPUT_TEST) alias_output "
but I encounter this error:
A nested INSERT, UPDATE, DELETE, or MERGE statement is not allowed in a SELECT statement that is not the immediate source of rows for an INSERT statement.
The other way I tried to rewrite it was:
caseoutputUpdateQuery = "(INSERT INTO dbo.UpdateOutput(OldCaseID,NotifiedOld) SELECT * FROM( UPDATE dbo.CASEOUTPUT_TEST SET NOTIFIED = 'YES' OUTPUT deleted.OldCaseID,DELETED.NotifiedOld ) AS tbl) alias_output "
but I've got this error:
A nested INSERT, UPDATE, DELETE, or MERGE statement is not allowed inside another nested INSERT, UPDATE, DELETE, or MERGE statement.
I've literally tried everything I found on the internet but without luck. Do you have any suggestion on how I can fix this and run my update statement?
I think Spark is not designed for that UPDATE statement use case. That's not the scenario where Spark can help to deal with RDBMS. I suggest to use a direct connection using a JDBC from the code you are writing (I mean calling that JDBC directly). If you are using Scala you can use as suggested here (for example, but there are other multiple ways) or from Python as explained here. Those samples reach Oracle engine, but please change the driver/connector if you are using MySQL, SQL Server, Postgres or any other RDMBS
spark.read under the covers does a select * from the source jdbc table. if you pass a query, spark translates it to
select your query
from ( their query select *)
Sql complains because you are trying to do an update on a view "select * from"
The database is in Azure cloud and not being used in production currently. There are 80.000 rows and a uprn is a VARCHAR(100);
I'm already using JOI to validate each UPRN as well;
I'm using KNEX with a SQL Server database with the following whereIn query:
knex(LOCATIONS.table).whereIn(LOCATIONS.uprn, req.body.uprns)
but this takes 8-12s to complete and sometimes timesout. if I use .toQuery() on the same thing, SSMS will return the result within 1-2.
If I do a raw query, the resulting .toQuery() or toString() works in SSMS and returns results. But if I try to use the raw directly, it will return 0 results.
I'm looking to either fix what's making whereIn so slow or get the raw query working.
EDIT 1:
After much debugging and trying -- it seems that the bug is due to how knex deals with arrays, so I made a for-of loop to add ? ? ? for each array element and then inputed the array for all params.
This led me to realizing the performance issue is due to SQL server way of parameterising.
I ended up building a raw query string with all of the parameters and validating the input with Joi string/regex config:
Joi.string()
.min(1)
.max(35)
.regex(/^[a-z\d\-_\s]+$/i)
allowing only for alphanumeric, dashes and spaces which should prevent sql injection.
I'm going to look deeper into security issues with this and might make a separate login that can only SELECT data from that table and nothing more to run with these queries.
Needed to just handle it raw and validate separately.
Working with KnexJS on a project and have been using the .raw() method throughout the project with out a single issue.
I now have one case where the .raw is just not being built into the SQL, and so I end up with the syntax error at end of input error. If I dump the SQL string, I can see why as it's failing
'update "mytable" set "a" = ? returning '
I can see why it's having an issue, the problem I'm having is that the returning is a Raw value as such, and so I just can't figure out why it's not being compiled with the SQL.
knexQuery.update(data).into('mytable').returning( knexQuery.raw('mytable::json') );`
If I use a string, in place of the raw in the returning() method, it will compile the string into it.
If I console out the knexQuery.raw('mytable::json') part, it shows as a Raw object, with the right object data...
Raw {
client:
Client_PG {
... },
sql: 'mytable::json',
bindings: undefined,
_wrappedBefore: undefined,
_wrappedAfter: undefined,
_debug: undefined } }
I know that the SQL works, as I've filled in the missing part and it works, but the SQL is not the issue, I just can't figure out why the raw() method is not being compiled with the SQL string.
I also have another piece of code (and INSERT one) somewhere else in my code that is using the same returning( knexQuery.raw(....) ) and that works perfectly fine.
I'm starting to think this is a bug in the code, but after spending an hour going through the KnexJS library code, I can't see any reason why it would not work.
So why does this code not build the raw into the query, while my other one has no problem and works?
I'm getting to the point where I just want to use something else to get around the problem, but it's just not possible without using this method.
This is bug in knex https://runkit.com/mikaelle/592412c3a631940012a51928 please open issue in knex github. Looks like handling raw input just haven't been implemented to returning builder method.
Anyways looks like this works as a workaround:
knex('TestTable')
.insert({ foo: 'bar' })
.returning([knex.raw('mytable::json')]) // raw wrapped in array
.toSQL();
I'm doing an app with express in node. I thought about different DBMS before coming up with the idea of using PostgreSQL (I haven't got any NoSQL experience) but it's bugging me out due to some unexpected errors.
I'm trying to check if a user exist when he logs, but the Select statement doesn't work, don't know why:
var query = client.query('SELECT "name" FROM "User" WHERE "pass" = $1', [req.body.password]);
console.log(query.toString());
query.on('row', function(row) {
console.log('user "%s"', row.name);
});
After this the initial page should render (that actually works). I'm using the node-postgres driver, by the way.
If I put something like client.query('SELECT * FROM "User"'); the database works perfectly (and I don't need that behaviour either). I've read about the PostgreSQL identifier problems, but nevertheless it keeps happening.
Okay, turns out user table exists, and the table I created is case sensitive. It works know.
I tried select * from user on psql and it returned the current user. Tried select * from "User" and turned out I was wrong. Funny thing.