I'm using a raw sequelize query in node to execute a postgresql function which returns a query. The problem I'm having is that the result set obtained from running the query in pgAdmin does not match the result set returned in node when using the .query() method, even though the query in pgAdmin and the query in node are both exactly the same.
here's the query I'm performing:
SELECT * FROM foo('2015-07-01','2015-07-31');
This query returns 91875 rows in pgAdmin, however, in node using sequelize, running the following only returns 87500 rows:
models.sequelize.query("SELECT * FROM foo('2015-07-01','2015-07-31');")
.spread(function (results, metadata) {
console.log(results.length);
});
output: 87500
The rows which are excluded correspond to the last date 2015-07-31.
I found that I can obtain all the rows in node by increasing the 2nd date parameter to go beyond the end date. For example, if my goal is to get results which include 2015-07-31, setting the end date to be 2015-08-03 works. Although, this is masking the problem and not ideal.
I thought there must be something wrong with the WHERE clause in my stored procedure function. However, if that was the case, why do I get back the desired number of rows when I run the query though pgAdmin?
The reason for the difference in results had to do with timezone. I had sequelize using UTC but my DB was using "US/Eastern".
After I set the DB to use UTC using the command, "SET time zone utc" I was able to replicate the results that sequelize was seeing.
Related
UPDATE 2
Using the suggestions from #Bergi, I added a db execution helper to set the timezone before executing the query. This allows me to control the timezone based on the logged in user's preferences.
async function execWithTZ(sql, params, timezone) {
const client = await this.pool.connect();
await client.query<any>(`SET TIMEZONE TO '${timezone}'`, []);
return client.query<any>(sql, params);
}
const result = await execWithTZ('SELECT ...', [], 'Australia/Sydney');
QUESTION
I am trying to get Postgresql to return the day of the week of a specific date in a view. I later use this to display values by day.
The data is in a table with a column named updated which is a timestamptz field.
When running the following query using DBeaver SQL client against a PostgreSQL 12 database I get the values as I expect, with records updated on Thursday local time (just after midnight) showing up as Thursday
SELECT
count(v.id) AS count,
btrim(to_char(v.updated, 'DAY'::text)) AS day,
date_trunc('DAY'::text, v.updated) AS "updatedDay"
FROM table v
GROUP BY (to_char(v.updated, 'DAY'::text)), (date_trunc('DAY'::text, v.updated))
I put this in a view and when I query this view using node-postgres the counts are for the previous day (Wednesday), which I presume is because the database thinks it should interpret the dates in the UTC timezone.
To further prove the point, when I change the above query to use DAYTZ instead of just DAY in DBeaver I get THURSDAYAEST but in nodejs as result I get WEDNESDAYUTC
Changing all my dates to be without timezone and forcing everything to UTC on the way in is not an option for me.
How can I make node-postgres tell the database what timezone I want these dates interpreted as so that I can get the correct day?
UPDATE 1
I managed to get PostgreSQL to return the correct values to postgres node by setting the database user's timezone.
ALTER ROLE visuo_ingest SET TIMEZONE TO 'Australia/Sydney';
Now the counts for things that happened on Thursday Sydney time is counted for Thursday and not Wednesday.
Still interested in a way to do this on the connection rather than the database user level.
Still interested in a way to do this on the connection rather than the database user level.
You already found the right setting, there's no reason to alter it on a role only. You can also change the setting in a client session by running the SET command
SET TIMEZONE TO 'Australia/Sydney';
Just put that in a pgClient.query("…"); right after connecting the client.
The database is in Azure cloud and not being used in production currently. There are 80.000 rows and a uprn is a VARCHAR(100);
I'm already using JOI to validate each UPRN as well;
I'm using KNEX with a SQL Server database with the following whereIn query:
knex(LOCATIONS.table).whereIn(LOCATIONS.uprn, req.body.uprns)
but this takes 8-12s to complete and sometimes timesout. if I use .toQuery() on the same thing, SSMS will return the result within 1-2.
If I do a raw query, the resulting .toQuery() or toString() works in SSMS and returns results. But if I try to use the raw directly, it will return 0 results.
I'm looking to either fix what's making whereIn so slow or get the raw query working.
EDIT 1:
After much debugging and trying -- it seems that the bug is due to how knex deals with arrays, so I made a for-of loop to add ? ? ? for each array element and then inputed the array for all params.
This led me to realizing the performance issue is due to SQL server way of parameterising.
I ended up building a raw query string with all of the parameters and validating the input with Joi string/regex config:
Joi.string()
.min(1)
.max(35)
.regex(/^[a-z\d\-_\s]+$/i)
allowing only for alphanumeric, dashes and spaces which should prevent sql injection.
I'm going to look deeper into security issues with this and might make a separate login that can only SELECT data from that table and nothing more to run with these queries.
Needed to just handle it raw and validate separately.
i am facing very strange behavior when running a query on server side(nodeJS with mongoose and mongoosePaginate)
I am querying a data base for a range of dates,
During the whole process I can see that I am sending the correct range,
But in execution time, it is actually sending an incorrect date as you can see below :
now the $gte is what i am looking for
but i keept hetting values from 28/2/2018.
what i found is if i am printing the query object to the console
this is what $gte value is :
what am i doing wrong ?
Thanks
So I have a NodeJS+KnexJS setup on a PostgreSQL DB, and am using the .whereRaw() method so I can use a CASE statement in my WHERE clause.
The query was tested in my CLI before migrating to code. Here is the code that is being used.
var qry = knex.select(....); // ignore the select, not important.
qry.with('daspecs', function(qy) {
qy.select('spec_id').from('drawings').where('uid', query.d);
}).whereRaw('CASE WHEN (select "spec_id" from "daspecs") IS NULL THEN true ELSE c.spec_id = (select "spec_id" from "daspecs") END');
The SQL that KnexJS is generating (output using qry.toString()) is correct, and I can even copy and paste this to my psql CLI and it returns the results I want (12 records), but for some wierd reason the KnexJS query seems to return a completely different set of results (1106 records).
Not sure where to go next, since KnexJS is giving me the right SQL, but seems like it's executing something else, and not sure how else to diagnose what it is actually doing (I've tried the knex.on('query'...) event).
Any alteration on the final SQL would result in an error (i've tested), so at the point of ruling out missing pieces.
Has anyone had any experience or issues with KnexJS saying one thing, but doing another, in particular, with the whereRaw command?
I am using Node for fetching data from MySQL. In database, i got record like : 2013-08-13 15:44:53 . But when Node fetches from Database , it assigns value like 2013-08-19T07:54:33.000Z.
I just need to get time format as in MySQL table. Btw ( My column format is DateTime in MySQL)
In Node :
connection.query(post, function(error, results, fields) {
userSocket.emit('history :', {
'dataMode': 'history',
msg: results,
});
});
When retrieving it from the database you most likely get a Date object which is exactly what you should work with (strings are only good to display dates, but working on a string representation of a date is nothing you want to do).
If you need a certain string representation, create it based on the data stored in the Date object - or even better, get some library that adds a proper strftime-like method to its prototype.
The best choice for such a library is moment.js which allows you to do this to get the string format you want:
moment('2013-08-19T07:54:33.000Z').format('YYYY-MM-DD hh:mm:ss')
// output (in my case on a system using UTC+2 as its local timezone):
// "2013-08-19 09:54:33"
However, when sending it through a socket (which requires a string representation) it's a good idea to use the default one since you can pass it to the new Date(..) constructor on the client side and get a proper Date object again.