I'm sending a JSON with some data to Google Big Query, all string and numeric columns are accepting my data just fine, however, whenever I try to send a data to the Date column it returns a error saying:
'invalid',
location: 'creationdate',
debugInfo: '',
message: "Invalid date: '2020-10-05T00:00:00.000Z'"
I'm using Node.js to work with the data and sending it to Google Big Query through the insertAll route, following the format told by them on their docs: https://cloud.google.com/bigquery/docs/reference/rest/v2/tabledata/insertAll
If I send the data as Null it works fine, I've also tried to send it as a string and as an integer, but it returns the same error.
To confirm is the data type on the BigQuery schema Date or Datetime?
You may need to truncate to fit the date format or switch to using datetime.
For some reason the Date format in Big Query only accepts YYYY-MM-DD, without hours or timezone, I had to create another column in my table so I would be able to store the hours and timezone.
Related
MongoDB:
I have a table in mongo with timestamps attribute set. The following dates are automatically created by the DB while saving the documents.

Request:
Now when i am querying for the documents form the UI, the request values are like follows (date)

Backend Logic:
if (filters.date) {
startDate = new Date(filters.date);
startDate.setHours(0, 0, 0, 0);
endDate = new Date(filters.date);
endDate.setDate(endDate.getDate() + 1);
endDate.setHours(0, 0, 0, 0);
}
datePredicate = {'createdAt': {"$gte": startDate, "$lt": endDate}};
Response:
The response received ( notice the the date is one day old )

In UI:
Here also it shows one day old date from the record fetched. Indeed the query fetches the objects saved on the previous day.

Kindly let me know why I am not getting the records for the chosen date, but one day before.
Note:
However this logic works perfectly in my localhost, just not in the prod instance.
Also when i pass the request as below, it works as expected.

What am i missing? I am confused with formatting dates. Kindly suggest.
If you have a server which is not located at the same timezone you are hitting from, then you'll run into this issue, this is especially true if you are using string to communicate in between. That is also the reason why you don't see this in your localhost.
When you save the date in the database, if you are using a string, then you need to make sure it's timezone immune, which means you have to choose A) certain string format, "Z" or B) date format if it carries with the timezone. Basically you have to make sure what you stored in the database can be safely converted back to any time format in the future, ex. your browser timezone.
Simply put, "6/27/2020" is very misleading, it could mean anything. But if you are using UTC time format (ex. "Z"), then you know this date has been uniquely locked to a particular timezone. Even if you switch timezone, you'll find ways to convert in between. The base reference timezone does matter here.
Hope this helps, sorry I didn't show you any code, but this is more like a understanding question than coding problem.
My database is SQL Server. I have the following sql query
SELECT RQI,RQIOver,PCI,PCIOver,PQI,PQIOver,SR,SROver,IRI,IRIOver,RUT,RUTOver,cast(DateCollected as date) as DateCollected,cast(DateCollectedOver as date) as DateCollectedOver from dbo.PF_Condition where SegmentId=12665
The result I am getting for DateCollected is just the date, which I need, when run it in SQL Management Studio. However when I put the same SQL query in a nodejs service which I created the result is not the same.
I needed to extract YYYY-MM-DD from datetime field DateCollected and DateCollectedOver as mentioned in the query above.
Can someone help me with it?
I made it work with the slice operator of javascript and sliced the date part only and displayed in the frontend where I need to. The results returned are of type string so the slice operator worked.
If someone can help why the cast function is not working in the sql query when I put it in the nodejs service it will be really helpful
After casting as DATE, you can then use the LEFT function to truncate the result to 10 characters:
LEFT(cast(DateCollected as date), 10) as DateCollected
which would return a string with the format 'YYYY-MM-DD'
I'm using node-mssql to get rows from a table that includes a date column (YYYY-MM-DD). I want to pass the date to a client application as a string in that same format. node-mssql is creating date objects, which I'm having to convert and slice at significant cost to get the format that I started with. sql.map.register(String, sql.Date) doesn't seem to work here (or I'm using it wrong). Is there a way to change how node-mssql handles the SQL date data type?
I'm having the same problem. sql.map.register is only for converting JS types to SQL types and I don't think there is any way to avoid SQL Date fields being converted into JS Date objects with node-mssql alone. I just changed my SQL query to convert to a formatted varchar instead of a Date type, e.g.: convert(varchar, birthday, 105) as dob.
In a Generic Inquiry, I'm trying to format the time part of a DateTime field in the results. I don't currently see any way to do this without parsing the date as a string, but I must be missing something. Using the Format() function, running the query tells me "The method or operation is not implemented". Using the Minute() function gets the minutes part, but using the Hour() function says "Unsupported formula operator Hour".
Add CRCase table in your tables and don't join with any tables under Relations tab, and in result grid, under Schema field use CRCase.CreatedDateTime, with this you will get the result in DateTime format.
let me know if this is my wrong assumption to your question.
I have an excel with a date field but the first row in the excel is blank and few other rows are having a date format as MM/dd/yyyy HH:mm:ss.
The data to be loaded into a Postgresql table with the field of data type timestamp yyyy-mm-dd HH:mm:ss.
The excel cannot be modified as it is being downloaded from the cloud and the data is loaded straight away into the table.
I tried using tConvert type but it cannot accept null or " " values in timestamp. I am facing a Null tMap error during runtime in talend. Even if I try to convert from string to date format in order to pass null in tmap, it is changing the date format and showing error.How can this be handled ?
The talend structure is : tFileInputExcel - > TMAP(date field : MM/dd/yyyy HH:mm:ss) -> tConvertType(date field : yyyy-mm-dd HH:mm:ss) ->TMAP(yyyy-mm-dd HH:mm:ss) -> Postgresql Table
Here is the Excel screenshot:
At first, I do not quite understand why do you want to use tConvertType component. After defining a proper schema Talend is changing your data into Java Date object and from that moment format is not important and you don't have to convert it when you want to put it into Postgres table. At least it should not cause NullPointerException.
Consider following steps:
Sample input file
I've prepared some file with date value/space/empty string, solution I'm describing works also with nulls.
Configure tFileInputExcel component
You have to allow taking null values in by checking the Nullable check box. You should also check trim option.
Examine output
After connecting input component to tLogRow null/empty/space values are handled properly.
I hope this will be helpful.
You can capture date format or null handling in variable within tMAP component
that is
var :TalendDate.formatDate("yyyy-mm-dd HH:mm:ss",row1.columnname)
so data flow would be
tFileInputExcel --->tMAP --->Postgresql Table