BigQuery convert unix timestamp struct to struct of datetime - struct

I have a BigQuery table that contains a struct column called daySliderTimes in the following form:
daySliderTimes STRUCT<_field_1 STRUCT<_seconds INT, _nanoseconds INT>, _field_1 STRUCT<_seconds INT, _nanoseconds INT>.
_field_1 and _field_2 represent two different timestamps. _seconds and _nanoseconds represent time since the unix epoch.
I want to convert the data into a new STRUCT with the following form:
daySlidertimes STRUCT<startTime DATETIME, endTime DATETIME>
This is the table as seen in the BigQuery UI:

If you want to create a new table from the old one with the format daySlidertimes STRUCT<startTime DATETIME, endTime DATETIME>, you can cast the data in milliseconds and so then transform it to TIMESTAMP with the function "TIMESTAMP_MICROS", check this link to see the amount of functions to parse timestamp [1].
An example of the query should look something like this:
CREATE TABLE `project.dataset.new_table` AS
SELECT searchDocId,
STRUCT(TIMESTAMP_MICROS(CAST(
((daySliderTimes.field1.seconds * 1e+6) +
ROUND(daySliderTimes.field1.nanoseconds * 0.001)) AS INT64)) as
startTime,
TIMESTAMP_MICROS(CAST( ((daySliderTimes.field2.seconds * 1e+6) +
ROUND(daySliderTimes.field2.nanoseconds * 0.001)) AS INT64)) as endTime)
as daySliderTimes,
enabledDaySliders
FROM `project.dataset.old_table`
[1] https://cloud.google.com/bigquery/docs/reference/standard-sql/functions-and-operators#parse_timestamp

You can use TIMESTAMP_SECONDS() function. This function converts the seconds to DATETIME format.
Therefore, you are ale to transform daySliderTimes._field_1.seconds to a date using TIMESTAMP_SECONDS() function. As well as, for _field_2, then aggregate them in a new struct format.
During the creation of the view or table, in your select you can do the following:
WITH table_newStruct as(
SELECT
#Select all the desired fields
searchDocId,
STRUCT(TIMESTAMP_SECONDS(daySliderTimes._field_1.seconds) as startTime,
TIMESTAMP_SECONDS(daySliderTimes._field_.seconds) as endTime) as new_daySlidertimes
FROM 'table_source')
SELECT searchDocId, new_daySlidertimes
FROM 'table_newStruct'
In addition, the returned TIMESTAMP should be in the following format 1970-01-01 00:00:00 UTC. You can format it using the FORMAT_DATE() function.

Related

convert and Insert date format in nodejs into Oracle timestamp

let date = moment(new Date()).format("YYYY-MM-DD hh:mm:ss.000000000 A");
// when i tried to insert date in table it is null
// TImestamp format in OracleDB is 14-03-22 3:53:08.901008000 PM
INSERT INTO STUDENT VALUES(join_date) ( '14-03-22 3:53:08.901008000 PM')
How can get date format like YYYY-MM-DD HH:MM:SS.FF3 AM/PM because in oracle it supports this kind of timestamp
In Oracle, a TIMESTAMP is a binary data type that consists of 7 - 13 bytes (century, year-of-century, month, day, hour, minute, second and between zero and six bytes for fractional seconds). It ALWAYS contains those components and it is NEVER stored in a particular format.
The client application you are using (i.e. SQL/Plus, SQL Developer, NodeJS, Java, etc.) may chose to DISPLAY the binary value with a default format but this is a function of the client application and NOT a function of the database. (Some client applications may use the NLS_TIMESTAMP_FORMAT session parameter from the database as their default format model but the implicit conversion from binary-to-string for display purposes is still something that the client application does, not the database, and not all clients use the database session variables for their defaults.)
You should either:
Use a timestamp literal:
INSERT INTO STUDENT (join_date) VALUES (TIMESTAMP '2022-03-14 15:53:08.901008000');
Explicitly convert your formatted string to a timestamp binary data type using the TO_TIMESTAMP function with a format model:
INSERT INTO STUDENT (join_date)
VALUES (
TO_TIMESTAMP('14-03-22 3:53:08.901008000 PM', 'DD-MM-RR HH12:MI:SS.FF9 AM')
)

How to convert an azure data factory string type variable into datetime format

** I had a string type variable in azure data factory which is storing datetime format from a lookup activity**
but after that i need to compare that value inside the variable with a datetime. how can i convert it into datetime format
i tried this but i am getting an error i will post the code and error below
varible--string(activity('Lookup1').output.value[1].CREATED_DATE) variable i created which converts datetime into string variable
query-select * from sampletable where modified_date >= formatDateTime(variables('createddate'),"o")```
this is the code i tried for comparing and to convert it into datetime format
ERROR
Failure happened on 'Source' side. ErrorCode=SqlOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=A database operation failed with the following error: ''variables' is not a recognized built-in function name.',Source=,''Type=System.Data.SqlClient.SqlException,Message='variables' is not a recognized built-in function name.,Source=.Net SqlClient Data Provider,SqlErrorNumber=195,Class=15,ErrorCode=-2146232060,State=10,Errors=[{Class=15,Number=195,State=10,Message='variables' is not a recognized built-in function name.,},],'
You can try as per the below sample
#{concat('SELECT TOP (10) * FROM [SalesLT].[Customer] WHERE ModifiedDate <=', formatDateTime(variables('createddate'),'yyyy-MM-dd'))}
Equivalent to:
SELECT TOP (10) * FROM [SalesLT].[Customer] WHERE ModifiedDate <=2021-10-27
See official doc: Functions in expressions
But If you try as per default format 'o' in formatDateTime() function
#{concat('SELECT TOP (10) * FROM [SalesLT].[Customer] WHERE ModifiedDate <=', formatDateTime(variables('createddate'),'o'))}
You might see the below error:
Try to refer formatDateTime and generate a query sutable for datetime format in your database.

Query date on datetime stored within jsonfield in Postgres through Django?

I have a Postgres table with a jsonb column containing UTC timestamp data in ISO format like the following:
{
"time": "2021-04-13T20:14:56Z"
}
The Django model for this table looks like:
class DateModel(models.Model):
values = models.JSONField(default=dict)
I need to query the table for all records with a timestamp on a certain date (ignoring time)
I'm looking for a solution similar to the following:
DateModel.objects.filter(values__time__date='2021-04-13')
The other solution I have found is to query for records with date greater than the previous day and less than the next one. This works but I am looking for a way to do it with a single query so the code would be more concise.
Any suggestions?
There's a couple of annotations you need to perform on the queryset to extract the time field and convert it to a datetime.
First you need to extract the time string by using django.contrib.postgres.fields.jsonb.KeyTextTransform
from django.contrib.postgres.fields.jsonb import KeyTextTransform
query = DateModel.objects.annotate(time_str=KeyTextTransform('time', 'values'))
Then you need to convert that string to a datetime using Cast
from django.db.models.functions import Cast
from django.db.models import DateTimeField
query = query.annotate(time=Cast('time_str', output_field=DateTimeField()))
Then you can filter by that annotation
query = query.filter(time__date='2021-04-13')

Inserting Timestamp Into Snowflake Using Python 3.8

I have an empty table defined in snowflake as;
CREATE OR REPLACE TABLE db1.schema1.table(
ACCOUNT_ID NUMBER NOT NULL PRIMARY KEY,
PREDICTED_PROBABILITY FLOAT,
TIME_PREDICTED TIMESTAMP
);
And it creates the correct table, which has been checked using desc command in sql. Then using a snowflake python connector we are trying to execute following query;
insert_query = f'INSERT INTO DATA_LAKE.CUSTOMER.ACT_PREDICTED_PROBABILITIES(ACCOUNT_ID, PREDICTED_PROBABILITY, TIME_PREDICTED) VALUES ({accountId}, {risk_score},{ct});'
ctx.cursor().execute(insert_query)
Just before this query the variables are defined, The main challenge is getting the current time stamp written into snowflake. Here the value of ct is defined as;
import datetime
ct = datetime.datetime.now()
print(ct)
2021-04-30 21:54:41.676406
But when we try to execute this INSERT query we get the following errr message;
ProgrammingError: 001003 (42000): SQL compilation error:
syntax error line 1 at position 157 unexpected '21'.
Can I kindly get some help on ow to format the date time value here? Help is appreciated.
In addition to the answer #Lukasz provided you could also think about defining the current_timestamp() as default for the TIME_PREDICTED column:
CREATE OR REPLACE TABLE db1.schema1.table(
ACCOUNT_ID NUMBER NOT NULL PRIMARY KEY,
PREDICTED_PROBABILITY FLOAT,
TIME_PREDICTED TIMESTAMP DEFAULT current_timestamp
);
And then just insert ACCOUNT_ID and PREDICTED_PROBABILITY:
insert_query = f'INSERT INTO DATA_LAKE.CUSTOMER.ACT_PREDICTED_PROBABILITIES(ACCOUNT_ID, PREDICTED_PROBABILITY) VALUES ({accountId}, {risk_score});'
ctx.cursor().execute(insert_query)
It will automatically assign the insert time to TIME_PREDICTED
Educated guess. When performing insert with:
insert_query = f'INSERT INTO ...(ACCOUNT_ID, PREDICTED_PROBABILITY, TIME_PREDICTED)
VALUES ({accountId}, {risk_score},{ct});'
It is a string interpolation. The ct is provided as string representation of datetime, which does not match a timestamp data type, thus error.
I would suggest using proper variable binding instead:
ctx.cursor().execute("INSERT INTO DATA_LAKE.CUSTOMER.ACT_PREDICTED_PROBABILITIES "
"(ACCOUNT_ID, PREDICTED_PROBABILITY, TIME_PREDICTED) "
"VALUES(:1, :2, :3)",
(accountId,
risk_score,
("TIMESTAMP_LTZ", ct)
)
);
Avoid SQL Injection Attacks
Avoid binding data using Python’s formatting function because you risk SQL injection. For example:
# Binding data (UNSAFE EXAMPLE)
con.cursor().execute(
"INSERT INTO testtable(col1, col2) "
"VALUES({col1}, '{col2}')".format(
col1=789,
col2='test string3')
)
Instead, store the values in variables, check those values (for example, by looking for suspicious semicolons inside strings), and then bind the parameters using qmark or numeric binding style.
You forgot to place the quotes before and after the {ct}. The code should be :
insert_query = "INSERT INTO DATA_LAKE.CUSTOMER.ACT_PREDICTED_PROBABILITIES(ACCOUNT_ID, PREDICTED_PROBABILITY, TIME_PREDICTED) VALUES ({accountId}, {risk_score},'{ct}');".format(accountId=accountId,risk_score=risk_score,ct=ct)
ctx.cursor().execute(insert_query)

convert scientific notation to datetime

How can I convert date from seconds to date format.
I have a table containing information about lat, long and time.
table
f_table['dt'] = pd.to_datetime(f_table['dt'])
f_table["dt"]
it results like this:
output
but the output is wrong actually the date is 20160628 but it converted to 1970.
My desired output:
24-April-2014
The unit needs to be nanoseconds, so you need to multiply with 1e9
f_table['dt'] = pd.to_datetime(f_table['dt'] * 1e9)
This should work.
#Split your string to extract timestamp, I am assuming a single space between each float
op = "28.359062 69.693673 5.204486e+08"
ts = float(op.split()[2])
from datetime import datetime
#Timestamp to datetime object
dt = datetime.fromtimestamp(ts)
#Datetime object to string
dt_str = dt.strftime('%m-%B-%Y')
print(dt_str)
#06-June-1986

Resources