I am trying to find a way to bulk load a csv file into postgresql. However, the data has datetime column with the format of "YYYY-MM-DD HH24:MI:SS". I couldn't find any documentation on how to bulk load date column using psycopg2 package in python 3.x. Can I get some help on this? Thanks in advance.
I am able to load the data using the below code:
cur.copy_from(dataIterator,'cmodm.patient_visit',sep=chr(31),size=8192,null='') conn.commit()
However, only the date part got loaded in the table. The time part was initialized:
2017-04-13 00:00:00 2017-04-13 00:00:00 2017-04-12 00:00:00
After discussing with #Belayer, it was concluded that copy_from takes the timestamp input value in the format 'YYYY-MM-DD HH:MI:SS'. If the source has some other format, then that needs to be converted to the desired format mentioned before in this response before feeding it into copy_from.
Related
I am trying to convert timestamp string 2021-09-29T05:32:48.000+0000 to time stamp using moment, but I am getting NaN as output. I am using this command to convert
(moment("2021-09-29T05:32:48.000+0000"))
Any suggestion on how I can handle the timestamp of this format in the node?
Currently having a field with the below date format.
3/2/2021 18:48
I need to convert it to 2021-03-02. I tried taking a substring and converting to date format. But it is not providing the desired output. Any suggestions will be helpful
Below if you are using spark SQL:
from_unixtime(unix_timestamp('3/2/2021 18:48', 'M/d/yyyy'), 'yyyy-MM-dd')
Same functions are available in Dataframe API as well:
https://spark.apache.org/docs/2.4.0/api/sql/index.html#from_unixtime
https://spark.apache.org/docs/2.4.0/api/sql/index.html#unix_timestamp
I'm able to get the current timestamp using now() in NIFI. But I want to get the yesterday's timestamp in the below format and could not achieve it. what expression should be given for getting yesterday's timestamp at NIFI processor.
Egg: "superDate":"2021-03-07 23:59:00"
Thanks in advance!
${now():toNumber():minus(86400000):format('yyyy-MM-dd HH:mm:ss')}
I'm trying to access Oracle table using cx_oracle module and convert that as a dataframe, every thing is fine except couple of date columns has date format like "01-JAN-01" Python considering it as datetime.datetime(1,1,1,0,0) and after creating dataframe it's showing as 0001-01-01 00:00:00. I am expecting output as 2001-01-01 00:00:00. Please help me on this. Thanks in advance.
You have a couple of choices. You could
* Retrieve it from the Oracle database with [read_sql](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_sql.html specifying the date in a format (TO_CHAR) more appropriate for the default date format of pandas
* Retrieve it from the database as a string (as above) and then convert it into a date in the pandas framework.
I have a dataframe in python where date columns in datetime64[ns] data type. Now I am trying to write this data frame to redshift. I am getting following stl_load_errors:
Invalid date format - length must be 10 or more
All my dates are 2016-10-21 format, thus have length of 10. More over, I have ensured that no row has any messed up format like 2016-1-8 where it can have only 8 character. So the error is not making sense.
Any one faced similar error while writing data to redshift ? Any explanation ?
Note:
Here's some context. I am running the python script from EC2. This script writes the data in json format to S3 bucket and then this json is uploaded to an empty redshift table. The redshift table describes the date columns as 'date' format. I know there's another way which uses boto3/copy but for now I am stuck to this method.