I have a column which is a date in the following format : 2018-04-28.
I try to convert it in Impala but it seems there is no way it lets me operate on it at all. I know it doesn't support date formats but it won't even let me convert it.
I've tried to use things like the following but nothing works :
SELECT unix_timestamp(cast(t1.`date` as string),'yyyy-MM-dd') FROM example
Even cast(t1.dateas string) won't work. I always get the same error :
AnalysisException: Unsupported type 'DATE' in 't1.`date`'.
Would anyone know a way to convert this ?
Thanks,
First use DATE_FORMAT to convert datetime
if you use impala please set your date-column type to timestamp impala support timestamp much well than date . the time stamp String format should be like yyyy-mm-dd hh:mm:ss
I tried your SQL it works well when change date to timeStamp
the date type might be a bug in impala.
Related
I have a requirement to fetch the quarter from a string field using hive version less than <1.3 so cant use quarter feature
sample data format in string field say abc--20210415
I tried doing it by using
CONCAT('T',(INT((MONTH(from_unixtime(unix_timestamp(abc,'yyyymmdd')))-1)/3)+1))
but its showing T1 instead it should show T2 (since april comes in second quarter 20210415) but this expression does not convert string to date before calculating quarter
So, i converted it to date using casting
cast(to_date(from_unixtime(unix_timestamp(date, 'yyyymmdd'))) as date)
it converted but output was 2021-01-15 and the same was converted to quarter using
CONCAT('T',(INT((MONTH(cast(to_date(from_unixtime(unix_timestamp(qc_creation_date, 'yyyymmdd'))) as date))-1)/3)+1))
Is there a way around to resolve this issue where in for ex: string 20210415 should give the output as T2 .Kindly help
Regards
i got the solution
CONCAT('Q',(INT((MONTH(cast(to_date(from_unixtime(unix_timestamp(abc, 'yyyyMMdd'))) as date))-1)/3)+1))
Thanks
Your date format yyyyMMdd can be efficiently converted to Hive format yyyy-MM-dd using regexp_replace function instead of unix_timestamp+from_unixtime functions which are using SimpleDateFormat. Bug in your query is wrong format template, it should be 'yyyyMMdd', see SimpleDateFormat for reference. But regexp_replace is much simpler, use unix_timestamp+from_unixtime only if not possible to do the same using regexp_replace.
Do not need to cast it to Date, string works fine.
For Hive < 1.3.0 extract month and use ceil(month/3.0):
select ceil(month(regexp_replace('20210415','(\\d{4})(\\d{2})(\\d{2})','$1-$2-$3'))/3.0) -- returns 2
Even simpler method:
select ceil(regexp_extract('20210415','^(\\d{4})(\\d{2})',2)/3) --returns 2
And since Hive 1.3.0 there is quarter(date/timestamp/string) function.
Demo:
quarter(regexp_replace('20210415','(\\d{4})(\\d{2})(\\d{2})','$1-$2-$3')); --Returns 2
Reading from Oracle database using python sqlalchemy - for a "Date" field stored as "varchar2" in the source .
Sample : Test_col format in Oracle "2019-12-20". This is stored as varchar2 in the source DB.
FYI: Converting "varchar2 to date" in python:
var_1: datetime.strptime('2019-12-20','%Y-%m-%d').date()
Passing var_1 to below sql which finally looks like below:
select *
FROM stg_test_table
WHERE to_date(Test_col,'YYYY-MM-DD') = '2019-12-20'
This is running fine if i run from Oracle developer but running same from Python code, gives below error:
sqlalchemy.exc.DatabaseError: (cx_Oracle.DatabaseError)
ORA-01861: literal does not match format string
Tried multiple date combinations but none of them seems to be working while running from Python.
Can this be linked to cx_oracle version issue?
That's what happens when dates are stored as strings. Although most "dates" have correct format, the one you predicted: YYYY-MM-DD, not all of them do.
For example, if someone entered today's date as 08042020 and you applied the mentioned format mask, you'd get
SQL> select to_date('08042020', 'yyyy-mm-dd') from dual;
select to_date('08042020', 'yyyy-mm-dd') from dual
*
ERROR at line 1:
ORA-01861: literal does not match format string
SQL>
What to do? Expect (a lot?) of pain to fix data. Hopefully, it'll taught you (and the others involved in this) to store dates into DATE columns.
You could, for example, "intercept" invalid formats by using regular expressions (expecting 4 digits - 2 digits - 2 digits), but it won't help with values as 2020-34-87 because it fits the format, but is nonsense.
Or, you could loop through those values (which is row-by-row meaning slow-by-slow) and discard values for which TO_DATE fails.
You could try different format masks, e.g. YYYYMMDD, but - as of 20200408, what is 04 and what is 08? Both can be days or months.
As I said, there's no easy way out of it.
ok so i was able to resolve this, key thing here is to pass the format your DB is expecting, you can check this by:
select value from v$nls_parameters where parameter = 'NLS_DATE_FORMAT';
Considering you are passing correct format, covert using usual TO_DATE conversions and this should work.
I am trying to convert date from '2019-12-12' to '2019/12/12' in my mapping dataflow.
But i cannot find dataflow expression which can convert to this format.
I want a function similar to formatDateTime() which is available in datafactory expression and not in dataflow expression.
Tried toDate() ->doesnt take yyyy/MM/dd
Tried toTimestamp() -> doesnt take yyyy/MM/dd
Your first conversion results in a timestamp which doesn't have a format. To output with your desired format use the below which wraps your code with an additional toString() to format as desired.
toString(toTimestamp(toString(byName('PostedTransactionDate')), 'yyyy-MM-dd') , 'yyyy/MM/dd')
You could using this expression:
toString(toDate('2019-12-12'), 'yyyy/MM/dd')
Result:
Hope this helps
I used this approach to get a date column format, Used derived column and used this the expression.
coalesce(toDate(YourDate,'MM/dd/yyyy'),toDate(YourDate,'yyyy/MM/dd'),toDate(YourDate,'dd/MM/yyyy'),toDate(YourDate,'MMddyyyy'),toDate(YourDate,'yyyyddMM'),toDate(YourDate,'MMddyyyy'),toDate(YourDate,'yyyyMMdd'))
I hope it works for you as well, Cheers
Date stored as a string in Access. I'm not able to apply the suggested solutions such as DateValue and Cdate. I tried to reformat to add a delimiter but it gives me a mismatched error. The data is stored as such: "201704100705"
You need to cast it to a datetime value first:
DateSerial(Left(MyField,4), Mid(MyField,5,2), Mid(MyField,8,2)) + TimeSerial(Mid(MyField,10,2), Right(MyField,2))
After that, you can use the usual date/time functions.
I have a timestamp (1394475248) when I translates using =(((Column/60)/60)/24)+DATE(1970,1,1) and the date format gives the timestamp (YYYY-MM-DD hh:mm:ss ) but I have to store it in a timestamp which looks like this "2014-03-10T23:11:09.000+0100" as all the previous data was in this format and changing would require a great deal of work/time.
Can someone tell what is the timestamp format and how do i convert the number to the exact format? many thanks
W
Try using TEXT function to convert to s strng in required format, e.g. with your original timestamp in A2 use this formula to convert and format
=TEXT(A2/86400+DATE(1970,1,1),"YYYY-MM-DDThh:mm:ss.000")&"+0100"