I have a requirement to fetch the quarter from a string field using hive version less than <1.3 so cant use quarter feature
sample data format in string field say abc--20210415
I tried doing it by using
CONCAT('T',(INT((MONTH(from_unixtime(unix_timestamp(abc,'yyyymmdd')))-1)/3)+1))
but its showing T1 instead it should show T2 (since april comes in second quarter 20210415) but this expression does not convert string to date before calculating quarter
So, i converted it to date using casting
cast(to_date(from_unixtime(unix_timestamp(date, 'yyyymmdd'))) as date)
it converted but output was 2021-01-15 and the same was converted to quarter using
CONCAT('T',(INT((MONTH(cast(to_date(from_unixtime(unix_timestamp(qc_creation_date, 'yyyymmdd'))) as date))-1)/3)+1))
Is there a way around to resolve this issue where in for ex: string 20210415 should give the output as T2 .Kindly help
Regards
i got the solution
CONCAT('Q',(INT((MONTH(cast(to_date(from_unixtime(unix_timestamp(abc, 'yyyyMMdd'))) as date))-1)/3)+1))
Thanks
Your date format yyyyMMdd can be efficiently converted to Hive format yyyy-MM-dd using regexp_replace function instead of unix_timestamp+from_unixtime functions which are using SimpleDateFormat. Bug in your query is wrong format template, it should be 'yyyyMMdd', see SimpleDateFormat for reference. But regexp_replace is much simpler, use unix_timestamp+from_unixtime only if not possible to do the same using regexp_replace.
Do not need to cast it to Date, string works fine.
For Hive < 1.3.0 extract month and use ceil(month/3.0):
select ceil(month(regexp_replace('20210415','(\\d{4})(\\d{2})(\\d{2})','$1-$2-$3'))/3.0) -- returns 2
Even simpler method:
select ceil(regexp_extract('20210415','^(\\d{4})(\\d{2})',2)/3) --returns 2
And since Hive 1.3.0 there is quarter(date/timestamp/string) function.
Demo:
quarter(regexp_replace('20210415','(\\d{4})(\\d{2})(\\d{2})','$1-$2-$3')); --Returns 2
Related
I need to convert the YYYYMM to YYYY-MM-01 using azure data factory pipeline expression.
I tried the below expression but it giving me error that date value should follow the ISO 8601 format.
#formatDateTime(concat('202301','01'),'YYYY-MM-DD')
It should return in this format '2023-01-01'.
Thanks
I used multiple substring to get this as shown below, if any better answer please let me know.
#concat(substring(concat('202301','01'),0,4 ),'-',substring(concat('202301','01'),4,2),'-',substring(concat('202301','01'),6,2))
You can supply a format string including numbers to formatDateTime, for example:
#formatDateTime(utcnow(), 'yyyy-MM-01')
NB You have the case of your argument wrong, it should be lower-case y for Year, upper-case M for month and lower-case d for Day.
Instead of concatenating 202301 with 01 (three times) and then taking substring, you can use the below expression to achieve the same.
#concat(substring('202301',0,4 ),'-',substring('202301',4,2),'-','01')
Reading from Oracle database using python sqlalchemy - for a "Date" field stored as "varchar2" in the source .
Sample : Test_col format in Oracle "2019-12-20". This is stored as varchar2 in the source DB.
FYI: Converting "varchar2 to date" in python:
var_1: datetime.strptime('2019-12-20','%Y-%m-%d').date()
Passing var_1 to below sql which finally looks like below:
select *
FROM stg_test_table
WHERE to_date(Test_col,'YYYY-MM-DD') = '2019-12-20'
This is running fine if i run from Oracle developer but running same from Python code, gives below error:
sqlalchemy.exc.DatabaseError: (cx_Oracle.DatabaseError)
ORA-01861: literal does not match format string
Tried multiple date combinations but none of them seems to be working while running from Python.
Can this be linked to cx_oracle version issue?
That's what happens when dates are stored as strings. Although most "dates" have correct format, the one you predicted: YYYY-MM-DD, not all of them do.
For example, if someone entered today's date as 08042020 and you applied the mentioned format mask, you'd get
SQL> select to_date('08042020', 'yyyy-mm-dd') from dual;
select to_date('08042020', 'yyyy-mm-dd') from dual
*
ERROR at line 1:
ORA-01861: literal does not match format string
SQL>
What to do? Expect (a lot?) of pain to fix data. Hopefully, it'll taught you (and the others involved in this) to store dates into DATE columns.
You could, for example, "intercept" invalid formats by using regular expressions (expecting 4 digits - 2 digits - 2 digits), but it won't help with values as 2020-34-87 because it fits the format, but is nonsense.
Or, you could loop through those values (which is row-by-row meaning slow-by-slow) and discard values for which TO_DATE fails.
You could try different format masks, e.g. YYYYMMDD, but - as of 20200408, what is 04 and what is 08? Both can be days or months.
As I said, there's no easy way out of it.
ok so i was able to resolve this, key thing here is to pass the format your DB is expecting, you can check this by:
select value from v$nls_parameters where parameter = 'NLS_DATE_FORMAT';
Considering you are passing correct format, covert using usual TO_DATE conversions and this should work.
I am trying to convert date from '2019-12-12' to '2019/12/12' in my mapping dataflow.
But i cannot find dataflow expression which can convert to this format.
I want a function similar to formatDateTime() which is available in datafactory expression and not in dataflow expression.
Tried toDate() ->doesnt take yyyy/MM/dd
Tried toTimestamp() -> doesnt take yyyy/MM/dd
Your first conversion results in a timestamp which doesn't have a format. To output with your desired format use the below which wraps your code with an additional toString() to format as desired.
toString(toTimestamp(toString(byName('PostedTransactionDate')), 'yyyy-MM-dd') , 'yyyy/MM/dd')
You could using this expression:
toString(toDate('2019-12-12'), 'yyyy/MM/dd')
Result:
Hope this helps
I used this approach to get a date column format, Used derived column and used this the expression.
coalesce(toDate(YourDate,'MM/dd/yyyy'),toDate(YourDate,'yyyy/MM/dd'),toDate(YourDate,'dd/MM/yyyy'),toDate(YourDate,'MMddyyyy'),toDate(YourDate,'yyyyddMM'),toDate(YourDate,'MMddyyyy'),toDate(YourDate,'yyyyMMdd'))
I hope it works for you as well, Cheers
I have a column which is a date in the following format : 2018-04-28.
I try to convert it in Impala but it seems there is no way it lets me operate on it at all. I know it doesn't support date formats but it won't even let me convert it.
I've tried to use things like the following but nothing works :
SELECT unix_timestamp(cast(t1.`date` as string),'yyyy-MM-dd') FROM example
Even cast(t1.dateas string) won't work. I always get the same error :
AnalysisException: Unsupported type 'DATE' in 't1.`date`'.
Would anyone know a way to convert this ?
Thanks,
First use DATE_FORMAT to convert datetime
if you use impala please set your date-column type to timestamp impala support timestamp much well than date . the time stamp String format should be like yyyy-mm-dd hh:mm:ss
I tried your SQL it works well when change date to timeStamp
the date type might be a bug in impala.
I have a one query explaind below....,
1.My DB2 table column is in INTEGER format (it has the date value in Julian date format - YYYYDDD)
2. After unload this date into a flat file, the result is in hexa decimal format.(4 bytes occupaid)
3. I have a requirement to compare this date with (Current Julian date+7days) and write the corresponding recods into a File.
4. So, when I am comparing the Hexa decimal Input date with Current Julian date+7days (DATE3P+7), I am getting MAxcc=0 but I am not getting required output.
5. What I identified the problem is that the input date should also be in packed decimal format.
Can anybody please let me know how to convert the table value in Packed decimal while unloading or please suggest alternate way to get it.
Ex:- 1. Unload Julian date (YYYYDDD) [After unload this will be in hexa decimal format].
2. compare with Current Julian date +7 days.
SORT FIELDS=COPY
OUTFIL FILES=1,
INCLUDE=(1,4,PD,LT,DATE3P+7)
OUTFIL FILES=2,SAVE
Thanks in Advance, Rajasekhar Jannu.
Believe these alternate ways, work...
Instead of getting the date in hex type, unload it as date in YYYYDDD
format and the DATE3 will give you the julian date in YYYYDDD. Now
these both are in compatible format to compare and condition.
There are many scalar functions of DB2 to provide Julian date and as
well date in numbers. I believe DIGITS on JULIAN scalar function
would do...
Refer this DB2 Scalar Functions
If you couldnt unload the table in the way described above, another
way would be manipulating the unload file, so that date from hex
format needs to be converted into YYYYDDD numeric format using
DFSORT. Then DATE3 will help u to get one week old data. DFSORT
reference has been already shared.