I am using sql as language for my notebook in databricks.
Want to get the day of week from the date given.
For doing this i used to_char(date,'fmday'). Getting error as function is not registered as temporary or permanant in databricks. Is there a way to get the name of day by other means.
Date is in format yyyymmdd
You are getting that error because to_char is not a SparkSQL function. You can see the list of functions in the ScalaDocs here: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html
If your date is a DateType, you can do dayofweek(date) in SparkSQL.
get the name of the day
Being as you want to get the name of the day, you can use the date_format function with the argument 'EEEE' to get the day name, eg Monday. If you want to pass in an integer (eg numbers between 1 and 7) then you could just code a CASE statement, something like:
%sql
SELECT
dayofweek( CAST( '2018-12-31' AS DATE ) ) AS d,
date_format( CAST( '2018-12-31' AS DATE ), 'EEEE' ) AS dayname,
CASE dayofweek( CAST( '2018-12-31' AS DATE ) )
WHEN 1 THEN 'Monday'
WHEN 2 THEN 'Tuesday'
WHEN 3 THEN 'Wednesday'
WHEN 4 THEN 'Thursday'
WHEN 5 THEN 'Friday'
WHEN 6 THEN 'Saturday'
WHEN 7 THEN 'Sunday'
ELSE 'Unknown'
END AS caseTest
NB I have coded the CASE to start the week from Day 1 - Monday, which is different to the dayofweek default; this might be one reason to do that, ie you want a different default.
My Results:
I got a way to get the name of day of week as below
date_format(to_date('20170821','yyyyMMdd'),'EEEE')
Now i want to pass a column of integer datatype, but when i pass it to query getting null as output. Could someone please help
Related
Can someone show me how to extract the year from the String date in databricks SQL.
I am based in the UK and our date format is normally as follows:
dd/mm/yyyy
The field containing the dates is set as StringType()
I am trying to extract the year from the string as follows:
select year(cast(financials_0_accountsDate as Date)) from `financiallimited_csv`
I'm using the following the code to extract the quarter
select quarter(cast(financials_0_accountsDate as Date)) from `financiallimited_csv`
However, both result in NULL values.
Any thoughts on how to extract the year and quarter from dates with StringType() dd/mm/yyyy?
The table looks like the following:
Could you try the to_date function?
select year(to_date(financials_0_accountsDate, 'dd/MM/yyyy')) from `financiallimited_csv`
so our DBA's setup our hive table with the date column as the partition column, but as a "string" YYYYMMDD format.
How can I WHERE filter this "date" column for something like last 30 days?
Please use date_format to format systemdate - 30 days into YYYYMMDD and then compare with your partition column. Please note to use partition column as is so hive can choose correct partitions.
When you want to pick previous 30th days data -
select *
from mytable
where partition_col = date_format( current_date() - interval '30' days, 'yyyyMMdd')
If you want all data since last 30 days -
select *
from mytable
wherecast(partition_col as INT) >= cast(date_format( current_date() - interval '30' days, 'yyyyMMdd') as INT)
casting shouldnt impact partition benefits but you need to check the performance before using it. Please get back in such scenario.
I have a table with the following (with other fields removed)
CREATE TABLE if NOT EXISTS request_audit (
user_id text,
request_body text,
lookup_timestamp TIMESTAMP
PRIMARY KEY ((user_id), lookup_timestamp)
) WITH CLUSTERING ORDER BY ( lookup_timestamp DESC);
I create a record with the following
INSERT INTO request_audit (user_id, lookup_timestamp, request_body) VALUES (?, ?, toTimestamp(now()))
I am trying to retrieve all rows within the last 24 hours, but I am having trouble with the timestamp,
I have tried
SELECT * from request_audit WHERE user_id = '1234' AND lookup_timestamp > toTimestamp(now() - "1 day" )
and various other ways of trying to take a day away from the query.
Cassandra has a very limited date operation support. What you need is a custom function to do date math calculation.
Inspired from here.
How to get Last 6 Month data comparing with timestamp column using cassandra query?
you can write a UDF (user defined function) to date operation.
CREATE FUNCTION dateAdd(date timestamp, day int)
CALLED ON NULL INPUT
RETURNS timestamp
LANGUAGE java
AS
$$java.util.Calendar c = java.util.Calendar.getInstance();
c.setTime(date);
c.add(java.util.Calendar.DAY_OF_MONTH, day);
return c.getTime();$$ ;
remember that you would have to enable UDF in config. Cassandra.yml. Hope that is possible.
enable_user_defined_functions: true
once done this query works perfectly.
SELECT * from request_audit WHERE user_id = '1234' AND lookup_timestamp > dateAdd(dateof(now()), -1)
You couldn't do it directly from CQL, as it doesn't support this kind of expressions. If you're running this query from cqlsh, then you can try to substitute the desired date with something like this:
date --date='-1 day' '+%F %T%z'
and execute this query.
If you're invoking this from your program, just use corresponding date/time library to get date corresponding -1 day, but this depends on the language that you're using.
I'm trying to cast a CHARACTER VARYING column to a DATE but I need a date format like this : DD/MM/YYYY. I use the following SQL query :
ALTER TABLE test
ALTER COLUMN date TYPE DATE using to_date(date, 'DD/MM/YYYY');
The result is a date like this : YYYY-MM-DD.
How can I get the DD/MM/YYYYformat ?
Thanks a lot in advance !
Thomas
A DATE column does not have a format. You cannot specify a format for it.
You can use DateStyle to control how PostgreSQL emits dates, but it's global and a bit limited.
Instead, you should use to_char to format the date when you query it, or format it in the client application. Like:
SELECT to_char("date", 'DD/MM/YYYY') FROM mytable;
e.g.
regress=> SELECT to_char(DATE '2014-04-01', 'DD/MM/YYYY');
to_char
------------
01/04/2014
(1 row)
https://www.postgresql.org/docs/8.4/functions-formatting.html
SELECT to_char(date_field, 'DD/MM/YYYY')
FROM table
The documentation says
The output format of the date/time types can be set to one of the four
styles ISO 8601, SQL (Ingres), traditional POSTGRES (Unix date
format), or German. The default is the ISO format.
So this particular format can be controlled with postgres date time output, eg:
t=# select now();
now
-------------------------------
2017-11-29 09:15:25.348342+00
(1 row)
t=# set datestyle to DMY, SQL;
SET
t=# select now();
now
-------------------------------
29/11/2017 09:15:31.28477 UTC
(1 row)
t=# select now()::date;
now
------------
29/11/2017
(1 row)
Mind that as #Craig mentioned in his answer, changing datestyle will also (and in first turn) change the way postgres parses date.
In case you need to convert the returned date of a select statement to a specific format you may use the following:
select to_char(DATE (*date_you_want_to_select*)::date, 'DD/MM/YYYY') as "Formated Date"
Depends on which type you require as output, but here are 2 quick examples based on an intervals:
SELECT (now() - interval '15 DAY')::date AS order_date -> 2021-07-29
SELECT to_char(now() - interval '15 DAY', 'YYYY-MM-DD') -> 2021-07-29
Let's say your date column is order_date:
SELECT (
RIGHT(order_date, 4)
|| '-'
|| SUBSTRING(order_date, 4, 2)
|| '-'
|| LEFT(order_date, 2)
)::DATE
FROM test
OR
SELECT CAST(
RIGHT(order_date, 4)
|| '-'
|| SUBSTRING(order_date, 4, 2)
|| '-'
|| LEFT(order_date, 2)
AS DATE )
FROM test
What I need is a data item expression which outputs the starting date of the current quarter of last year. Finding the current year is easy, as is subtracting 1 year from that date. But beyond that I get stuck.
I currently have an ugly if expression for each quarter like:
if (extract(month,current_date) in (10,12,12))
then ((extract(year,_add_years (current_date,-1))||'-10-01'))
But no matter what I do I can't concatonate the year and date into string I can convert to a date object. The above code gives the error:
The operation "add" is invalid for the following combination of data types: "integer" and "character"
Trying to cast the integer as a character using cast() I get this error. I also get this error when trying to turn a character array into a date:
The operation "condexp" is invalid for the following combination of data types: "character" and "integer"
Trying to use SQL Server specific functions (it is a SQL Server database) just gives me an error that those functions are unavailable for local processing, so I can't seem to use SS date arithmatic, and I can't find anything particularly applicable in Cognos' built in date functions.
How can I manipulate a date to add a year to a known day/month combination and use that as a date object?
I'm using cognos 10.1.1, but it would work.
You can get the value by using the following code:
_make_timestamp(
extract(year, _add_years(current_date, -1)),
(floor((extract(month,current_date)-1)/3)*3+1),
01
)
The following is the simple way to get the starting month of the quarter.
(floor((extract(month,current_date)-1)/3)*3+1),
and you can convert timestamp to date or string.
cast(
_make_timestamp(
extract(year, _add_years(current_date, -1)),
(floor((extract(month,current_date)-1)/3)*3+1),
01
),
date
)
,
cast(
cast(
_make_timestamp(
extract(year, _add_years(current_date, -1)),
(floor((extract(month,current_date)-1)/3)*3+1),
01
),
date),
varchar(10)
)
plus, you can use extract function when you get a day value.
extract(day, _first_of_month (current_date))
I would go with SQLSERVER built-in functions tht exists in Cognos.
Here is an expression that works for me:
DATEADD({month},(3)*((DATEPART({quarter},[FullDateAlternateKey]))-1),
DATEADD({YEAR}, DATEDIFF({YEAR}, 0, dateadd({year},-1,[FullDateAlternateKey])), 0))
The FullDateAlternateKey field is my date field (from ADVERTUREWORKS DW DB).
If you still have problems, try to isolate the problem by trying this expression on new simple list report.