Spark SQL weekofyear function - apache-spark

Question

I am using spark sql's weekofyear function to calculate the week number for the given date.
I am using the following code,
test("udf - week number of the year") {
val spark = SparkSession.builder().master("local").appName("udf - week number of the year").getOrCreate()
import spark.implicits._
val data1 = Seq("20220101", "20220102", "20220103", "20220104", "20220105", "20220106", "20220107", "20220108", "20220109", "20220110", "20220111", "20220112")
data1.toDF("day").createOrReplaceTempView("tbl_day")
spark.sql("select day, to_date(day, 'yyyyMMdd') as date, weekofyear(to_date(day, 'yyyyMMdd')) as week_num from tbl_day").show(truncate = false)
/*
+--------+----------+--------+
|day |date |week_num|
+--------+----------+--------+
|20220101|2022-01-01|52 |
|20220102|2022-01-02|52 |
|20220103|2022-01-03|1 |
|20220104|2022-01-04|1 |
|20220105|2022-01-05|1 |
|20220106|2022-01-06|1 |
|20220107|2022-01-07|1 |
|20220108|2022-01-08|1 |
|20220109|2022-01-09|1 |
|20220110|2022-01-10|2 |
|20220111|2022-01-11|2 |
|20220112|2022-01-12|2 |
+--------+----------+--------+
*/
spark.stop
}
I am surprised to find that 20220101's week number is 52, but it is the first day of 2022, so that it should be 1.
I instigate the source code of weekofyear and find:
It is using the following code to create the Calendar instance so that it gives the result above
#transient private lazy val c = {
val c = Calendar.getInstance(DateTimeUtils.getTimeZone("UTC"))
c.setFirstDayOfWeek(Calendar.MONDAY)
c.setMinimalDaysInFirstWeek(4)
c
}
I would ask why spark sql treat the first few days of the year in this way.
As a comparison,
I using the following oracle sql to get the week number which gives me 1
select to_number(to_char(to_date('01/01/2022','MM/DD/YYYY'),'WW')) from dual
In hive, the result is the same as spark sql.

Answer 1

I will post my findings here:
Spark SQL and Hive are following ISO-8601 standard to calculate the week number of the year for a given date.
One point to note: Spark SQL internally is using java.util.Calendar API to do the work , java 8' java.time API has been natively supporting ISO-8601 standard,using java.time API, we don't have to do the trick(c.setMinimalDaysInFirstWeek(4))

Answer 2

On Spark 3.0 you can use EXTRACT function. A few examples:
> SELECT extract(YEAR FROM TIMESTAMP '2019-08-12 01:00:00.123456');
2019
> SELECT extract(week FROM timestamp'2019-08-12 01:00:00.123456');
33
> SELECT extract(doy FROM DATE'2019-08-12');
224
> SELECT extract(SECONDS FROM timestamp'2019-10-01 00:00:01.000001');
1.000001
> SELECT extract(days FROM interval 1 year 10 months 5 days);
5
> SELECT extract(seconds FROM interval 5 hours 30 seconds 1 milliseconds 1 microseconds);
30.001001
Documentation here