I want to replicate sql OUTER APPLY functinally in pyspark - apache-spark

Question

I want to replicate what "OUTER APPLY" function in pyspark.
Here are my example data frames
## Deparment table
data = [
(1,'Engineering'),
(2,'Administration'),
(3,'Sales'),
(4,'Marketing'),
(5,'Finance')
]
schema = StructType([
StructField('DepartmentID', IntegerType(), True),
StructField('Name', StringType(), True)
])
Department = spark.createDataFrame(data=data, schema =schema)
Department.show()
+------------+--------------+
|DepartmentID| Name|
+------------+--------------+
| 1| Engineering|
| 2|Administration|
| 3| Sales|
| 4| Marketing|
| 5| Finance|
+------------+--------------+
## Employee table
data = [
(1,'Orlando', 'Gee', 1),
(2,'Keith', 'Harris', 2),
(3,'Donna', 'Carreras', 3),
(4,'Janet', 'Gates', 3),
]
schema = StructType([
StructField('EmployeeID', IntegerType(), True),
StructField('FirstName', StringType(), True),
StructField('LastName', StringType(), True),
StructField('DepartmentID', IntegerType(), True),
])
Employee = spark.createDataFrame(data=data, schema =schema)
Employee.show()
+----------+---------+--------+------------+
|EmployeeID|FirstName|LastName|DepartmentID|
+----------+---------+--------+------------+
| 1| Orlando| Gee| 1|
| 2| Keith| Harris| 2|
| 3| Donna|Carreras| 3|
| 4| Janet| Gates| 3|
+----------+---------+--------+------------+
I tried creating a temp table and use a spark SQL command to query as we normally do on temporary tables... but I keep getting
`[PARSE_SYNTAX_ERROR] Syntax error at or near 'OUTER'(line 3, pos 2)
== SQL ==
SELECT * FROM Department D
OUTER APPLY
--^^^
(
SELECT * FROM Employee E
WHERE E.DepartmentID = D.DepartmentID
) A
`
error. Any help is appreciated.
Employee.createOrReplaceTempView("Employee")
Department.createOrReplaceTempView("Department")
sql_query = """
SELECT * FROM Department D
OUTER APPLY
(
SELECT * FROM Employee E
WHERE E.DepartmentID = D.DepartmentID
) A
"""
result_df = sqlContext.sql(sql_query)

Answer 1

OUTER APPLY is not an option in the Spark SQL Syntax
However an OUTER APPLY command will produce the same results as an LEFT OUTER JOIN.
LEFT OUTER JOIN is an option in the Spark SQL Syntax.
Using LEFT OUTER JOIN for your example would look like this in the Spark SQL Syntax,
sql_query = """
SELECT * FROM Department D
LEFT OUTER JOIN Employee E ON E.DepartmentID = D.DepartmentID
"""
Using LEFT OUTER JOIN for your example would look like this in the PySpark Syntax,
Department.join(Employee, Employee.DepartmentID == Department.DepartmentID, "left_outer") \
.show(truncate=False)