Databricks SQL: Why is subquery in left join causing error msg - databricks

Question

I am attempting to use a subquery in a left join condition, but am getting an error message that reads: "Error in SQL statement: AnalysisException: Table or view not found: TableD;" and points to the FROM TableD D2 statement in my subquery.
SELECT D1.Code, D1.Description, C.InstanceKey
FROM TableA A
INNER JOIN TableB B
ON A.Key = B.Key
INNER JOIN TableC C
ON B.DetailKey = C.DetailKey
LEFT JOIN TableD D1
ON C.InstanceKey = D1.InstanceKey
AND D1.RankCnt = (SELECT MIN(D2.RankCnt)
FROM TableD D2
WHERE C.InstanceKey = D2.InstanceKey);
If I remove the subquery and hardcode D1.RankCnt = [anyValidRankCnt], the query runs without issue.
This question has also been posted on the Databricks Community Forum at https://forums.databricks.com/questions/14588/why-is-subquery-in-left-join-causing-error-msg.html.

Answer 1

I'm not sure if that particular type of correlated subquery is supported in Spark at this time, although I was able to rewrite it in a couple of different ways, including using ROW_NUMBER. Please check these queries are semantically equivalent to yours with your data:
%sql
-- Rewrite 1: CTE
WITH cte AS
(
SELECT D1.Code, D1.Description, C.InstanceKey, ROW_NUMBER() OVER ( PARTITION BY c.InstanceKey ORDER BY D1.RankCnt ) xrank
FROM TableA A
INNER JOIN TableB B
ON A.Key = B.Key
INNER JOIN TableC C
ON B.DetailKey = C.DetailKey
LEFT JOIN TableD D1
ON C.InstanceKey = D1.InstanceKey
)
SELECT *
FROM cte
WHERE xrank = 1
-- Rewrite 2: subquery
SELECT x.Code, x.Description, C.InstanceKey
FROM TableA A
INNER JOIN TableB B
ON A.Key = B.Key
INNER JOIN TableC C
ON B.DetailKey = C.DetailKey
LEFT JOIN
(
SELECT D1.InstanceKey, D1.Code, D1.Description, D1.RankCnt
FROM TableD D1
INNER JOIN
(
SELECT InstanceKey, MIN(RankCnt) RankCnt
FROM TableD
GROUP BY InstanceKey
) D2 ON D1.InstanceKey = D2.InstanceKey
AND D1.RankCnt = D2.RankCnt
) x
ON c.InstanceKey = x.InstanceKey;
-- Rewrite 3: UNION ALL
SELECT D1.Code, D1.Description, C.InstanceKey
FROM TableA A
INNER JOIN TableB B
ON A.Key = B.Key
INNER JOIN TableC C
ON B.DetailKey = C.DetailKey
INNER JOIN TableD D1
ON C.InstanceKey = D1.InstanceKey
INNER JOIN
(
SELECT D2.InstanceKey, MIN(D2.RankCnt) RankCnt
FROM TableD D2
GROUP BY D2.InstanceKey
) x ON C.InstanceKey = x.InstanceKey
AND D1.RankCnt = x.RankCnt
UNION ALL
SELECT NULL AS Code, NULL AS Description, C.InstanceKey
FROM TableA A
INNER JOIN TableB B
ON A.Key = B.Key
INNER JOIN TableC C
ON B.DetailKey = C.DetailKey
WHERE NOT EXISTS
(
SELECT *
FROM TableD D1
WHERE C.InstanceKey = D1.InstanceKey
);