Spark SQL get lineage of parent tables - apache-spark

Question

I have created 3 temporary tables from loading from case class. And let's call it
ABC
BCD
EFG
Then I proceed to created 3 more tables by performing join.
ABC join with BCD which gives XYZ
XYZ join with EFG which gives LMN
LMN join with ABC which gives PQR
Does spark allow me in some way to see the lineage of dependent registered temporary table?? How can I extract the information where it knows LMN depends on XYZ and PQR depends on LMN, and use it programmatically to build a lineage tree (without all the plan information).
etc:
|-PQR
|-LMN
|-XYZ
|- ABC
|-EFG
|-ABC
Thanks.

Answer 1

There is an open source tool you can use to visualize the lineage: https://github.com/AbsaOSS/spline
It harvests the lineages at the runtime and captures it to display as a graph.

Answer 2

Let's take this SQL for example, which exactly creates the table/view as you mentioned.
CREATE TABLE ABC(abc_column1 int, abc_column2 string);
CREATE TABLE BCD(bcd_column1 int, bcd_column2 string);
CREATE TABLE EFG(efg_column1 int, efg_column2 string);
create view XYZ(xyz_column1,xyz_column2)
as select abc.abc_column1, bcd.bcd_column2
from abc
left join bcd on abc.abc_column1 = bcd.bcd_column1;
create view LMN (lmn_column1,lmn_column2)
as select xyz.xyz_column1, efg.efg_column2
from xyz
left join efg on xyz.xyz_column1 = efg.efg_column1;
create view PQR (pqr_column1, pqr_column2)
as select lmn.lmn_column1, abc.abc_column2
from LMN
left join ABC on abc.abc_column1 = lmn.lmn_column1;
And this is the data lineage that you are asking:
You may try your own SQL to get the data lineage here:
https://sqlflow.gudusoft.com/#/