pyspark rdd map is not calling function - apache-spark

Question

I am trying to do few transformations on my RDD and for that, I am calling a function using map. However, this function is not getting invoked. Someone please let me know what I am doing wrong here?
I can see test function getting invoked but not store_past_info
def store_past_info(row):
print "------------------- store_past_info ------------------------------"
if row["transactiontype"] == "Return":
global prv_transaction_number
prv_transaction_number = row["transnumber"]
global return_occured
return_occured = True
global group_id
group_id.append(row["transnumber"])
if row["transactiontype"] == "Purchase":
if return_occured:
global group_id
group_id.append(prv_transaction_number)
else:
global group_id
group_id.append(row["transnumber"])
print group_id
def test(rdd):
print "------------------- test ------------------------------"
rdd.map(store_past_info).collect()
print group_id
This is how it works in store:
if some item is purchased an id is generated.
if you want to return few items from your purchase, two entries are been made
Return entry with new id for the return of all the products, with org_id as id of your purchase order you want to return
New Purchase entry with the same id as your last purchase id for things you want to keep
Input
Date Type Id org_id
25-03-2018 Purchase 111
25-03-2018 Purchase 112
26-03-2018 Return 113 111
26-03-2018 Purchase 111
Output
I want to add a new column group_id, which will show the same id for Return and Corresponding Purchase happened after return ( customer don't do this purchase, this is how system keeps entry for every return) step 2.1
Date Type Id org_id group_id
25-03-2018 Purchase 111 111
25-03-2018 Purchase 112 112
26-03-2018 Return 113 111 113
26-03-2018 Purchase 111 113

Answer 1

IIUC, I believe you can get your output using DataFrames, a pyspark.sql.Window function, and crossJoin()
First convert your rdd to a DataFrame using
df = rdd.toDF() # you may have to specify the column names
df.show()
#+----------+--------+---+------+
#| Date| Type| Id|org_id|
#+----------+--------+---+------+
#|25-03-2018|Purchase|111| null|
#|25-03-2018|Purchase|112| null|
#|26-03-2018| Return|113| 111|
#|26-03-2018|Purchase|111| null|
#+----------+--------+---+------+
Then we will need to add an Index column to keep track of the order of the rows. We can use pyspark.sql.functions.monotonically_increasing_id(). This will guarantee that the values will be increasing (so they can be ordered), but does not mean that they will be sequential.
import pyspark.sql.functions as f
df = df.withColumn('Index', f.monotonically_increasing_id())
df.show()
#+----------+--------+---+------+-----------+
#| Date| Type| Id|org_id| Index|
#+----------+--------+---+------+-----------+
#|25-03-2018|Purchase|111| null| 8589934592|
#|25-03-2018|Purchase|112| null|17179869184|
#|26-03-2018| Return|113| 111|34359738368|
#|26-03-2018|Purchase|111| null|42949672960|
#+----------+--------+---+------+-----------+
The ordering is important because you want to look for rows that come after a Return.
Next use crossJoin to join the DataFrame to itself.
Since this returns the Cartesian product, we will filter it to just the rows that meet either of the following conditions:
l.Index = r.Index (essentially join a row to itself)
(l.Id = r.org_id) AND (l.Index > r.Index) (an Id is equal to a org_id from an earlier row- this is where the Index column is helpful)
Then we add a column for group_id and set it equal to r.Id if the second condition is met. Otherwise we set this column to None.
df1 = df.alias('l').crossJoin(df.alias('r'))\
.where('(l.Index = r.Index) OR ((l.Id = r.org_id) AND (l.Index > r.Index))')\
.select(
'l.Index',
'l.Date',
'l.Type',
'l.Id',
'l.org_id',
f.when(
(f.col('l.Id') == f.col('r.org_id')) & (f.col('l.Index') > f.col('r.Index')),
f.col('r.Id')
).otherwise(f.lit(None)).alias('group_id')
)
df1.show()
#+-----------+----------+--------+---+------+--------+
#| Index| Date| Type| Id|org_id|group_id|
#+-----------+----------+--------+---+------+--------+
#| 8589934592|25-03-2018|Purchase|111| null| null|
#|17179869184|25-03-2018|Purchase|112| null| null|
#|34359738368|26-03-2018| Return|113| 111| null|
#|42949672960|26-03-2018|Purchase|111| null| 113|
#|42949672960|26-03-2018|Purchase|111| null| null|
#+-----------+----------+--------+---+------+--------+
We are almost there but as you can see there are two things that still need to be done.
We need to eliminate the duplicate row for Index = 42949672960
We need to fill in the group_id for rows where it is null using the value from Id.
For the first step, we will use a Window function to create a temporary column called rowNum. This will be the pyspark.sql.functions.row_number() for each Index ordered by the boolean condition group_id IS NULL.
For the Index values where there are multiple rows, the one where the group_id has already been set will sort first. Thus we just need to select the rows where the rowNum is equal to 1 (row_number() starts at 1, not 0).
After this is done, the second step is trivial- just replace the remaining null values with the value from Id.
from pyspark.sql import Window
w = Window.partitionBy(f.col('Index')).orderBy(f.isnull('group_id'))
df2 = df1.withColumn('rowNum', f.row_number().over(w))\
.where(f.col('rowNum')==1)\
.sort('Index')\
.select(
'Date',
'Type',
'Id',
'org_id',
f.when(
f.isnull('group_id'),
f.col('Id')
).otherwise(f.col('group_id')).alias('group_id')
)
df2.show()
#+----------+--------+---+------+--------+
#| Date| Type| Id|org_id|group_id|
#+----------+--------+---+------+--------+
#|25-03-2018|Purchase|111| null| 111|
#|25-03-2018|Purchase|112| null| 112|
#|26-03-2018| Return|113| 111| 113|
#|26-03-2018|Purchase|111| null| 113|
#+----------+--------+---+------+--------+