How the check the Boolean condition between the column in Dataframe - apache-spark

Question

I have the Dataframe I want to check the condition in between the column:
+---+----+------+---------+------+
| ID|Name|Salary|Operation|Points|
+---+----+------+---------+------+
| 1| A| 10000| a AND b| 100|
| 1| A| 10000| a OR b| 200|
| 1| A| 10000|otherwise| 0|
| 2| B| 200| a AND b| 100|
| 2| B| 200| a OR b| 200|
| 2| B| 200|otherwise| 0|
| 3| C| 700| a AND b| 100|
| 3| C| 700| a OR b| 200|
| 3| C| 700|otherwise| 0|
| 4| D| 1000| a AND b| 100|
| 4| D| 1000| a OR b| 200|
| 4| D| 1000|otherwise| 0|
| 5| E| 650| a AND b| 100|
| 5| E| 650| a OR b| 200|
| 5| E| 650|otherwise| 0|
+---+----+------+---------+------+
Where:
a='salary==1000'
b='salary>500'
If the operation will true so points will assign and new column will add in the dataframe by name reward
for eg
if first entry occur A having salary is 10000 check the condition a if salary is equal to 1000 and salary is greater then 500 so a AND b false so 0 point will asign
result:
+---+----+------+------+
| ID|Name|Salary|Reward|
+---+----+------+------+
| 1| A| 10000| 200|
| 2| B| 200| 0|
| 3| C| 700| 200|
| 4| D| 1000| 200|
| 5| E| 650| 200|
+---+----+------+------+

Answer 1

You can piece something together with a filter expression and a groupby:
import pyspark.sql.functions as F
l = [
( 1, 'A', 10000, 'a AND b', 100),
( 1, 'A', 10000, 'a OR b', 200),
( 1, 'A', 10000,'otherwise', 0),
( 2, 'B', 200, 'a AND b', 100),
( 2, 'B', 200, 'a OR b', 200),
( 2, 'B', 200,'otherwise', 0),
( 3, 'C', 700, 'a AND b', 100),
( 3, 'C', 700, 'a OR b', 200),
( 3, 'C', 700,'otherwise', 0),
( 4, 'D', 1000, 'a AND b', 100),
( 4, 'D', 1000, 'a OR b', 200),
( 4, 'D', 1000,'otherwise', 0),
( 5, 'E', 650, 'a AND b', 100),
( 5, 'E', 650, 'a OR b', 200),
( 5, 'E', 650,'otherwise', 0)]
columns = ['ID','Name','Salary','Operation','Points']
df=spark.createDataFrame(l, columns)
df.filter(
(df.Operation.contains('AND') & (df.Salary == 1000) & (df.Salary > 500)) |
(df.Operation.contains('OR') & ((df.Salary == 1000) | (df.Salary > 500))) |
df.Operation.contains('otherwise')
).groupBy('ID', 'Name', 'Salary').agg(F.max('Points').alias('Rewards')).show()
Output:
+---+----+------+-------+
| ID|Name|Salary|Rewards|
+---+----+------+-------+
| 1| A| 10000| 200|
| 2| B| 200| 0|
| 3| C| 700| 200|
| 5| E| 650| 200|
| 4| D| 1000| 200|
+---+----+------+-------+
Please also have a look at a similar question and the answer of Shan.