I have a PySpark Dataframe input_dataframe as shown below:
**cust_id** **source_id** **value**
10 11 test_value
10 12 test_value2
i have another dataframe delta_dataframe which have updated records from input_dataframe and some new records as shown below:
**cust_id** **source_id** **value**
10 11 update_value
10 15 new_value2
In Both dataframe, primary key is combination of cust_id and source_id.
I have to generate a new dataframe output_dataframe, which will have records from input_dataframe with updated records from delta_dataframe, so my final dataframe is as below:
**cust_id** **source_id** **value**
10 11 update_value
10 12 test_value2
10 15 new_value2
Can someone please suggest me, how can i achieve it in PySpark. Any help will be appreciated on this.
Subtract the two dataframes based on primary key. Make inner join of output with input_dataframe. Then take Uion of it with Delta_dataframe. You will get proper output.
You need to join input_dataframe and delta_dataframe using join on two columns
output_df = input_df.join(delta_df, input_df['cust_id'] = delta_df['cust_id'] & input_df['source_id'] = delta_df['source_id'], 'left_outer')
And then select only the required fields from output_df
We can use Outer join and select the required dataframe value,
>>> input_dataframe.join(delta_dataframe,['custid','sourceid'],'outer').select('custid','sourceid',F.coalesce(delta_dataframe['value'],input_dataframe['value']).alias('value')).show()
+------+--------+-------------+
|custid|sourceid| value|
+------+--------+-------------+
| 10| 15| new_value2|
| 10| 11|updated_value|
| 10| 12| test_value2|
+------+--------+-------------+
Related
I have a spark dataframe containing businesses with their contact numbers in 2 columns, however some of my businesses are repeated with different contact info, for example:
Name:
Phone:
bus1
082...
bus1
087...
bus2
076...
bus3
081...
bus3
084...
bus3
086...
I want to have 3 lines, 1 for each business with varying phone numbers in each, for example:
Name:
Phone1:
Phone2:
Phone3:
bus1
082...
087...
bus2
076...
bus3
081...
084...
086...
I have tried using select('Name','Phone').distinct(), but I don't know how to pivot it to a single row matching on the 'Name' column... please help
First construct the phone array based on name, and then split the array into multiple columns.
df = df.groupBy('Name').agg(F.collect_list('Phone').alias('Phone'))
df = df.select('Name', *[F.col('Phone')[i].alias(f'Phone{str(i+1)}') for i in range(3)])
df.show(truncate=False)
Try something as below -
Input DataFrame
df = spark.createDataFrame([('bus1', '082...'), ('bus1', '087...'), ('bus2', '076...'), ('bus3', '081...'),('bus3', '084...'),('bus3', '086...')], schema=["Name", "Phone"])
df.show()
+----+------+
|Name| Phone|
+----+------+
|bus1|082...|
|bus1|087...|
|bus2|076...|
|bus3|081...|
|bus3|084...|
|bus3|086...|
+----+------+
Collecting all the Phone values into an array using collect_list
from pyspark.sql.functions import *
from pyspark.sql.types import *
df1 = df.groupBy("Name").agg(collect_list(col("Phone")).alias("Phone")).select( "Name", "Phone")
df1.show(truncate=False)
+----+------------------------+
|Name|Phone |
+----+------------------------+
|bus1|[082..., 087...] |
|bus2|[076...] |
|bus3|[081..., 084..., 086...]|
+----+------------------------+
Splitting Phone into multiple columns
df1.select(['Name'] + [df1.Phone[x].alias(f"Phone{x+1}") for x in range(0,3)]).show(truncate=False)
+----+------+------+------+
|Name|Phone1|Phone2|Phone3|
+----+------+------+------+
|bus1|082...|087...|null |
|bus2|076...|null |null |
|bus3|081...|084...|086...|
+----+------+------+------+
I am having two dataframe like described below
Dataframe 1
P_ID P_Name P_Description P_Size
100 Moto Mobile 16
200 Apple Mobile 15
300 Oppo Mobile 18
Dataframe 2
P_ID List_Code P_Amount
100 ALPHA 20000
100 BETA 60000
300 GAMMA 15000
Requirement :
Need to join the two dataframe by P_ID.
Information about the dataframe :
In dataframe 1 P_ID is a primary key and dataframe 2 does't have any primary attribute.
How to join the dataframe
Need to create new columns in dataframe 1 from the value of dataframe 2 List_Code appends with "_price". If dataframe 2 List_Code contains 20 unique values we need to create 20 column in dataframe 1. Then, we have fill the value in newly created column in dataframe 1 from the dataframe 2 P_Amount column based on P_ID if present else fills with zero. After creation of dataframe we need to join the dataframe based on the P_ID. If we add the column with the expected value in dataframe 1 we can join the dataframe. My problem is creating new columns with the expected value.
The expected dataframe is shown below
Expected dataframe
P_ID P_Name P_Description P_Size ALPHA_price BETA_price GAMMA_price
100 Moto Mobile 16 20000 60000 0
200 Apple Mobile 15 0 0 0
300 Oppo Mobile 18 0 0 15000
Can you please help me to solve the problem, thanks in advance.
For you application, you need to pivot the second dataframe and then join the first dataframe on to the pivoted result on P_ID using left join.
See the code below.
df_1 = pd.DataFrame({'P_ID' : [100, 200, 300], 'P_Name': ['Moto', 'Apple', 'Oppo'], 'P_Size' : [16, 15, 18]})
sdf_1 = sc.createDataFrame(df_1)
df_2 = pd.DataFrame({'P_ID' : [100, 100, 300], 'List_Code': ['ALPHA', 'BETA', 'GAMMA'], 'P_Amount' : [20000, 60000, 10000]})
sdf_2 = sc.createDataFrame(df_2)
sdf_pivoted = sdf_2.groupby('P_ID').pivot('List_Code').agg(f.sum('P_Amount')).fillna(0)
sdf_joined = sdf_1.join(sdf_pivoted, on='P_ID', how='left').fillna(0)
sdf_joined.show()
+----+------+------+-----+-----+-----+
|P_ID|P_Name|P_Size|ALPHA| BETA|GAMMA|
+----+------+------+-----+-----+-----+
| 300| Oppo| 18| 0| 0|10000|
| 200| Apple| 15| 0| 0| 0|
| 100| Moto| 16|20000|60000| 0|
+----+------+------+-----+-----+-----+
You can change the column names or ordering of the dataframe as needed.
I have the following Spark dataframe that is created dynamically
| name| number |
+--------+---------+
| Andy | (20,10,30)|
|Berta | (30,40,20)|
| Joe | (40,90,60)|
+-------+---------+
Now, I need to iterate each row and column in Spark to print the following output, How to do this?
Andy 20
Andy 10
Andy 30
Berta 30
Berta 40
Berta 20
Joe 40
Joe 90
Joe 60
Assuming the number column is of string Data Type, you can achieve the desired results by following below steps.
Original Data Frame:
val df = Seq(("Andy", "20,10,30"), ("Berta", "30,40,20"), ("Joe", "40,90,60"))
.toDF("name", "number")
Then Create an intermediate Data Frame having 3 number columns by splitting the number column with comma.
val Interim_Df = df.withColumn("n1", split(col("number"), ",").getItem(0))
.withColumn("n2", split(col("number"), ",").getItem(1))
.withColumn("n3", split(col("number"), ",").getItem(2))
.drop("number")
Then generate the final result data frame by doing union with oneIndexDfs.
val columnIndexes = Seq(1, 2, 3)
val onlyOneIndexDfs = columnIndexes.map(x =>
Interim_Df.select(
$"name",
col(s"n$x").alias("number")))
val resultDF = onlyOneIndexDfs.reduce(_ union _)
You need explode function.
Here samples of its usage.
I am completely baffled with the following problem:
When I join 2 data frames and return the row count, I get a slightly different count on each try. Here are the details:
I would like to join the data frames: 'df_user_ids' and 'df_conversions':
df_user_ids.show()
>>>
+--------------------+
| user_id|
+--------------------+
|AMsySZY-cqcufnXst...|
|AMsySZY1Oo75A6vKU...|
|AMsySZY4nbqZiuEMR...|
|AMsySZY5RSfgj6Xvi...|
|AMsySZY5geAmTx0er...|
|AMsySZY6Gskv_kEAv...|
|AMsySZY6MIOyPWM4U...|
|AMsySZYCEZYS00UB9...|
df_conversions.show()
>>>
+--------------------+----------------------+---------+
| user_id|time_activity_observed|converted|
+--------------------+----------------------+---------+
|CAESEAl1YPOZpaWVx...| 2018-03-23 12:15:37| 1|
|CAESEAuvSBzmfc_f3...| 2018-03-23 21:58:25| 1|
|CAESEBXWsSYm4ntvR...| 2018-03-30 12:16:53| 1|
|CAESEC-5uPwWGFdnv...| 2018-03-23 08:52:48| 1|
|CAESEDB3Z-NNvz7zL...| 2018-03-24 21:37:05| 1|
|CAESEDu7S7rGTVlj2...| 2018-04-01 17:00:12| 1|
|CAESEE4s6g1-JlUEt...| 2018-03-23 19:32:23| 1|
|CAESEELlJt0mE2xjn...| 2018-03-24 18:26:15| 1|
Both data frames have the key column named: "user_id",
and both are created using ".sampleBy()" with a fixed seed:
.sampleBy("converted", fractions={0: 0.035, 1: 1}, seed=0)
Before I join the data frames I persist them to disk:
df_user_ids.persist(StorageLevel.DISK_ONLY)
df_conversions.persist(StorageLevel.DISK_ONLY)
Then I verify that the row count of both data frames is consistent:
df_user_ids.count()
>>> 584309
df_user_ids.count()
>>> 584309
df_conversions.count()
>>> 5830
df_conversions.count()
>>> 5830
And check that the key column of both data frames does not contain duplicates:
df_user_ids.count()
>>> 584309
df_user_ids.select('user_id').distinct().count()
>>> 584309
df_conversions.count()
>>> 5830
df_conversions.select('user_id').distinct().count()
>>> 5830
Then I get the inconsistent row counts when I join them!
df_user_ids.join(df_conversions, ["user_id"], "left").count()
>>> 584314
df_user_ids.join(df_conversions, ["user_id"], "left").count()
>>> 584317
df_user_ids.join(df_conversions, ["user_id"], "left").count()
>>> 584304
How is this possible??
Sometimes this joined count is higher than "df_user_ids.count()" and sometimes it is lower. I am using a Zeppelin notebook in AWS EMR on an EMR cluster to run this code.
I already tried what is suggested in the link below:
".persist(StorageLevel.DISK_ONLY)" doesn't help.
I don't use monotonically_increasing_id.
spark inconsistency when running count command
By looking at the series of operations you are doing on DataFrames, i think the issue is due to Join. Join operation results shuffle, where every node talks to every other node and they share data according to which node has a certain key or set of keys (on which you are joining). When sharing data across executors, if executor doesnt have the dataframe persisted on Disk, it will re-compute the DAG and sampleBy is not guaranteed to return the same fraction of rows in dataframe.
I have 2 dataframes each one having Array[String] as one of the columns. For each entry in one dataframe, I need to find out subsets, if any, in the other dataframe. An example is here:
DF1:
----------------------------------------------------
id : Long | labels : Array[String]
---------------------------------------------------
10 | [label1, label2, label3]
11 | [label4, label5]
12 | [label6, label7]
DF2:
----------------------------------------------------
item : String | labels : Array[String]
---------------------------------------------------
item1 | [label1, label2, label3, label4, label5]
item2 | [label4, label5]
item3 | [label4, label5, label6, label7]
After the subset operation I described, the expected o/p should be
DF3:
----------------------------------------------------
item : String | id : Long
---------------------------------------------------
item1 | [10, 11]
item2 | [11]
item3 | [11, 12]
It is guaranteed that the DF2, will always have corresponding subsets in DF1, so there won't be any left over elements.
Can someone please help with the right approach here ? It looks like for each element in DF2, I need to scan DF1 and do subset operation (or set subtraction) on the 2nd column until I find all the subsets and exhaust the labels in that row and while doing that accumulate the list of "id" fields. How do I do this in compact and efficient manner ? Any help is greatly appreciated. Realistically, I may have 100s of elements in DF1 and 1000s of elements in DF2.
I'm not aware of any way to perform this kind of operation in an efficient way. However, here is one possible solution using UDF as well as Cartesian join.
The UDF takes two sequences and checks if all strings in the first exists in the second:
val matchLabel = udf((array1: Seq[String], array2: Seq[String]) => {
array1.forall{x => array2.contains(x)}
})
To use Cartesian join, it needs to be enabled as it is computationally expensive.
val spark = SparkSession.builder.getOrCreate()
spark.conf.set("spark.sql.crossJoin.enabled", true)
The two dataframes are joined together utilizing the UDF. Afterwards the resulting dataframe is grouped by the item column to collect a list of all ids. Using the same DF1 and DF2 as in the question:
val DF3 = DF2.join(DF1, matchLabel(DF1("labels"), DF2("labels")))
.groupBy("item")
.agg(collect_list("id").as("id"))
The result is as follows:
+-----+--------+
| item| id|
+-----+--------+
|item3|[11, 12]|
|item2| [11]|
|item1|[10, 11]|
+-----+--------+