Endless execution with spark udf - apache-spark

Question

I'm want to get the country with lat and long, so i used geopy and create a sample dataframe
data = [{"latitude": -23.558111, "longitude": -46.64439},
{"latitude": 41.877445, "longitude": -87.723846},
{"latitude": 29.986801, "longitude": -90.166314}
]
then create a udf
#F.udf("string")
def city_state_country(lat,lng):
geolocator = Nominatim(user_agent="geoap")
coord = f"{lat},{lng}"
location = geolocator.reverse(coord, exactly_one=True)
address = location.raw['address']
country = address.get('country', '')
return country
and it works this is the result
df2 = df.withColumn("contr",city_state_country("latitude","longitude"))
+----------+----------+-------------+
| latitude| longitude| contr|
+----------+----------+-------------+
|-23.558111| -46.64439| Brasil|
| 41.877445|-87.723846|United States|
| 29.986801|-90.166314|United States|
+----------+----------+-------------+
, but when I want to use my data with the schema
root
|-- id: integer (nullable = true)
|-- open_time: string (nullable = true)
|-- starting_lng: float (nullable = true)
|-- starting_lat: float (nullable = true)
|-- user_id: string (nullable = true)
|-- date: string (nullable = true)
|-- lat/long: string (nullable = false)
and 4 million rows, so I use limit and select
df_open_app3= df_open_app2.select("starting_lng","starting_lat").limit(10)
Finally, use the same udf
df_open_app4= df_open_app3.withColumn('con', city_state_country("starting_lat","starting_lng"))
The problem is that when I execute a display the process is endless, I don't know why but theorically should be process only 10 rows

Answer 1

I tried the similar scenario in my environment and it's working perfectly fine with around million records.
Created sample UDF function and dataframe with around million records.
selected particular columns and executed function on that.
As Derek O suggested try by using .cache() while creating a dataframe co you don't need to re-read the dataframe. so, that you can reuse the cached dataframe. When you have billions of records. Since action triggers the transformations, display is the first action hence it triggers the execution of all above dataframes creations ang might causing abnormal behavior.