Create new "timezone" column from TimezoneFinder() with longitude and latitude columns as input in PySpark - apache-spark

Question

I want to create a new column that contains the timezone of the equivalent longitude and latitude. The longitude and latitude from already existing columns is the input in the timezonefinder function i.e. get_timezone(). I keep getting TypeError: an integer is required (got type Column)
Thanks.
from timezonefinder import TimezoneFinder
def get_timezone(longitude, latitude):
tzf = TimezoneFinder()
return tzf.timezone_at(lng=longitude, lat=latitude)
location_table = location_table.withColumn("timezone", get_timezone(location_table["location_longitude"], location_table["location_latitude"]))
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<command-253463262459944> in <module>
8
9 # df = sqlContext.read.parquet(INPUT)
---> 10 location_table.withColumn("timezone", get_timezone(location_table["location_longitude"].cast(IntegerType()), location_table["location_latitude"].cast(IntegerType())))
11 # .write.parquet(OUTPUT)
<command-253463262459944> in get_timezone(longitude, latitude)
3 def get_timezone(longitude, latitude):
4 tzf = TimezoneFinder()
----> 5 return tzf.timezone_at(lng=longitude, lat=latitude)
6
7 # udf_timezone = F.udf(get_timezone, StringType())
/databricks/python/lib/python3.7/site-packages/timezonefinder/timezonefinder.py in timezone_at(self, lng, lat)
657 :return: the timezone name of the matched timezone polygon. possibly "Etc/GMT+-XX" in case of an ocean timezone.
658 """
--> 659 lng, lat = rectify_coordinates(lng, lat)
660
661 shortcut_id_x, shortcut_id_y = coord2shortcut(lng, lat)
TypeError: an integer is required (got type Column)

Answer 1

You need to convert the function to a UDF first:
import pyspark.sql.functions as F
from timezonefinder import TimezoneFinder
#F.udf('string')
def get_timezone(longitude, latitude):
if longitude is None or latitude is None:
return None
tzf = TimezoneFinder()
return tzf.timezone_at(lng=longitude, lat=latitude)
location_table = location_table.withColumn("timezone", get_timezone(location_table["location_longitude"], location_table["location_latitude"]))