How to cast string to timestamp with nanoseconds in pyspark - apache-spark

Question

I am working with data with timestamps that contain nanoseconds and am trying to convert the string to timestamp format.
Here is what the 'Time' column looks like:
+---------------+
| Time |
+---------------+
|091940731349000|
|092955002327000|
|092955004088000|
+---------------+
I would like to cast it to:
+------------------+
| Timestamp |
+------------------+
|09:19:40.731349000|
|09:29:55.002327000|
|09:29:55.004088000|
+------------------+
From what I have found online, I don't need to use a udf to do this and there should be a native function that I can use.
I have tried cast and to_timestamp but got 'null' values:
df_new = df.withColumn('Timestamp', df.Time.cast("timestamp"))
df_new.select('Timestamp').show()
+---------+
|Timestamp|
+---------+
| null|
| null|
+---------+

Answer 1

There are two problems in your code:
Input is not a valid timestamp representation.
Spark doesn't provide type that can represent time without date component
The closest you can get to the required output is to convert input to JDBC compliant java.sql.Timestamp format:
from pyspark.sql.functions import col, regexp_replace
df = spark.createDataFrame(
["091940731349000", "092955002327000", "092955004088000"],
"string"
).toDF("time")
df.select(regexp_replace(
col("time"),
"^(\\d{2})(\\d{2})(\\d{2})(\\d{9}).*",
"1970-01-01 $1:$2:$3.$4"
).cast("timestamp").alias("time")).show(truncate = False)
# +--------------------------+
# |time |
# +--------------------------+
# |1970-01-01 09:19:40.731349|
# |1970-01-01 09:29:55.002327|
# |1970-01-01 09:29:55.004088|
# +--------------------------+
If you want just a string skip cast and limit output to:
df.select(regexp_replace(
col("time"),
"^(\\d{2})(\\d{2})(\\d{2})(\\d{9}).*",
"$1:$2:$3.$4"
).alias("time")).show(truncate = False)
# +------------------+
# |time |
# +------------------+
# |09:19:40.731349000|
# |09:29:55.002327000|
# |09:29:55.004088000|
# +------------------+