vertica sql in pyspark not running - apache-spark

Question

I'm new to use pyspark running sql to vertica. I have vertica_python module imported, I have sql to vertica table, I need to convert the query result into pyspark dataframe. I use spark-submit command to run, but got error saying 'MemoryError'. Can someone help?
I tried to increase and decrease the drive and executor memory in spark-submit command, didn't work, got same error message.
from pyspark.sql import SQLContext
from pyspark.sql import HiveContext
sc=SparkContext(appName="build_history_pyspark")
sqlContext=SQLContext(sc)
start_load_time_UTC='2019-06-01 00:00:00'
start_load_time_UTC=datetime.strptime(start_load_time_UTC,'%Y-%m-%d %H:%M:%S')
url="jdbc:vertica://144.60.100.141:5433/SRVVERTICA" . #This is the vertica database host IP and port
properties={"user":"hhhhh","password":"yyyyy","driver": "com.vertica.jdbc.Driver"}
sql_for_kafka="SELECT STARTTIME, LONGITUDE,LATITUDE FROM my_table WHERE (VERTICA_LOAD_TIME >'%s')" %(start_load_time_UTC)
sdf_orig=sqlContext.read.format("JDBC").options(url=url, query=sql_for_kafka, **properties).load()
sdf_orig.show(10)
error saying:
MemoryError
my command:
spark-submit --driver-memory 10g --num-executors 10 --executor-cores 4 --executor-memory 10g build_history_pyspark.py
I did further debugging, I think the error is due to the variable start_load_time_UTC. I really need to use a variable here not the straight value. But don't know the correct syntax.
I also changed the query statement to:
sql_for_kafka="SELECT STARTTIME, LONGITUDE,LATITUDE FROM my_table WHERE VERTICA_LOAD_TIME >$start_load_time_UTC"
I got different error:
Py4JJavaError: An error occurred while calling o57.load.
: java.sql.SQLDataException: [Vertica][VJDBC](3679) ERROR: Invalid
input syntax for timestamp: "$start_load_time_UTC"
at com.vertica.util.ServerErrorData.buildException(Unknown Source)
at com.vertica.io.ProtocolStream.readExpectedMessage(Unknown Source)
at com.vertica.dataengine.VDataEngine.prepareImpl(Unknown Source)
at com.vertica.dataengine.VDataEngine.prepare(Unknown Source)
at com.vertica.dataengine.VDataEngine.prepare(Unknown Source)
at com.vertica.jdbc.common.SPreparedStatement.<init>(Unknown Source)
at com.vertica.jdbc.jdbc4.S4PreparedStatement.<init>(Unknown Source)
at com.vertica.jdbc.VerticaJdbc4PreparedStatementImpl.<init>(Unknown Source)
at com.vertica.jdbc.VJDBCObjectFactory.createPreparedStatement(Unknown Source)
at com.vertica.jdbc.common.SConnection.prepareStatement(Unknown Source)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.resolveTable(JDBCRDD.scala:58)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation.<init>(JDBCRelation.scala:115)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:52)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:340)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:239)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:227)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:164)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.vertica.support.exceptions.DataException: [Vertica]VJDBC ERROR: Invalid input syntax for timestamp: "$start_load_time_UTC"
... 28 more
I finally solved the problem by changing the query statement to below:
sql_for_kafka="(SELECT STARTTIME, LONGITUDE,LATITUDE FROM my_table WHERE (VERTICA_LOAD_TIME >'{0}') )temp" .format(start_load_time_UTC)