I have a rudimentary spark streaming word count and its just not working.
import sys
from pyspark import SparkConf, SparkContext
from pyspark.streaming import StreamingContext
sc = SparkContext(appName='streaming', master="local[*]")
scc = StreamingContext(sc, batchDuration=5)
lines = scc.socketTextStream("localhost", 9998)
words = lines.flatMap(lambda line: line.split())
counts = words.map(lambda word: (word, 1)).reduceByKey(lambda x, y: x + y)
counts.pprint()
print 'Listening'
scc.start()
scc.awaitTermination()
I have on a another terminal running nc -lk 9998 and I pasted some text. It prints out the typical logs (no exceptions) but it ends up queuing the job for some weird time (45 yrs) and it keeps on printing this...
15/06/19 18:53:30 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:874
15/06/19 18:53:30 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 2 (PythonRDD[7] at RDD at PythonRDD.scala:43)
15/06/19 18:53:30 INFO TaskSchedulerImpl: Adding task set 2.0 with 1 tasks
15/06/19 18:53:35 INFO JobScheduler: Added jobs for time 1434754415000 ms
15/06/19 18:53:40 INFO JobScheduler: Added jobs for time 1434754420000 ms
15/06/19 18:53:45 INFO JobScheduler: Added jobs for time 1434754425000 ms
...
...
What am I doing wrong?
Spark Streaming requires multiple executors to work. Try using local[4] for the master.
Related
I am try to send code to Spark master from intellij directly:
def main(args: Array[String]) = {
println("***hello spark-test***")
val spark = SparkSession
.builder()
.master("spark://172.22.208.1:7077")
.appName("Spark-Test Application")
.getOrCreate()
import spark.implicits._
val rawData = Array(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
val rdd = spark.sparkContext.parallelize(rawData)
val df = rdd.toDF()
val result = df.filter($"value" % 2 === 1).count()
println(s"***Result odd numbers count: $result ***")
spark.stop()
}
result:
the application log hangs at connecting to master:
22/12/29 12:38:55 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://172.22.208.1:7077...
22/12/29 12:38:55 INFO TransportClientFactory: Successfully created connection to /172.22.208.1:7077 after 38 ms (0 ms spent in bootstraps)
22/12/29 12:39:15 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://172.22.208.1:7077...
driver log:
22/12/29 12:39:35 ERROR TransportRequestHandler: Error while invoking RpcHandler#receive() for one-way message.
java.io.InvalidClassException: scala.collection.immutable.ArraySeq; local class incompatible: stream classdesc serialVersionUID = -8615987390676041167, local class serialVersionUID = 2701977568115426262
spark version : spark-3.3.0-hadoop3-scala2.13
intellij's scala version: v2.13.5
However when I remove the line 'master("spark://172.22.208.1:7077")' and build a jar and send it via spark-submit it works fine.
Usually this error means that you are not running the same Spark version or the same Scala version.
You need to make sure that the code you are running on Intellij is running on the same Scala and Spark version than the one you are running on your cluster.
Since you mentionned you are running Scala 2.13 on IntelliJ, I guess that your local Spark version is not the same as the one you have on the cluster.
I am trying to read kinesis data stream with spark streaming. I am not getting any records in the output. My code is not giving any error, just that it does not print anything on the console even after feeding data to kinesis. I have also tried playing around with trim horizon and latest but
still no luck. Please find the code below:
from pyspark import SparkContext
from pyspark.streaming import StreamingContext
from pyspark.streaming.kinesis import KinesisUtils, InitialPositionInStream
import time
from pyspark import StorageLevel
appName="cdc_application"
sc = SparkContext(appName=appName)
ssc = StreamingContext(sc, 1)
streamName = 'cdc-stream'
endpointUrl = 'https://kinesis.ap-south-1.amazonaws.com'
regionName = 'ap-south-1'
checkpointInterval = 5
kinesisstream = KinesisUtils.createStream(ssc, appName,
streamName, endpointUrl, regionName,
InitialPositionInStream.LATEST,
checkpointInterval,StorageLevel.MEMORY_AND_DISK_2
)
kinesisstream.pprint()
ssc.start()
time.sleep(10) # Run stream for 10 minutes just in case no detection of producer
ssc.stop(stopSparkContext=True,stopGraceFully=True)
Version - Spark 2.4.8 on EMR
Package used - org.apache.spark:spark-streaming-kinesis-asl_2.11:2.4.8
More details: My Kinesis stream is the target of AWS DMS which has been connected to the postgreSQL database. I am getting the change records(CDC) in Kinesis stream which i am trying to process through spark on EMR.
My output for this looks like below:
Time: 2022-05-05 05:19:33
Time: 2022-05-05 05:19:34
Time: 2022-05-05 05:19:35
Time: 2022-05-05 05:19:36
Time: 2022-05-05 05:19:37
Time: 2022-05-05 05:19:38
In console, it gives following output, but does not show anything:
Time: 2022-05-05 07:02:26
22/05/05 07:02:26 INFO JobScheduler: Finished job streaming job 1651734146000 ms.0 from job set of time 1651734146000 ms
22/05/05 07:02:26 INFO JobScheduler: Total delay: 0.014 s for time 1651734146000 ms (execution: 0.003 s)
22/05/05 07:02:26 INFO KinesisBackedBlockRDD: Removing RDD 844 from persistence list
22/05/05 07:02:26 INFO KinesisInputDStream: Removing blocks of RDD KinesisBackedBlockRDD[844] at createStream at NativeMethodAccessorImpl.java:0 of time 1651734146000 ms
22/05/05 07:02:26 INFO ReceivedBlockTracker: Deleting batches: 1651734144000 ms
22/05/05 07:02:26 INFO InputInfoTracker: remove old batch metadata: 1651734144000 ms
22/05/05 07:02:27 INFO JobScheduler: Added jobs for time 1651734147000 ms
22/05/05 07:02:27 INFO JobScheduler: Starting job streaming job 1651734147000 ms.0 from job set of time 1651734147000 ms
Time: 2022-05-05 07:02:27
I am working in a databricks cluster that have 240GB of memory and 64 cores. This the settings I defined.
import pyspark
from pyspark.sql import SparkSession
from pyspark.sql.types import *
import pyspark.sql.functions as fs
from pyspark.sql import SQLContext
from pyspark import SparkContext
from pyspark.sql.functions import count
from pyspark.sql.functions import col, countDistinct
from pyspark import SparkContext
from geospark.utils import GeoSparkKryoRegistrator, KryoSerializer
from geospark.register import upload_jars
from geospark.register import GeoSparkRegistrator
spark.conf.set("spark.sql.shuffle.partitions", 1000)
#Recommended settings for using GeoSpark
spark.conf.set("spark.driver.memory", "20g")
spark.conf.set("spark.network.timeout", "1000s")
spark.conf.set("spark.driver.maxResultSize", "10g")
spark.conf.set("spark.serializer", KryoSerializer.getName)
spark.conf.set("spark.kryo.registrator", GeoSparkKryoRegistrator.getName)
upload_jars()
SparkContext.setSystemProperty("geospark.global.charset","utf8")
spark.conf.set
I am working with large datasets and this is the error I get after hours of running.
org.apache.spark.SparkException: Job aborted due to stage failure: Task 3 in stage 10.0 failed 4 times, most recent failure: Lost task 3.3 in stage 10.0 (TID 6054, 10.17.21.12, executor 7):
ExecutorLostFailure (executor 7 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 170684 ms
Let the heartbeat Interval be default(10s) and increase the network time out interval(default -120 s) to 300s (300000ms) and see. Use set and get .
spark.conf.set("spark.sql.<name-of-property>", <value>)
spark.conf.set("spark.network.timeout", 300000 )
or run this script in the notebook .
%scala
dbutils.fs.put("dbfs:/databricks/init/set_spark_params.sh","""
|#!/bin/bash
|
|cat << 'EOF' > /databricks/driver/conf/00-custom-spark-driver-defaults.conf
|[driver] {
| "spark.network.timeout" = "300000"
|}
|EOF
""".stripMargin, true)
The error tells you that the worker has timed out because it took too long.
There is probably some bottleneck happening in the background. Check the spark UI for executor 7, task 3 and stage 10. You also want to check the query that you have been running.
You also want to check these setting for better configuration:
spark.conf.set("spark.databricks.io.cache.enabled", True) # delta caching
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", True) # adaptive query execution for skewed data
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", -1) # setting treshhold on broadcasting
spark.conf.set("spark.databricks.optimizer.rangeJoin.binSize", 20) #range optimizer
Feel free to give us more info on the Spark UI, we can better help you find the problem that way. Also, what kind of query were you doing?
Can you please try the following options ,
repartition the dataframe you work in more numbers for example df.repartition(1000)
--conf spark.network.timeout 10000000
--conf spark.executor.heartbeatInterval=10000000
I'm attempting to run a pyspark script on BigInsights on Cloud 4.2 Enterprise that accesses a Hive table.
First I create the hive table:
[biadmin#bi4c-xxxxx-mastermanager ~]$ hive
hive> CREATE TABLE pokes (foo INT, bar STRING);
OK
Time taken: 2.147 seconds
hive> LOAD DATA LOCAL INPATH '/usr/iop/4.2.0.0/hive/doc/examples/files/kv1.txt' OVERWRITE INTO TABLE pokes;
Loading data to table default.pokes
Table default.pokes stats: [numFiles=1, numRows=0, totalSize=5812, rawDataSize=0]
OK
Time taken: 0.49 seconds
hive>
Then I create a simple pyspark script:
[biadmin#bi4c-xxxxxx-mastermanager ~]$ cat test_pokes.py
from pyspark import SparkContext
sc = SparkContext()
from pyspark.sql import HiveContext
hc = HiveContext(sc)
pokesRdd = hc.sql('select * from pokes')
print( pokesRdd.collect() )
I attempt to execute with:
[biadmin#bi4c-xxxxxx-mastermanager ~]$ spark-submit \
--master yarn-cluster \
--deploy-mode cluster \
--jars /usr/iop/4.2.0.0/hive/lib/datanucleus-api-jdo-3.2.6.jar, \
/usr/iop/4.2.0.0/hive/lib/datanucleus-core-3.2.10.jar, \
/usr/iop/4.2.0.0/hive/lib/datanucleus-rdbms-3.2.9.jar \
test_pokes.py
However, I encounter the error:
Traceback (most recent call last):
File "test_pokes.py", line 8, in <module>
pokesRdd = hc.sql('select * from pokes')
File "/disk6/local/usercache/biadmin/appcache/application_1477084339086_0481/container_e09_1477084339086_0481_01_000001/pyspark.zip/pyspark/sql/context.py", line 580, in sql
File "/disk6/local/usercache/biadmin/appcache/application_1477084339086_0481/container_e09_1477084339086_0481_01_000001/py4j-0.9-src.zip/py4j/java_gateway.py", line 813, in __call__
File "/disk6/local/usercache/biadmin/appcache/application_1477084339086_0481/container_e09_1477084339086_0481_01_000001/pyspark.zip/pyspark/sql/utils.py", line 51, in deco
pyspark.sql.utils.AnalysisException: u'Table not found: pokes; line 1 pos 14'
End of LogType:stdout
If I run spark-submit standalone, I can see the table exists ok:
[biadmin#bi4c-xxxxxx-mastermanager ~]$ spark-submit test_pokes.py
…
…
16/12/21 13:09:13 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 18962 bytes result sent to driver
16/12/21 13:09:13 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 168 ms on localhost (1/1)
16/12/21 13:09:13 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
16/12/21 13:09:13 INFO DAGScheduler: ResultStage 0 (collect at /home/biadmin/test_pokes.py:9) finished in 0.179 s
16/12/21 13:09:13 INFO DAGScheduler: Job 0 finished: collect at /home/biadmin/test_pokes.py:9, took 0.236558 s
[Row(foo=238, bar=u'val_238'), Row(foo=86, bar=u'val_86'), Row(foo=311, bar=u'val_311')
…
…
See my previous question related to this issue: hive spark yarn-cluster job fails with: "ClassNotFoundException: org.datanucleus.api.jdo.JDOPersistenceManagerFactory"
This question is similar to this other question: Spark can access Hive table from pyspark but not from spark-submit. However, unlike that question I am using HiveContext.
Update: see here for the final solution https://stackoverflow.com/a/41272260/1033422
This is because the spark-submit job is unable to find the hive-site.xml, so it cannot connect to the Hive metastore. Please add --files /usr/iop/4.2.0.0/hive/conf/hive-site.xml to your spark-submit command.
It looks like you are affected by this bug: https://issues.apache.org/jira/browse/SPARK-15345.
I had a similar issue with Spark 1.6.2 and 2.0.0 on HDP-2.5.0.0:
My goal was to create a dataframe from a Hive SQL query, under these conditions:
python API,
cluster deploy-mode (driver program running on one of the executor nodes)
use YARN to manage the executor JVMs (instead of a standalone Spark master instance).
The initial tests gave these results:
spark-submit --deploy-mode client --master local ... =>
WORKING
spark-submit --deploy-mode client --master yarn ... => WORKING
spark-submit --deploy-mode cluster --master yarn .... => NOT WORKING
In case #3, the driver running on one of the executor nodes could find the database. The error was:
pyspark.sql.utils.AnalysisException: 'Table or view not found: `database_name`.`table_name`; line 1 pos 14'
Fokko Driesprong's answer listed above worked for me.
With, the command listed below, the driver running on the executor node was able to access a Hive table in a database which is not default:
$ /usr/hdp/current/spark2-client/bin/spark-submit \
--deploy-mode cluster --master yarn \
--files /usr/hdp/current/spark2-client/conf/hive-site.xml \
/path/to/python/code.py
The python code I have used to test with Spark 1.6.2 and Spark 2.0.0 is:
(Change SPARK_VERSION to 1 to test with Spark 1.6.2. Make sure to update the paths in the spark-submit command accordingly)
SPARK_VERSION=2
APP_NAME = 'spark-sql-python-test_SV,' + str(SPARK_VERSION)
def spark1():
from pyspark.sql import HiveContext
from pyspark import SparkContext, SparkConf
conf = SparkConf().setAppName(APP_NAME)
sc = SparkContext(conf=conf)
hc = HiveContext(sc)
query = 'select * from database_name.table_name limit 5'
df = hc.sql(query)
printout(df)
def spark2():
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName(APP_NAME).enableHiveSupport().getOrCreate()
query = 'select * from database_name.table_name limit 5'
df = spark.sql(query)
printout(df)
def printout(df):
print('\n########################################################################')
df.show()
print(df.count())
df_list = df.collect()
print(df_list)
print(df_list[0])
print(df_list[1])
print('########################################################################\n')
def main():
if SPARK_VERSION == 1:
spark1()
elif SPARK_VERSION == 2:
spark2()
if __name__ == '__main__':
main()
For me the accepted answer did not work.
(--files /usr/iop/4.2.0.0/hive/conf/hive-site.xml)
Adding the below code on top of the code file solved it.
import findspark
findspark.init('/usr/share/spark-2.4') # for 2.4
tldr; Can't use Kinesis Spark Streaming integration, because it receives no data.
Testing stream is set up, nodejs app sends 1 simple record per second.
Standard Spark 1.5.2 cluster is set up with master and worker nodes (4 cores) with docker-compose, AWS credentials in environment
spark-streaming-kinesis-asl-assembly_2.10-1.5.2.jar is downloaded and added to classpath
job.py or job.jar (just reads and prints) submitted.
Everything seems to be okay, but no records what-so-ever are received.
From time to time the KCL Worker thread says "Sleeping ..." - it might be broken silently (I checked all the stderr I could find, but no hints). Maybe swallowed OutOfMemoryError... but I doubt that, because of the amount of 1 record per second.
-------------------------------------------
Time: 1448645109000 ms
-------------------------------------------
15/11/27 17:25:09 INFO JobScheduler: Finished job streaming job 1448645109000 ms.0 from job set of time 1448645109000 ms
15/11/27 17:25:09 INFO KinesisBackedBlockRDD: Removing RDD 102 from persistence list
15/11/27 17:25:09 INFO JobScheduler: Total delay: 0.002 s for time 1448645109000 ms (execution: 0.001 s)
15/11/27 17:25:09 INFO BlockManager: Removing RDD 102
15/11/27 17:25:09 INFO KinesisInputDStream: Removing blocks of RDD KinesisBackedBlockRDD[102] at createStream at NewClass.java:25 of time 1448645109000 ms
15/11/27 17:25:09 INFO ReceivedBlockTracker: Deleting batches ArrayBuffer(1448645107000 ms)
15/11/27 17:25:09 INFO InputInfoTracker: remove old batch metadata: 1448645107000 ms
15/11/27 17:25:10 INFO JobScheduler: Added jobs for time 1448645110000 ms
15/11/27 17:25:10 INFO JobScheduler: Starting job streaming job 1448645110000 ms.0 from job set of time 1448645110000 ms
-------------------------------------------
Time: 1448645110000 ms
-------------------------------------------
<----- Some data expected to show up here!
15/11/27 17:25:10 INFO JobScheduler: Finished job streaming job 1448645110000 ms.0 from job set of time 1448645110000 ms
15/11/27 17:25:10 INFO JobScheduler: Total delay: 0.003 s for time 1448645110000 ms (execution: 0.001 s)
15/11/27 17:25:10 INFO KinesisBackedBlockRDD: Removing RDD 103 from persistence list
15/11/27 17:25:10 INFO KinesisInputDStream: Removing blocks of RDD KinesisBackedBlockRDD[103] at createStream at NewClass.java:25 of time 1448645110000 ms
15/11/27 17:25:10 INFO BlockManager: Removing RDD 103
15/11/27 17:25:10 INFO ReceivedBlockTracker: Deleting batches ArrayBuffer(1448645108000 ms)
15/11/27 17:25:10 INFO InputInfoTracker: remove old batch metadata: 1448645108000 ms
15/11/27 17:25:11 INFO JobScheduler: Added jobs for time 1448645111000 ms
15/11/27 17:25:11 INFO JobScheduler: Starting job streaming job 1448645111000 ms.0 from job set of time 1448645111000 ms
Please let me know any hints, I'd really like to use Spark for real time analytics... everything but this small detail of not receiving data :) seems to be ok.
PS: I find strange that somehow Spark ignores my settings of Storage level (mem and disk 2) and Checkpoint interval (20,000 ms)
15/11/27 17:23:26 INFO KinesisInputDStream: metadataCleanupDelay = -1
15/11/27 17:23:26 INFO KinesisInputDStream: Slide time = 1000 ms
15/11/27 17:23:26 INFO KinesisInputDStream: Storage level = StorageLevel(false, false, false, false, 1)
15/11/27 17:23:26 INFO KinesisInputDStream: Checkpoint interval = null
15/11/27 17:23:26 INFO KinesisInputDStream: Remember duration = 1000 ms
15/11/27 17:23:26 INFO KinesisInputDStream: Initialized and validated org.apache.spark.streaming.kinesis.KinesisInputDStream#74b21a6
Source code (java):
public class NewClass {
public static void main(String[] args) {
SparkConf conf = new SparkConf().setAppName("appname").setMaster("local[3]");
JavaStreamingContext ssc = new JavaStreamingContext(conf, new Duration(1000));
JavaReceiverInputDStream kinesisStream = KinesisUtils.createStream(
ssc, "webassist-test", "test", "https://kinesis.us-west-1.amazonaws.com", "us-west-1",
InitialPositionInStream.LATEST,
new Duration(20000),
StorageLevel.MEMORY_AND_DISK_2()
);
kinesisStream.print();
ssc.start();
ssc.awaitTermination();
}
}
Python code (tried both pprinting before and sending to MongoDB):
from pyspark.streaming.kinesis import KinesisUtils, InitialPositionInStream
from pyspark import SparkContext, StorageLevel
from pyspark.streaming import StreamingContext
from sys import argv
sc = SparkContext(appName="webassist-test")
ssc = StreamingContext(sc, 5)
stream = KinesisUtils.createStream(ssc,
"appname",
"test",
"https://kinesis.us-west-1.amazonaws.com",
"us-west-1",
InitialPositionInStream.LATEST,
5,
StorageLevel.MEMORY_AND_DISK_2)
stream.pprint()
ssc.start()
ssc.awaitTermination()
Note: I also tried sending data to MongoDB with stream.foreachRDD(lambda rdd: rdd.foreachPartition(send_partition)) but not pasting it here, since you'd need a MongoDB instance and it's not related to the problem - no records come in on the input already.
One more thing - the KCL never commits. The corresponding DynamoDB looks like this:
leaseKey checkpoint leaseCounter leaseOwner ownerSwitchesSinceCheckpoint
shardId-000000000000 LATEST 614 localhost:d92516... 8
The command used for submitting:
spark-submit --executor-memory 1024m --master spark://IpAddress:7077 /path/test.py
In the MasterUI I can see:
Input Rate
Receivers: 1 / 1 active
Avg: 0.00 events/sec
KinesisReceiver-0
Avg: 0.00 events/sec
...
Completed Batches (last 76 out of 76)
Thanks for any help!
I've had issues with no record activity being shown in Spark Streaming in the past when connecting with Kinesis.
I'd try these things to get more feedback/a different behaviour from Spark:
Make sure that you force the evaluation of your DStream transformation operations with output operations like foreachRDD, print, saveas...
Create a new KCL Application in DynamoDB using a new name for the "Kinesis app name" parameter when creating the stream or purge the existing one.
Switch between TRIM_HORIZON and LATEST for initial position when creating the stream.
Restart the context when you try these changes.
EDIT after code was added:
Perhaps I'm missing something obvious, but I cannot spot anything wrong with your source code. Do you have n+1 cpus running this application (n is the number of Kinesis shards)?
If you run a KCL application (Java/Python/...) reading from the shards in your docker instance, does it work? Perhaps there's something wrong with your network configuration, but I'd expect some error messages pointing it out.
If this is important enough / you have a bit of time, you can quickly implement kcl reader in your docker instance and will allow you to compare with your Spark Application. Some urls:
Python
Java
Python example
Another option is to run your Spark Streaming application in a different cluster and to compare.
P.S.: I'm currently using Spark Streaming 1.5.2 with Kinesis in different clusters and it processes records / shows activity as expected.
I was facing this issue when I used the suggested documentation and examples for the same, the following scala code works fine for me(you can always use java instead)--
val conf = ConfigFactory.load
val config = new SparkConf().setAppName(conf.getString("app.name"))
val ssc = new StreamingContext(config, Seconds(conf.getInt("app.aws.batchDuration")))
val stream = if (conf.hasPath("app.aws.key") && conf.hasPath("app.aws.secret")){
logger.info("Specifying AWS account using credentials.")
KinesisUtils.createStream(
ssc,
conf.getString("app.name"),
conf.getString("app.aws.stream"),
conf.getString("app.aws.endpoint"),
conf.getString("app.aws.region"),
InitialPositionInStream.LATEST,
Seconds(conf.getInt("app.aws.batchDuration")),
StorageLevel.MEMORY_AND_DISK_2,
conf.getString("app.aws.key"),
conf.getString("app.aws.secret")
)
} else {
logger.info("Specifying AWS account using EC2 profile.")
KinesisUtils.createStream(
ssc,
conf.getString("app.name"),
conf.getString("app.aws.stream"),
conf.getString("app.aws.endpoint"),
conf.getString("app.aws.region"),
InitialPositionInStream.LATEST,
Seconds(conf.getInt("app.aws.batchDuration")),
StorageLevel.MEMORY_AND_DISK_2
)
}
stream.foreachRDD((rdd: RDD[Array[Byte]], time) => {
val rddstr: RDD[String] = rdd
.map(arrByte => new String(arrByte))
rddstr.foreach(x => println(x))
}