I'm trying to run a spark yarn job on BigInsights on Cloud 4.2 Basic cluster but I'm hitting the following issue:
Stack trace: ExitCodeException exitCode=1: /data/hadoop-swap/yarn/local/usercache/snowch/appcache/application_1480680664469_0038/container_1480680664469_0038_01_000004/launch_container.sh: line 24: $PWD:$PWD/__spark__.jar:/etc/hadoop/conf:/usr/iop/current/hadoop-client/*:/usr/iop/current/hadoop-client/lib/*:/usr/iop/current/hadoop-hdfs-client/*:/usr/iop/current/hadoop-hdfs-client/lib/*:/usr/iop/current/hadoop-yarn-client/*:/usr/iop/current/hadoop-yarn-client/lib/*:/usr/lib/hadoop-lzo/lib/*:/etc/hadoop/conf/:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/iop/${iop.version}/hadoop/lib/hadoop-lzo-0.5.1.jar:/etc/hadoop/conf/secure:/usr/lib/hadoop-lzo/lib/*: bad substitution
Digging deeper in to the error, I see:
Error: Could not find or load main class org.apache.spark.executor.CoarseGrainedExecutorBackend
The solution was here: http://www-01.ibm.com/support/docview.wss?uid=swg21987053
However, I just had to set the correct iop.version:
--conf "spark.driver.extraJavaOptions=-Diop.version=4.2.0.0"
Related
I am running my first PySpark application on EMR. My step fails with error "Unknown error". I checked the stderr log file and all it says is "User application exited with status 1". After researching a bit, I found out this happens when you don't pas your arguments correctly. Below is the kist of arguments I am passing. Can someone take a look and advise what's wrong here?
Thanks!
Jatin
spark-submit --deploy-mode cluster --master yarn s3://dpsdl-184210370075-prod-us-east-1-appobjects/scripts/spectre_matchedevent/scm_case_alert_data_sanitized_emr.py -source_bucket dpsdl-184210370075-prod-us-east-1-transformed -source_prefix_match spectre_matchedevent/output/spectre_match.parquet/ -source_prefix_alert spectre_matchedevent/output/spectre_case.parquet/ -source_prefix_ap spectre_matchedevent/output/spectre_amazonparty.parquet/ -source_prefix_dp spectre_matchedevent/output/spectre_deniedparty.parquet/ -dest_bucket dpsdl-184210370075-prod-us-east-1-enriched -dest_db_table read_out_secure.jk_scm_case_alert_data_new -dest_db_table_stg read_out_secure.jk_scad_updated_data_stg -dest_db_table_updated_cases_stg read_out_secure.jk_scm_case_alert_data_updated_cases_stg -dest_work_prefix_scad spectre_matchedevent/workdir/scm_case_alert_data/ -dest_work_prefix_updated_cases spectre_matchedevent/workdir/scad_updated_cases/ -dest_prefix_scad spectre_matchedevent/datamart-tables/scm_case_alert_data/ -iam_role arn:aws:iam::184210370075:role/as-redshift-load-for-scheduled -update_mode overwrite -default_matchtime_lower_threshold 2022-12-09
When I submit the spark-shell command, I see the following error:
# spark-shell
> SPARK_MAJOR_VERSION is set to 2, using Spark2
File "/usr/bin/hdp-select", line 249
print "Packages:"
^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print(t "Packages:")?
ls: cannot access /usr/hdp//hadoop/lib: No such file or directory
Exception in thread "main" java.lang.IllegalStateException: hdp.version is not set while running Spark under HDP, please set through HDP_VERSION in spark-env.sh or add a java-opts file in conf with -Dhdp.version=xxx
at org.apache.spark.launcher.Main.main(Main.java:118)
The problem is that the HDP script /usr/bin/hdp-select is apparently run under Python3, whereas it contains incompatible Python2 specific code.
You may port /usr/bin/hdp-select to Python3 by:
adding parentheses to the print statements
replacing the line "packages.sort()" by "list(package).sort()")
replacing the line "os.mkdir(current, 0755)" by "os.mkdir(current, 0o755)"
You may also try to force HDP to run /usr/bin/hdp-select under Python2:
PYSPARK_DRIVER_PYTHON=python2 PYSPARK_PYTHON=python2 spark-shell
Had the same problem: I set HDP_VERSION before running spark.
export HDP_VERSION=<your hadoop version>
spark-shell
I'm using Spark version 1.3. I have a job that's taking forever to finish.
To fix it, I made some optimizations to the code, and started the job again. Unfortunately, I launched the optimized code before stopping the earlier version, and now I cannot stop the earlier job.
Here are the things I've tried to kill this app:
Through the Web UI
result: The spark UI has no "kill" option for apps (I'm assuming they have not enabled the "spark.ui.killEnabled", I'm not the owner of this cluster).
Through the command line: spark-class org.apache.spark.deploy.Client kill mymasterURL app-XXX
result: I get this message:
Driver app-XXX has already finished or does not exist
But I see in the web UI that it is still running, and the resources are still occupied.
Through the command line via spark-submit: spark-submit --master mymasterURL --deploy-mode cluster --kill app-XXX
result: I get this error:
Error: Killing submissions is only supported in standalone mode!
I tried to retrieve the spark context to stop it (via SparkContext.stop(), or cancelAllJobs() ) but have been unsuccessful as ".getOrCreate" is not available in 1.3. I have not been able to retrieve the spark context of the initial app.
I'd appreciate any ideas!
Edit: I've also tried killing the app through yarn by executing: yarn application -kill app-XXX
result: I got this error:
Exception in thread "main" java.lang.IllegalArgumentException:
Invalid ApplicationId prefix: app-XX. The valid ApplicationId should
start with prefix application
When im trying to deploy at Windows this error comes. Im using apache-spark 2.0.
Command: ./bin/spark-class org.apache.spark.deploy.master.Master
Error: ./bin/spark-class: line 84: [: too many arguments
Its the same error reported here
The command is wrong, i forgot the ".cmd". The right command is:
./bin/spark-class.cmd org.apache.spark.deploy.master.Master
I'm trying to start a spark jobserver, here are the steps I'm following:
I configure the local.sh based on the template.
Then I run ./bin/server_deploy.sh and it finishes without any error.
Configure local.conf.
Run ./bin/server_start.sh in the deploy server.
But when I do the last step I get the following error:
Error: Exception thrown by the agent : java.lang.NullPointerException
Note: I'm using spark 1.4.1. I'm using version 0.5.2 from jobserver (https://github.com/spark-jobserver/spark-jobserver/tree/v0.5.2)
Any idea in how I can fix this (or at least debug it).
Thanks
The error log does not provide much information.
I encountered the same error. For my case, I had another instance of the JobServer running (and somehow ./bin/server_stop.sh did not catch it). It works after I manually killed the other process.
Hint : Error: Exception thrown by the agent : java.lang.NullPointerException when starting Java application