Spark提交失败 -/opt/cloudera/parcels/CDH/bin/spark-class:没有这样的文件或目录
问题描述:
我在做Cloudera教程,并执行“4.使用spark-submit提交应用程序”。我做错了什么,以便运行教程失败?我从/ bin文件夹中找到spark-shell和spark-submit,但没有Spark-slass。Spark提交失败 -/opt/cloudera/parcels/CDH/bin/spark-class:没有这样的文件或目录
https://www.cloudera.com/documentation/enterprise/5-5-x/topics/spark_streaming.html#streaming
export SPARK_HOME="/opt/cloudera/parcels/CDH"
spark-submit --master local[2] --conf
"spark.dynamicAllocation.enabled=false" --jars
$SPARK_HOME/lib/spark/lib/spark-examples.jar kafka_wordcount_keke.py k
localhost:2181 POCTopicKeke1
[[email protected] lib]$ spark-submit --master local[2] --jars $SPARK_HOME/lib/spark/lib/spark-examples.jar kafka_wordcount_keke.py localhost:2181 POCTopicKeke1
/log/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/bin/../lib/spark/bin/spark-submit: line 27: /opt/cloudera/parcels/CDH/bin/spark-class: No such file or directory
/log/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/bin/../lib/spark/bin/spark-submit: line 27: exec: /opt/cloudera/parcels/CDH/bin/spark-class: cannot execute: No such file or directory
[[email protected] lib]$
答
我遇到了类似的问题与CDH 5.13和Spark2.2
/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/bin/../lib/spark2/bin/pyspark: line 77: /opt/cloudera/parcels/SPARK2/bin/spark-submit: No such file or directory
在调查我发现,我在/etc/profile
手动设置SPARK_HOME到
export SPARK_HOME=/opt/cloudera/parcels/SPARK2
和即使在发表评论并重新加载/etc/profile
之后没有工作。
解决方案:
env
命令显示的SPARK_HOME
仍然设置(怪),所以我用解封SPARK_HOME
以下命令
unset SPARK_HOME
,并开始工作。
您的链接用于CDH 5.5 ...您正在使用5.12 –
您可以在https://www.cloudera.com/documentation/enterprise/latest/topics/spark_streaming.html#streaming –
中找到最新文档你的意思是SPARK_HOME指向错误的路径?如果CHH版本是5.12,它是否与spark-submit命令有所不同? – Keke