Spark提交失败 -/opt/cloudera/parcels/CDH/bin/spark-class:没有这样的文件或目录

问题描述:

我在做Cloudera教程,并执行“4.使用spark-submit提交应用程序”。我做错了什么,以便运行教程失败?我从/ bin文件夹中找到spark-shell和spark-submit,但没有Spark-slass。Spark提交失败 -/opt/cloudera/parcels/CDH/bin/spark-class:没有这样的文件或目录

https://www.cloudera.com/documentation/enterprise/5-5-x/topics/spark_streaming.html#streaming

export SPARK_HOME="/opt/cloudera/parcels/CDH" 

spark-submit --master local[2] --conf 
"spark.dynamicAllocation.enabled=false" --jars 
$SPARK_HOME/lib/spark/lib/spark-examples.jar kafka_wordcount_keke.py k 
localhost:2181 POCTopicKeke1 


[[email protected] lib]$ spark-submit --master local[2] --jars $SPARK_HOME/lib/spark/lib/spark-examples.jar kafka_wordcount_keke.py localhost:2181 POCTopicKeke1 
/log/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/bin/../lib/spark/bin/spark-submit: line 27: /opt/cloudera/parcels/CDH/bin/spark-class: No such file or directory 
/log/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/bin/../lib/spark/bin/spark-submit: line 27: exec: /opt/cloudera/parcels/CDH/bin/spark-class: cannot execute: No such file or directory 
[[email protected] lib]$ 
+0

您的链接用于CDH 5.5 ...您正在使用5.12 –

+0

您可以在https://www.cloudera.com/documentation/enterprise/latest/topics/spark_streaming.html#streaming –

+0

中找到最新文档你的意思是SPARK_HOME指向错误的路径?如果CHH版本是5.12,它是否与spark-submit命令有所不同? – Keke

我遇到了类似的问题与CDH 5.13和Spark2.2

/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/bin/../lib/spark2/bin/pyspark: line 77: /opt/cloudera/parcels/SPARK2/bin/spark-submit: No such file or directory 

在调查我发现,我在/etc/profile手动设置SPARK_HOME到

export SPARK_HOME=/opt/cloudera/parcels/SPARK2 

和即使在发表评论并重新加载/etc/profile之后没有工作。

解决方案

env命令显示的SPARK_HOME仍然设置(怪),所以我用解封SPARK_HOME以下命令

unset SPARK_HOME

,并开始工作。