pyspark模块不可用于spark正在使用的python实例

问题描述：

我正在使用我自己没有在AWS中设置的EMR。pyspark模块不可用于spark正在使用的python实例

我想了解Python解释器的火花是使用在我的.bashrc我有以下设置export PYSPARK_PYTHON=/mnt/anaconda/bin/python

当我运行我们的火花提交命令我用sys.executable打印到路径python解释器，这确实是它使用的解释器。

然而，当我专门去到该文件夹，运行Python的该实例与./python，然后尝试我得到：

>>> import pyspark 
Traceback (most recent call last): 
    File "<stdin>", line 1, in <module> 
ImportError: No module named pyspark

这是怎么回事？这个解释器不应该安装pyspark模块吗？

你的'PYTHONPATH'环境变量是什么？ –

@JohnGordon'：/ mnt/anaconda/bin' – timbram

python模块似乎不太可能安装在'bin'目录下，所以这可能是你的问题。如何设置PYTHONPATH？ –

答

当你调用spark-submit时，它会为你自己添加它自己的python文件夹和py4j库到PYTHONPATH。他们住在spark安装到的文件夹中，它们不会直接安装到您正在使用的python的站点包中。例如，如果我pyspark的打印出来的蟒蛇路径：

$ pyspark 
Python 2.6.6 (r266:84292, Aug 18 2016, 15:13:37) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-17)] on linux2 
Type "help", "copyright", "credits" or "license" for more information. 
17/10/17 15:34:04 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
17/10/17 15:34:04 WARN Utils: Your hostname, rwidmaier resolves to a loopback address: 127.0.0.1; using 10.4.248.126 instead (on interface em1) 
17/10/17 15:34:04 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address 
/home/rwidmaier/apps/spark/python/pyspark/context.py:195: UserWarning: Support for Python 2.6 is deprecated as of Spark 2.0.0 
    warnings.warn("Support for Python 2.6 is deprecated as of Spark 2.0.0") 
17/10/17 15:34:09 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0 
17/10/17 15:34:09 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException 
17/10/17 15:34:09 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException 
Welcome to 
     ____    __ 
    /__/__ ___ _____/ /__ 
    _\ \/ _ \/ _ `/ __/ '_/ 
    /__/.__/\_,_/_/ /_/\_\ version 2.2.0 
     /_/ 

Using Python version 2.6.6 (r266:84292, Aug 18 2016 15:13:37) 
SparkSession available as 'spark'. 
>>> import sys 
>>> from pprint import pprint 
>>> pprint(sys.path) 
['', 
u'/tmp/spark-0a08efa7-714a-498b-b5e9-bba48684d52d/userFiles-52012e25-d7af-4599-a214-9637141ed4ec', 
'/usr/lib64/python2.6/site-packages/deluge-1.3.11-py2.6-linux-x86_64.egg', 
'/home/ryan/apps/spark/python/lib/py4j-0.10.4-src.zip', 
'/home/ryan/apps/spark/python', 
'/usr/lib64/python26.zip', 
'/usr/lib64/python2.6', 
'/usr/lib64/python2.6/plat-linux2', 
'/usr/lib64/python2.6/lib-tk', 
'/usr/lib64/python2.6/lib-old', 
'/usr/lib64/python2.6/lib-dynload', 
'/usr/lib64/python2.6/site-packages', 
'/usr/lib64/python2.6/site-packages/gst-0.10', 
'/usr/lib64/python2.6/site-packages/gtk-2.0', 
'/usr/lib64/python2.6/site-packages/webkit-1.0', 
'/usr/lib/python2.6/site-packages']

您可以在列表中看到它有手动添加：

/家庭/瑞安/应用/火花/ Python的/ lib目录/py4j-0.10.4-src.zip
/家庭/瑞安/应用/火花/蟒蛇

为了确保解释总是设置与正确的库，你只需要设置PYSPARK_PYTHON点到python安装你会可以使用，然后运行“pyspark”而不是直接运行python。或者，如果您想提供驱动程序，则可以使用spark-submit。

pyspark模块不可用于spark正在使用的python实例

相关推荐