Spark 集群搭建从零开始之3 Spark Standalone集群安装、配置与测试

1、进入pc1的Spark的conf目录，创建或者修改spark-env.sh

export JAVA_HOME=/usr/lib/jvm/java-8-oracle
export SCALA_HOME=/chenjie/scala-2.10.4
export HADOOP_HOME=/chenjie/hadoop-2.6.5
export HADOOP_CONF_DIR=/chenjie/hadoop-2.6.5/etc/hadoop
export SPARK_MASTER_IP=pc1
export SPARK_WORKER_MEMORY=2g
export SPARK_EXECUTOR_MEMORY=2g
export SPARK_DRIVER_MEMORY=2g
export SPARK_WORKER_CORES=2

2、配置spark-defaults.conf

spark.eventLog.enabled true
spark.eventLog.dir hdfs://pc1:9000/historyserverforSpark
spark.yarn.historyServer.address pc1:18080
spark.history.fs.logDirectory hdfs://pc1:9000/historyserverforSpark
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.executor.extraJavaOptions -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"

3、使用scp -r /chenjie/spark-1.6.0-bin-hadoop2.6 [email protected]:/chenjie命令将spark文件夹拷贝给pc2 pc3同理

4、pc1上进入/chenjie/spark-1.6.0-bin-hadoop2.6/sbin目录使用./start-all.sh命令启动集群

5、pc1上使用jps看到Master进程，pc2上看到Worker进程，pc3上看到Worker进程

6、使用http://192.168.153.131:8080/访问spark看到两个Worker结点

Spark 集群搭建从零开始之3 Spark Standalone集群安装、配置与测试