您的位置: 首页  >  文章  >  Hadoop全分布式部署

Hadoop全分布式部署

分类: 文章 • 2025-06-08 14:26:28

1、安装VMWare（略）

2、安装Ubuntu（略）

3、配置静态IP

3.1设置虚拟机上网模式为NAT

Hadoop全分布式部署

将多余的VMnet0、1删除

子网IP为想设置的IP段，如：192.168.157.0

子网掩码设置为：255.255.255.0

取消勾选"使用本地DHCP服务将IP地址分配给虚拟机"

3.2进入Ubuntu系统中配置网络

（1）修改文件/etc/network/interfaces

[email protected]:~$ sudo vim /etc/network/interfaces

内容如下：

#interfaces(5) file used by ifup(8) and ifdown(8)

auto lo

iface lo inet loopback

auto ens33

iface ens33 inet static

address 192.168.157.100

netmask 255.255.255.0

gateway 192.168.157.2

dns-nameservers 8.8.8.8

 

（2）修改文件/etc/resolv.conf

内容如下：

nameserver 8.8.8.8

(3)重启虚拟机网络

[email protected]:~$ sudo service networking restart

(4)重启虚拟机

把虚拟机重启下就好了

4、安装jdk并配置jdk环境变量

在oracle官网下载jdk1.8版本以上，jdk-8u161-linux-x64.tar.gz

将压缩包放在/usr/local/java目录下（java目录需手动建立）

执行命令

[email protected]:~$ /usr/local/java$tar -zxvf jdk-8u161-linux-x64.tar.gz

发现有个jdk1.8.0_161的文件夹

修改/etc/profile

export JAVA_HOME=/usr/local/jdk1.8.0_161

export JRE_HOME=$JAVA_HOME/jre

export CLASSPATH=$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH

 

export PATH=$JAVA_HOME/bin:$JRE_HOME/bin

 

[email protected]:~/etc$source /etc/profile

5、新建hadoop用户，增加管理员权限

[email protected]:~$ sudo useradd -m hadoop -s /bin/bash

[email protected]:~$ sudo passwd hadoop

按提示输入两遍密码

为hadoop用户添加root权限

[email protected]:~$ sudo adduser hadoop sudo

注销用户，使用hadoop用户登录

6、安装SSH，并配置无密码登录

更新apt sudo apt-get update

[email protected]:~$ sudo apt-get install vim

[email protected]:~$ sudo apt-get install openssh-server

[email protected]:~$ ssh localhost

Hadoop全分布式部署

每次使用ssh登录时都要输入密码，比较麻烦

[email protected]:~$ exit

[email protected]:~$ cd ~/.ssh/

[email protected]:~$ ssh-****** -t rsa #一直按回车

[email protected]:~$ cat ./id_rsa.pub >> ./authorized_keys

[email protected]:~$ ssh localhost #已经可以无密码登录

7、安装Hadoop

下载hadoop-2.7.5.tar.gz

进入下载目录

[email protected]:~$ cd ~/下载

[email protected]:~/下载$ mv ./hadoop-2.7.5.tar.gz /home/hadoop/soft/

[email protected]:~/下载$ cd /home/hadoop/soft

[email protected]:~/home/hadoop/soft$ tar -zxvf hadoop-2.7.5.tar.gz

检查hadoop是否可用

[email protected]:~/home/hadoop/soft$ ./bin/hadoop version

8、配置Hadoop环境

8.1 修改hadoop-env.sh

[email protected]:~/soft/hadoop$ vim ./etc/hadoop/hadoop-env.sh

修改JAVA_HOME=/usr/bin/local/java/jdk1.8.0_161

8.2 修改yarn-env.sh

[email protected]:~/soft/hadoop$ vim ./etc/hadoop/yarn-env.sh

修改JAVA_HOME=/usr/bin/local/java

8.3 修改slaves

[email protected]:~/soft/hadoop$ vim ./etc/hadoop/slaves

添加数据

hadoop02

hadoop03

hadoop04

8.4 修改core-site.xml

[email protected]:~/soft/hadoop$ vim ./etc/hadoop/core-site.xml

<configuration>

<property>

<name>fs.defaultFS</name>

<value>hdfs://hadoop01:9000</value>

</property>

 

<property>

<name>hadoop.tmp.dir</name>

<value>file:/home/hadoop/hadoop/tmp</value>

</property>

</configuration>

8.5 修改hdfs-site.xml

[email protected]:~/soft/hadoop$ vim ./etc/hadoop/hdfs-site.xml

<configuration>

<property>

<name>dfs.namenode.secondary.http-address</name>

<value>hadoop01:9001</value>

</property>

<property>

<name>dfs.name.dir</name>

<value>/home/hadoop/hadoop/temp/name</value>

<final>true</final>

</property>

<property>

<name>dfs.data.dir</name>

<value>/home/hadoop/hadoop/temp/data</value>

<final>true</final>

</property>

<property>

<name>dfs.replication</name>

<value>1</value>

<final>true</final>

</property>

<property>

<name>dfs.permissions</name>

<value>false</value>

</property>

</configuration>

 

8.6 修改mapred-site.xml

[email protected]:~/soft/hadoop$ vim ./etc/hadoop/mapred-site.xml

 

<configuration>

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

<property>

<name>mapreduce.jobhistory.address</name>

<value>hadoop01:10020</value>

</property>

<property>

<name>mapreduce.jobhistory.webapp.address</name>

<value>hadoop01:19888</value>

</property>

</configuration>

 

8.7 修改yarn-site.xml

[email protected]:~/soft/hadoop$ vim yarn-site.xml

<configuration>

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<property>

<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

<property>

<name>yarn.resourcemanager.address</name>

<value>hadoop01:8032</value>

</property>

<property>

<name>yarn.resourcemanager.scheduler.address</name>

<value>hadoop01:8030</value>

</property>

<property>

<name>yarn.resourcemanager.resource-tracker.address</name>

<value>hadoop01:8031</value>

</property>

<property>

<name>yarn.resourcemanager.admin.address</name>

<value>hadoop01:8033</value>

</property>

<property>

<name>yarn.resourcemanager.webapp.address</name>

<value>hadoop01:8088</value>

</property>

</configuration>

 

8.8 复制到其他节点上

[email protected]:~$ sudo scp -r /home/hadoop/soft/hadoop2.7.5 [email protected]:~/soft/

[email protected]:~$ sudo scp -r /home/hadoop/soft/hadoop2.7.5 [email protected]:~/soft/

[email protected]:~$ sudo scp -r /home/hadoop/soft/hadoop2.7.5 [email protected]:~/soft/

8.9 修改Hadoop环境变量

[email protected]:~$ sudo vim /etc/profile

export HADOOP_HOME=/home/hadoop/soft/hadoop

export PATH=${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin

(每台slaver都配置环境变量)

9、启动Hadoop

9.1 格式化namenode

[email protected]:~$ hdfs namenode -format

9.2启动hdfs

[email protected]:~$ start-dfs.sh

9.3启动yarn

[email protected]:~$ start-yarn.sh

9.4 查看启动情况

[email protected]:~$jps

Hadoop全分布式部署

[email protected]:~$ hdfs dfsadmin -report

Hadoop全分布式部署

 

配置成功

 

 

 

相关推荐