【大数据入门实践】Hadoop hdfs伪分布式部署
1:更新系统
yum -y update
2:准备好ssh
3:安装好对应版本的jdk
4:创建hadoop用户1:安装jdk
[[email protected] usr]# mkdir /usr/java
2:进入java目录
[[email protected] usr]# cd /usr/java/
[[email protected] java]# ll
总用量 169212
-rw-r--r--. 1 root root 173271626 3月 9 10:56 jdk-8u45-linux-x64.gz
3:解压jdk 包
[[email protected] java]# tar -zxvf jdk-8u45-linux-x64.gz
[[email protected] java]# ll
总用量 169216
drwxr-xr-x. 8 uucp 143 4096 4月 11 2015 jdk1.8.0_45
-rw-r--r--. 1 root root 173271626 3月 9 10:56 jdk-8u45-linux-x64.gz
4:修改解压包权限
[[email protected] java]# chown -R root:root jdk1.8.0_45/
[[email protected] java]# chown -R root:root jdk1.8.0_45/*
5:配置环境变量
[[email protected] java]# vi /etc/profile
JAVA_HOME=/usr/java/jdk1.8.0_45
export PATH=$JAVA_HOME/bin:$PATH
6:刷新配置文件
[[email protected] jdk1.8.0_45]# source /etc/profile
[[email protected] jdk1.8.0_45]# javac
1:创建hadoop用户
[[email protected] ~]# useradd hadoop1:进入hadoop用户
[[email protected] ~]# su - hadoop
2:在hadoop家目录创建好需要的目录
[[email protected] ~]$ mkdir app data lib maven_reposhttp://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0/
http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0.tar.gz
1:解压hadoop
[[email protected] app]$ tar -zxvf hadoop-2.6.0-cdh5.7.0.tar.gz
2:查看权限
[[email protected] app]$ ll
total 304288
drwxr-xr-x. 14 hadoop hadoop 4096 Mar 24 2016 hadoop-2.6.0-cdh5.7.0
-rw-r--r--. 1 root root 311585484 Mar 9 11:19 hadoop-2.6.0-cdh5.7.0.tar.gz
3:编辑hadoop-env.sh
[[email protected] hadoop-2.6.0-cdh5.7.0]$ vi etc/hadoop/hadoop-env.sh
# The java implementation to use.
export JAVA_HOME=/usr/java/jdk1.8.0_45
export HADOOP_PREFIX=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0
4:配置core-site.xml
[[email protected] hadoop]$ pwd
/home/hadoop/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop
[[email protected] hadoop]$ vi core-site.xml
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
[[email protected] hadoop]$ vi hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
1:设计ssh 无密码信任
[[email protected] hadoop]$ which ssh-******
/usr/bin/ssh-******
[[email protected] home]# ssh-****** -t rsa -P '' -f ~/.ssh/id_rsa
[[email protected] ~]$ cd .ssh/
[[email protected] .ssh]$ ll
total 8
-rw-------. 1 hadoop hadoop 1675 Mar 9 11:57 id_rsa
-rw-r--r--. 1 hadoop hadoop 410 Mar 9 11:57 id_rsa.pub
2:将公钥放入认证文件
[[email protected] .ssh]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[[email protected] hadoop-2.6.0-cdh5.7.0]$ ssh localhost date
The authenticity of host 'localhost (::1)' can't be established.
RSA key fingerprint is 82:a0:b2:5c:7b:ae:83:6e:ce:9a:f2:b7:6d:82:98:d2.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
发现错误
[[email protected] .ssh]$ rm -rf known_hosts
[[email protected] .ssh]$ chmod 600 authorized_keys
[[email protected] .ssh]$ ssh localhost date
The authenticity of host 'localhost (::1)' can't be established.
RSA key fingerprint is 82:a0:b2:5c:7b:ae:83:6e:ce:9a:f2:b7:6d:82:98:d2.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
Sat Mar 9 12:04:46 CST 2019
1:格式化文件系统
[[email protected] bin]$ ./hdfs namenode -format
19/03/09 12:14:51 INFO common.Storage: Storage directory /tmp/hadoop-hadoop/dfs/name has been successfully formatted.
19/03/09 12:14:51 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
19/03/09 12:14:51 INFO util.ExitUtil: Exiting with status 0
19/03/09 12:14:51 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at localhost/127.0.0.1
************************************************************/
2:启动
[[email protected] sbin]$ ./start-dfs.sh
3:访问hadoop 50070端口
http://192.168.43.100:50070
4:jps
[[email protected] sbin]$ jps
4384 DataNode
4533 SecondaryNameNode
4267 NameNode
4637 Jps