您的位置: 首页  >  文章  >  Hadoop单机模式以及伪分布模式

Hadoop单机模式以及伪分布模式

分类: 文章 • 2023-12-17 12:49:16
  1. jdk安装

wget --no-check-certificate --no-cookies --header "Cookie:oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u131-bll/d54c1d3a095b4ff2b6607d096fa80163/jdk-8u131-linux-x64.rpm

存在java8：openjdk

[email protected]:~# java -version

openjdk version "1.8.0_212"

OpenJDK Runtime Environment (build 1.8.0_212-8u212-b03-0kord1.16.04.1-b03)

OpenJDK 64-Bit Server VM (build 25.212-b03, mixed mode)

 

  1. ssh生成公钥免密登录（集群模式需要互相免密登录）

#查看ssh安装包情况

dpkg -l | grep ssh

Hadoop单机模式以及伪分布模式

#查看是否启动ssh

ps –e | grep ssh

Hadoop单机模式以及伪分布模式

如果没有ssh服务，则安装ssh

$sudo apt-get install openssh-server

安装完成，开启服务

$sudo /etc/init.d/ssh start

 

测试登陆本机 ssh localhost  输入yes，再输入密码，就应该可以登录了。但是每次输入比较繁琐，如果是集群那就是灾难了，所以要配置成免密码登陆的方式。

Hadoop单机模式以及伪分布模式

生成公钥私钥，将在~/.ssh文件夹下生成文件id_rsa：私钥，id_rsa.pub：公钥

生成公钥：$ ssh-****** -t rsa

导入公钥到认证文件，更改权限：

[email protected]:~/.ssh# cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

[email protected]:~/.ssh# chmod 700 ~/.ssh

[email protected]:~/.ssh# chmod 600 ~/.ssh/authorized_keys

Hadoop单机模式以及伪分布模式

（PS：集群模式需要将公钥匙复制到其他节点$ scp ~/.ssh/id_rsa.pub [email protected]:/home/xxx/id_rsa.pub）

  1. 配置主机名

{

cat /etc/passwd 可以查看所有用户的列表

cat /etc/group 查看用户组

userdel 用户

userdel -f 用户

 

lsb_release -a 查看操作系统信息

hostnamectl set-hostname hadoop1 修改主机名

hostnamectl  查看主机名

}

# vi /etc/hosts

 

10.11.16.85 hadoop

 

vi /etc/hostname

 

hadoop

 

  1. Hadoop安装：

下载安装包：

$wget --no-check-certificate https://archive.apache.org/dist/hadoop/common/hadoop-3.0.0/hadoop-3.0.0.tar.gz

（ps:下载的hadoop文件大小只有16K，不知是啥原因。源文件大小293M。最终查明原因因为服务器上网需要认证，认证后，下载正常）

找寻一个比较大的目录，创建文件夹安装解压：

596  df -h

  601  df -h /

605  mkdir software

  606  ll

  607  cd software/

  608  mkdir hadoop

  609  mv ~/hadoop-3.0.0.tar.gz ./hadoop/

  610  ls

  611  cd hadoop/

  612  ls

  613  tar -zxvf hadoop-3.0.0.tar.gz

  创建Hadoop用户和组，并授予执行权限

  $ sudo addgroup hadoop

  $ sudo usermod -a -G hadoop root #将当前用户加入到hadoop组

  $ vi /etc/sudoers

在 root ALL=(ALL) ALL   后

 

添加一行 hadoop ALL=(ALL) ALL

 

759  vi /etc/sudoers

760  sudo chmod -R 755 /software/hadoop/

761  sudo chown -R root:hadoop /software/hadoop/ 否则ssh会拒绝访问

 

  1. 查看Java安装路径：

[email protected]:/usr/share/java# whereis java

java: /usr/bin/java /usr/lib/java /usr/share/java /usr/share/man/man1/java.1.gz

[email protected]:/usr/share/java# which java

/usr/bin/java

（PS：找到java的指引的最终路径）

[email protected]:/usr/share/java# java -version

openjdk version "1.8.0_212"

OpenJDK Runtime Environment (build 1.8.0_212-8u212-b03-0kord1.16.04.1-b03)

OpenJDK 64-Bit Server VM (build 25.212-b03, mixed mode)

[email protected]:/usr/share/java# ls -lrt /usr/bin/java

lrwxrwxrwx 1 root root 22 1月  20 15:59 /usr/bin/java -> /etc/alternatives/java

[email protected]:/usr/share/java# ls -lrt /etc/alternatives/java

lrwxrwxrwx 1 root root 46 1月  20 15:59 /etc/alternatives/java -> /usr/lib/jvm/java-8-openjdk-arm64/jre/bin/java

[email protected]:/usr/share/java# cd /usr/lib/jvm/java-8-openjdk-arm64/

[email protected]:/usr/lib/jvm/java-8-openjdk-arm64# ls

ASSEMBLY_EXCEPTION  bin  docs  include  jre  lib  man  src.zip  THIRD_PARTY_README

 

配置全局/全用户环境变量：

$vi /etc/profile

添加

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-arm64/

export HADOOP_HOME=/software/hadoop/hadoop-3.0.0/

export PATH=.:$JAVA_HOME/bin:${HADOOP_HOME}/bin:$PATH

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

 

  1. 测试是否配置成功

$ hadoop version

Hadoop单机模式以及伪分布模式

 

 

  1. Hadoop单机配置（非分布式模式）

[email protected]:/etc# cd /software/hadoop/

[email protected]:/software/hadoop# ls

hadoop-3.0.0  hadoop-3.0.0.tar.gz

[email protected]:/software/hadoop# cd hadoop-3.0.0

[email protected]:/software/hadoop/hadoop-3.0.0# ls

bin  etc  include  lib  libexec  LICENSE.txt  NOTICE.txt  README.txt  sbin  share

[email protected]:/software/hadoop/hadoop-3.0.0# mkdir input

[email protected]:/software/hadoop/hadoop-3.0.0# cp README.txt input/

[email protected]:/software/hadoop/hadoop-3.0.0# hadoop jar share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-3.0.0-test-sources.jar org.apache.hadoop.examples.WordCount input output

执行一个Word Count案例：

Hadoop单机模式以及伪分布模式

2020-07-08 14:06:53,083 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

2020-07-08 14:06:53,737 INFO beanutils.FluentPropertyBeanIntrospector: Error when creating PropertyDescriptor for public final void org.apache.commons.configuration2.AbstractConfiguration.setProperty(java.lang.String,java.lang.Object)! Ignoring this property.

2020-07-08 14:06:53,800 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties

2020-07-08 14:06:53,893 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).

2020-07-08 14:06:53,893 INFO impl.MetricsSystemImpl: JobTracker metrics system started

2020-07-08 14:06:54,097 INFO input.FileInputFormat: Total input files to process : 1

2020-07-08 14:06:54,153 INFO mapreduce.JobSubmitter: number of splits:1

2020-07-08 14:06:54,352 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local191892762_0001

2020-07-08 14:06:54,354 INFO mapreduce.JobSubmitter: Executing with tokens: []

2020-07-08 14:06:54,561 INFO mapreduce.Job: The url to track the job: http://localhost:8080/

2020-07-08 14:06:54,562 INFO mapreduce.Job: Running job: job_local191892762_0001

2020-07-08 14:06:54,563 INFO mapred.LocalJobRunner: OutputCommitter set in config null

2020-07-08 14:06:54,571 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2

2020-07-08 14:06:54,572 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false

2020-07-08 14:06:54,573 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter

2020-07-08 14:06:54,624 INFO mapred.LocalJobRunner: Waiting for map tasks

2020-07-08 14:06:54,624 INFO mapred.LocalJobRunner: Starting task: attempt_local191892762_0001_m_000000_0

2020-07-08 14:06:54,661 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2

2020-07-08 14:06:54,662 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false

2020-07-08 14:06:54,683 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]

2020-07-08 14:06:54,856 INFO mapred.MapTask: Processing split: file:/software/hadoop/hadoop-3.0.0/input/README.txt:0+1366

2020-07-08 14:06:54,899 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)

2020-07-08 14:06:54,899 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100

2020-07-08 14:06:54,900 INFO mapred.MapTask: soft limit at 83886080

2020-07-08 14:06:54,900 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600

2020-07-08 14:06:54,900 INFO mapred.MapTask: kvstart = 26214396; length = 6553600

2020-07-08 14:06:54,907 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer

2020-07-08 14:06:54,928 INFO mapred.LocalJobRunner:

2020-07-08 14:06:54,929 INFO mapred.MapTask: Starting flush of map output

2020-07-08 14:06:54,929 INFO mapred.MapTask: Spilling map output

2020-07-08 14:06:54,929 INFO mapred.MapTask: bufstart = 0; bufend = 2055; bufvoid = 104857600

2020-07-08 14:06:54,929 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26213684(104854736); length = 713/6553600

2020-07-08 14:06:54,967 INFO mapred.MapTask: Finished spill 0

2020-07-08 14:06:54,986 INFO mapred.Task: Task:attempt_local191892762_0001_m_000000_0 is done. And is in the process of committing

2020-07-08 14:06:54,990 INFO mapred.LocalJobRunner: map

2020-07-08 14:06:54,990 INFO mapred.Task: Task 'attempt_local191892762_0001_m_000000_0' done.

2020-07-08 14:06:55,005 INFO mapred.Task: Final Counters for attempt_local191892762_0001_m_000000_0: Counters: 18

         File System Counters

                   FILE: Number of bytes read=317460

                   FILE: Number of bytes written=790256

                   FILE: Number of read operations=0

                   FILE: Number of large read operations=0

                   FILE: Number of write operations=0

         Map-Reduce Framework

                   Map input records=31

                   Map output records=179

                   Map output bytes=2055

                   Map output materialized bytes=1836

                   Input split bytes=116

                   Combine input records=179

                   Combine output records=131

                   Spilled Records=131

                   Failed Shuffles=0

                   Merged Map outputs=0

                   GC time elapsed (ms)=163

                   Total committed heap usage (bytes)=1587019776

         File Input Format Counters

                   Bytes Read=1366

2020-07-08 14:06:55,006 INFO mapred.LocalJobRunner: Finishing task: attempt_local191892762_0001_m_000000_0

2020-07-08 14:06:55,007 INFO mapred.LocalJobRunner: map task executor complete.

2020-07-08 14:06:55,011 INFO mapred.LocalJobRunner: Waiting for reduce tasks

2020-07-08 14:06:55,011 INFO mapred.LocalJobRunner: Starting task: attempt_local191892762_0001_r_000000_0

2020-07-08 14:06:55,023 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2

2020-07-08 14:06:55,023 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false

2020-07-08 14:06:55,024 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]

2020-07-08 14:06:55,028 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: [email protected]

2020-07-08 14:06:55,030 WARN impl.MetricsSystemImpl: JobTracker metrics system already initialized!

2020-07-08 14:06:55,053 INFO reduce.MergeManagerImpl: The max number of bytes for a single in-memory shuffle cannot be larger than Integer.MAX_VALUE. Setting it to Integer.MAX_VALUE

2020-07-08 14:06:55,053 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=20022505472, maxSingleShuffleLimit=2147483647, mergeThreshold=13214854144, ioSortFactor=10, memToMemMergeOutputsThreshold=10

2020-07-08 14:06:55,056 INFO reduce.EventFetcher: attempt_local191892762_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events

2020-07-08 14:06:55,092 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local191892762_0001_m_000000_0 decomp: 1832 len: 1836 to MEMORY

2020-07-08 14:06:55,096 INFO reduce.InMemoryMapOutput: Read 1832 bytes from map-output for attempt_local191892762_0001_m_000000_0

2020-07-08 14:06:55,098 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 1832, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->1832

2020-07-08 14:06:55,100 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning

2020-07-08 14:06:55,101 INFO mapred.LocalJobRunner: 1 / 1 copied.

2020-07-08 14:06:55,101 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs

2020-07-08 14:06:55,113 INFO mapred.Merger: Merging 1 sorted segments

2020-07-08 14:06:55,113 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 1823 bytes

2020-07-08 14:06:55,118 INFO reduce.MergeManagerImpl: Merged 1 segments, 1832 bytes to disk to satisfy reduce memory limit

2020-07-08 14:06:55,119 INFO reduce.MergeManagerImpl: Merging 1 files, 1836 bytes from disk

2020-07-08 14:06:55,120 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce

2020-07-08 14:06:55,120 INFO mapred.Merger: Merging 1 sorted segments

2020-07-08 14:06:55,120 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 1823 bytes

2020-07-08 14:06:55,121 INFO mapred.LocalJobRunner: 1 / 1 copied.

2020-07-08 14:06:55,133 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords

2020-07-08 14:06:55,143 INFO mapred.Task: Task:attempt_local191892762_0001_r_000000_0 is done. And is in the process of committing

2020-07-08 14:06:55,145 INFO mapred.LocalJobRunner: 1 / 1 copied.

2020-07-08 14:06:55,145 INFO mapred.Task: Task attempt_local191892762_0001_r_000000_0 is allowed to commit now

2020-07-08 14:06:55,147 INFO output.FileOutputCommitter: Saved output of task 'attempt_local191892762_0001_r_000000_0' to file:/software/hadoop/hadoop-3.0.0/output

2020-07-08 14:06:55,148 INFO mapred.LocalJobRunner: reduce > reduce

2020-07-08 14:06:55,148 INFO mapred.Task: Task 'attempt_local191892762_0001_r_000000_0' done.

2020-07-08 14:06:55,149 INFO mapred.Task: Final Counters for attempt_local191892762_0001_r_000000_0: Counters: 24

         File System Counters

                   FILE: Number of bytes read=321164

                   FILE: Number of bytes written=793418

                   FILE: Number of read operations=0

                   FILE: Number of large read operations=0

                   FILE: Number of write operations=0

         Map-Reduce Framework

                   Combine input records=0

                   Combine output records=0

                   Reduce input groups=131

                   Reduce shuffle bytes=1836

                   Reduce input records=131

                   Reduce output records=131

                   Spilled Records=131

                   Shuffled Maps =1

                   Failed Shuffles=0

                   Merged Map outputs=1

                   GC time elapsed (ms)=0

                   Total committed heap usage (bytes)=1587019776

         Shuffle Errors

                   BAD_ID=0

                   CONNECTION=0

                   IO_ERROR=0

                   WRONG_LENGTH=0

                   WRONG_MAP=0

                   WRONG_REDUCE=0

         File Output Format Counters

                   Bytes Written=1326

2020-07-08 14:06:55,150 INFO mapred.LocalJobRunner: Finishing task: attempt_local191892762_0001_r_000000_0

2020-07-08 14:06:55,150 INFO mapred.LocalJobRunner: reduce task executor complete.

2020-07-08 14:06:55,569 INFO mapreduce.Job: Job job_local191892762_0001 running in uber mode : false

2020-07-08 14:06:55,570 INFO mapreduce.Job:  map 100% reduce 100%

2020-07-08 14:06:55,572 INFO mapreduce.Job: Job job_local191892762_0001 completed successfully

2020-07-08 14:06:55,581 INFO mapreduce.Job: Counters: 30

         File System Counters

                   FILE: Number of bytes read=638624

                   FILE: Number of bytes written=1583674

                   FILE: Number of read operations=0

                   FILE: Number of large read operations=0

                   FILE: Number of write operations=0

         Map-Reduce Framework

                   Map input records=31

                   Map output records=179

                   Map output bytes=2055

                   Map output materialized bytes=1836

                   Input split bytes=116

                   Combine input records=179

                   Combine output records=131

                   Reduce input groups=131

                   Reduce shuffle bytes=1836

                   Reduce input records=131

                   Reduce output records=131

                   Spilled Records=262

                   Shuffled Maps =1

                   Failed Shuffles=0

                   Merged Map outputs=1

                   GC time elapsed (ms)=163

                   Total committed heap usage (bytes)=3174039552

         Shuffle Errors

                   BAD_ID=0

                   CONNECTION=0

                   IO_ERROR=0

                   WRONG_LENGTH=0

                   WRONG_MAP=0

                   WRONG_REDUCE=0

         File Input Format Counters

                   Bytes Read=1366

         File Output Format Counters

                   Bytes Written=1326

 

 

  1. Hadoop伪分布式模式

伪分布式只需要更改两个文件就够了，core-site.xml和hdfs-site.xml。这两个文件都在hadoop目录下的etc/hadoop中。

 

[email protected]:/software/hadoop/hadoop-3.0.0/etc/hadoop# vi core-site.xml

内容为：

<configuration>

 

<property>

    <name>fs.default.name</name>

    <value>hdfs://hadoop:9000</value>

</property>

<property>

    <name>fs.defaultFS</name>

    <value>hdfs://hadoop:9000</value>

</property>

<property>

    <name>hadoop.tmp.dir</name>

    <value>/software/hadoop/hadoop-3.0.0/tmp</value>

</property>

</configuration>

 

[email protected]:/software/hadoop/hadoop-3.0.0/etc/hadoop# vi hdfs-site.xml

内容为：

<configuration>

     <property>

                <name>dfs.replication</name>

                <value>1</value>

        </property>

        <property>

                <name>dfs.name.dir</name>

                <value>/software/hadoop/hadoop-3.0.0/hdfs/name</value>

        </property>

        <property>

                <name>dfs.data.dir</name>

                <value>/software/hadoop/hadoop-3.0.0/hdfs/data</value>

        </property>

</configuration>

 

[email protected]:/software/hadoop/hadoop-3.0.0/etc/hadoop# vi  mapred-site.xml

内容为：

<configuration>

<property>

    <name>mapreduce.framework.name</name>

    <value>yarn</value>

</property>

</configuration>

 

[email protected]:/software/hadoop/hadoop-3.0.0/etc/hadoop# vi  yarn-site.xml

内容为：

<property>

    <name>yarn.resourcemanager.hostname</name>

    <value>10.11.16.85</value>

</property>

<property>

    <name>yarn.nodemanager.aux-services</name>

    <value>mapreduce_shuffle</value>

</property>

 

</configuration>

配置完成后在 ./hadoop-3.0.0 (注意是自己的hadoop目录) 下使用以下命令 执行format命令，格式化名称节点

$ ./bin/hdfs namenode –format

Hadoop单机模式以及伪分布模式

Hadoop单机模式以及伪分布模式

Hadoop单机模式以及伪分布模式

Hadoop单机模式以及伪分布模式

  问题一：启动失败，

ERROR:but there is no HDFS_NAMENODE_USER defined . Aborting operation.

ERROR: but there is no HDFS_DATANODE_USER defined . Aborting operation.

but there is no HDFS_SECONDARYNAMENODE_USER defined . Aborting operation.

 

Hadoop单机模式以及伪分布模式

在Hadoop安装目录下找到sbin文件夹

 

在里面修改四个文件

 

对于start-dfs.sh和stop-dfs.sh文件，添加下列参数：

 

#!/usr/bin/env bash

HDFS_DATANODE_USER=root

HADOOP_SECURE_DN_USER=hdfs

HDFS_NAMENODE_USER=root

HDFS_SECONDARYNAMENODE_USER=root

对于start-yarn.sh和stop-yarn.sh文件，添加下列参数：

 

#!/usr/bin/env bash

YARN_RESOURCEMANAGER_USER=root

HADOOP_SECURE_DN_USER=yarn

YARN_NODEMANAGER_USER=root

重新开始start...就可以了。 

 

 

 

 

 

 

问题二：JAVA_HOME未配置：

vi etc/hadoop/hadoop-env.sh

增加：

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-arm64/

Hadoop单机模式以及伪分布模式

 

Hadoop单机模式以及伪分布模式

Hadoop单机模式以及伪分布模式

问题三：系统架构差异导致：

[email protected]:~/hadoop-3.0.0-src# hadoop checknative -a

2020-07-08 16:36:43,455 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...

2020-07-08 16:36:43,457 DEBUG util.NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: /software/hadoop/hadoop-3.0.0/lib/native/libhadoop.so.1.0.0: /software/hadoop/hadoop-3.0.0/lib/native/libhadoop.so.1.0.0: 无法打开共享对象文件: 没有那个文件或目录 (Possible cause: can't load AMD 64-bit .so on a AARCH64-bit platform)

2020-07-08 16:36:43,457 DEBUG util.NativeCodeLoader: java.library.path=/software/hadoop/hadoop-3.0.0//lib/native

2020-07-08 16:36:43,457 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

2020-07-08 16:36:43,670 DEBUG util.Shell: setsid exited with exit code 0

Native library checking:

hadoop:  false

zlib:    false

zstd  :  false

snappy:  false

lz4:     false

bzip2:   false

openssl: false

ISA-L:   false

2020-07-08 16:36:43,690 DEBUG util.ExitUtil: Exiting with status 1: ExitException

1: ExitException

         at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:304)

         at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:292)

         at org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:160)

2020-07-08 16:36:43,693 INFO util.ExitUtil: Exiting with status 1: ExitException

 

2020-07-08 15:02:21,900 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

 

（注：可以学习的BUG调试方式）

使用hadoop命令警告提示：Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

我的系统版本：centos6.5 64位

我的hadoop版本：2.1.7 64位

 

网上查看了很多问题解决方法，有的说系统位数不对，有的说hadoop版本是32位，等等，因为我的都是64位，所以这些和我的问题不一致

方法是：打开hadoop调试，跟踪错误信息

hadoop调试信息开启关闭：

开启：export HADOOP_ROOT_LOGGER=DEBUG,console

关闭：export HADOOP_ROOT_LOGGER=INFO,console

 

查看内核版本命令：cat /proc/version

查看linux版本： lsb_release -a

查看hadoop版本和位数：进入到hadoop目录/lib/native

使用命令：file libhadoop so1.0.0

hadoop checknative -a 查看lib是否有错误

 

参考资料http://blog.sina.com.cn/s/blog_4eca88390102vn86.html

http://www.aboutyun.com/blog-9721-878.html#userconsent#

http://www.aboutyun.com/blog-9721-878.html#userconsent#

http://www.linuxidc.com/Linux/2012-04/59200.htm

 

解决过程：

 

1,进入到hadoop目录下，lib，native

ll /lib64/libc.so.6，查看库的版本

lrwxrwxrwx. 1 root root 12 11月 3 23:06 /lib64/libc.so.6 -> libc-2.12.so 2.12的版本，现在要把它升级到2.14

 

2，下载glibc-2.14.tar.bz2

下载glibc-linuxthreads-2.5.tar.bz2

拷贝到 /home/hadoop/software目录下

 

3，tar -jxvf /home/hadoop/software/glibc-2.14.tar.bz2 解压

cd glibc-2.14/

tar -jxvf /home/hadoop/software/glibc-linuxthreads-2.5.tar.bz2

export CFLAGS="-g -O2" #加上优化开关，否则会出现错误

cd .. #必须返回上级目录

 

export CFLAGS="-g -O2" #加上优化开关，否则会出现错误

（下面这句出错）

./glibc-2.14/configure --prefix=/usr --disable-profile --enable-add-ons --with-headers=/usr/include --with-binutils=/usr/bin

[[email protected] native]# ./glibc-2.14/configure --prefix=/usr --disable-profile --enable-add-ons --with-headers=/usr/include --with-binutils=/usr/bin

checking build system type... x86_64-unknown-linux-gnu

checking host system type... x86_64-unknown-linux-gnu

checking for gcc... no

checking for cc... no

checking for cl.exe... no

configure: error: in `/root/soft/hadoop-2.7.1/lib/native':

configure: error: no acceptable C compiler found in $PATH

See `config.log' for more details

4，现在提示没有编译工具，那么安装；

 

1.安装gcc|gc++

yum install gcc

yum install gcc-c++

验证

 

现在再来执行这句话；

./glibc-2.14/configure --prefix=/usr --disable-profile --enable-add-ons --with-headers=/usr/include --with-binutils=/usr/bin

可以成功运行了

 

make（需要很长时间），如果这句话报错，安装make

yum install make

验证： make --version

 

sudo make install #安装，必须root用户执行

 

make: *** [install] 错误 2 显示错误两个，先不理他

ll /lib64/libc.so.6 查看现在我的libc的版本

lrwxrwxrwx. 1 root root 12 11月 18 21:13 /lib64/libc.so.6 -> libc-2.14.so 2.14，说明升级成功了

增加调试信息

export HADOOP_ROOT_LOGGER=DEBUG,console

hadoop fs -text /test/data/origz/access.log.gz 验证本机的版本

15/11/18 13:15:30 DEBUG util.NativeCodeLoader: Loaded the native-hadoop library 说明也是对的，有报错，先不管这个

 

,5，然后我把hadoop集群重启一遍；

/root/soft/hadoop-2.7.1/sbin 进入到这个目录

./start-all.sh

./stop-all.sh

export HADOOP_ROOT_LOGGER=INFO,console 关闭调试

 

,6，hadoop fs -ls / 测试效果，发现hadoop01拒绝连接，把它打开

警告消失了，说明升级libc是有用的。

至此Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

问题得到解决

之前 hadoop checknative -a显示全部是失败，现在都变好了

(ps.问题三编译不通过，未解决)

 

问题四：HDFS处于安全模式

Hadoop单机模式以及伪分布模式

解决方法 退出安全模式hdfs dfsadmin -safemode leave

 

问题五：错误: 找不到或无法加载主类org.apache.hadoop.mapreduce.v2.app.MRAppMaster

Hadoop单机模式以及伪分布模式

解决方法：

查看hadoop classpath：    $ hadoop classpath

 

<property>

  <name>mapreduce.application.classpath</name>

      <value>

       /software/hadoop/hadoop-3.0.0/etc/*,

           /software/hadoop/hadoop-3.0.0/etc/hadoop/*,

           /software/hadoop/hadoop-3.0.0/lib/*,

           /software/hadoop/hadoop-3.0.0/share/hadoop/common/*,

           /software/hadoop/hadoop-3.0.0/share/hadoop/common/lib/*,

           /software/hadoop/hadoop-3.0.0/share/hadoop/mapreduce/*,

           /software/hadoop/hadoop-3.0.0/share/hadoop/mapreduce/lib-examples/*,

           /software/hadoop/hadoop-3.0.0/share/hadoop/hdfs/*,

           /software/hadoop/hadoop-3.0.0/share/hadoop/hdfs/lib/*,

           /software/hadoop/hadoop-3.0.0/share/hadoop/yarn/*,

           /software/hadoop/hadoop-3.0.0/share/hadoop/yarn/lib/*,

      </value>

</property> # 编辑yarn-site.xml

 

vim yarn-site.xml

添加如下内容

 

<property>

  <name>mapreduce.application.classpath</name>

      <value>

       /software/hadoop/hadoop-3.0.0/etc/*,

           /software/hadoop/hadoop-3.0.0/etc/hadoop/*,

           /software/hadoop/hadoop-3.0.0/lib/*,

           /software/hadoop/hadoop-3.0.0/share/hadoop/common/*,

           /software/hadoop/hadoop-3.0.0/share/hadoop/common/lib/*,

           /software/hadoop/hadoop-3.0.0/share/hadoop/mapreduce/*,

           /software/hadoop/hadoop-3.0.0/share/hadoop/mapreduce/lib-examples/*,

           /software/hadoop/hadoop-3.0.0/share/hadoop/hdfs/*,

           /software/hadoop/hadoop-3.0.0/share/hadoop/hdfs/lib/*,

           /software/hadoop/hadoop-3.0.0/share/hadoop/yarn/*,

           /software/hadoop/hadoop-3.0.0/share/hadoop/yarn/lib/*,

      </value>

</property>重启yarn

Ps: /software/hadoop/hadoop-3.0.0/etc/*,

不要写错：/software/hadoop/hadoop-3.0.0/etc/hadooop/*,

否则失败提醒依旧如下错误六

错误六：Application application_1594361965641_0002 failed 2 times due to AM Container for appattempt_1594361965641_0002_000002 exited with  exitCode: 1

[email protected]:/software/hadoop/hadoop-3.0.0# hadoop jar share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-3.0.0-test-sources.jar org.apache.hadoop.examples.WordCount hdfs://hadoop:9000/logs/ hdfs://hadoop:9000/output

2020-07-10 14:20:29,154 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

2020-07-10 14:20:30,421 INFO client.RMProxy: Connecting to ResourceManager at /10.11.16.85:8032

2020-07-10 14:20:31,130 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1594361965641_0002

2020-07-10 14:20:32,057 INFO input.FileInputFormat: Total input files to process : 25

2020-07-10 14:20:32,538 INFO mapreduce.JobSubmitter: number of splits:25

2020-07-10 14:20:32,585 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled

2020-07-10 14:20:32,719 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1594361965641_0002

2020-07-10 14:20:32,722 INFO mapreduce.JobSubmitter: Executing with tokens: []

2020-07-10 14:20:32,995 INFO conf.Configuration: resource-types.xml not found

2020-07-10 14:20:32,995 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.

2020-07-10 14:20:33,390 INFO impl.YarnClientImpl: Submitted application application_1594361965641_0002

2020-07-10 14:20:33,448 INFO mapreduce.Job: The url to track the job: http://hadoop:8088/proxy/application_1594361965641_0002/

2020-07-10 14:20:33,449 INFO mapreduce.Job: Running job: job_1594361965641_0002

2020-07-10 14:20:41,539 INFO mapreduce.Job: Job job_1594361965641_0002 running in uber mode : false

2020-07-10 14:20:41,540 INFO mapreduce.Job:  map 0% reduce 0%

2020-07-10 14:20:41,561 INFO mapreduce.Job: Job job_1594361965641_0002 failed with state FAILED due to: Application application_1594361965641_0002 failed 2 times due to AM Container for appattempt_1594361965641_0002_000002 exited with  exitCode: 1

Failing this attempt.Diagnostics: [2020-07-10 14:20:41.308]Exception from container-launch.

Container id: container_1594361965641_0002_02_000001

Exit code: 1

 

[2020-07-10 14:20:41.314]Container exited with a non-zero exit code 1. Error file: prelaunch.err.

Last 4096 bytes of prelaunch.err :

Last 4096 bytes of stderr :

log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster).

log4j:WARN Please initialize the log4j system properly.

log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

 

 

[2020-07-10 14:20:41.315]Container exited with a non-zero exit code 1. Error file: prelaunch.err.

Last 4096 bytes of prelaunch.err :

Last 4096 bytes of stderr :

log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster).

log4j:WARN Please initialize the log4j system properly.

log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

 

 

For more detailed output, check the application tracking page: http://hadoop:8088/cluster/app/application_1594361965641_0002 Then click on links to logs of each attempt.

. Failing the application.

2020-07-10 14:20:41,597 INFO mapreduce.Job: Counters: 0

解决方式同第五点

  1. Hadoop伪集群模式Word Count测试

HDFS退出安全模式：hdfs dfsadmin -safemode leave

MR运行：

[email protected]:/software/hadoop/hadoop-3.0.0# hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0.jar wordcount /logs/ /output

Hadoop单机模式以及伪分布模式

Hadoop单机模式以及伪分布模式

相关推荐