hadoop本地库版本问题
问题如下:配置完hadoop-env.sh,装好native库后,hadoop checknative仍然无法找到任何库
错误如下:推荐一个实用技巧,手动设置hadoop shell debug模式:
export HADOOP_ROOT_LOGGER=DEBUG,console(直接输入到命令行就行)
18/08/22 17:12:28 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
18/08/22 17:12:28 DEBUG util.NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: /opt/beh/core/hadoop/lib/native/libhadoop.so
: /lib64/libc.so .6: version `GLIBC_2.14' not found (required by /opt/beh/core/hadoop/lib/native/libhadoop.so )
18/08/22 17:12:28 DEBUG util.NativeCodeLoader: java.library.path=/opt/beh/core/hadoop/lib/native
18/08/22 17:12:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/08/22 17:12:28 DEBUG util.Shell: setsid exited with exit code 0
Native library checking:
hadoop: false
zlib: false
snappy: false
lz4: false
bzip2: false
openssl: false
18/08/22 17:12:28 INFO util.ExitUtil: Exiting with status 1
问题分析:看报错一时间没反应过来为啥,也没有去搜索,旁边同事告诉我可能是centos版本的问题,这个错也符合这个情况,就没有搜索,开始尝试修改
软件版本:hadoop-2.6.0-cdh5.13.1,Centos6
解决:刚开始的native包是CDH-5.13.1-1.cdh5.13.1.p0.2-el7.parcel里面找的native库,然后我就用以前的cdh5.1.3.-el6的hadoop里面的native替代这个native(就在一台机器,方便拿),结果如下:
18/08/22 18:50:50 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
18/08/22 18:50:50 DEBUG util.NativeCodeLoader: Loaded the native-hadoop library
18/08/22 18:50:50 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
18/08/22 18:50:50 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
18/08/22 18:50:50 DEBUG crypto.OpensslCipher: Failed to load OpenSSL Cipher.
java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsOpenssl()Z
at org.apache.hadoop.util.NativeCodeLoader.buildSupportsOpenssl(Native Method)
at org.apache.hadoop.crypto.OpensslCipher.<clinit>(OpensslCipher.java:84)
at org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:87)
18/08/22 18:50:50 DEBUG util.Shell: setsid exited with exit code 0
Native library checking:
hadoop: true /opt/beh/core/hadoop/lib/native/libhadoop.so
zlib: true /lib64/libz.so.1
snappy: true /opt/beh/core/hadoop/lib/native/libsnappy.so.1
lz4: true revision:99
bzip2: true /lib64/libbz2.so.1
openssl: false org.apache.hadoop.util.NativeCodeLoader.buildSupportsOpenssl()Z
当时以为没事,后来hbase一直起不了,报错跟这个一样,且hdfs 不能put文件,都是这个错。。。。
最后的手段就是着手下载CDH-5.13.1-1.cdh5.13.1.p0.2-el6.parcel,然后传到linux上解压(Windows解压不全,而且正好没有需要的native库(=|,|=))
完整路径:CDH/lib/hadoop/lib/native
把native拷贝到集群上,替换掉原有的native
然后就是一切ok了,有几台机器出现:openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)!
这个直接从好的机器拷贝过来到响应目录下就ok了
教训:版本真重要!