hive的DDL操作,报错提示0 datanode(s) running and no node(s) are excluded in this operation.

1、在对hive进行DDL操作时,如下错错误:

FAILED: SemanticException [Error 10293]: Unable to create temp file for insert values File /tmp/hive/hadoop/d3007212-b33a-41c4-9bb2-3dbd86c8bb06/_tmp_space.db/Values__Tmp__Table__2/data_file could only be replicated to 0 nodes instead of minReplication (=1).  There are 0 datanode(s) running and no node(s) are excluded in this operation.

hive的DDL操作,报错提示0 datanode(s) running and no node(s) are excluded in this operation.

2、原因:

      没有datanode开启;根本原因是在开启HA集群时,多次执行hdfs namenode -format对namenode格式化造成;而每次格式化时,namenode会更新clusterID,但是datanode只会在首次格式化时确定,因此就造成不一致现象。从而导致namenode的VERSION文件中的clusterID也会随之变化,而datanode的VERSION文件中的clusterID保持不变,造成两个clusterID不一致,从而造成启动集群时无法启动DataNode,最终页面显示节点1和节点的状态都是standby状态。yarn页面显示拒绝连接请求。

hive的DDL操作,报错提示0 datanode(s) running and no node(s) are excluded in this operation.

hive的DDL操作,报错提示0 datanode(s) running and no node(s) are excluded in this operation.

hive的DDL操作,报错提示0 datanode(s) running and no node(s) are excluded in this operation.

3、解决办法:

(1)查看namenode中的 version文件中的ClusterIddatanode下的version文件中的ClusterId是否一致;经查看,两者id确实不同。

进入目录install/hadoop-2.6.0-cdh5.14.2/hadoopDatas/tempDatas/dfs/data/current下,

vim VERSION文件:

storageID=DS-cb620da7-ef8a-4cd6-a97d-1c3bd293fe30

clusterID=CID-19e05b08-2f12-4e0f-8996-046303f82434

cTime=0

datanodeUuid=b42f2c24-95d1-49ae-aaff-c9b5981b3135

storageType=DATA_NODE

layoutVersion=-56

进入目录/install/hadoop-2.6.0-cdh5.14.2/hadoopDatas/tempDatas/dfs/name/current

namespaceID=1817146503

clusterID=CID-f08aabde-cf9a-4d0b-ace0-1bbc56ca861a

cTime=0

storageType=NAME_NODE

blockpoolID=BP-1385319452-192.168.52.120-1585065097522

layoutVersion=-60

(2)删掉相关文件

将三台机器中如下目录中的文件全部删除:

dfs目录:

/install/hadoop-2.6.0-cdh5.14.2/hadoopDatas/dfs/jn/hann/current

/install/hadoop-2.6.0-cdh5.14.2/hadoopDatas/dfs/nn/snn/edits/current

/install/hadoop-2.6.0-cdh5.14.2/hadoopDatas/dfs/snn/name/current

datanodeDatas目录:

/install/hadoop-2.6.0-cdh5.14.2/hadoopDatas/datanodeDatas/current

namenodeDatas目录:

/install/hadoop-2.6.0-cdh5.14.2/hadoopDatas/namenodeDatas/current

tempDatas目录:

/install/hadoop-2.6.0-cdh5.14.2/hadoopDatas/tempDatas

(3)在三台机器上启动JournalNode

hadoop-daemon.sh start journalnode

(4)在第一个NameNode上进行格式化(注意,本文出现的原因就是多次执行了此步操作;此步从搭建HA集群时,只能操作一次,切记切记!!)

hadoop namenode -format

(5)在第一个NameNode上启动NameNode

hadoop-daemon.sh start namenode

6第二个namenode同步第一个namenode

hdfs namenode -bootstrapStandby

7启动第二个namenode

hadoop-daemon.sh start namenode

8启动zookeeper集群,并进行格式化(先jps查看zookeeper是否已启动

在三台机器上执行

zkServer.sh start

jps查看出现 QuorumPeerMain说明zookeeper已启动

9然后在node01上执行

hdfs zkfc -formatZK

10)启动hdfs集群

cd /install/hadoop-2.6.0-cdh5.14.2/sbin

start-dfs.sh

start-yarn.sh

 mr-jobhistory-daemon.sh start historyserver

11)网页查看

node01可查看页面:http://node01:50070/

hive的DDL操作,报错提示0 datanode(s) running and no node(s) are excluded in this operation.

Node02节点可查看hdfs集群yarn集群:http://node02:50070/ 和 http://node02:8088/

hive的DDL操作,报错提示0 datanode(s) running and no node(s) are excluded in this operation.

hive的DDL操作,报错提示0 datanode(s) running and no node(s) are excluded in this operation.

Node03可查看jobhistory:http://node03:19888/jobhistory

hive的DDL操作,报错提示0 datanode(s) running and no node(s) are excluded in this operation.