es磁盘满处理

curl -XGET 'http://192.168.0.131:9200/_cat/indices/?v' -uelastic

查询当前所有索引

could not store triggered watch with id [uJHz02SGQsmgOmEK1QkLdg_xpack_license_expiration_7447c0a1-2186-49fb-b7e1-332c47ada93d-2019-01-13T08:46:20.194Z]: [ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]

 

那么报这个错的原因是:存储不足,当es存储不足es磁盘满处理时候, kibana会将其索引的读写权限设置为只读read-only mode

 

  1. PUT .kibana/_settings
  2. {
  3. "index": {
  4. "blocks": {
  5. "read_only_allow_delete": "false"
  6. }
  7. }
  8. }

curl http://192.168.0.131:9200/_cluster/health?pretty -u elastic

Enter host password for user 'elastic':

{

"cluster_name" : "tk-log",

"status" : "green",

"timed_out" : false,

"number_of_nodes" : 3,

"number_of_data_nodes" : 3,

"active_primary_shards" : 1137,

"active_shards" : 2274,

"relocating_shards" : 1,

"initializing_shards" : 0,

"unassigned_shards" : 0,

"delayed_unassigned_shards" : 0,

"number_of_pending_tasks" : 0,

"number_of_in_flight_fetch" : 0,

"task_max_waiting_in_queue_millis" : 0,

"active_shards_percent_as_number" : 100.0

}

curl -XPUT -H "Content-Type: application/json" http://192.168.0.131:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}' -uelastic

Enter host password for user 'elastic':

{"acknowledged":true}[[email protected] home]# tail -f /var/log/elasticsearch/tk-log.log

服务器执行如上命令,看日志[tk-01] [.watcher-history-9-2019.01.13] creating index, cause [auto(bulk api)], templates [.watch-history-9], shards [1]/[0], mappings [doc]

[2019-01-13T16:51:10,703][INFO ][o.e.c.r.a.AllocationService] [tk-01] updating number_of_replicas to [1] for indices [.watcher-history-9-2019.01.13]

恢复正常

!不能在这些设置中混合使用百分比值和字节值。全部都设置为百分比值,或全部设置为字节值。

设定:cluster.routing.allocation.disk.watermark.low

控制磁盘使用的低水位线。它默认为85%,这意味着一旦使用超过85%的磁盘,ES将不会为节点分配新的分片。它也可以设置为绝对字节值(如500mb),以防止ES在小于配置的可用空间量时分配分片。

设定:cluster.routing.allocation.disk.watermark.high

控制高水印。默认为90%,这意味着如果节点磁盘使用率超过90%,ES将尝试将分片重定位到另一个节点。它也可以设置为绝对字节值(类似于低水印),以便重新定位分片,一次小于节点上可用的配置空间量。

设置 ::cluster.routing.allocation.disk.watermark.flood_stage

控制洪水阶段水印。它默认为95%,这意味着Elasticsearch对每个索引强制执行只读索引块(index.blocks.read_only_allow_delete),该索引在至少有一个磁盘超过泛洪阶段的节点上分配了一个或多个分片。这是防止节点耗尽磁盘空间的最后手段。一旦有足够的可用磁盘空间允许索引操作继续,就必须手动释放索引块