Elasticsearch 总结

Downloading .........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................DONE

Verifying https://github.com/mobz/elasticsearch-head/archive/master.zip checksums if available ...

NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify)

Installed head into /home/elasticsearch/elasticsearch-2.1.1/plugins/head

或者在elasticsearch的目录下创建以下目录，把https://github.com/mobz/elasticsearch-head/archive/master.zip下载后的压缩包解压到_site目录下即可，操作如下:

[[email protected] elasticsearch-2.3.1]# mkdir -p plugins/head/

[[email protected] elasticsearch-2.3.1]# unzip elasticsearch-head-master.zip -C /opt/elasticsearch-2.3.1/plugins/head/

[[email protected] head]# cd elasticsearch-head-master

[[email protected] elasticsearch-head-master]# mv * ../

[[email protected] elasticsearch-head-master]# cd ../

[[email protected] head]# rm -rf elasticsearch-head-master

[[email protected] head]# chmod -R es:elsearch *

B）、license 安装

[[email protected] bin]# ./plugin install license

-> Installing license...

Trying https://download.elastic.co/elasticsearch/release/org/elasticsearch/plugin/license/2.3.1/license-2.3.1.zip ...

Downloading .......DONE

Verifying https://download.elastic.co/elasticsearch/release/org/elasticsearch/plugin/license/2.3.1/license-2.3.1.zip checksums if available ...

Downloading .DONE

Installed license into /usr/local/elasticsearch-2.3.1/plugins/license

C）、

1-6）、配置集群

[[email protected] config]# vi elasticsearch.yml

4.修改配置

vi /bigdata/elasticsearch-2.3.1/config/elasticsearch.yml

#集群名称，通过组播的方式通信，通过名称判断属于哪个集群

cluster.name: bigdata

#节点名称，要唯一

node.name: es-1

#数据存放位置

path.data: /data/to/data

#日志存放位置

path.logs: /data/to/logs

#es绑定的ip地址

network.host: 172.16.0.14

# 端口

http.port: 9200

#初始化时可进行选举的节点

discovery.zen.ping.unicast.hosts: ["hadoop1", "hadoop2", "hadoop3"]

1-7）、传送到其他的机器上

[[email protected] elasticsearch]# scp elasticsearch-2.1.1 hadoop2:$PWD

[[email protected] elasticsearch]# scp elasticsearch-2.1.1 hadoop3:$PWD

1-8）、修改配置文件

[[email protected] elasticsearch-2.3.1]# cd config/

[[email protected] config]# vi elasticsearch.yml

# ======================== Elasticsearch Configuration =========================

# NOTE: Elasticsearch comes with reasonable defaults for most settings.

# Before you set out to tweak and tune the configuration, make sure you

# understand what are you trying to accomplish and the consequences.

# The primary way of configuring a node is via this file. This template lists

# the most important settings you may want to configure for a production cluster.

# Please see the documentation for further information on configuration options:

# <http://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html>

# ---------------------------------- Cluster -----------------------------------

# Use a descriptive name for your cluster:

cluster.name: my-application

# ------------------------------------ Node ------------------------------------

# Use a descriptive name for the node:

node.name: node-1

# Add custom attributes to the node:

# node.rack: r1

# ----------------------------------- Paths ------------------------------------

# Path to directory where to store the data (separate multiple locations by comma):

path.data: /path/to/data

# Path to log files:

path.logs: /path/to/logs

# ----------------------------------- Memory -----------------------------------

# Lock the memory on startup:

# bootstrap.mlockall: true

# Make sure that the `ES_HEAP_SIZE` environment variable is set to about half the memory

# available on the system and that the owner of the process is allowed to use this limit.

# Elasticsearch performs poorly when the system is swapping the memory.

# ---------------------------------- Network -----------------------------------

# Set the bind address to a specific IP (IPv4 or IPv6):

network.host: 192.168.215.134

# Set a custom port for HTTP:

http.port: 9200

# For more information, see the documentation at:

# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html>

# --------------------------------- Discovery ----------------------------------

# Pass an initial list of hosts to perform discovery when new node is started:

# The default list of hosts is ["127.0.0.1", "[::1]"]

discovery.zen.ping.unicast.hosts: ["hadoop1", "hadoop2", "hadoop3"]

# Prevent the "split brain" by configuring the majority of nodes (total number of nodes / 2 + 1):

# discovery.zen.minimum_master_nodes: 3

# For more information, see the documentation at:

# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery.html>

# ---------------------------------- Gateway -----------------------------------

# Block initial recovery after a full cluster restart until N nodes are started:

# gateway.recover_after_nodes: 3

# For more information, see the documentation at:

# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-gateway.html>

# ---------------------------------- Various -----------------------------------

# Disable starting multiple nodes on a single system:

# node.max_local_storage_nodes: 1

# Require explicit names when deleting indices:

# action.destructive_requires_name: true

1-9）、运行Elasticsearch

A）、前台运行

[[email protected] bin]$ ./elasticsearch

[2016-08-27 14:18:15,854][INFO ][node ] [Trader] version[2.1.1], pid[7894], build[40e2c53/2015-12-15T13:05:55Z]

[2016-08-27 14:18:15,855][INFO ][node ] [Trader] initializing ...

[2016-08-27 14:18:16,743][INFO ][plugins ] [Trader] loaded [], sites []

[2016-08-27 14:18:16,993][INFO ][env ] [Trader] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [11gb], net total_space [17.6gb], spins? [unknown], types [rootfs]

[2016-08-27 14:18:22,613][INFO ][node ] [Trader] initialized

[2016-08-27 14:18:22,613][INFO ][node ] [Trader] starting ...

[2016-08-27 14:18:22,856][INFO ][transport ] [Trader] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}

[2016-08-27 14:18:22,874][INFO ][discovery ] [Trader] elasticsearch/HW-4e50uSkeSFEeNgEkMJQ

[2016-08-27 14:18:26,020][INFO ][cluster.service ] [Trader] new_master {Trader}{HW-4e50uSkeSFEeNgEkMJQ}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)

[2016-08-27 14:18:26,242][INFO ][http ] [Trader] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}, {[::1]:9200}

[2016-08-27 14:18:26,242][INFO ][node ] [Trader] started

[2016-08-27 14:18:26,257][INFO ][gateway ] [Trader] recovered [0] indices into cluster_state

^C[2016-08-27 14:21:54,274][INFO ][node ] [Trader] stopping ...

[2016-08-27 14:21:54,361][INFO ][node ] [Trader] stopped

[2016-08-27 14:21:54,362][INFO ][node ] [Trader] closing ...

[2016-08-27 14:21:54,371][INFO ][node ] [Trader] closed

B）、后端运行

[[email protected] bin]$ ./elasticsearch -d

C）、查看进程

bash-4.1$ jps

21177 Elasticsearch

21234 Jps

[[email protected] bin]$ jps

14643 Jps

14606 Elasticsearch

[[email protected] bin]$ jps

12896 Jps

12875 Elasticsearch

D）、查看界面

http://hadoop1:9200/_plugin/head/

从中可以看出node-2是主节点，其他的副节点，其中0,1,2,3,4是leader其他的都是flowller是同步数据的。

1-10）、停止ES

若是要停止ES服务，则输入sh elasticsearch stop，则输出如下图，则ES成功停止。

kill `ps -ef | grep Elasticsearch | grep -v grep | awk '{print $2}'`

1-11）、查看Zookeeper保存的信息

1-12）、一键启动脚本

请用es用户执行以下脚本，注意脚本的权限

[[email protected] start-single]$ ls -l

-rwxr-xr-x. 1 es elsearch 166 Apr 23 11:14 start-elasticsearch.sh

执行脚本

[[email protected] start-single]$ sh start-elasticsearch.sh

cat /opt/start-script/slave |while read line

{

echo "$line"

ssh $line "source /etc/profile;su es;cd /opt/elasticsearch-2.3.1/bin;sh elasticsearch -d"

wait

Done

Curl 常用操作

1-1）、检测集群是否健康

[[email protected] root]$ curl 'hadoop1:9200/_cat/health?v'

epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent

1493390489 22:41:29 my-application green 3 3 12 6 0 0 0 0 - 100.0%

1-2）、查看集群列表

[[email protected] root]$ curl 'hadoop1:9200/_cat/nodes?v'

host ip heap.percent ram.percent load node.role master name

192.168.132.148 192.168.132.148 5 40 0.00 d m node-

192.168.132.139 192.168.132.139 2 38 0.00 d m node-3

192.168.132.140 192.168.132.140 5 35 0.00 d * node-1

1-3）、查看所有的索引

[[email protected] root]$ curl 'hadoop1:9200/_cat/indices?v'

health status index pri rep docs.count docs.deleted store.size pri.store.size

green open .kibana 1 1 1 0 6.3kb 3.1kb

green open customer 5 1 45 0 117.7kb 58.8kb

1-4）、创建索引

[[email protected] root]$ curl -XPUT 'hadoop1:9200/customer?pretty'

会看到以下信息

{

"acknowledged" : true

}

1-5）、插入数据

[[email protected] root]$ curl -XPUT 'hadoop1:9200/customer/external/1?pretty' -d '{"name":"xiaozhang","age":"18"}'

回车可以看到以下信息

{

"_index" : "customer",

"_type" : "external",

"_id" : "1",

"_version" : 1,

"_shards" : {

"total" : 2,

"successful" : 2,

"failed" : 0

"created" : true

}

1-6）、获取数据

[[email protected] root]$ curl -XGET 'hadoop1:9200/customer/external/1?pretty'

回车即可看到以下信息

{

"_index" : "customer",

"_type" : "external",

"_id" : "1",

"_version" : 1,

"found" : true,

"_source" : {

"name" : "xiaozhang",

"age" : "18"

}

1-7）、删除索引

[[email protected] root]$ curl -XDELETE 'hadoop1:9200/customer?pretty'

{

"acknowledged" : true

}

1-8）、修改数据

先查看id为3的数据

[[email protected] root]$ curl -XPUT 'hadoop1:9200/customer/external/3?pretty' -d '{"name":"xiaoxu","age":"23"}'

1-9）、更新数据

查看修改ID为1的数据

[[email protected] root]$ curl -XPOST 'hadoop1:9200/customer/external/1/_update?pretty' -d '{"doc":{"name":"Jane Han"}}'

1-10）、删除数据

删除ID为1的数据

[[email protected] root]$ curl -XDELETE 'hadoop1:9200/customer/external/1?pretty'

1-11）、批处理数据

JSON文件下载地址：链接：http://pan.baidu.com/s/1bLZqn8 密码：m6he

[[email protected] testData]$ curl -XPOST 'hadoop1:9200/bank/account/_bulk?pretty' --data-binary "@/opt/testData/accounts.json"

1-12）、常见错误处理

问题描述

elasticsearch6.x {"error":"Content-Type header [application/x-www-form-urlencoded] is not supported" }

问题解答

[[email protected] root]$ curl -H "Content-Type: application/json" -XDELETE 'hadoop1:9200/customer/external/1?pretty'

代码实例

在项目中导入项目中/elasticsearch-2.3.1/lib

1-1）、实体类

package com.sdcet;

public class Medicine {

private Integer id;

private String name;

private String function;

public Integer getId() {

return id;

}

public void setId(Integer id) {

this.id = id;

}

public String getName() {

return name;

}

public void setName(String name) {

this.name = name;

}

public String getFunction() {

return function;

}

public void setFunction(String function) {

this.function = function;

}

public Medicine() {

}

public Medicine(Integer id, String name, String function) {

this.id = id;

this.name = name;

this.function = function;

}

@Override

public String toString() {

return "Medicine [id=" + id + ", name=" + name + ", function="

+ function + "]";

}

1-2）、工具类

package com.sdcet;

import java.io.IOException;

import org.elasticsearch.common.xcontent.XContentBuilder;

import org.elasticsearch.common.xcontent.XContentFactory;

public class JsonUtil {

/**

* 实现将实体对象转换成json对象

* @param medicine

* Medicine对象

* @return

public static String obj2JsonData(Medicine medicine) {

String jsonData = null;

try {

// 使用XContentBuilder创建json数据

XContentBuilder jsonBuild = XContentFactory.jsonBuilder();

jsonBuild.startObject().field("id", medicine.getId())

.field("name", medicine.getName())

.field("funciton", medicine.getFunction()).endObject();

jsonData = jsonBuild.string();

System.out.println(jsonData);

} catch (IOException e) {

e.printStackTrace();

}

return jsonData;

}

1-3）、工厂类

package com.sdcet;

import java.util.ArrayList;

import java.util.List;

public class DataFactory {

public static DataFactory dataFactory = new DataFactory();

private DataFactory() {

}

public DataFactory getInstance() {

return dataFactory;

}

public static List<String> getInitJsonData() {

List<String> list = new ArrayList<String>();

for (int i = 0; i < 10; i++) {

String data1 = JsonUtil.obj2JsonData(new Medicine(i, "银花感冒颗粒",

"功能主治：银花感冒颗粒，头痛,清热，解表，利咽。"));

list.add(data1);

}

return list;

}

1-4）、主类

package com.sdcet;

import java.net.InetAddress;

import java.net.UnknownHostException;

import java.util.ArrayList;

import java.util.List;

import org.elasticsearch.action.index.IndexRequestBuilder;

import org.elasticsearch.action.index.IndexResponse;

import org.elasticsearch.action.search.SearchResponse;

import org.elasticsearch.client.Client;

import org.elasticsearch.client.transport.TransportClient;

import org.elasticsearch.common.transport.InetSocketTransportAddress;

import org.elasticsearch.index.query.QueryBuilder;

import org.elasticsearch.index.query.QueryBuilders;

import org.elasticsearch.search.SearchHit;

import org.elasticsearch.search.SearchHits;

public class ElasticSearchHandler {

private Client client;

public ElasticSearchHandler() {

// 使用本机做为节点

this("127.0.0.1");

}

public ElasticSearchHandler(String ipAddress) {

try {

client = TransportClient

.builder()

.build()

.addTransportAddress(

new InetSocketTransportAddress(InetAddress

.getByName("127.0.0.1"), 9300));

} catch (UnknownHostException e) {

e.printStackTrace();

}

// 集群连接超时设置

* Settings settings =

* ImmutableSettings.settingsBuilder().put("client.transport.ping_timeout"

* , "10s").build(); client = new TransportClient(settings);

// client = new TransportClient()

// .addTransportAddress(new InetSocketTransportAddress(ipAddress,

// 9200));

}

/**

* 建立索引,索引建立好之后,会在elasticsearch-0.20.6\data\elasticsearch\nodes\0创建所以你看

* @param indexName

* 为索引库名，一个es集群中可以有多个索引库。名称必须为小写

* @param indexType

* Type为索引类型，是用来区分同索引库下不同类型的数据的，一个索引库下可以有多个索引类型。

* @param jsondata

* json格式的数据集合

* @return

public void createIndexResponse(String indexname, String type,

List<String> jsondata) {

// 创建索引库需要注意的是.setRefresh(true)这里一定要设置,否则第一次建立索引查找不到数据

IndexRequestBuilder requestBuilder = client.prepareIndex(indexname,

type).setRefresh(true);

for (int i = 0; i < jsondata.size(); i++) {

System.out.println(jsondata.get(i) + "-");

requestBuilder.setSource(jsondata.get(i)).execute().actionGet();

}

/**

* 创建索引

* @param client

* @param jsondata

* @return

public IndexResponse createIndexResponse(String indexname, String type,

String jsondata) {

IndexResponse response = client.prepareIndex(indexname, type)

.setSource(jsondata).execute().actionGet();

return response;

}

/**

* 执行搜索

* @param queryBuilder

* @param indexname

* @param type

* @return

public List<Medicine> searcher(QueryBuilder queryBuilder, String indexname,

String type) {

List<Medicine> list = new ArrayList<Medicine>();

SearchResponse searchResponse = client.prepareSearch(indexname)

.setTypes(type).setQuery(queryBuilder).execute().actionGet();

// SearchHits hits = searchResponse.hits();

SearchHits hits = searchResponse.getHits();

System.out.println("查询到记录数=" + hits.getTotalHits());

SearchHit[] searchHists = hits.getHits();

if (searchHists.length > 0) {

for (SearchHit hit : searchHists) {

Integer id = (Integer) hit.getSource().get("id");

String name = (String) hit.getSource().get("name");

String function = (String) hit.getSource().get("funciton");

list.add(new Medicine(id, name, function));

}

return list;

}

public static void main(String[] args) {

ElasticSearchHandler esHandler = new ElasticSearchHandler();

List<String> jsondata = DataFactory.getInitJsonData();

String indexname = "indexdemo";

String type = "typedemo";

// 创建索引

// esHandler.createIndexResponse(indexname, type, jsondata);

// 查询条件F

// QueryBuilder queryBuilder = QueryBuilders.fuzzyQuery("name", "感冒");

QueryBuilder queryBuilder = QueryBuilders.functionScoreQuery();

// QueryBuilder queryBuilder = QueryBuilders.fieldQuery("name", "感冒");

// QueryBuilders.se

* QueryBuilder queryBuilder = QueryBuilders.boolQuery()

* .must(QueryBuilders.termQuery("id", 1));

List<Medicine> result = esHandler.searcher(queryBuilder, indexname,

type);

for (int i = 0; i < result.size(); i++) {

Medicine medicine = result.get(i);

System.out.println("(" + medicine.getId() + ")药品名称:"

+ medicine.getName() + "\t\t" + medicine.getFunction());

}

查看效果

查看执行时间以及条数，感觉还可以

查看保存的数据结构

注意以上数据保存的路径以及数据的后缀。还有一个后缀为lock的文件，是咋写入数据时的对数据的处理的进度。

查询语句

must: 文档必须完全匹配条件
should: should下面会带一个以上的条件，至少满足一个条件，这个文档就符合shouldmust_not: 文档必须不匹配条件

详细请见http://www.cnblogs.com/yjf512/p/4897294.html

1-1）、精确查找

{

"query": {

"match_phrase": {

"name": "小青"

}

1-2）、模糊查找

{

"query": {

"match_phrase": {

"name": "小青"

}

1-3）、查找多个字段的数据

{

"query": {

"multi_match": {

"query": "小张感冒",

"fields": [

"name",

"funciton"

]

}

1-4）、我们希望完全匹配的文档占的评分比较高，则需要使用best_fields

{

"query": {

"multi_match": {

"query": "小张感冒",

"type": "best_fields",

"fields": [

"name",

"funciton"

"tie_breaker": 0.1

}

意思就是完全匹配"宝马发动机"的文档评分会比较靠前，如果只匹配宝马的文档评分乘以0.3的系数

1-5）、多字段匹配

{

"query": {

"multi_match": {

"query": "小张感冒",

"type": "most_fields",

"fields": [

"name",

"funciton"

"tie_breaker": 0.1

}

1-6）、我们会希望这个词条的分词词汇是分配到不同字段中的，那么就使用cross_fields

{

"query": {

"term": {

"content": "1"

}

1-7）、高亮设置

{

"query": {

"match_phrase": {

"name": "大青叶"

}

"highlight": {

"fields": {

"name": {}

}

1-8）、URL 查询

http://localhost:9200/indexdemo/typedemo/AVbVXZ5YQST3amZyeBjV?_source=id,name

Elasticsearch 数据映射给Hive

1-1）、下载相对应的JAR

链接：http://pan.baidu.com/s/1bp9vU2N 密码：1nbz

1-2）、加载JAR包

hive > ADD JAR /home/xiaoxu/elasticsearch-hadoop-2.3.4.jar;

1-3)、映射数据

hive > create EXTERNAL table es_hive_sp_address(

regtime string,

uid int,

mobile string,

username string

)

STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'

TBLPROPERTIES('es.resource' = 'sp_address/sp_address', 'es.nodes'='192.168.31.122', 'es.port'='9003','es.nodes.wan.only'='true');

1-4）、查看数据

hive > select * from es_hive_sp_address;

1 01 110000 北京市

2 01 120000 天津市

3 01 130000 河北省

4 01 140000 山西省

5 01 150000 内蒙古自治区

6 01 210000 辽宁省

7 01 220000 吉林省

8 01 230000 黑龙江省

9 01 310000 上海市

***************************

elasticsearch-sql使用

github介绍:https://github.com/NLPchina/elasticsearch-sql

1-1）、软件安装

$ ./plugin install https://github.com/NLPchina/elasticsearch-sql/releases/download/2.3.1.1/elasticsearch-sql-2.3.1.1.zip

1-2）、软件使用

http://192.168.31.47:9200/_plugin/sql/

1-3）、API 调用elasticsearch-sql

public void testJDBC() throws Exception {

Properties properties = new Properties();

properties.put("url", "jdbc:elasticsearch://192.168.31.47:9200/" + TestsConstants.TEST_INDEX);

DruidDataSource dds = (DruidDataSource) ElasticSearchDruidDataSourceFactory.createDataSource(properties);

Connection connection = dds.getConnection();

PreparedStatement ps = connection.prepareStatement("SELECT gender,lastname,age from " + TestsConstants.TEST_INDEX + " where lastname='Heath'");

ResultSet resultSet = ps.executeQuery();

List<String> result = new ArrayList<String>();

while (resultSet.next()) {

System.out.println(resultSet.getString("lastname") + "," + resultSet.getInt("age") + "," + resultSet.getString("gender"))

}

ps.close();

connection.close();

dds.close();

}

快学Big Data -- Elasticsearch 总结（二十六)

Elasticsearch 总结

Elasticsearch 安装

1-5）、安装Elasticsearch的插件

1-9）、运行Elasticsearch

1-10）、停止ES

1-12）、常见错误处理

1-1）、实体类

1-2）、工具类

1-3）、工厂类

1-4）、主类

1-1）、精确查找

1-4）、我们希望完全匹配的文档占的评分比较高，则需要使用best_fields

1-6）、我们会希望这个词条的分词词汇是分配到不同字段中的，那么就使用cross_fields

Elasticsearch 数据映射给Hive

1-1）、下载相对应的JAR

1-2）、加载JAR包

1-3)、映射数据

1-4）、查看数据

相关推荐