当前位置：首页 > news >正文

elk快速部署、集成、调优

news 2025/8/4 5:38:21

elk快速部署、集成、调优

1.部署

1.1 docker部署

参考：https://blog.csdn.net/taotao_guiwang/article/details/135508643
在这里插入图片描述

1.2 elk部署

资源见，百度网盘：https://pan.baidu.com/s/1qlabJ7m8BDm77GbDuHmbNQ?pwd=41ac
修改elasticsearch.yml，将IP修改为实际IP：
执行elk_install.sh，开始部署：

# 赋权，在elk_install目录，执行：
chmod -R 777 .
# 执行
./elk_install.sh.sh

安装分词器

# 默认安装在/home/install/elk
docker cp /home/install/elk/elasticsearch-analysis-ik-7.17.3.zip elasticsearch:/
# 进入容器，执行以下命令，然后重启容器
docker exec -it elasticsearch bash
elasticsearch-plugin install file:/elasticsearch-analysis-ik-7.17.3.zip

出现如图所示页面输入 y
在这里插入图片描述

# 在容器内执行
bin/elasticsearch-setup-passwords interactive
# 设置密码为：test@123456(PS:不建议修改为其他密码，因为配置文件里配的都是这个密码)

在这里插入图片描述

# 退出容器：
exit
# 重启容器：
docker restart elasticsearch logstash kibana

elastic访问地址：
http://10.86.97.210:9200/
账号：elastic
密码：test@123456
在这里插入图片描述
kibana访问地址：
http://10.86.97.210:5601/
账号：elastic
密码：test@123456

2. spring boot集成

2.1 创建索引

转到，kibana：Management->开发工具：
执行：

PUT blog
{"mappings": {"properties": {"id":{"type":"long"},"title": {"type": "text"},"content": {"type": "text"},"author":{"type": "text","fields": {"keyword": {"type": "keyword"}}},"category":{"type": "keyword"},"createTime": {"type": "date","format":"yyyy-MM-dd HH:mm:ss.SSS||yyyy-MM-dd'T'HH:mm:ss.SSS||yyyy-MM-dd HH:mm:ss||epoch_millis"},"updateTime": {"type": "date","format":"yyyy-MM-dd HH:mm:ss.SSS||yyyy-MM-dd'T'HH:mm:ss.SSS||yyyy-MM-dd HH:mm:ss||epoch_millis"},"status":{"type":"integer"},"serialNum": {"type": "keyword"}}}
}

执行效果：
查看blog信息：

GET /blog

在这里插入图片描述

2.2 构造数据

数据，每个文档必须独占一行，不能换行：

{"index":{"_index":"blog","_id":1}}
{"blogId":1,"title":"Spring Data ElasticSearch学习教程1","content":"这是批量添加的文档1","author":"Iron Man","category":"ElasticSearch","status":1,"serialNum":"1","createTime":"2021-10-10 11:52:01.249","updateTime":null}
{"index":{"_index":"blog","_id":2}}
{"blogId":2,"title":"Spring Data ElasticSearch学习教程2","content":"这是批量添加的文档2","author":"Iron Man","category":"ElasticSearch","status":1,"serialNum":"2","createTime":"2021-10-10 11:52:02.249","updateTime":null}
{"index":{"_index":"blog","_id":3}}
{"blogId":3,"title":"Spring Data ElasticSearch学习教程3","content":"这是批量添加的文档3","author":"Captain America","category":"ElasticSearch","status":1,"serialNum":"3","createTime":"2021-10-10 11:52:03.249","updateTime":null}
{"index":{"_index":"blog","_id":4}}
{"blogId":4,"title":"Spring Data ElasticSearch学习教程4","content":"这是批量添加的文档4","author":"Captain America","category":"ElasticSearch","status":1,"serialNum":"4","createTime":"2021-10-10 11:52:04.249","updateTime":null}
{"index":{"_index":"blog","_id":5}}
{"blogId":5,"title":"Spring Data ElasticSearch学习教程5","content":"这是批量添加的文档5","author":"Spider Man","category":"ElasticSearch","status":1,"serialNum":"5","createTime":"2021-10-10 11:52:05.249","updateTime":null}
{"index":{"_index":"blog","_id":6}}
{"blogId":6,"title":"Java学习教程6","content":"这是批量添加的文档6","author":"Spider Man","category":"ElasticSearch","status":1,"serialNum":"6","createTime":"2021-10-10 11:52:06.249","updateTime":null}
{"index":{"_index":"blog","_id":7}}
{"blogId":7,"title":"Java学习教程7","content":"这是批量添加的文档7","author":"Iron Man","category":"ElasticSearch","status":1,"serialNum":"7","createTime":"2021-10-10 11:52:07.249","updateTime":null}
{"index":{"_index":"blog","_id":8}}
{"blogId":8,"title":"Java学习教程8","content":"这是批量添加的文档8","author":"Iron Man","category":"ElasticSearch","status":1,"serialNum":"8","createTime":"2021-10-10 11:52:08.249","updateTime":null}
{"index":{"_index":"blog","_id":9}}
{"blogId":9,"title":"Java学习教程9","content":"这是批量添加的文档9","author":"Captain America","category":"ElasticSearch","status":1,"serialNum":"9","createTime":"2021-10-10 11:52:09.249","updateTime":null}
{"index":{"_index":"blog","_id":10}}
{"blogId":10,"title":"Java学习教程10","content":"这是批量添加的文档10","author":"God Of Thunder","category":"ElasticSearch","status":1,"serialNum":"10","createTime":"2021-10-10 11:52:10.249","updateTime":null}

执行效果：
查询数据：

GET /blog/_search
{"query": {"match_all": {}}
}

2.3 核心代码

2.3.1 常见操作

包括：添加单个文档、添加多个文档、修改单个文档数据、修改单个文档部分数据、修改多个文档部分数据、查看单个文档、删除单个文档（根据id）、删除单个文档（根据条件）、删除所有文档。

package com.elk;import com.elk.entity.Blog;
import org.elasticsearch.index.query.QueryBuilders;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate;
import org.springframework.data.elasticsearch.core.document.Document;
import org.springframework.data.elasticsearch.core.mapping.IndexCoordinates;
import org.springframework.data.elasticsearch.core.query.NativeSearchQuery;
import org.springframework.data.elasticsearch.core.query.NativeSearchQueryBuilder;
import org.springframework.data.elasticsearch.core.query.UpdateQuery;
import org.springframework.data.elasticsearch.core.query.UpdateResponse;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;//增删改查（RestTemplate方式)
@RestController
@RequestMapping("crudViaRestTemplate")
public class CrudViaRestTemplateController {@Autowiredprivate ElasticsearchRestTemplate elasticsearchRestTemplate;@PostMapping("addDocument")public Blog addDocument() {Long id = 1L;Blog blog = new Blog();blog.setBlogId(id);blog.setTitle("Spring Data ElasticSearch学习教程" + id);blog.setContent("这是添加单个文档的实例" + id);blog.setAuthor("Tony");blog.setCategory("ElasticSearch");blog.setCreateTime(new Date());blog.setStatus(1);blog.setSerialNum(id.toString());return elasticsearchRestTemplate.save(blog);}@PostMapping("addDocuments")public Object addDocuments(Integer count) {List<Blog> blogs = new ArrayList<>();for (int i = 1; i <= count; i++) {Long id = (long)i;Blog blog = new Blog();blog.setBlogId(id);blog.setTitle("Spring Data ElasticSearch学习教程" + id);blog.setContent("这是添加单个文档的实例" + id);blog.setAuthor("Tony");blog.setCategory("ElasticSearch");blog.setCreateTime(new Date());blog.setStatus(1);blog.setSerialNum(id.toString());blogs.add(blog);}return elasticsearchRestTemplate.save(blogs);}/*** 跟新增是同一个方法。若id已存在，则修改。* 无法只修改某个字段，只能覆盖所有字段。若某个字段没有值，则会写入null。* @return 成功写入的数据*/@PostMapping("editDocument")public Blog editDocument() {Long id = 1L;Blog blog = new Blog();blog.setBlogId(id);blog.setTitle("Spring Data ElasticSearch学习教程" + id);blog.setContent("这是修改单个文档的实例" + id);return elasticsearchRestTemplate.save(blog);}@PostMapping("editDocumentPart")public UpdateResponse editDocumentPart() {long id = 1L;Document document = Document.create();document.put("title", "修改后的标题" + id);document.put("content", "修改后的内容" + id);UpdateQuery updateQuery = UpdateQuery.builder(Long.toString(id)).withDocument(document).build();UpdateResponse response = elasticsearchRestTemplate.update(updateQuery, IndexCoordinates.of("blog"));return response;}@PostMapping("editDocumentsPart")public void editDocumentsPart(int count) {List<UpdateQuery> updateQueryList = new ArrayList<>();for (int i = 1; i <= count; i++) {long id = (long) i;Document document = Document.create();document.put("title", "修改后的标题" + id);document.put("content", "修改后的内容" + id);UpdateQuery updateQuery = UpdateQuery.builder(Long.toString(id)).withDocument(document).build();updateQueryList.add(updateQuery);}elasticsearchRestTemplate.bulkUpdate(updateQueryList, IndexCoordinates.of("blog"));}@GetMapping("findById")public Blog findById(Long id) {return elasticsearchRestTemplate.get(id.toString(), Blog.class, IndexCoordinates.of("blog"));}@PostMapping("deleteDocumentById")public String deleteDocumentById(Long id) {return elasticsearchRestTemplate.delete(id.toString(), Blog.class);}@PostMapping("deleteDocumentByQuery")public void deleteDocumentByQuery(String title) {NativeSearchQuery nativeSearchQuery = new NativeSearchQueryBuilder().withQuery(QueryBuilders.matchQuery("title", title)).build();elasticsearchRestTemplate.delete(nativeSearchQuery, Blog.class, IndexCoordinates.of("blog"));}@PostMapping("deleteDocumentAll")public void deleteDocumentAll() {NativeSearchQuery nativeSearchQuery = new NativeSearchQueryBuilder().withQuery(QueryBuilders.matchAllQuery()).build();elasticsearchRestTemplate.delete(nativeSearchQuery, Blog.class, IndexCoordinates.of("blog"));}
}

2.3.2 查询的方法

Query接口有3个抽象实现：
在这里插入图片描述
此处使用NativeSearchQuery。因为它更贴近ES，语法更偏向ES原来的命令。CriteriaQuery的语法跟JPA的差不多。

2.3.3构建Query

可通过new NativeSearchQueryBuilder()来构建NativeSearchQuery对象。NativeSearchQuery中有众多的方法来为我们实现复杂的查询与筛选等操作。其中build()返回NativeSearchQuery。
在这里插入图片描述

2.3.4 QueryBuilders构造复杂查询条件

NativeSearchQueryBuilder中接收QueryBuilder。

public NativeSearchQueryBuilder withQuery(QueryBuilder queryBuilder) {this.queryBuilder = queryBuilder;return this;
}

可以用QueryBuilders构造QueryBuilder对象：
在这里插入图片描述

2.3.5 基本查询

通过标题和内容来查找博客：

package com.elk;import com.elk.entity.Blog;
import org.apache.commons.lang3.StringUtils;
import org.elasticsearch.index.query.BoolQueryBuilder;
import org.elasticsearch.index.query.QueryBuilders;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate;
import org.springframework.data.elasticsearch.core.SearchHit;
import org.springframework.data.elasticsearch.core.SearchHits;
import org.springframework.data.elasticsearch.core.query.NativeSearchQueryBuilder;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;import java.util.ArrayList;
import java.util.List;
// 动态查询
@RestController
@RequestMapping("dynamicQuery")
public class DynamicQueryController {@Autowiredprivate ElasticsearchRestTemplate elasticsearchRestTemplate;// 简单查询@GetMapping("simple")public List<Blog> simple(String title, String content) {NativeSearchQueryBuilder query = new NativeSearchQueryBuilder();BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();if (StringUtils.isNotBlank(title)) {boolQueryBuilder.must(QueryBuilders.matchQuery("title", title));}if (StringUtils.isNotBlank(content)) {boolQueryBuilder.must(QueryBuilders.matchQuery("content", content));}query.withQuery(boolQueryBuilder);SearchHits<Blog> searchHits = elasticsearchRestTemplate.search(query.build(), Blog.class);List<Blog> blogs = new ArrayList<>();for (SearchHit<Blog> searchHit : searchHits) {blogs.add(searchHit.getContent());}return blogs;}
}

postman调用：

在这里插入图片描述

2.3.6 分页排序

通过标题和作者来查找博客，分页，并根据创建时间倒序排序。

// 分页排序@GetMapping("pageAndSort")public Page<Blog> pageAndSort(String title, String author) {PageRequest pageRequest = PageRequest.of(0, 2);NativeSearchQueryBuilder query = new NativeSearchQueryBuilder();BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();if (StringUtils.isNotBlank(title)) {boolQueryBuilder.must(QueryBuilders.matchQuery("title", title));}if (StringUtils.isNotBlank(author)) {boolQueryBuilder.must(QueryBuilders.matchQuery("author", author));}query.withQuery(boolQueryBuilder);query.withPageable(pageRequest);query.withSort(SortBuilders.fieldSort("createTime").order(SortOrder.DESC));SearchHits<Blog> searchHits = elasticsearchRestTemplate.search(query.build(), Blog.class);List<Blog> blogs = new ArrayList<>();for (SearchHit<Blog> searchHit : searchHits) {blogs.add(searchHit.getContent());}return new PageImpl<Blog>(blogs, pageRequest, searchHits.getTotalHits());}

postman调用：
在这里插入图片描述

2.3.7 去重

通过标题和作者来查找博客，根据作者来去重。

// 去重@GetMapping("collapse")public List<Blog> collapse(String title, String author) {PageRequest pageRequest = PageRequest.of(0, 2);NativeSearchQueryBuilder query = new NativeSearchQueryBuilder();BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();if (StringUtils.isNotBlank(title)) {boolQueryBuilder.must(QueryBuilders.matchQuery("title", title));}if (StringUtils.isNotBlank(author)) {boolQueryBuilder.must(QueryBuilders.matchQuery("author", author));}query.withQuery(boolQueryBuilder);query.withPageable(pageRequest);query.withSort(SortBuilders.fieldSort("createTime").order(SortOrder.DESC));// 去重的字段不能是text类型。所以，author的mapping要有keyword，且通过author.keyword去重。query.withCollapseField("author.keyword");//query.withCollapseField("category");SearchHits<Blog> searchHits = elasticsearchRestTemplate.search(query.build(), Blog.class);List<Blog> blogs = new ArrayList<>();for (SearchHit<Blog> searchHit : searchHits) {blogs.add(searchHit.getContent());}return blogs;}

postman调用：
在这里插入图片描述

2.3.8 聚合

查询标题中带有“java”的每个作者的文章数量。

//聚合@GetMapping("aggregation")public Map<String, Long> aggregation() {NativeSearchQueryBuilder query = new NativeSearchQueryBuilder();BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();boolQueryBuilder.must(QueryBuilders.matchQuery("title", "java"));query.withQuery(boolQueryBuilder);// 作为聚合的字段不能是text类型。所以，author的mapping要有keyword，且通过author.keyword聚合。query.addAggregation(AggregationBuilders.terms("per_count").field("author.keyword"));// 不需要获取source结果集，在aggregation里可以获取结果query.withSourceFilter(new FetchSourceFilterBuilder().build());SearchHits<Blog> searchHits = elasticsearchRestTemplate.search(query.build(), Blog.class);Aggregations aggregations = searchHits.getAggregations();assert aggregations != null;//因为结果为字符串类型 所以用ParsedStringTerms。其他还有ParsedLongTerms、ParsedDoubleTerms等ParsedStringTerms per_count = aggregations.get("per_count");Map<String, Long> map = new HashMap<>();for (Terms.Bucket bucket : per_count.getBuckets()) {map.put(bucket.getKeyAsString(), bucket.getDocCount());}return map;}

postman调用：
在这里插入图片描述

2.3.9 嵌套聚合

查询标题中带有“java”的每个作者的文章数量，再查出各个作者的最新的发文时间。

// 嵌套聚合@GetMapping("subAggregation")public Map<String, Map<String, Object>> subAggregation() {NativeSearchQueryBuilder query = new NativeSearchQueryBuilder();BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();boolQueryBuilder.must(QueryBuilders.matchQuery("title", "java"));query.withQuery(boolQueryBuilder);query.addAggregation(AggregationBuilders.terms("per_count").field("author.keyword").subAggregation(AggregationBuilders.max("latest_create_time").field("createTime")));// 不需要获取source结果集，在aggregation里可以获取结果query.withSourceFilter(new FetchSourceFilterBuilder().build());SearchHits<Blog> searchHits = elasticsearchRestTemplate.search(query.build(), Blog.class);Aggregations aggregations = searchHits.getAggregations();assert aggregations != null;//因为结果为字符串类型 所以用ParsedStringTerms。其他还有ParsedLongTerms、ParsedDoubleTerms等ParsedStringTerms per_count = aggregations.get("per_count");Map<String, Map<String, Object>> map = new HashMap<>();for (Terms.Bucket bucket : per_count.getBuckets()) {Map<String, Object> objectMap = new HashMap<>();objectMap.put("docCount", bucket.getDocCount());ParsedMax parsedMax = bucket.getAggregations().get("latest_create_time");objectMap.put("latestCreateTime", parsedMax.getValueAsString());map.put(bucket.getKeyAsString(), objectMap);}return map;}

postman调用：
在这里插入图片描述

2.3.10 application.yml

server:port: 8080spring:elasticsearch:rest:# 如果是集群，用逗号隔开uris: http://10.86.97.210:9200username: elasticpassword: test@123456connection-timeout: 1read-timeout: 30# 上边是客户端High Level REST Client的配置，推荐使用。# 下边是reactive客户端的配置，非官方，不推荐。
#  data:
#    elasticsearch:
#      client:
#        reactive:
#          endpoints: 127.0.0.1:9200
#          username: xxx
#          password: yyy
#          connection-timeout: 1
#          socket-timeout: 30
#          use-ssl: false

2.3.11 pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"><modelVersion>4.0.0</modelVersion><groupId>com.elk</groupId><artifactId>spring-boot-elk</artifactId><version>0.0.1-SNAPSHOT</version><packaging>jar</packaging><name>spring-boot-elk</name><description>Demo project for Spring Boot</description><parent><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-parent</artifactId><version>2.3.12.RELEASE</version><relativePath /> <!-- lookup parent from repository --></parent><properties><project.build.sourceEncoding>UTF-8</project.build.sourceEncoding><project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding><java.version>1.8</java.version></properties><dependencies><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-web</artifactId></dependency><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-data-elasticsearch</artifactId></dependency><dependency><groupId>org.projectlombok</groupId><artifactId>lombok</artifactId><optional>true</optional></dependency><dependency><groupId>org.apache.commons</groupId><artifactId>commons-lang3</artifactId><version>3.12.0</version> <!-- 推荐稳定版本 --></dependency></dependencies><build><plugins><plugin><groupId>org.springframework.boot</groupId><artifactId>spring-boot-maven-plugin</artifactId></plugin></plugins></build>
</project>