Elasticsearch常用代码

来自ling
跳转至: 导航搜索

链接

位置搜索、GeoPoint

SpringBoot整合ElasticSearch(控制度查询,父子关系建立)

索引api

elasticsearch查询示例

Elasticsearch索引重建

fielddata

ElasticSearch基本查询(Query查询)

elasticsearch 基础 —— Query String

Elasticsearch——使用Java API实现ES中的索引、映射、文档操作

elasticsearch学习笔记

ElasticSearch系列(六)springboot中使用QueryBuilders、NativeSearchQuery实现复杂查询

常用代码

{
  "from": 0,
  "size": 10,
  "query": {
    "bool": {
      "must": [
        {
          "query_string": {
            "query": "b5b7883e-82f9-46e5-8a9c-63e2678bd82d",
            "fields": [
              "bookId"
            ]
          }
        }
      ]
    }
  }
}

BoolQueryBuilder

must的性能要低一些,为什么?因为他要进行打分评估,也就是说要进行_score,而filter则不会。

TermQueryBuilder: 词条查询是ElasticSearch的一个简单查询。它仅匹配在给定字段中含有该词条的文档,而且是确切的、未经分析的词条。term查询会查找我们设定的准确值。term查询本身很简单,它接受一个字段名和我们希望查找的值。

TermsQueryBuilder: 词条查询(Term Query)允许匹配单个未经分析的词条,多词条查询(Terms Query)可以用来匹配多个这样的词条。只要指定字段包含任一我们给定的词条,就可以查询到该文档。

MatchQueryBuilder: 与TermQueryBuilder类似。

踩坑:使用Term(s)QueryBuilder查询内容需要转换成全小写,而MatchQueryBuilder不需要转换。 原因:Term(s)QueryBuilder输入的词条不会经过处理;而MatchQueryBuilder输入的词条 会被es解析并进行分词,在此过程中就已经转换成全小写。(es在存储字段时,已做解析、分词和小写处理。看见的是大写的字段内容,实际在es里已经被当作小写进行处理)

 /**
     * 拼装查询条件
     * @param query
     * @return
     */
    public static BoolQueryBuilder buildEsParam(QueryGatewayTradeEsVo query) {
 
        BoolQueryBuilder queryBuilder = QueryBuilders.boolQuery();
        
        //filter 效率比 must高的多 
        if (StringUtils.isNotBlank(query.getRouterDatabaseNo())) {
            queryBuilder.filter(QueryBuilders.termQuery("routerDatabaseNo", query.getRouterDatabaseNo()));
        }
 
        if (StringUtils.isNotBlank(query.getTradeNo())) {
            queryBuilder.filter(QueryBuilders.termQuery("tradeNo", query.getTradeNo()));
        }
 
        if (null != query.getSystemId()) {
            queryBuilder.filter(QueryBuilders.termQuery("systemId", query.getSystemId()));
        }
 
        if (null != query.getRefundType()) {
            queryBuilder.filter(QueryBuilders.termQuery("refundType", query.getRefundType()));
        }
 
        if (null != query.getRefundStatus()) {
            queryBuilder.filter(QueryBuilders.termQuery("refundStatus", query.getRefundStatus()));
        }
 
        //时间段 一定要有头有尾 不然会出现慢查询
        if (null != query.getCreateTime() && null != query.getUpdateTime()) {
            queryBuilder.filter(QueryBuilders.rangeQuery("createTime").from( query.getCreateTime()).to(query.getUpdateTime()));
        }
 
        return queryBuilder;
    }

    /**
     * 模糊查找标题含有 “战”的,内容中必须没有“星球”的
     *
     */
    @Test
    public void searchMutil1(){
        SearchRequestBuilder srb = client.prepareSearch("firm").setTypes("dongzuo");
        QueryBuilder qb1 = QueryBuilders.matchPhraseQuery("title", "战");
        QueryBuilder qb2 = QueryBuilders.matchQuery("content","星球");
        SearchResponse response = srb.setQuery(
                QueryBuilders.boolQuery().must(qb1).mustNot(qb2)
        ).execute().actionGet();
        SearchHits hits = response.getHits();
        for(SearchHit hit : hits){
            System.out.println(hit.getSourceAsString());
        }

    }


    /**
     * should 的使用 , 提高得分
     * 标题含有战的
     * 得分:1.含有星球的得分 2. 出版日期在指定区间范围的得分
     */
    @Test
    public void shouldSearch(){
        SearchRequestBuilder srb = client.prepareSearch("firm").setTypes("dongzuo");

        MatchPhraseQueryBuilder mqb = QueryBuilders.matchPhraseQuery("title", "战");
        MatchPhraseQueryBuilder mqb2 = QueryBuilders.matchPhraseQuery("content", "星球");
        RangeQueryBuilder range = QueryBuilders.rangeQuery("publishDate").gt("2018-01-01");

        SearchResponse response = srb.setQuery(
                QueryBuilders.boolQuery().must(mqb).should(mqb2).should(range)
        ).execute().actionGet();

        SearchHits hits = response.getHits();
        for(SearchHit hit:hits){
            System.out.println(hit.getScore() + ":" +hit.getSourceAsString());
        }
    }

    /**
     * 标题含有战的
     * 过滤: 票价必须小于40的
     *
     */
    @Test
    public void filterSearch(){
        SearchRequestBuilder srb = client.prepareSearch("firm").setTypes("dongzuo");

        MatchPhraseQueryBuilder q1 = QueryBuilders.matchPhraseQuery("title", "战");
        RangeQueryBuilder priceRange = QueryBuilders.rangeQuery("price").lt(40);

        SearchResponse response = srb.setQuery(
                QueryBuilders.boolQuery().must(q1).filter(priceRange)
        ).execute().actionGet();
        SearchHits hits = response.getHits();
        for(SearchHit hit: hits){
            System.out.println(hit.getSourceAsString());
        }
    }

sort

https://blog.csdn.net/weixin_26800111/article/details/112084457

 public Map<String, Object> findByQuestion(String content, int page) {
        Map<String, Object> results = new HashMap<>();
        MatchQueryBuilder matchQueryBuilder = new MatchQueryBuilder("contents", content);
        matchQueryBuilder.analyzer("ik");
        TermQueryBuilder termQueryType = new TermQueryBuilder("type", "question");
        Page<BookPage> questionQa = bookPageRepository.search(new NativeSearchQueryBuilder()
                .withQuery(new BoolQueryBuilder().filter(matchQueryBuilder).filter(termQueryType))
                .withPageable(PageRequest.of(0, 10))
                .withSort(new ScoreSortBuilder().order(SortOrder.DESC))
                .withSort(new ScriptSortBuilder(new Script("'qyer'==doc['source'].value?0:('mafengwo'==doc['source'].value?1:2)"), ScriptSortBuilder.ScriptSortType.NUMBER).order(SortOrder.ASC))
                .withSort(new FieldSortBuilder("view_cnt").order(SortOrder.DESC))
                .withSort(new FieldSortBuilder("view_cnt").order(SortOrder.DESC))
                .build());
        System.err.println("----------------------------------------");
        System.err.println("总问题数:" + questionQa.getTotalElements());
        System.err.println("总页数:" + questionQa.getTotalPages());
        System.err.println("----------------------------------------");
        Map<String, Object> map = new HashMap<>();
        List<Map<String, Object>> listmap = new ArrayList<>();
//        for (BookPage qa : questionQa.getContent()) {
//            map.put("question", qa);
//            Page<BookPage> answerQas = bookPageRepository.search(new NativeSearchQueryBuilder()
//                    .withQuery(new AndQueryBuilder().add(new TermQueryBuilder("qid", qa.getQid())).add(new TermQueryBuilder("type", "answer")))
//                    .withSort(new FieldSortBuilder("view_cnt").order(SortOrder.DESC))
//                    .build());
//            List<BookPage> qas = new ArrayList<BookPage>();
//            for (BookPage answerQa : answerQas.getContent()) {
//                qas.add(answerQa);
//            }
//            map.put("answer", qas);
//            listmap.add(map);
//        }
        results.put("results", listmap);
        return results;
    }

    public void testSort () {
        String scoreScript = "if (doc['type'].value == 3) {" + "   return 100;" + "} else {" + "   return 1;" + "}";
        FunctionScoreQueryBuilder.FilterFunctionBuilder[] filterFunctionBuilders = new FunctionScoreQueryBuilder.FilterFunctionBuilder[]{new FunctionScoreQueryBuilder.FilterFunctionBuilder(QueryBuilders.matchAllQuery(), ScoreFunctionBuilders.scriptFunction(new Script(scoreScript)))};
        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
        BoolQueryBuilder boolQuery = QueryBuilders.boolQuery();
        boolQuery.must(QueryBuilders.termQuery("userId", 1));
        FunctionScoreQueryBuilder functionScoreQueryBuilder = QueryBuilders.functionScoreQuery(boolQuery, filterFunctionBuilders);
        searchSourceBuilder.query(functionScoreQueryBuilder).sort("_score", SortOrder.DESC).sort("heat", SortOrder.DESC);
        SearchRequest searchRequest = new SearchRequest(elasticSearchIndexConfig.getArticleSearchIndexName());
        searchRequest.types(EsConstant.DEFAULT_TYPE);
        searchRequest.source(searchSourceBuilder);
        List searchResults = kittyRestHighLevelClient.search(searchRequest, ArticleDocument.class);
        searchResults.forEach(doc -> {
            System.out.println(doc.getId() + "\t" + doc.getType() + "\t" + doc.getHeat());
        });
    }

query_string

query_string:支持通配符*、AND、OR、必须包含+ 不包含-

GET _search
{
  "query":{
    "query_string": {
      "default_field": "message",
      "query": "*Exception AND -*exception=null"
    }
  }
}

您可以使用query_string query字段的变体在keyword中执行相同的操作。

{
  "query": {
    "query_string": {
      "query": "some value",
      "default_field": "some_field.value.keyword"
    }
  }
}

bool

{
  "bool": {
    "must": [
      {
        "multi_match": {
          "query": "?0",
          "type": "most_fields",
          "fields": [
            "title",
            "content"
          ]
        }
      },
      {
        "match": {
          "catory": "?1"
        }
      }
    ]
  }
}

Elasticsearch中的一些概念整理

https://blog.csdn.net/u013041642/article/details/93799337

文档:

Elasticsearch是面向文档(document oriented)的,这意味着它可以存储整个对象或文档(document)。然而它不仅仅是存储,还会索引(index,注意这里是动词的意思)每个文档的内容使之可以被搜索。在Elasticsearch中,你可以对文档(而非成行成列的数据)进行索引、搜索、排序、过滤。这种理解数据的方式与以往完全不同,这也是Elasticsearch能够执行复杂的全文搜索的原因之一。

索引:

在Elasticsearch中存储数据的行为就叫做索引(indexing, 动词),不过在索引之前,我们需要明确数据应该存储在哪里,同样,存储在索引(indices, 名词)中。

「索引」含义的区分

你可能已经注意到索引(index)这个词在Elasticsearch中有着不同的含义,所以有必要在此做一下区分:

索引(名词) 如上文所述,一个索引(index)就像是传统关系数据库中的数据库,它是相关文档存储的地方,index的复数是indices 或indexes。 索引(动词) 「索引一个文档」表示把一个文档存储到索引(名词)里,以便它可以被检索或者查询。这很像SQL中的INSERT关键字,差别是,如果文档已经存在,新的文档将覆盖旧的文档。 倒排索引 传统数据库为特定列增加一个索引,例如B-Tree索引来加速检索。Elasticsearch和Lucene使用一种叫做倒排索引(inverted index)的数据结构来达到相同目的。 类型 文档归属于一种类型(type),而这些类型存在于索引(index)中,我们可以画一些简单的对比图来类比传统关系型数据库:

Relational DB -> Databases -> Tables -> Rows -> Columns
Elasticsearch -> Indices   -> Types  -> Documents -> Fields

Elasticsearch集群可以包含多个索引(indices)(数据库),每一个索引可以包含多个类型(types)(表),每一个类型包含多个文档(documents)(行),然后每个文档包含多个字段(Fields)(列)。

分析

分析(analysis)是这样一个过程: 首先, 标记化一个文本块为适用于倒排索引单独的词(term) 然后标准化这些词为标准形式, 提高它们的“可搜索性”或“查全率”

要快速检索的字段,分析以后,生成倒排索引。检索的时候,输入的字符串也会经过同样规则的分析,这样可以提高“可搜索性”

倒排索引

Elasticsearch使用一种叫做倒排索引(inverted index)的结构来做快速的全文搜索。 倒排索引由在文档中出现的唯一的单词列表, 以及对于每个单词在文档中的位置组成。

spring data elasticsearch

三、代码实现

1. 实体注解
注:Spring Data通过注解来声明字段的映射属性,有下面的三个注解:

@Document:作用在类,标记实体类为文档对象,一般有两个属性

indexName:对应索引库名称
type:对应在索引库中的类型
shards:分片数量,默认5
replicas:副本数量,默认1
@Id:作用在成员变量,标记一个字段作为id主键

@Fie


ld:作用在成员变量,标记为文档的字段,并指定字段映射属性:

type:字段类型,是枚举:FieldType,可以是text、long、short、date、integer、object等

text:存储数据时候,会自动分词,并生成索引
keyword:存储数据时候,不会分词建立索引
Numerical:数值类型,分两类
基本数据类型:long、interger、short、byte、double、float、half_float
浮点数的高精度类型:scaled_float。
需要指定一个精度因子,比如10或100。elasticsearch会把真实值乘以这个因子后存储,取出时再还原。
Date:日期类型。elasticsearch可以对日期格式化为字符串存储,但是建议我们存储为毫秒值,存储为long,节省空间

index:是否索引,布尔类型,默认是true

store:是否存储,布尔类型,默认是false

analyzer:分词器名称,这里的ik_max_word即使用ik分词器

2. 创建实体

此处的实体为提供搜索用的实体,应与业务实体区分开,如果有相同实体,为避免混淆应写两份。正常情况下全局搜索接口可单独部署服务器。 实体的配置方式有多种,除了下方每个字段使用注解外,spring boot还为我们提供了@Setting及@Mapping注解方式。如对字段设置要求较高的建议使用后者,后者可在配置文件处写原生DDL语句,虽然麻烦些,但是较为清晰。

常用代码

实现全局高亮分页搜索

springboot整合Elasticsearch - 实现全局高亮分页搜索

elasticsearch基本操作之--使用QueryBuilders进行查询

spring-data-elasticsearch 中QueryBuilders的用法

NativeSearchQueryBuilder

    public Page<ProductDoc> search(String keyword, Boolean isSelfRun, Pageable pageable) {
        BoolQueryBuilder queryBuilder = QueryBuilders.boolQuery();
        if (StringUtils.isNotEmpty(keyword)) {
            queryBuilder.should(QueryBuilders.matchQuery(ProductDoc._name, keyword).boost(3)); // 给name字段更高的权重
            queryBuilder.should(QueryBuilders.matchQuery(ProductDoc._description, keyword));   // description 默认权重 1
            queryBuilder.minimumNumberShouldMatch(1); // 至少一个should条件满足
        }

        if (isSelfRun!=null && isSelfRun) {
            queryBuilder.must(QueryBuilders.matchQuery(ProductDoc._isSelfRun, Boolean.TRUE)); // 精准值条件查询
        }


        SearchQuery searchQuery = new NativeSearchQueryBuilder().withQuery(queryBuilder)
                .withPageable(pageable).build();
        LOGGER.info("\n search(): searchContent [" + keyword + "] \n DSL  = \n " + searchQuery.getQuery().toString());
        return productDocRespository.search(searchQuery);
    }

基于注解

https://www.pomit.cn/p/1959424408619521#105

public interface QuestionElasticsearchRepository extends ElasticsearchRepository<FQuestionElasticssearch, Long> {

	Page<FQuestionElasticssearch> findByCatory(String catory, Pageable pageable);

	@Query("{ \"bool\":{ \"must\":[ { \"multi_match\": { \"query\": \"?0\", \"type\": \"most_fields\", \"fields\": [ \"title\", \"content\" ] } }, { \"match\": { \"catory\": \"?1\" } } ] } } ")
	Page<FQuestionElasticssearch> searchByKeyWordsAndCatory(String keyword, String catory, Pageable pageable);

}

springBoot+elasticSearch 使用function_score自定义评分

@Override
    public SearchDto improveSearch(SearchDto searchDto) {

        String text = searchDto.getTerm();
        String type = searchDto.getType();
        HighSearchParam searchParam = searchDto.getSearchParam();

        // 搜索请求对象
        SearchRequest searchRequest = new SearchRequest(BwbdType.ES_INDEX);
        // 指定类型
        searchRequest.types(BwbdType.ES_TYPE);
        // 搜索源构建对象
        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
        // 搜索方式
        // 首先构造多关键字查询条件
        BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
        if (StringUtils.isNotEmpty(text)) {
            text = QueryParser.escape(text);  // 主要就是这一句把特殊字符都转义,那么lucene就可以识别
            MultiMatchQueryBuilder matchQueryBuilder = QueryBuilders
                    .multiMatchQuery(text, BwbdType.PROPERTY_NUMBERS
                            , BwbdType.PROPERTY_TITLES, BwbdType.PROPERTY_CONTENTS)
                    .field(BwbdType.PROPERTY_NUMBERS, 1f)
                    .field(BwbdType.PROPERTY_TITLES, 0.1f)
                    .field(BwbdType.PROPERTY_CONTENTS, 0.01f)
                    .minimumShouldMatch(BwbdType.MATCH_LEVEL_THREE);
            // 添加条件到布尔查询
            boolQueryBuilder.must(matchQueryBuilder);
        } else {
            if (null == searchDto.getSearchParam() && StringUtils.isEmpty(type)) {
                searchDto.setType(BwbdType.DATA_TYPE_FG);
            }
        }

//        // 通过布尔查询来构造过滤查询
//        boolQueryBuilder.filter(QueryBuilders.matchQuery("economics","L"));
        if (StringUtils.isNotEmpty(type)) {
            boolQueryBuilder.filter(QueryBuilders
                    .matchQuery(BwbdType.PROPERTY_DATA_TYPE, type));
        }

        addFilterProperties(text,searchParam, boolQueryBuilder, searchSourceBuilder);

        FunctionScoreQueryBuilder.FilterFunctionBuilder[] filterFunctionBuilders = buildFilterFunctionBuilders();

        FunctionScoreQueryBuilder query = QueryBuilders.functionScoreQuery(boolQueryBuilder,filterFunctionBuilders)
                .boostMode(CombineFunction.SUM)
                .scoreMode(FunctionScoreQuery.ScoreMode.SUM);

        // 将查询条件封装给查询对象
        searchSourceBuilder.query(query);
        if (searchDto.getSize() > 20) {
            searchDto.setSize(20);
        }
        searchSourceBuilder.size(searchDto.getSize());
        searchSourceBuilder.from(searchDto.getPage() - 1);

        // ***********************

        // 高亮查询
        HighlightBuilder highlightBuilder = new HighlightBuilder();
        highlightBuilder.preTags(CommonConstraint.LIGHT_TAG_START); // 高亮前缀
        highlightBuilder.postTags(CommonConstraint.LIGHT_TAG_END); // 高亮后缀
        List<HighlightBuilder.Field> fields = highlightBuilder.fields();
        fields.add(new HighlightBuilder
                .Field(BwbdType.PROPERTY_NUMBERS)); // 高亮字段
        fields.add(new HighlightBuilder
                .Field(BwbdType.PROPERTY_TITLES)); // 高亮字段
        fields.add(new HighlightBuilder
                .Field(BwbdType.PROPERTY_CONTENTS).fragmentSize(100000)); // 高亮字段
        // 添加高亮查询条件到搜索源
        searchSourceBuilder.highlighter(highlightBuilder);

        // ***********************

//        // 设置源字段过虑,第一个参数结果集包括哪些字段,第二个参数表示结果集不包括哪些字段
//        searchSourceBuilder.fetchSource(new String[]{"name","studymodel","price","timestamp"},new String[]{});
        // 向搜索请求对象中设置搜索源
        searchRequest.source(searchSourceBuilder);
        // 执行搜索,向ES发起http请求
        SearchResponse searchResponse = null;
        try (RestHighLevelClient client = new RestHighLevelClient(restClientBuilder)) {
            searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
            obtainFgType(searchDto, searchResponse);
        } catch (IOException e) {
            e.printStackTrace();
        }
        return searchDto;
    }

    private FunctionScoreQueryBuilder.FilterFunctionBuilder[] buildFilterFunctionBuilders() {
        FunctionScoreQueryBuilder.FilterFunctionBuilder[] filterFunctionBuilders = new FunctionScoreQueryBuilder.FilterFunctionBuilder[3];
        // 时间相关
        ScoreFunctionBuilder<?> dateFieldValueScoreFunction = ScoreFunctionBuilders.fieldValueFactorFunction("contentDate")
                .missing(946656000000d)
                .modifier(FieldValueFactorFunction.Modifier.LN1P).factor(0.1f);
        FunctionScoreQueryBuilder.FilterFunctionBuilder date = new FunctionScoreQueryBuilder.FilterFunctionBuilder(dateFieldValueScoreFunction);
        filterFunctionBuilders[0] = date;
        // 类型相关
        ScoreFunctionBuilder<?> dataTypeFieldValueScoreFunction = ScoreFunctionBuilders.fieldValueFactorFunction("dataTypeRelation")
                .missing(10d)
                .modifier(FieldValueFactorFunction.Modifier.LN1P).factor(1f);
        FunctionScoreQueryBuilder.FilterFunctionBuilder dataType = new FunctionScoreQueryBuilder.FilterFunctionBuilder(dataTypeFieldValueScoreFunction);
        filterFunctionBuilders[1] = dataType;
        // 来源相关
        ScoreFunctionBuilder<?> originFieldValueScoreFunction = ScoreFunctionBuilders.fieldValueFactorFunction("originTypeRelation")
                .missing(10d)
                .modifier(FieldValueFactorFunction.Modifier.LN1P).factor(1f);
        FunctionScoreQueryBuilder.FilterFunctionBuilder origin = new FunctionScoreQueryBuilder.FilterFunctionBuilder(originFieldValueScoreFunction);
        filterFunctionBuilders[2] = origin;
        return filterFunctionBuilders;
    }

    private void addFilterProperties(String text, HighSearchParam searchParam, BoolQueryBuilder boolQueryBuilder, SearchSourceBuilder searchSourceBuilder) {
        if (null != searchParam) {

            if (StringUtils.isNotEmpty(searchParam.getYearStr())) {
                boolQueryBuilder.filter(QueryBuilders.matchPhraseQuery(BwbdType.PROPERTY_YEARS, searchParam.getYearStr()));
            }
            if (StringUtils.isNotEmpty(searchParam.getReasonName())) {
                boolQueryBuilder.filter(QueryBuilders.matchPhraseQuery("reasonName", searchParam.getReasonName()));
            }
            if (StringUtils.isNotEmpty(searchParam.getCaseType())) {
                boolQueryBuilder.filter(QueryBuilders.matchPhraseQuery("caseType", searchParam.getCaseType()));
            }
            if (StringUtils.isNotEmpty(searchParam.getTrialRoundText())) {
                boolQueryBuilder.filter(QueryBuilders.matchPhraseQuery("trialRoundText", searchParam.getTrialRoundText()));
            }
            if (StringUtils.isNotEmpty(searchParam.getJudgementType())) {
                boolQueryBuilder.filter(QueryBuilders.matchPhraseQuery("judgementType", searchParam.getJudgementType()));
            }
            if (StringUtils.isNotEmpty(searchParam.getAreaCode())) {
                boolQueryBuilder.filter(QueryBuilders.matchPhraseQuery("areaId", searchParam.getAreaCode()));
            }
            if (StringUtils.isNotEmpty(searchParam.getIndustry())) {
                boolQueryBuilder.filter(QueryBuilders.matchPhraseQuery("economics", searchParam.getIndustry()));
            }
            if (StringUtils.isNotEmpty(searchParam.getTaxType())) {
                boolQueryBuilder.filter(QueryBuilders.matchPhraseQuery("stypes", searchParam.getTaxType()));
            }

            // 排序
            // 根据 years 降序排列
            if (BwbdType.ORDER_TYPE_DATE.equals(searchParam.getOrderType())) {
                searchSourceBuilder.sort(new FieldSortBuilder("contentDate").order(SortOrder.DESC));
            }
        }
        // 如果没有检索内容 默认时间排序
        if (StringUtils.isEmpty(text)) {
            searchSourceBuilder.sort(new FieldSortBuilder("contentDate").order(SortOrder.DESC));
        }
        // 根据分数 _score 降序排列 (默认行为)
//        searchSourceBuilder.sort(new ScoreSortBuilder().order(SortOrder.DESC));
    }

    private void obtainFgType(AbstractTxjDto searchDto, SearchResponse searchResponse) {

        // 搜索结果
        SearchHits hits = searchResponse.getHits();
        // 匹配到的总记录数
        long totalHits = hits.getTotalHits();
        searchDto.setTotal(totalHits);
        // 得到匹配度高的文档
        SearchHit[] searchHits = hits.getHits();

        List<BwbdType> bwbdTypes = new ArrayList<>();

        for (SearchHit hit : searchHits) {
            String content = hit.getSourceAsString();//使用ES的java接口将实体类对应的内容转换为json字符串
            BwbdType bwbdType = JSONObject.parseObject(content, BwbdType.class); //生成pojo对象
            // 获取高亮查询的内容。如果存在,则替换原来的name
            Map<String, HighlightField> highlightFields = hit.getHighlightFields();
            if (highlightFields != null) {
                HighlightField nameField = highlightFields.get(bwbdType.PROPERTY_NUMBERS);
                if (nameField != null) {
                    Text[] fragments = nameField.getFragments();
                    StringBuffer stringBuffer = new StringBuffer();
                    for (Text str : fragments) {
                        stringBuffer.append(str.string());
                    }
                    String numbers = stringBuffer.toString();
                    bwbdType.setNumbers(numbers);
                }

                HighlightField titlesField = highlightFields.get(bwbdType.PROPERTY_TITLES);
                if (titlesField != null) {
                    Text[] fragments = titlesField.getFragments();
                    StringBuffer stringBuffer = new StringBuffer();
                    for (Text str : fragments) {
                        stringBuffer.append(str.string());
                    }
                    String titles = stringBuffer.toString();
                    bwbdType.setTitles(titles);
                }

                HighlightField contentsField = highlightFields.get(bwbdType.PROPERTY_CONTENTS);
                if (contentsField != null) {
                    Text[] fragments = contentsField.getFragments();
                    StringBuffer stringBuffer = new StringBuffer();
                    for (Text str : fragments) {
                        stringBuffer.append(str.string());
                    }
                    bwbdType.setContents(stringBuffer.toString());
                }
                // 处理内容
                handleResult(bwbdType);
            }
            bwbdTypes.add(bwbdType);
        }
        searchDto.setRows(bwbdTypes);
    }

spring data

<colgroup> <col style="width: 16.6666%;"> <col style="width: 33.3333%;"> <col style="width: 50.0001%;"> </colgroup> <thead> </thead> <tbody> </tbody>
Table 2. Supported keywords inside method names
Keyword Sample Elasticsearch Query String

And

findByNameAndPrice

{ "query" : {

"bool" : { "must" : [ { "query_string" : { "query" : "?", "fields" : [ "name" ] } }, { "query_string" : { "query" : "?", "fields" : [ "price" ] } } ] }

}}

Or

findByNameOrPrice

{ "query" : {

"bool" : { "should" : [ { "query_string" : { "query" : "?", "fields" : [ "name" ] } }, { "query_string" : { "query" : "?", "fields" : [ "price" ] } } ] }

}}

Is

findByName

{ "query" : {

"bool" : { "must" : [ { "query_string" : { "query" : "?", "fields" : [ "name" ] } } ] }

}}

Not

findByNameNot

{ "query" : {

"bool" : { "must_not" : [ { "query_string" : { "query" : "?", "fields" : [ "name" ] } } ] }

}}

Between

findByPriceBetween

{ "query" : {

"bool" : { "must" : [ {"range" : {"price" : {"from" : ?, "to" : ?, "include_lower" : true, "include_upper" : true } } } ] }

}}

LessThan

findByPriceLessThan

{ "query" : {

"bool" : { "must" : [ {"range" : {"price" : {"from" : null, "to" : ?, "include_lower" : true, "include_upper" : false } } } ] }

}}

LessThanEqual

findByPriceLessThanEqual

{ "query" : {

"bool" : { "must" : [ {"range" : {"price" : {"from" : null, "to" : ?, "include_lower" : true, "include_upper" : true } } } ] }

}}

GreaterThan

findByPriceGreaterThan

{ "query" : {

"bool" : { "must" : [ {"range" : {"price" : {"from" : ?, "to" : null, "include_lower" : false, "include_upper" : true } } } ] }

}}

GreaterThanEqual

findByPriceGreaterThan

{ "query" : {

"bool" : { "must" : [ {"range" : {"price" : {"from" : ?, "to" : null, "include_lower" : true, "include_upper" : true } } } ] }

}}

Before

findByPriceBefore

{ "query" : {

"bool" : { "must" : [ {"range" : {"price" : {"from" : null, "to" : ?, "include_lower" : true, "include_upper" : true } } } ] }

}}

After

findByPriceAfter

{ "query" : {

"bool" : { "must" : [ {"range" : {"price" : {"from" : ?, "to" : null, "include_lower" : true, "include_upper" : true } } } ] }

}}

Like

findByNameLike

{ "query" : {

"bool" : { "must" : [ { "query_string" : { "query" : "?*", "fields" : [ "name" ] }, "analyze_wildcard": true } ] }

}}

StartingWith

findByNameStartingWith

{ "query" : {

"bool" : { "must" : [ { "query_string" : { "query" : "?*", "fields" : [ "name" ] }, "analyze_wildcard": true } ] }

}}

EndingWith

findByNameEndingWith

{ "query" : {

"bool" : { "must" : [ { "query_string" : { "query" : "*?", "fields" : [ "name" ] }, "analyze_wildcard": true } ] }

}}

Contains/Containing

findByNameContaining

{ "query" : {

"bool" : { "must" : [ { "query_string" : { "query" : "*?*", "fields" : [ "name" ] }, "analyze_wildcard": true } ] }

}}

In

findByNameIn(Collection<String>names)

{ "query" : {

"bool" : { "must" : [ {"bool" : {"must" : [ {"terms" : {"name" : ["?","?"]}} ] } } ] }

}}

NotIn

findByNameNotIn(Collection<String>names)

{ "query" : {

"bool" : { "must" : [ {"bool" : {"must_not" : [ {"terms" : {"name" : ["?","?"]}} ] } } ] }

}}

Near

findByStoreNear

Not Supported Yet !

True

findByAvailableTrue

{ "query" : {

"bool" : { "must" : [ { "query_string" : { "query" : "true", "fields" : [ "available" ] } } ] }

}}

False

findByAvailableFalse

{ "query" : {

"bool" : { "must" : [ { "query_string" : { "query" : "false", "fields" : [ "available" ] } } ] }

}}

OrderBy

findByAvailableTrueOrderByNameDesc

{ "query" : {

"bool" : { "must" : [ { "query_string" : { "query" : "true", "fields" : [ "available" ] } } ] } }, "sort":[{"name":{"order":"desc"}}]

}

<colgroup> <col style="width: 25%;"> <col style="width: 25%;"> <col style="width: 25%;"> <col style="width: 25%;"> </colgroup> <thead> </thead> <tbody> </tbody>
Table 3. Supported Entity Callbacks
Callback Method Description Order

Reactive/BeforeConvertCallback

onBeforeConvert(T entity, IndexCoordinates index)

Invoked before a domain object is converted to org.springframework.data.elasticsearch.core.document.Document. Can return the entity or a modified entity which then will be converted.

Ordered.LOWEST_PRECEDENCE

Reactive/AfterConvertCallback

onAfterConvert(T entity, Document document, IndexCoordinates indexCoordinates)

Invoked after a domain object is converted from org.springframework.data.elasticsearch.core.document.Document on reading result data from Elasticsearch.

Ordered.LOWEST_PRECEDENCE

Reactive/AuditingEntityCallback

onBeforeConvert(Object entity, IndexCoordinates index)

Marks an auditable entity created or modified

100

Reactive/AfterSaveCallback

T onAfterSave(T entity, IndexCoordinates index)

Invoked after a domain object is saved.

Ordered.LOWEST_PRECEDENCE

elasticsearch中的Suggest用法

elasticsearch中的Suggest用法 Elasticsearch搜索Suggest功能优化 elasticsearch拼音插件安装以及(IK+pinyin使用)

Elasticsearch学习笔记5: suggest实现搜索补全

Spring Data Elasticsearch 调用分词器和搜索建议接口 Elasticsearch自动补全-ngram

高级搜索(中文+拼音+首字母+简繁转换+补全)

elasticsearch中的Suggest用法

settings

创建特殊字符的映射关系 https://www.cnblogs.com/clonen/p/6674888.html

touch /alidata/dockerdata/elasticsearch/config/char_filter.txt

[Elasticsearch Index Template(索引模板)]

PUT _template/zy

{
    "index_patterns":"zy_*",
    "settings":{
        "index.number_of_replicas":"1",
        "index.number_of_shards":"5",
        "index.translog.flush_threshold_size":"512mb",
        "index.translog.sync_interval":"60s",
        "index.codec":"best_compression",
        "analysis":{
            "filter":{
                "edge_ngram_filter":{
                    "type":"edge_ngram",
                    "min_gram":1,
                    "max_gram":50
                },
                "pinyin_simple_filter":{
                    "type":"pinyin",
                    "keep_first_letter":true,
                    "keep_separate_first_letter":true,
                    "keep_full_pinyin":false,
                    "keep_original":false,
                    "limit_first_letter_length":50,
                    "lowercase":true
                },
                "pinyin_full_filter":{
                    "type":"pinyin",
                    "keep_first_letter":false,
                    "keep_separate_first_letter":false,
                    "keep_full_pinyin":true,
                    "none_chinese_pinyin_tokenize":true,
                    "keep_original":false,
                    "limit_first_letter_length":50,
                    "lowercase":true
                },
                "t2s_convert":{
                    "type":"stconvert",
                    "delimiter":",",
                    "convert_type":"t2s"
                }
            },
            "char_filter":{
                "charconvert":{
                    "type":"mapping",
                    "mappings_path":"char_filter.txt"
                },
                "tsconvert":{
                    "type":"stconvert",
                    "convert_type":"t2s"
                }
            },
            "analyzer":{
                "ngramIndexAnalyzer":{
                    "type":"custom",
                    "tokenizer":"keyword",
                    "filter":[
                        "edge_ngram_filter",
                        "lowercase"
                    ],
                    "char_filter":[
                        "charconvert",
                        "tsconvert"
                    ]
                },
                "ngramSearchAnalyzer":{
                    "type":"custom",
                    "tokenizer":"keyword",
                    "filter":[
                        "lowercase"
                    ],
                    "char_filter":[
                        "charconvert",
                        "tsconvert"
                    ]
                },
                "ikIndexAnalyzer":{
                    "type":"custom",
                    "tokenizer":"ik_max_word",
                    "char_filter":[
                        "charconvert",
                        "tsconvert"
                    ]
                },
                "ikSearchAnalyzer":{
                    "type":"custom",
                    "tokenizer":"ik_smart",
                    "char_filter":[
                        "charconvert",
                        "tsconvert"
                    ]
                },
                "pinyiSimpleIndexAnalyzer":{
                    "tokenizer":"keyword",
                    "filter":[
                        "pinyin_simple_filter",
                        "edge_ngram_filter",
                        "lowercase"
                    ]
                },
                "pinyiSimpleSearchAnalyzer":{
                    "tokenizer":"keyword",
                    "filter":[
                        "pinyin_simple_filter",
                        "lowercase"
                    ]
                },
                "pinyiFullIndexAnalyzer":{
                    "tokenizer":"keyword",
                    "filter":[
                        "pinyin_full_filter",
                        "lowercase"
                    ]
                },
                "pinyiFullSearchAnalyzer":{
                    "tokenizer":"keyword",
                    "filter":[
                        "pinyin_full_filter",
                        "lowercase"
                    ]
                },
                "tsconvert":{
                    "tokenizer":"tsconvert"
                }
            },
            "tokenizer":{
                "tsconvert":{
                    "type":"stconvert",
                    "delimiter":"#",
                    "keep_both":false,
                    "convert_type":"t2s"
                }
            }
        }
    }
}

mapping

@Document(type = "book", _timestamp = true, _ttl = @TTL(enabled = true, _default = "5m"))
public class Book {
    /*ID,只能是Long或者String类型*/
    @Id
    private Long id;

    /*数值类型*/
    @Field(type = FieldType.Double, ignoreMalformed = true)
    private Double price;

    /*数值类型*/
    @Field(type = FieldType.Integer)
    private Integer pageCount;

    /*未分词String型*/
    @Field(type = FieldType.String, index = FieldIndex.not_analyzed)
    private String isnNo;

    /*bool型*/
    @Field(type = FieldType.Boolean, nullValue = "false")
    private Boolean isValid;

    /*日期类型*/
    @Field(type = FieldType.Date, format = DateFormat.basic_time_no_millis)
    private Date publishDate;

    /*分词String类型,并设置fielddata加载限制(当然也可不设置用默认)*/
    @Field(
            type = FieldType.String,
            index = FieldIndex.analyzed,
            analyzer = "ik_max_word",
            searchAnalyzer = "ik_smart",
            termVector = TermVector.with_positions_offsets,
            fielddata = @Fielddata(
                    format = FielddataFormat.paged_bytes,
                    frequency = @FielddataFrequencyFilter(
                            enable = true,
                            min = 0.001,
                            max = 1.2,
                            minSegmentSize = 500
                    ),
                    loading = FielddataLoading.eager_global_ordinals
            )

    )
    private String author;

    /*multi field 类型(用于多字段搜索)*/
    @MultiField(
            mainField = @Field(type = FieldType.String, index = FieldIndex.analyzed, analyzer = "ik_max_word", searchAnalyzer = "ik_smart"),
            otherFields = {
                    @MultiNestedField(dotSuffix = "pinyin", nestedField = @Field(
                            type = FieldType.String,
                            index = FieldIndex.analyzed,
                            analyzer = "lc_index",
                            searchAnalyzer = "lc_search")
                    ),
                    @MultiNestedField(dotSuffix = "english", nestedField = @Field(
                            type = FieldType.String,
                            index = FieldIndex.analyzed,
                            analyzer = "standard")
                    )
            }
    )
    private String title;

    /*Completion Context Suggester配置(如果不配置CompletionContext则是Completion Suggester)*/
    @CompletionField(analyzer = "ik", payloads = true, context = {
            @CompletionContext(name = "bookType", type = CompletionContextType.category, defaultVal = {"algorithm"}),
            @CompletionContext(name = "bookColor", type = CompletionContextType.category, defaultVal = {"red"})
    })
    private String suggestContextField;

    /*二进制类型*/
    @Field(type = FieldType.Binary)
    private byte[] pdf;

    /*内嵌类型*/
    @NestedObject(clazz = SalesArea.class)
    private SalesArea salesArea;

}

排序

public Map<String, Object> findByQuestion(String content,int page){
		Map<String, Object> results=new HashMap<>();
		MatchQueryBuilder matchQueryBuilder=new MatchQueryBuilder("contents", content);
		matchQueryBuilder.analyzer("ik");
		TermQueryBuilder termQueryType=new TermQueryBuilder("type", "question");
		Page<QA> questionQa=qaRepository.search(new NativeSearchQueryBuilder()
				.withQuery(new AndQueryBuilder().add(matchQueryBuilder).add(termQueryType))
				.withPageable(new PageRequest(page, 10))
				.withSort(new ScoreSortBuilder().order(SortOrder.DESC))
				.withSort(new ScriptSortBuilder("'qyer'==doc['source'].value?0:('mafengwo'==doc['source'].value?1:2)","number").order(SortOrder.ASC))
				.withSort(new FieldSortBuilder("view_cnt").order(SortOrder.DESC))
				.build());
		System.err.println("----------------------------------------");
		System.err.println("总问题数:"+questionQa.getTotalElements());
		System.err.println("总页数:"+questionQa.getTotalPages());
		System.err.println("----------------------------------------");
		Map<String, Object> map=new HashMap<>();
		List<Map<String, Object>> listmap=new ArrayList<>();
		for(QA qa:questionQa.getContent()){
			
			map.put("question",qa);
			Page<QA> answerQas=qaRepository.search(new NativeSearchQueryBuilder()
			.withQuery(new AndQueryBuilder().add(new TermQueryBuilder("qid", qa.getQid())).add(new TermQueryBuilder("type", "answer")))
			.withSort(new FieldSortBuilder("view_cnt").order(SortOrder.DESC))
			.build());
			List<QA> qas=new ArrayList<QA>();
			for(QA answerQa:answerQas.getContent()){
				qas.add(answerQa);
			}
			map.put("answer", qas);
			listmap.add(map);
		}
		results.put("results", listmap);
		return results;
	}

————————————————
版权声明:本文为CSDN博主「张小竟」的原创文章,遵循CC 4.0 BY-SA版权协议,转载请附上原文出处链接及本声明。
原文链接:https://blog.csdn.net/zhanglu1236789/article/details/56678443


{
  "from": 0,
  "size": "100",
  "min_score": 1,
  "query": {
    "function_score": {
      "filter": {
        "bool": {
          "should": {
            "multi_match": {
              "query": "bmw",
              "fields": [
                "name",
                "host"
              ]
            }
          }
        }
      },
      "functions": [
        {
          "filter": {
            "match": {
              "name": {
                "query": " bmw",
                "minimum_should_match": "100%"
              }
            }
          },
          "weight": "4"
        },
        {
          "filter": {
            "match": {
              "host": "bmw"
            }
          },
          "weight": "10"
        },
        {
          "script_score": {
            "script": "width=doc['rank'].value;if(width==0){return -4};return 0"
          }
        }
      ],
      "score_mode": "sum"
    }
  },
  "sort": [
    {
      "_score": {
        "order": "desc"
      }
    }
  ]
}
 
备注
function_score  自定义 score 得分使用 函数
 
min_score 最小score 的值, 显示大于当前score 的值
 
score_mode score 得分计算方式 ; 
 
script_score script 的作用 ,如果 rank 字段的值等于 0 该记录的score 得分 = score得分 - 4; 比如 原来 得分 10 现在的得分是6 
 
sort 按照 score 得分进行排序
————————————————
版权声明:本文为CSDN博主「记忆的残缺」的原创文章,遵循CC 4.0 BY-SA版权协议,转载请附上原文出处链接及本声明。
原文链接:https://blog.csdn.net/u014017121/article/details/69951202

问题解决

Rejecting mapping update to [zhongyao] as the final mapping would have more than 1 type: [_doc, zhongyao]

已存在所有,所以手工设置索引时,需要createIndex = false

No mapping found for [id] in order to sort on

去掉order