Elasticsearch

来自ling
跳转至: 导航搜索

Docker常用服务#elasticsearch

http://cloud.ling2.cn:9800/

IK分词

elasticsearch常用代码 ESService elasticsearch reindex

迁移

ESWorkOrderController

query和filter

https://blog.csdn.net/laoyang360/article/details/80468757

迁移

第一种方法 (使用elasticdump)这是一个nodejs的插件,安装方式直接使用 npm即可

npm install elasticdump
cd /volume3/node_modules/elasticdump/bin

导出索引的格式

elasticdump --input {protocol}://{host}:{port}/{index} --output ./test_index.json

例子:将ES中的test_index 中的索引导出

导出当前索引的mapping结构

elasticdump --input http://www.ling2.cn:9200/test_index --output ./test_index_mapping.json --type=mapping

导出当前索引下的所有真实数据

elasticdump --input http://www.ling2.cn:9200/test_index --output ./test_index.json --type=data

向新的es服务导入索引

创建索引

curl -XPUT http:192.168.56.104:9200/test_index

因为导入的是mapping,所以设置type为mapping

elasticdump --input ./test_index_mapping.json --output http://192.168.56.105:9200/ --type=mapping

因为导入的是data(真实数据)所以设置type为data

elasticdump --input ./test_index.json --output http://192.168.56.105:9200/ --type=data

脚本备份

 cd /volume3/node_modules/elasticdump/bin
./elasticdump --input http://www.ling2.cn:9290/szzy --output /alidata/backup/elasticsearch/szzy_mapping.json --type=mapping
./elasticdump --input http://www.ling2.cn:9290/szzy --output /alidata/backup/elasticsearch/szzy.json --type=data
./elasticdump --input http://www.ling2.cn:9290/ocrpage --output /alidata/backup/elasticsearch/ocrpage_mapping.json --type=mapping
./elasticdump --input http://www.ling2.cn:9290/ocrpage --output /alidata/backup/elasticsearch/ocrpage.json --type=data
./elasticdump --input http://www.ling2.cn:9290/book --output /alidata/backup/elasticsearch/book_mapping.json --type=mapping
./elasticdump --input http://www.ling2.cn:9290/book --output /alidata/backup/elasticsearch/book.json --type=data
./elasticdump --input http://www.ling2.cn:9290/fangji --output /alidata/backup/elasticsearch/fangji_mapping.json --type=mapping
./elasticdump --input http://www.ling2.cn:9290/fangji --output /alidata/backup/elasticsearch/fangji.json --type=data
./elasticdump --input http://www.ling2.cn:9290/zhongyao --output /alidata/backup/elasticsearch/zhongyao_mapping.json --type=mapping
./elasticdump --input http://www.ling2.cn:9290/zhongyao --output /alidata/backup/elasticsearch/zhongyao.json --type=data
./elasticdump --input http://www.ling2.cn:9290/ocrresponse --output /alidata/backup/elasticsearch/ocrresponse_mapping.json --type=mapping
./elasticdump --input http://www.ling2.cn:9290/ocrresponse --output /alidata/backup/elasticsearch/ocrresponse.json --type=data
curl -XPUT http://qunhui.ling2.cn:9290/szzy
./elasticdump --input /alidata/backup/elasticsearch/szzy_mapping.json --output http://qunhui.ling2.cn:9290/ --type=mapping
./elasticdump --input /alidata/backup/elasticsearch/szzy.json --output http://qunhui.ling2.cn:9290/ --type=data

curl -XPUT http://qunhui.ling2.cn:9290/ocrpage
./elasticdump --input /alidata/backup/elasticsearch/ocrpage_mapping.json --output http://qunhui.ling2.cn:9290/ --type=mapping
./elasticdump --input /alidata/backup/elasticsearch/ocrpage.json --output http://qunhui.ling2.cn:9290/ --type=data

curl -XPUT http://qunhui.ling2.cn:9290/book
./elasticdump --input /alidata/backup/elasticsearch/book_mapping.json --output http://qunhui.ling2.cn:9290/ --type=mapping
./elasticdump --input /alidata/backup/elasticsearch/book.json --output http://qunhui.ling2.cn:9290/ --type=data

curl -XPUT http://qunhui.ling2.cn:9290/fangji
./elasticdump --input /alidata/backup/elasticsearch/fangji_mapping.json --output http://qunhui.ling2.cn:9290/ --type=mapping
./elasticdump --input /alidata/backup/elasticsearch/fangji.json --output http://qunhui.ling2.cn:9290/ --type=data

curl -XPUT http://qunhui.ling2.cn:9290/zhongyao
./elasticdump --input /alidata/backup/elasticsearch/zhongyao_mapping.json --output http://qunhui.ling2.cn:9290/ --type=mapping
./elasticdump --input /alidata/backup/elasticsearch/zhongyao.json --output http://qunhui.ling2.cn:9290/ --type=data

curl -XPUT http://qunhui.ling2.cn:9290/ocrresponse
./elasticdump --input /alidata/backup/elasticsearch/ocrresponse_mapping.json --output http://qunhui.ling2.cn:9290/ --type=mapping
./elasticdump --input /alidata/backup/elasticsearch/ocrresponse.json --output http://qunhui.ling2.cn:9290/ --type=data

脚本到笔记本

 cd /root/node_modules/elasticdump/bin
 touch /alidata/dockerdata/elasticsearch/config/char_filter.txt
./elasticdump --input http://192.168.31.2:9290/zy_szzy --output /alidata/backup/elasticsearch/szzy_mapping.json --type=mapping
./elasticdump --input http://192.168.31.2:9290/zy_szzy --output /alidata/backup/elasticsearch/szzy.json --type=data
./elasticdump --input http://192.168.31.2:9290/zy_ocrpage --output /alidata/backup/elasticsearch/ocrpage_mapping.json --type=mapping
./elasticdump --input http://192.168.31.2:9290/zy_ocrpage --output /alidata/backup/elasticsearch/ocrpage.json --type=data
./elasticdump --input http://192.168.31.2:9290/zy_book --output /alidata/backup/elasticsearch/book_mapping.json --type=mapping
./elasticdump --input http://192.168.31.2:9290/zy_book --output /alidata/backup/elasticsearch/book.json --type=data
./elasticdump --input http://192.168.31.2:9290/zy_fangji --output /alidata/backup/elasticsearch/fangji_mapping.json --type=mapping
./elasticdump --input http://192.168.31.2:9290/zy_fangji --output /alidata/backup/elasticsearch/fangji.json --type=data
./elasticdump --input http://192.168.31.2:9290/zy_zhongyao --output /alidata/backup/elasticsearch/zhongyao_mapping.json --type=mapping
./elasticdump --input http://192.168.31.2:9290/zy_zhongyao --output /alidata/backup/elasticsearch/zhongyao.json --type=data
./elasticdump --input http://192.168.31.2:9290/zy_ocrresponse --output /alidata/backup/elasticsearch/ocrresponse_mapping.json --type=mapping
./elasticdump --input http://192.168.31.2:9290/zy_ocrresponse --output /alidata/backup/elasticsearch/ocrresponse.json --type=data
curl --location --request PUT '192.168.74.134:9290/_template/zy' \
--header 'Content-Type: application/json' \
--data-raw '{
  "index_patterns": "zy_*",
  "settings": {
    "index.number_of_replicas": "1",
    "index.number_of_shards": "5",
    "index.translog.flush_threshold_size": "512mb",
    "index.translog.sync_interval": "60s",
    "index.codec": "best_compression",
    "analysis": {
      "filter": {
        "edge_ngram_filter": {
          "type": "edge_ngram",
          "min_gram": 1,
          "max_gram": 50
        },
        "pinyin_simple_filter": {
          "type": "pinyin",
          "keep_first_letter": true,
          "keep_separate_first_letter": true,
          "keep_full_pinyin": false,
          "keep_original": false,
          "limit_first_letter_length": 50,
          "lowercase": true
        },
        "pinyin_full_filter": {
          "type": "pinyin",
          "keep_first_letter": false,
          "keep_separate_first_letter": false,
          "keep_full_pinyin": true,
          "none_chinese_pinyin_tokenize": true,
          "keep_original": false,
          "limit_first_letter_length": 50,
          "lowercase": true
        },
        "t2s_convert": {
          "type": "stconvert",
          "delimiter": ",",
          "convert_type": "t2s"
        },
        "auto_complete_filter": {
          "type": "edge_ngram",
          "min_gram": 1,
          "max_gram": 15
        },
        "pinyin_filter": {
          "type": "pinyin",
          "keep_first_letter": true,
          "keep_full_pinyin": false,
          "keep_joined_full_pinyin": true,
          "keep_none_chinese": true,
          "keep_original": true,
          "limit_first_letter_length": 16,
          "lowercase": true,
          "trim_whitespace": true,
          "keep_none_chinese_in_first_letter": true
        }
      },
      "char_filter": {
        "charconvert": {
          "type": "mapping",
          "mappings_path": "char_filter.txt"
        },
        "tsconvert": {
          "type": "stconvert",
          "convert_type": "t2s"
        }
      },
      "analyzer": {
        "ngramIndexAnalyzer": {
          "type": "custom",
          "tokenizer": "keyword",
          "filter": [
            "edge_ngram_filter",
            "lowercase"
          ],
          "char_filter": [
            "charconvert",
            "tsconvert"
          ]
        },
        "ngramSearchAnalyzer": {
          "type": "custom",
          "tokenizer": "keyword",
          "filter": [
            "lowercase"
          ],
          "char_filter": [
            "charconvert",
            "tsconvert"
          ]
        },
        "ikIndexAnalyzer": {
          "type": "custom",
          "tokenizer": "ik_max_word",
          "char_filter": [
            "charconvert",
            "tsconvert"
          ]
        },
        "ikSearchAnalyzer": {
          "type": "custom",
          "tokenizer": "ik_smart",
          "char_filter": [
            "charconvert",
            "tsconvert"
          ]
        },
        "pinyiSimpleIndexAnalyzer": {
          "tokenizer": "keyword",
          "filter": [
            "pinyin_simple_filter",
            "edge_ngram_filter",
            "lowercase"
          ]
        },
        "pinyiSimpleSearchAnalyzer": {
          "tokenizer": "keyword",
          "filter": [
            "pinyin_simple_filter",
            "lowercase"
          ]
        },
        "pinyiFullIndexAnalyzer": {
          "tokenizer": "keyword",
          "filter": [
            "pinyin_full_filter",
            "lowercase"
          ]
        },
        "pinyiFullSearchAnalyzer": {
          "tokenizer": "keyword",
          "filter": [
            "pinyin_full_filter",
            "lowercase"
          ]
        },
        "tsconvert": {
          "tokenizer": "tsconvert"
        },
        "chinese_pinyin_prefix_analyzer": {
          "type": "custom",
          "char_filter": [
            "html_strip"
          ],
          "tokenizer": "keyword",
          "filter": [
            "lowercase",
            "pinyin_filter",
            "auto_complete_filter"
          ]
        },
        "chinese_prefix_analyzer": {
          "type": "custom",
          "char_filter": [
            "html_strip"
          ],
          "tokenizer": "keyword",
          "filter": [
            "lowercase",
            "auto_complete_filter"
          ]
        }
      },
      "tokenizer": {
        "tsconvert": {
          "type": "stconvert",
          "delimiter": "#",
          "keep_both": false,
          "convert_type": "t2s"
        }
      }
    }
  }
}'
curl -XPUT http://192.168.74.134:9290/zy_szzy
./elasticdump --input /alidata/backup/elasticsearch/szzy_mapping.json --output http://192.168.74.134:9290/ --type=mapping
./elasticdump --input /alidata/backup/elasticsearch/szzy.json --output http://192.168.74.134:9290/ --type=data

curl -XPUT http://192.168.74.134:9290/zy_ocrpage
./elasticdump --input /alidata/backup/elasticsearch/ocrpage_mapping.json --output http://192.168.74.134:9290/ --type=mapping
./elasticdump --input /alidata/backup/elasticsearch/ocrpage.json --output http://192.168.74.134:9290/ --type=data

curl -XPUT http://192.168.74.134:9290/zy_book
./elasticdump --input /alidata/backup/elasticsearch/book_mapping.json --output http://192.168.74.134:9290/ --type=mapping
./elasticdump --input /alidata/backup/elasticsearch/book.json --output http://192.168.74.134:9290/ --type=data

curl -XPUT http://192.168.74.134:9290/zy_fangji
./elasticdump --input /alidata/backup/elasticsearch/fangji_mapping.json --output http://192.168.74.134:9290/ --type=mapping
./elasticdump --input /alidata/backup/elasticsearch/fangji.json --output http://192.168.74.134:9290/ --type=data

curl -XPUT http://192.168.74.134:9290/zy_zhongyao
./elasticdump --input /alidata/backup/elasticsearch/zhongyao_mapping.json --output http://192.168.74.134:9290/ --type=mapping
./elasticdump --input /alidata/backup/elasticsearch/zhongyao.json --output http://192.168.74.134:9290/ --type=data

curl -XPUT http://192.168.74.134:9290/zy_ocrresponse
./elasticdump --input /alidata/backup/elasticsearch/ocrresponse_mapping.json --output http://192.168.74.134:9290/ --type=mapping
./elasticdump --input /alidata/backup/elasticsearch/ocrresponse.json --output http://192.168.74.134:9290/ --type=data

错误处理

版本问题

https://docs.spring.io/spring-data/elasticsearch/docs/4.2.4/reference/html/#preface.versions

备份

cluster.name: "docker-cluster"
network.host: 0.0.0.0

http.port: 9290
http.cors.allow-origin: http://localhost:1358,http://127.0.0.1:1358,http://qunhui.ling2.cn:1358
http.cors.enabled: true
http.cors.allow-headers : X-Requested-With,X-Auth-Token,Content-Type,Content-Length,Authorization
http.cors.allow-credentials: true

None of the configured nodes are available

  • 报错原因

在 elasticsearch/config/elasticsearch.yml配置文件中 tcp 端口与ES 客户端配置不一致导致

elasticsearch 服务端与 客户端版本不一致导致

  • 解决方法

统一 ES 版本

修改 elasticsearch/config/elasticsearch.yml 与 客户端配置统一

network.host: 0.0.0.0
# Set a custom port for HTTP:
transport.tcp.port: 9300
http.port: 9200

客户端连接配置

  data:
    elasticsearch:
      ####集群名称
      cluster-name: elasticsearch
      ####地址
      cluster-nodes: 192.168.112.143:9300

重启 ES


as the final mapping would have more than 1 type

备份旧索引-->删除旧索引-->重建索引