Elasticsearch
来自ling
elasticsearch常用代码 ESService elasticsearch reindex
目录
query和filter
https://blog.csdn.net/laoyang360/article/details/80468757
迁移
第一种方法 （使用elasticdump）这是一个nodejs的插件，安装方式直接使用 npm即可
npm install elasticdump cd /volume3/node_modules/elasticdump/bin
导出索引的格式
elasticdump --input {protocol}://{host}:{port}/{index} --output ./test_index.json
例子：将ES中的test_index 中的索引导出
导出当前索引的mapping结构
elasticdump --input http://www.ling2.cn:9200/test_index --output ./test_index_mapping.json --type=mapping
导出当前索引下的所有真实数据
elasticdump --input http://www.ling2.cn:9200/test_index --output ./test_index.json --type=data
向新的es服务导入索引
创建索引
curl -XPUT http:192.168.56.104:9200/test_index
因为导入的是mapping，所以设置type为mapping
elasticdump --input ./test_index_mapping.json --output http://192.168.56.105:9200/ --type=mapping
因为导入的是data（真实数据）所以设置type为data
elasticdump --input ./test_index.json --output http://192.168.56.105:9200/ --type=data
脚本备份
cd /volume3/node_modules/elasticdump/bin
./elasticdump --input http://www.ling2.cn:9290/szzy --output /alidata/backup/elasticsearch/szzy_mapping.json --type=mapping ./elasticdump --input http://www.ling2.cn:9290/szzy --output /alidata/backup/elasticsearch/szzy.json --type=data
./elasticdump --input http://www.ling2.cn:9290/ocrpage --output /alidata/backup/elasticsearch/ocrpage_mapping.json --type=mapping ./elasticdump --input http://www.ling2.cn:9290/ocrpage --output /alidata/backup/elasticsearch/ocrpage.json --type=data
./elasticdump --input http://www.ling2.cn:9290/book --output /alidata/backup/elasticsearch/book_mapping.json --type=mapping ./elasticdump --input http://www.ling2.cn:9290/book --output /alidata/backup/elasticsearch/book.json --type=data
./elasticdump --input http://www.ling2.cn:9290/fangji --output /alidata/backup/elasticsearch/fangji_mapping.json --type=mapping ./elasticdump --input http://www.ling2.cn:9290/fangji --output /alidata/backup/elasticsearch/fangji.json --type=data
./elasticdump --input http://www.ling2.cn:9290/zhongyao --output /alidata/backup/elasticsearch/zhongyao_mapping.json --type=mapping ./elasticdump --input http://www.ling2.cn:9290/zhongyao --output /alidata/backup/elasticsearch/zhongyao.json --type=data
./elasticdump --input http://www.ling2.cn:9290/ocrresponse --output /alidata/backup/elasticsearch/ocrresponse_mapping.json --type=mapping ./elasticdump --input http://www.ling2.cn:9290/ocrresponse --output /alidata/backup/elasticsearch/ocrresponse.json --type=data
curl -XPUT http://qunhui.ling2.cn:9290/szzy ./elasticdump --input /alidata/backup/elasticsearch/szzy_mapping.json --output http://qunhui.ling2.cn:9290/ --type=mapping ./elasticdump --input /alidata/backup/elasticsearch/szzy.json --output http://qunhui.ling2.cn:9290/ --type=data curl -XPUT http://qunhui.ling2.cn:9290/ocrpage ./elasticdump --input /alidata/backup/elasticsearch/ocrpage_mapping.json --output http://qunhui.ling2.cn:9290/ --type=mapping ./elasticdump --input /alidata/backup/elasticsearch/ocrpage.json --output http://qunhui.ling2.cn:9290/ --type=data curl -XPUT http://qunhui.ling2.cn:9290/book ./elasticdump --input /alidata/backup/elasticsearch/book_mapping.json --output http://qunhui.ling2.cn:9290/ --type=mapping ./elasticdump --input /alidata/backup/elasticsearch/book.json --output http://qunhui.ling2.cn:9290/ --type=data curl -XPUT http://qunhui.ling2.cn:9290/fangji ./elasticdump --input /alidata/backup/elasticsearch/fangji_mapping.json --output http://qunhui.ling2.cn:9290/ --type=mapping ./elasticdump --input /alidata/backup/elasticsearch/fangji.json --output http://qunhui.ling2.cn:9290/ --type=data curl -XPUT http://qunhui.ling2.cn:9290/zhongyao ./elasticdump --input /alidata/backup/elasticsearch/zhongyao_mapping.json --output http://qunhui.ling2.cn:9290/ --type=mapping ./elasticdump --input /alidata/backup/elasticsearch/zhongyao.json --output http://qunhui.ling2.cn:9290/ --type=data curl -XPUT http://qunhui.ling2.cn:9290/ocrresponse ./elasticdump --input /alidata/backup/elasticsearch/ocrresponse_mapping.json --output http://qunhui.ling2.cn:9290/ --type=mapping ./elasticdump --input /alidata/backup/elasticsearch/ocrresponse.json --output http://qunhui.ling2.cn:9290/ --type=data
脚本到笔记本
cd /root/node_modules/elasticdump/bin touch /alidata/dockerdata/elasticsearch/config/char_filter.txt
./elasticdump --input http://192.168.31.2:9290/zy_szzy --output /alidata/backup/elasticsearch/szzy_mapping.json --type=mapping ./elasticdump --input http://192.168.31.2:9290/zy_szzy --output /alidata/backup/elasticsearch/szzy.json --type=data
./elasticdump --input http://192.168.31.2:9290/zy_ocrpage --output /alidata/backup/elasticsearch/ocrpage_mapping.json --type=mapping ./elasticdump --input http://192.168.31.2:9290/zy_ocrpage --output /alidata/backup/elasticsearch/ocrpage.json --type=data
./elasticdump --input http://192.168.31.2:9290/zy_book --output /alidata/backup/elasticsearch/book_mapping.json --type=mapping ./elasticdump --input http://192.168.31.2:9290/zy_book --output /alidata/backup/elasticsearch/book.json --type=data
./elasticdump --input http://192.168.31.2:9290/zy_fangji --output /alidata/backup/elasticsearch/fangji_mapping.json --type=mapping ./elasticdump --input http://192.168.31.2:9290/zy_fangji --output /alidata/backup/elasticsearch/fangji.json --type=data
./elasticdump --input http://192.168.31.2:9290/zy_zhongyao --output /alidata/backup/elasticsearch/zhongyao_mapping.json --type=mapping ./elasticdump --input http://192.168.31.2:9290/zy_zhongyao --output /alidata/backup/elasticsearch/zhongyao.json --type=data
./elasticdump --input http://192.168.31.2:9290/zy_ocrresponse --output /alidata/backup/elasticsearch/ocrresponse_mapping.json --type=mapping ./elasticdump --input http://192.168.31.2:9290/zy_ocrresponse --output /alidata/backup/elasticsearch/ocrresponse.json --type=data
curl --location --request PUT '192.168.74.134:9290/_template/zy' \
--header 'Content-Type: application/json' \
--data-raw '{
"index_patterns": "zy_*",
"settings": {
"index.number_of_replicas": "1",
"index.number_of_shards": "5",
"index.translog.flush_threshold_size": "512mb",
"index.translog.sync_interval": "60s",
"index.codec": "best_compression",
"analysis": {
"filter": {
"edge_ngram_filter": {
"type": "edge_ngram",
"min_gram": 1,
"max_gram": 50
},
"pinyin_simple_filter": {
"type": "pinyin",
"keep_first_letter": true,
"keep_separate_first_letter": true,
"keep_full_pinyin": false,
"keep_original": false,
"limit_first_letter_length": 50,
"lowercase": true
},
"pinyin_full_filter": {
"type": "pinyin",
"keep_first_letter": false,
"keep_separate_first_letter": false,
"keep_full_pinyin": true,
"none_chinese_pinyin_tokenize": true,
"keep_original": false,
"limit_first_letter_length": 50,
"lowercase": true
},
"t2s_convert": {
"type": "stconvert",
"delimiter": ",",
"convert_type": "t2s"
},
"auto_complete_filter": {
"type": "edge_ngram",
"min_gram": 1,
"max_gram": 15
},
"pinyin_filter": {
"type": "pinyin",
"keep_first_letter": true,
"keep_full_pinyin": false,
"keep_joined_full_pinyin": true,
"keep_none_chinese": true,
"keep_original": true,
"limit_first_letter_length": 16,
"lowercase": true,
"trim_whitespace": true,
"keep_none_chinese_in_first_letter": true
}
},
"char_filter": {
"charconvert": {
"type": "mapping",
"mappings_path": "char_filter.txt"
},
"tsconvert": {
"type": "stconvert",
"convert_type": "t2s"
}
},
"analyzer": {
"ngramIndexAnalyzer": {
"type": "custom",
"tokenizer": "keyword",
"filter": [
"edge_ngram_filter",
"lowercase"
],
"char_filter": [
"charconvert",
"tsconvert"
]
},
"ngramSearchAnalyzer": {
"type": "custom",
"tokenizer": "keyword",
"filter": [
"lowercase"
],
"char_filter": [
"charconvert",
"tsconvert"
]
},
"ikIndexAnalyzer": {
"type": "custom",
"tokenizer": "ik_max_word",
"char_filter": [
"charconvert",
"tsconvert"
]
},
"ikSearchAnalyzer": {
"type": "custom",
"tokenizer": "ik_smart",
"char_filter": [
"charconvert",
"tsconvert"
]
},
"pinyiSimpleIndexAnalyzer": {
"tokenizer": "keyword",
"filter": [
"pinyin_simple_filter",
"edge_ngram_filter",
"lowercase"
]
},
"pinyiSimpleSearchAnalyzer": {
"tokenizer": "keyword",
"filter": [
"pinyin_simple_filter",
"lowercase"
]
},
"pinyiFullIndexAnalyzer": {
"tokenizer": "keyword",
"filter": [
"pinyin_full_filter",
"lowercase"
]
},
"pinyiFullSearchAnalyzer": {
"tokenizer": "keyword",
"filter": [
"pinyin_full_filter",
"lowercase"
]
},
"tsconvert": {
"tokenizer": "tsconvert"
},
"chinese_pinyin_prefix_analyzer": {
"type": "custom",
"char_filter": [
"html_strip"
],
"tokenizer": "keyword",
"filter": [
"lowercase",
"pinyin_filter",
"auto_complete_filter"
]
},
"chinese_prefix_analyzer": {
"type": "custom",
"char_filter": [
"html_strip"
],
"tokenizer": "keyword",
"filter": [
"lowercase",
"auto_complete_filter"
]
}
},
"tokenizer": {
"tsconvert": {
"type": "stconvert",
"delimiter": "#",
"keep_both": false,
"convert_type": "t2s"
}
}
}
}
}'curl -XPUT http://192.168.74.134:9290/zy_szzy ./elasticdump --input /alidata/backup/elasticsearch/szzy_mapping.json --output http://192.168.74.134:9290/ --type=mapping ./elasticdump --input /alidata/backup/elasticsearch/szzy.json --output http://192.168.74.134:9290/ --type=data curl -XPUT http://192.168.74.134:9290/zy_ocrpage ./elasticdump --input /alidata/backup/elasticsearch/ocrpage_mapping.json --output http://192.168.74.134:9290/ --type=mapping ./elasticdump --input /alidata/backup/elasticsearch/ocrpage.json --output http://192.168.74.134:9290/ --type=data curl -XPUT http://192.168.74.134:9290/zy_book ./elasticdump --input /alidata/backup/elasticsearch/book_mapping.json --output http://192.168.74.134:9290/ --type=mapping ./elasticdump --input /alidata/backup/elasticsearch/book.json --output http://192.168.74.134:9290/ --type=data curl -XPUT http://192.168.74.134:9290/zy_fangji ./elasticdump --input /alidata/backup/elasticsearch/fangji_mapping.json --output http://192.168.74.134:9290/ --type=mapping ./elasticdump --input /alidata/backup/elasticsearch/fangji.json --output http://192.168.74.134:9290/ --type=data curl -XPUT http://192.168.74.134:9290/zy_zhongyao ./elasticdump --input /alidata/backup/elasticsearch/zhongyao_mapping.json --output http://192.168.74.134:9290/ --type=mapping ./elasticdump --input /alidata/backup/elasticsearch/zhongyao.json --output http://192.168.74.134:9290/ --type=data curl -XPUT http://192.168.74.134:9290/zy_ocrresponse ./elasticdump --input /alidata/backup/elasticsearch/ocrresponse_mapping.json --output http://192.168.74.134:9290/ --type=mapping ./elasticdump --input /alidata/backup/elasticsearch/ocrresponse.json --output http://192.168.74.134:9290/ --type=data
错误处理
版本问题
https://docs.spring.io/spring-data/elasticsearch/docs/4.2.4/reference/html/#preface.versions
备份
cluster.name: "docker-cluster"
network.host: 0.0.0.0
http.port: 9290
http.cors.allow-origin: http://localhost:1358,http://127.0.0.1:1358,http://qunhui.ling2.cn:1358
http.cors.enabled: true
http.cors.allow-headers : X-Requested-With,X-Auth-Token,Content-Type,Content-Length,Authorization
http.cors.allow-credentials: trueNone of the configured nodes are available
- 报错原因
在 elasticsearch/config/elasticsearch.yml配置文件中 tcp 端口与ES 客户端配置不一致导致
elasticsearch 服务端与 客户端版本不一致导致
- 解决方法
统一 ES 版本
修改 elasticsearch/config/elasticsearch.yml 与 客户端配置统一
network.host: 0.0.0.0
# Set a custom port for HTTP:
transport.tcp.port: 9300
http.port: 9200客户端连接配置
data:
elasticsearch:
####集群名称
cluster-name: elasticsearch
####地址
cluster-nodes: 192.168.112.143:9300重启 ES
as the final mapping would have more than 1 type
备份旧索引-->删除旧索引-->重建索引
