Use elasticsearch to delete index data regularly

1. Sometimes we use ES

Due to limited resources or business needs, we only want to save data from the most recent period, so it is necessary to delete data at a scheduled time.

2. Write a script

vim del_es_by_day.sh
#!/bin/bash
#Scheduled deletion of elasticsearch index#author menard 2019-3-25
date=`date -d "-7 days" "+%Y.%m.%d"`
/usr/bin/curl -v --user elastic:password -XDELETE "http://192.168.10.201:9200/*-$date"

Add executable permissions chmod +x del_es_by_day.sh

3. Create an index for testing

put test-2019.03.18
put index-2019.03.18 

4. Execute the script test results and you can see that the deletion is successful

5. Do scheduled tasks

crontab -e
00 01 * * * /workspace/script/del_es_by_day.sh

Supplement: Elasticsearch scheduled backup index data and recovery

Scheduled backup script

Linux scheduled tasks use cron service to perform

Writing cron expressions for scheduled tasks

crontab -e #Enter cron scheduled task editing

Scheduled tasks

*/1 * * * * /opt/scheduler/es_bk.sh >> /opt/scheduler/bk_log.txt 2>&1

The es_bk.sh script in the /opt/scheduler/ directory is executed every 1 minute, and the data content is written to the bk.log.txt file in the /opt/scheduler directory.

Check the contents of the es_bk.sh script

#!/bin/bash
echo '==================================start========================================'
#Delete the backup snapshot curl -i -X ​​DELETE localhost:9200/_snapshot/es_backup/snapshot01
#Backup again curl -i -X ​​PUT localhost:9200/_snapshot/es_backup/snapshot01
echo '===================================end=========================================='

ES backup data requires a snapshot of the index data to be backed up. A snapshot name needs to be specified, and the same snapshot cannot be used. Therefore, the old snapshot needs to be deleted before each backup and then backed up again.

ES backup and recovery

Create a backup repository (directory)

mkdir -p /bk/es/data
#Change permissions chmod -R 777 bk

Modify the elasticsearch.yml file and specify the warehouse location

Send a request to initialize the repository

curl -i -H ""'Content-Type:application/json;charset=UTF-8'"" -X PUT --data '{"type": "fs","settings": {"location": "/bk/es/data"}}' localhost:9200/_snapshot/es_backup

es_backup is the backup namespace and can be specified at will

Creating the first snapshot

curl -i -X ​​PUT localhost:9200/_snapshot/es_backup/snapshot01

We will use es_backup to back up all index data of es to snapshot01. Of course, we can also back up only the specified index.

curl -i -H ""'Content-Type:application/json;charset=UTF-8'"" -X PUT --data '{"indices": "'bk_user_index_server'"}' localhost:9200/_snapshot/es_backup/snapshot01

Restart scheduled tasks

systemctl restart cron

Restore Index

Restore Assignment

curl -i -H ""'Content-Type:application/json;charset=UTF-8'"" -X POST --data '{"indices": "'bk_user_index_server'"}' localhost:9200/_snapshot/es_backup/snapshot01/_restore

Restore All

curl -i -X ​​POST localhost:9200/_snapshot/es_backup/snapshot01/_restore

The above is my personal experience. I hope it can give you a reference. I also hope that you will support 123WORDPRESS.COM. If there are any mistakes or incomplete considerations, please feel free to correct me.