How to copy selective documents from one index to a new index in Elasticsearch? - Big Data In Real World

How to copy selective documents from one index to a new index in Elasticsearch?

How to copy an index and its contents to a new index in Elasticsearch?
February 23, 2022
What is the difference between map and flatMap functions in Spark?
March 9, 2022
How to copy an index and its contents to a new index in Elasticsearch?
February 23, 2022
What is the difference between map and flatMap functions in Spark?
March 9, 2022

In this post we have shown how to copy an index and all its contents to a new index in Elasticsearch.

In this post we are going to show how to copy selective documents from one index to a new index in Elasticsearch.

reindex with query

As shown in the other post, we still use a reindex by specifying the source and destination but this time we also specify a query in source along with the term which indicates that the documents with state = ‘ny’ will be filtered from the source.

So only documents with state ny will be copied to the new index account_v3 with this reindex operation.

 curl -X POST localhost:9200/_reindex  -H 'Content-Type: application/json' -d'
 {
   "source": {
     "index": "account",
     "query": {
       "term": {
         "state": "ny"
       }
     }
   },
   "dest": {
     "index": "account_v3"
   }
 }'
 {
 "took": 591,
 "timed_out": false,
 "total": 20,
 "updated": 0,
 "created": 20,
 "deleted": 0,
 "batches": 1,
 "version_conflicts": 0,
 "noops": 0,
 "retries": {
 "bulk": 0,
 "search": 0
 },
 "throttled_millis": 0,
 "requests_per_second": -1.0,
 "throttled_until_millis": 0,
 "failures": []
 } 

Let’s list the indices in elasticsearch. There you go we see 3 indices the new index account_v3 has only 20 documents because we only have 20 documents with state = ny in account.

 [osboxes@wk1 ~]$  curl http://localhost:9200/_cat/indices?v
 health status index      uuid                   pri rep docs.count docs.deleted store.size pri.store.size
 yellow open   account_v2 Kws5DlAdQuCv8845NZPUhQ   1   1        993            0    393.5kb        393.5kb
 yellow open   account_v3 vqWW9i3KSD-wjqFH5kFWwg   1   1         20            0     17.4kb         17.4kb
 yellow open   account    avtO6o3jTmWtgAyQwTTM6Q   1   1        993            7    400.9kb        400.9kb 
Big Data In Real World
Big Data In Real World
We are a group of Big Data engineers who are passionate about Big Data and related Big Data technologies. We have designed, developed, deployed and maintained Big Data applications ranging from batch to real time streaming big data platforms. We have seen a wide range of real world big data problems, implemented some innovative and complex (or simple, depending on how you look at it) solutions.

Comments are closed.

How to copy selective documents from one index to a new index in Elasticsearch?
This website uses cookies to improve your experience. By using this website you agree to our Data Protection Policy.

Hadoop In Real World is now Big Data In Real World!

X