How to search a file or objects by name inside an S3 bucket? - Big Data In Real World

How to search a file or objects by name inside an S3 bucket?

How to fail a Hive script based on a condition?
March 20, 2023
How to automatically add timestamp to documents and find the latest document in Elasticsearch?
March 27, 2023
How to fail a Hive script based on a condition?
March 20, 2023
How to automatically add timestamp to documents and find the latest document in Elasticsearch?
March 27, 2023

Each files we upload to S3 gets assigned a key and the key has the following structure [FOLDERNAME]/[FILENAME]

A common problem is to search the objects by the name of the file or object.

Solution

If you are on the console, simply type the name you are trying to search on the search box.

Use aws s3 ls command with recursive option followed by a grep and the file name pattern you are trying to match. Here is an example. Below we are trying to find the files or objects with “instance” in its name.

[osboxes@wk1 ~]$ aws s3 ls s3://hirw-workshop --recursive | grep instance
2015-11-18 12:26:23      35309 logs/j-3W1M8XUBXX40/node/i-37098987/daemons/instance-state/instance-state.log-2015-11-18-12-15.gz
2015-11-18 12:35:50      34855 logs/j-3W1M8XUBXX40/node/i-37098987/daemons/instance-state/instance-state.log-2015-11-18-12-30.gz
2015-11-18 12:22:35      20959 logs/j-3W1M8XUBXX40/node/i-3a09898a/daemons/instance-state/instance-state.log-2015-11-18-12-15.gz
2015-11-18 12:32:35         73 logs/j-3W1M8XUBXX40/node/i-3a09898a/daemons/instance-state/instance-state.log-2015-11-18-12-30.gz
2015-11-18 12:22:35      20840 logs/j-3W1M8XUBXX40/node/i-3b09898b/daemons/instance-state/instance-state.log-2015-11-18-12-15.gz
2015-11-18 12:32:35         73 logs/j-3W1M8XUBXX40/node/i-3b09898b/daemons/instance-state/instance-state.log-2015-11-18-12-30.gz
Big Data In Real World
Big Data In Real World
We are a group of Big Data engineers who are passionate about Big Data and related Big Data technologies. We have designed, developed, deployed and maintained Big Data applications ranging from batch to real time streaming big data platforms. We have seen a wide range of real world big data problems, implemented some innovative and complex (or simple, depending on how you look at it) solutions.

Comments are closed.

How to search a file or objects by name inside an S3 bucket?
This website uses cookies to improve your experience. By using this website you agree to our Data Protection Policy.

Hadoop In Real World is now Big Data In Real World!

X