How to fail a Hive script based on a condition?
March 20, 2023How to automatically add timestamp to documents and find the latest document in Elasticsearch?
March 27, 2023Each files we upload to S3 gets assigned a key and the key has the following structure [FOLDERNAME]/[FILENAME]
A common problem is to search the objects by the name of the file or object.
Solution
If you are on the console, simply type the name you are trying to search on the search box.
Use aws s3 ls command with recursive option followed by a grep and the file name pattern you are trying to match. Here is an example. Below we are trying to find the files or objects with “instance” in its name.
[osboxes@wk1 ~]$ aws s3 ls s3://hirw-workshop --recursive | grep instance 2015-11-18 12:26:23 35309 logs/j-3W1M8XUBXX40/node/i-37098987/daemons/instance-state/instance-state.log-2015-11-18-12-15.gz 2015-11-18 12:35:50 34855 logs/j-3W1M8XUBXX40/node/i-37098987/daemons/instance-state/instance-state.log-2015-11-18-12-30.gz 2015-11-18 12:22:35 20959 logs/j-3W1M8XUBXX40/node/i-3a09898a/daemons/instance-state/instance-state.log-2015-11-18-12-15.gz 2015-11-18 12:32:35 73 logs/j-3W1M8XUBXX40/node/i-3a09898a/daemons/instance-state/instance-state.log-2015-11-18-12-30.gz 2015-11-18 12:22:35 20840 logs/j-3W1M8XUBXX40/node/i-3b09898b/daemons/instance-state/instance-state.log-2015-11-18-12-15.gz 2015-11-18 12:32:35 73 logs/j-3W1M8XUBXX40/node/i-3b09898b/daemons/instance-state/instance-state.log-2015-11-18-12-30.gz
Big Data In Real World
We are a group of Big Data engineers who are passionate about Big Data and related Big Data technologies. We have designed, developed, deployed and maintained Big Data applications ranging from batch to real time streaming big data platforms. We have seen a wide range of real world big data problems, implemented some innovative and complex (or simple, depending on how you look at it) solutions.
