How to check size of a bucket in S3? - Big Data In Real World

How to check size of a bucket in S3?

How to transpose or convert columns to rows in Hive?
April 3, 2023
What is an alias and how to create an alias in Elasticsearch?
April 10, 2023
How to transpose or convert columns to rows in Hive?
April 3, 2023
What is an alias and how to create an alias in Elasticsearch?
April 10, 2023

This is a simple and common problem or question with a simple answer.

Solution

Do a directory listing with recursive, human-readable and summarize options. You will see the count of objects and size at the end.

[osboxes@wk1 ~]$ aws s3 ls s3://hirw-kickstarter --recursive --human-readable --summarize
2020-11-18 16:46:03    0 Bytes Output2/_SUCCESS2
2020-11-18 16:43:34  111 Bytes Output2/_committed_1768500173264775255
2020-11-18 16:43:34    0 Bytes Output2/_started_1768500173264775255
2020-11-18 16:43:34  166.1 KiB Output2/part-00000-tid-1768500173264775255-99c4578a-e958-4e22-869b-46bfaf1acbfa-611-c000.csv
2018-04-07 16:20:35   44.3 MiB ks-projects-201612.csv

Total Objects: 5
   Total Size: 44.5 MiB
Big Data In Real World
Big Data In Real World
We are a group of Big Data engineers who are passionate about Big Data and related Big Data technologies. We have designed, developed, deployed and maintained Big Data applications ranging from batch to real time streaming big data platforms. We have seen a wide range of real world big data problems, implemented some innovative and complex (or simple, depending on how you look at it) solutions.

Comments are closed.

How to check size of a bucket in S3?
This website uses cookies to improve your experience. By using this website you agree to our Data Protection Policy.

Hadoop In Real World is now Big Data In Real World!

X