How to view a message in Kafka?
April 17, 2023How to properly remove or decommission a node from an Elasticsearch cluster?
April 24, 2023There is no single command that can rename a S3 bucket. Mainly because S3 is not a filesystem and a bucket is not a typical folder in a filesystem.
Renaming a bucket is a 3 step operation – create bucket, sync between old and new bucket, deleting the old bucket.
Solution
We want to rename the bucket iot-activity-data to iot-activity-data-v2.
[osboxes@wk1 ~]$ aws s3 ls --recursive s3://iot-activity-data 2018-04-16 11:06:49 3819 1-stand.json 2018-04-23 11:16:06 6325 2-sit.json 2018-04-23 11:26:45 4474 3-stairsdown.json 2018-04-23 11:29:52 4340 4-bike.json
Create the new bucket with aws s3 mb (make bucket) command.
[osboxes@wk1 ~]$ aws s3 mb s3://iot-activity-data-v2 make_bucket: iot-activity-data-v2
sync will copy all the files and folder recursively under iot-activity-data bucket to iot-activity-data-v2.
[osboxes@wk1 ~]$ aws s3 sync s3://iot-activity-data s3://iot-activity-data-v2 copy: s3://iot-activity-data/1-stand.json to s3://iot-activity-data-v2/1-stand.json copy: s3://iot-activity-data/2-sit.json to s3://iot-activity-data-v2/2-sit.json copy: s3://iot-activity-data/3-stairsdown.json to s3://iot-activity-data-v2/3-stairsdown.json copy: s3://iot-activity-data/4-bike.json to s3://iot-activity-data-v2/4-bike.json
Forcefully remove the iot-activity-data bucket using rb command.
[osboxes@wk1 ~]$ aws s3 rb --force s3://iot-activity-data delete: s3://iot-activity-data/1-stand.json delete: s3://iot-activity-data/4-bike.json delete: s3://iot-activity-data/3-stairsdown.json delete: s3://iot-activity-data/2-sit.json remove_bucket: iot-activity-data
We can see and list the contents of the new bucket – iot-activity-data-v2.
[osboxes@wk1 ~]$ aws s3 ls --- --- 2020-11-18 17:16:42 iot-activity-data-v2 2018-03-29 15:44:13 iot-activity-stream-op --- --- [osboxes@wk1 ~]$ aws s3 ls --recursive s3://iot-activity-data-v2 2020-11-18 17:16:49 3819 1-stand.json 2020-11-18 17:16:49 6325 2-sit.json 2020-11-18 17:16:49 4474 3-stairsdown.json 2020-11-18 17:16:49 4340 4-bike.json
Big Data In Real World
We are a group of Big Data engineers who are passionate about Big Data and related Big Data technologies. We have designed, developed, deployed and maintained Big Data applications ranging from batch to real time streaming big data platforms. We have seen a wide range of real world big data problems, implemented some innovative and complex (or simple, depending on how you look at it) solutions.
