How to create a folder in Amazon S3 using AWS CLI? - Big Data In Real World

How to create a folder in Amazon S3 using AWS CLI?

How to control log settings in Spark and stop INFO messages?
January 25, 2021
How does Cartesian Product Join work in Spark?
January 29, 2021
How to control log settings in Spark and stop INFO messages?
January 25, 2021
How does Cartesian Product Join work in Spark?
January 29, 2021

You probably heard this – Amazon S3 is not a file system, it is an object storage. So in S3, there is no technical concept of a folder. 

If that’s the case, why do I see folders in AWS S3 console?

Folders in S3 are meant only for organization purposes. For us to organize the objects that make sense for us. The Amazon S3 implements folder object creation by creating a zero-byte object. 

aws-s3-folder

If you see a file in the console you will see the key of the file also has the folder reference in the key – test-folder/hdfs-0.0.1.jar.zip. test-folder is the folder name.

aws-s3-file

Solution

We are now going to create a new folder named new-folder and upload a file into that folder.

[osboxes@wk1 ~]$ aws s3 ls s3://hirw-test-bucket

                           PRE /
                           PRE test-folder/
2016-11-05 12:43:00       3411 block_-3863181236475038926

 

Here when we copy the file we mention the destination as new-folder/test-file eventhough new-folder doesn’t exist.

[osboxes@wk1 ~]$ aws s3 cp test-file s3://hirw-test-bucket/new-folder/test-file

upload: ./test-file to s3://hirw-test-bucket/new-folder/test-file

We can now see the new-folder when we do a listing on the bucket.

[osboxes@wk1 ~]$ aws s3 ls s3://hirw-test-bucket

                           PRE /
                           PRE new-folder/
                           PRE test-folder/
2016-11-05 12:43:00       3411 block_-3863181236475038926

 

We also see the folder created on the AWS S3 console also. Check out the key on the file as well.

aws-s3-new-folder

aws-s3-new-folder-file

 

Big Data In Real World
Big Data In Real World
We are a group of Big Data engineers who are passionate about Big Data and related Big Data technologies. We have designed, developed, deployed and maintained Big Data applications ranging from batch to real time streaming big data platforms. We have seen a wide range of real world big data problems, implemented some innovative and complex (or simple, depending on how you look at it) solutions.

Comments are closed.

How to create a folder in Amazon S3 using AWS CLI?
This website uses cookies to improve your experience. By using this website you agree to our Data Protection Policy.

Hadoop In Real World is now Big Data In Real World!

X