June 2, 2024
Hadoop Developer In Real World 3 node Hadoop cluster will be shutdown permanently on June 30th 2024. Reason for the shutdown Hadoop Developer In Real World […]
June 19, 2023
In this post we will see how to recursively delete files/objects, folders and bucket from S3. Recursively deleting a folder in S3 rm –recursive followed by […]
February 25, 2023
www.bigdatainrealworld.com is now fully live Hadoop In Real World is now Big Data In Real World! In-case you missed our communication from last week, you can […]
February 19, 2023
We got an announcement to make. But first, what we have been up to? We have big things planned for 2023 and we have been busy […]
October 18, 2021
As simple as this question sounds, we get this question a lot. So we decided to write a post on finding the version of Hadoop and […]
September 8, 2021
It is a quite common need to find the available free space in HDFS. There are a couple of ways to find this information. Using cluster […]
August 23, 2021
So you got this below execution and wondering how to fix it. In this post we are going to explain the possible causes for this exception which […]
August 13, 2021
When namenode is started or restarted, namenode will be in safemode for a period of time. At this time you will not be able to use […]
May 17, 2021
In a Big Data platform, when you manage a lot of data, it is quite inevitable at some point that data could be corrupted. This post […]
April 28, 2021
A common question with a simple solution. Solution Use the below YARN command to list all applications that are running in YARN. yarn application -appStates RUNNING […]
April 23, 2021
It is pretty common in certain Hadoop distributions to get the below error when you attempt to start resource manager service or other services. ERROR [main] […]
April 16, 2021
This post will list out the steps to properly remove a node from a Hadoop cluster. It is not advisable to just shut down the node […]
April 2, 2021
Common problem with a pretty simple solution. Solution When you are doing the directory listing use the -R option to recursively list the directories. If you […]
March 24, 2021
HDFS divides the files into blocks and stores the blocks locally in datanodes. The location varies from cluster to cluster based on the configuration in hdfs-site.xml […]
February 24, 2021
The answer to this question depends on the version of Hadoop you are using. Older version of Hadoop (1.x.x) The first argument of copyFromLocal is restricted […]
February 3, 2021
In Hadoop you will find Hadoop specific types for basic types. For eg. you will find Text for String. IntWritable instead of Integer. For all primitive […]
January 18, 2021
This is a very common need in the day to day Big Data world with a simple solution. Solution Use the hdfs du command to get […]
January 13, 2021
When you run a MapReduce or a Spark job the number of files will equal to the number of reducers involved in the MapReduce job or […]
January 6, 2021
There are so many properties in Spark that affect the way you can add jars to a Spark application. We understand it could be confusing and […]
December 25, 2020
If you are trying to access HDFS, you might have come across the below three types of commands. In this post we will see the difference […]
