February 16, 2022
You can not directly view or preview the content of a GZip file in HDFS. HDFS does not provide a direct command to do that. Solution […]
February 9, 2022
Simple problem with a simple solution. Solution Use hdfs dfs -count to get the count of files and directories inside the directory. [hirw@wk1 ~]$ hdfs dfs […]
September 13, 2021
Use cat followed by a head or tail to see few lines from the top or end of the file. Few lines from top of the […]
September 8, 2021
It is a quite common need to find the available free space in HDFS. There are a couple of ways to find this information. Using cluster […]
September 1, 2021
You are getting the below error during DataNode startup. This post talks about how to fix the issue. 2013-04-11 16:25:50,515 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting […]
August 23, 2021
So you got this below execution and wondering how to fix it. In this post we are going to explain the possible causes for this exception which […]
August 13, 2021
When namenode is started or restarted, namenode will be in safemode for a period of time. At this time you will not be able to use […]
May 28, 2021
NameNode NameNode is the heart of HDFS. NameNode maintains the metadata of HDFS – files, list of blocks, directories, permissions etc. The metadata is persisted on […]
May 17, 2021
In a Big Data platform, when you manage a lot of data, it is quite inevitable at some point that data could be corrupted. This post […]
April 2, 2021
Common problem with a pretty simple solution. Solution When you are doing the directory listing use the -R option to recursively list the directories. If you […]
March 24, 2021
HDFS divides the files into blocks and stores the blocks locally in datanodes. The location varies from cluster to cluster based on the configuration in hdfs-site.xml […]
March 10, 2021
Don’t run the command until you fully understand what you are trying to do. hadoop namenode -format will format or delete all files and folders in […]
February 24, 2021
The answer to this question depends on the version of Hadoop you are using. Older version of Hadoop (1.x.x) The first argument of copyFromLocal is restricted […]
January 18, 2021
This is a very common need in the day to day Big Data world with a simple solution. Solution Use the hdfs du command to get […]
January 13, 2021
When you run a MapReduce or a Spark job the number of files will equal to the number of reducers involved in the MapReduce job or […]
December 25, 2020
If you are trying to access HDFS, you might have come across the below three types of commands. In this post we will see the difference […]
